原始内容
name: talkies description: Self-hosted OpenAI-compatible speech service. /v1/audio/transcriptions fronts 12 open ASR models (Whisper, Parakeet, Nemotron-3.5-ASR, Canary, Sherpa-ONNX, Vosk); /v1/audio/transcriptions/stream accepts live PCM over WebSocket. /v1/audio/speech fronts 2 TTS engines / 3 backends — Kokoro-82M (41 baked voices, PyTorch + ONNX runtimes) and the CUDA-only Qwen3-TTS family (voice cloning, preset speakers, voice design). Stereo diarization, URL fetching, MCP endpoint, bearer auth. homepage: https://github.com/psyb0t/docker-talkies user-invocable: true permissions:
- network: outbound HTTP to the configured TALKIES_URL; the talkies SERVER also performs its own outbound fetch when a URL is passed as file_path (server-side download, not client-side)
- shell: documented setup/workflow examples invoke local curl / ffmpeg / docker
- filesystem: reads/writes server-side staged files via the /v1/files endpoints (list/put/get/delete); no local filesystem access by this skill itself metadata: { "openclaw": { "emoji": "🎙️", "primaryEnv": "TALKIES_URL", "requires": { "bins": ["docker", "curl"] } } }
talkies
Self-hosted speech service — ASR and TTS, one container. OpenAI-compatible wire shape on both endpoints; point an OpenAI client at it, change the model slug, done.
ASR (POST /v1/audio/transcriptions): twelve bundled slugs — whisper-large-v3, whisper-large-v3-turbo, parakeet-tdt-0.6b-v3, nemotron-3.5-asr-0.6b, canary-180m-flash, canary-1b-flash, canary-qwen-2.5b, four selectable English Sherpa Zipformer variants, and vosk-small-en-us-0.15.
TTS (POST /v1/audio/speech): 2 engines / 3 backends across 7 slugs — kokoro-82m (PyTorch) and kokoro-82m-nvidia (ONNX/ORT) with 41 baked voices across en/es/fr/hi/it/pt, plus the CUDA-only Qwen3-TTS family: qwen3-tts-0.6b / qwen3-tts-1.7b (voice cloning from reference clips), qwen3-tts-0.6b-custom / qwen3-tts-1.7b-custom (9 preset speakers), qwen3-tts-1.7b-design (voice from an NL description). Discover voices via GET /v1/audio/voices.
Extras: live PCM ASR over WebSocket, stereo diarization on transcription, URL file_path fetching, server-side file staging, MCP endpoint with 6 ASR-side tools, optional bearer-token auth.
For installation, configuration, and container setup, see references/setup.md.
Security & safety
This skill is not low-risk to orchestrate blindly — it issues local shell commands (curl, ffmpeg, docker in the typical deployment/setup path) and outbound HTTP requests to whatever $TALKIES_URL points at:
- Outbound HTTP to an operator-chosen host — data leaves your host. Every command in this skill, including TTS
inputtext and voice-cloning reference samples, is sent to whatever$TALKIES_URLpoints at — a server the skill does not control or vet. Point it only at an instance you run or explicitly trust; prefer HTTPS. file_pathURL fetches happen server-side, not client-side. Passing a URL causes the talkies server to download it — see URLfile_path(Download + Cache) below.- Staged files persist server-side until explicitly deleted, and are enumerable by anyone who can reach the API — see Server-Side File Staging below.
- Shared file staging — staged files (
/v1/files) live in one namespace with no per-caller ownership: any caller can see or remove any staged file. Only remove paths you staged in the current task, and treat file management as admin-only on a shared instance — see Server-Side File Staging below. - No auth by default —
TALKIES_AUTH_TOKENis opt-in; an unconfigured server is wide open to anyone who can reach the port. Require the token and least-privilege network exposure in any non-trusted environment. - Voice cloning requires consent. Only clone or synthesize a voice you have explicit authorization/consent to use — synthesized speech of a real person can be used for impersonation, fraud, or deception. See Qwen3-TTS Custom Voices.
- Local shell execution.
references/setup.mdand workflow examples rundocker,curl, andffmpegdirectly on the local host — review commands before running them against unfamiliar hosts/images.
When To Use
- Transcribe audio files (any format ffmpeg decodes — WAV, MP3, M4A, FLAC, OGG, WebM, Opus, MP4 audio).
- Generate SRT/VTT subtitles for video.
- Transcribe podcasts, lectures, interviews, voicemails, calls.
- Stream 16 kHz mono PCM from a microphone or decoded live-audio source through
WS /v1/audio/transcriptions/stream. - Stereo two-mic recordings → per-speaker diarized output (
L:/R:channel tagging). - German/French/Spanish ↔ English speech-to-text translation via Canary-1B-Flash.
- Synthesize speech from text via Kokoro-82M — English (American + British), Spanish, French, Hindi, Italian, Portuguese.
- Voice-clone speech via Qwen3-TTS-0.6B from a reference
.wavyou provide — drop into/data/custom-voices/, immediately appears underGET /v1/audio/voiceswithorigin=custom. Only clone voices you're authorized to use, with the speaker's consent — see Qwen3-TTS Custom Voices. - Drop-in replacement for
api.openai.com/v1/audio/transcriptionsandapi.openai.com/v1/audio/speechin existing client code.
When NOT To Use
- OpenAI-compatible live ASR —
/v1/audio/transcriptionsremains request/response only. Use talkies-specificWS /v1/audio/transcriptions/streamfor raw PCM. (TTS has one streaming exception:qwen3-tts-*+response_format=pcmstreams chunked PCM — see Streaming PCM (Qwen3-TTS).) - Speaker identification from voice (only stereo-channel diarization is supported, not voice clustering).
- Per-request
prompt/temperatureon/v1/audio/transcriptions— accepted for OpenAI compat, ignored. (instructionson/v1/audio/speechis different: Kokoro ignores it, but Qwen3-TTS honors it in most modes — see Request Body.) - Japanese / Chinese TTS — Kokoro upstream supports them but talkies filters those voices out (they need the
misaki[ja]/misaki[zh]extras). - Kokoro on OpenAI aliases (
alloy,echo,fable,onyx,nova,shimmer) — Kokoro exposes its native voice names only (af_*,bm_*, etc.). Map client-side. (Qwen3-TTS does shipalloy/echo/fableas builtin voice slugs, but they're voice-cloned samples, not OpenAI's voices — there's no audio compatibility.) qwen3-tts-0.6bon CPU — voice cloning hard-fails without CUDA at load time. Thefaster_qwen3_ttsupstream raisesValueErroron non-CUDA devices; talkies surfaces this as a load failure on the first request.qwen3-tts-0.6bspeedparameter — Qwen3-TTS has no playback-rate control. Field is accepted for OpenAI compat but ignored (only Kokoro honorsspeed).- arm64 hosts —
linux/amd64only.
Setup
The container should already be running. Set the base URL:
export TALKIES_URL=http://localhost:8000
If the server has TALKIES_AUTH_TOKEN set, export it too:
export TALKIES_AUTH_TOKEN=<your-token>
# every request below needs: -H "Authorization: Bearer $TALKIES_AUTH_TOKEN"
Verify: curl $TALKIES_URL/healthz returns {"ok": true, "device": "...", "models": [...]}.
For install / configuration / env vars / CPU vs CUDA images / custom model registry, see references/setup.md.
Quick Start
# Discover what's available.
curl -s $TALKIES_URL/v1/models | jq
# Simplest transcribe — file upload, JSON response.
curl -s $TALKIES_URL/v1/audio/transcriptions \
-F "file=@audio.mp3" \
-F "model=whisper-large-v3-turbo" | jq
# Same call, but the audio lives at a URL — talkies downloads + caches it.
# WARNING: the URL is fetched by the talkies SERVER, not this client, and the
# download is cached persistently on disk (see "URL file_path" below). Don't
# pass URLs to private/sensitive media unless you trust the server operator.
curl -s $TALKIES_URL/v1/audio/transcriptions \
-F "file_path=https://example.com/podcasts/ep-042.mp3" \
-F "model=whisper-large-v3-turbo" | jq
# Full Whisper-shape JSON with per-segment + per-word timestamps.
curl -s $TALKIES_URL/v1/audio/transcriptions \
-F "file=@audio.mp3" \
-F "model=whisper-large-v3-turbo" \
-F "response_format=verbose_json" | jq
# SRT subtitles.
curl -s $TALKIES_URL/v1/audio/transcriptions \
-F "file=@lecture.mp3" \
-F "model=whisper-large-v3" \
-F "response_format=srt" > lecture.srt
# Discover TTS voices, then synthesize an MP3.
curl -s $TALKIES_URL/v1/audio/voices | jq
curl -s $TALKIES_URL/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"model": "kokoro-82m",
"input": "Hello from talkies.",
"voice": "af_heart",
"response_format": "mp3"
}' \
--output hello.mp3
Live streaming ASR
WS /v1/audio/transcriptions/stream is talkies-specific rather than an OpenAI
endpoint. It accepts headerless 16 kHz, mono, signed 16-bit little-endian PCM.
Send one JSON start message, wait for ready, stream binary PCM frames, then
send {"type":"end"} for final and stats (or {"type":"cancel"} to
discard the stream). Use a configured ASR slug with streaming support; native
sessions are available through parakeet.cpp, Sherpa-ONNX, and Vosk, while
faster-whisper uses a bounded rolling window. When authentication is enabled,
send the bearer token in the WebSocket upgrade header, never in the URL.
See the repository's docs/streaming.md
for the exact start/event shapes, a Python client, limits, and custom Sherpa or
Vosk model-registry entries.
Supported Models
ASR
| Slug | Family | CPU | CUDA | Languages | Strength |
|---|---|---|---|---|---|
whisper-large-v3 |
faster-whisper | yes | yes | 99 auto-detect | best accuracy, slowest |
whisper-large-v3-turbo |
faster-whisper | yes | yes | 99 auto-detect | sweet spot — fast, accurate |
parakeet-tdt-0.6b-v3 |
NeMo TDT | no | yes | English only | very fast on GPU |
nemotron-3.5-asr-0.6b |
parakeet.cpp / ggml (CPU inference) | yes | yes | 40+ locales, auto-detect | CPU-optimized multilingual; pin via language= |
canary-180m-flash |
NeMo Canary | yes | yes | English only (small) | smallest, runs anywhere |
canary-1b-flash |
NeMo Canary | no | yes | en/de/fr/es + translation | multilingual, translation |
canary-qwen-2.5b |
NeMo SALM | no | yes | English only | best English accuracy (no timestamps) |
sherpa-zipformer-en-left-64 |
Sherpa-ONNX Zipformer | yes | yes | English only | native live ASR, lower left context |
sherpa-zipformer-en-left-128 |
Sherpa-ONNX Zipformer | yes | yes | English only | native live ASR, higher left context |
sherpa-zipformer-en-int8-left-64 |
Sherpa-ONNX Zipformer INT8 | yes | yes | English only | smaller native live ASR variant |
sherpa-zipformer-en-int8-left-128 |
Sherpa-ONNX Zipformer INT8 | yes | yes | English only | smaller native live ASR variant |
vosk-small-en-us-0.15 |
Vosk | yes | yes | English only | native live ASR; CPU decoder |
Pick by use case:
- General-purpose:
whisper-large-v3-turbo. - English-only, max accuracy on GPU:
canary-qwen-2.5b(but no per-segment timestamps). - Translation EN↔DE/FR/ES:
canary-1b-flash(requires custom model registry — see Translation). - Low-overhead live English ASR: one of the Sherpa Zipformer variants or
vosk-small-en-us-0.15; Sherpa uses CUDA in the CUDA image, while Vosk remains CPU-decoded.
TTS
2 engines / 3 backends. Kokoro ships in two runtimes (kokoro-82m PyTorch, kokoro-82m-nvidia ONNX/ORT) — same weights, same voice catalog, same wire format. Qwen3-TTS ships 5 CUDA-only slugs across three modes (base cloning / custom_voice preset speakers / voice_design). Mode is implicit in the slug — see Qwen3-TTS Modes.
| Slug | Family | Mode | CPU | CUDA | Languages | Voices |
|---|---|---|---|---|---|---|
kokoro-82m |
Kokoro (PyTorch in-process, 24 kHz) | — | yes | yes | en (US + UK), es, fr, hi, it, pt | 41 baked (discover via GET /v1/audio/voices) |
kokoro-82m-nvidia |
Kokoro (ONNX via ORT, 24 kHz) | — | yes | yes | en (US + UK), es, fr, hi, it, pt | 41 baked (same catalog as kokoro-82m) |
qwen3-tts-0.6b |
Qwen3-TTS (24 kHz) | base | no | yes | 17 (en, zh, ja, ko, fr, de, es, it, pt, ru, vi, th, id, ar, tr, pl, nl) | 3 builtin samples + any .wav under /data/custom-voices/ |
qwen3-tts-1.7b |
Qwen3-TTS (24 kHz) | base | no | yes | 10 (en, zh, ja, ko, fr, de, es, it, pt, ru) | 3 builtin samples + any .wav under /data/custom-voices/ |
qwen3-tts-0.6b-custom |
Qwen3-TTS (24 kHz) | custom_voice | no | yes | en, zh, ja, ko | 9 preset speakers (instructions dropped — 0.6B limitation) |
qwen3-tts-1.7b-custom |
Qwen3-TTS (24 kHz) | custom_voice | no | yes | en, zh, ja, ko | 9 preset speakers + emotion via instructions |
qwen3-tts-1.7b-design |
Qwen3-TTS (24 kHz) | voice_design | no | yes | en, zh, ja, ko | voice synthesized from NL description in instructions (required) |
Pick by use case:
- General-purpose multi-voice TTS:
kokoro-82m— fast, 41 baked voices, runs on CPU. Usekokoro-82m-nvidiafor the ONNX/ORT execution path (CUDA EP on the CUDA image, CPU EP otherwise). - Voice cloning from a reference clip:
qwen3-tts-0.6b/qwen3-tts-1.7b— drop a.wavinto/data/custom-voices/, immediately usable. CUDA required. - Preset speakers (no reference WAV):
qwen3-tts-0.6b-custom/qwen3-tts-1.7b-custom— 9 baked speakers; the 1.7B honoursinstructionsfor emotion. CUDA required. - Invent a voice from a description:
qwen3-tts-1.7b-design— the NL description goes ininstructions. CUDA required.
canary-qwen-2.5b produces no segment/word timestamps — verbose_json.segments and .words come back empty, srt/vtt collapse to a single full-duration cue. Transcription itself is whole-file. Use a Whisper or Canary multitask slug if you need timing.
API — POST /v1/audio/transcriptions
Multipart form. Same field names as OpenAI's transcription endpoint where they overlap.
Request Fields
| Field | Required | Default | Notes |
|---|---|---|---|
file |
one of file/file_path |
— | Audio file. Capped at TALKIES_MAX_UPLOAD_BYTES (default 100 MB). |
file_path |
one of file/file_path |
— | Either a path under the staging area (/v1/files) or an http(s):// URL (downloaded + cached server-side). Not subject to the 100 MB upload cap; URL downloads capped by TALKIES_MAX_DOWNLOAD_BYTES (default 1 GiB). |
model |
yes | — | One of the configured slugs (see GET /v1/models). Unknown → 404. |
language |
no | model default | ISO-639-1 code. Whisper auto-detects when omitted; Canary uses its default_source_lang. |
response_format |
no | json |
json / text / verbose_json / srt / vtt. |
timestamp_granularities[] |
no | — | Accepted for OpenAI compat; ignored — verbose_json always emits both segment + word. |
prompt |
no | — | Accepted, ignored. |
temperature |
no | — | Accepted, ignored. |
diarization |
no | false |
Stereo-channel diarization. Requires 2-channel input — mono returns 400. |
Exactly one of file or file_path must be set — passing both or neither returns 400.
Response Formats
response_format |
Content-Type | Shape |
|---|---|---|
json (default) |
application/json |
{"text": "..."} — just the transcript. |
text |
text/plain |
The transcript as plain text. |
verbose_json |
application/json |
Full Whisper shape — task, language, duration, text, segments[], words[]. |
srt |
application/x-subrip |
SubRip subtitle file, one cue per VAD-segmented chunk. |
vtt |
text/vtt |
WebVTT subtitle file, one cue per VAD-segmented chunk. |
json shape:
{ "text": " full transcript as a single string" }
verbose_json shape — segments and words are always present (empty arrays for backends with no alignment output):
{
"task": "transcribe",
"language": "en",
"duration": 6.42,
"text": " full transcript",
"segments": [{ "id": 0, "start": 0.0, "end": 2.31, "text": " ...", "tokens": [], "temperature": 0.0, "avg_logprob": null, "compression_ratio": null, "no_speech_prob": null }],
"words": [{ "word": " the", "start": 0.0, "end": 0.12 }]
}
Whisper-only confidence fields (avg_logprob, compression_ratio, no_speech_prob) are emitted as null regardless of backend so clients reading them don't crash. tokens is always [].
Stereo Diarization
Pass diarization=true and upload a 2-channel file. Left channel = speaker L, right channel = speaker R. Each channel is transcribed independently, the two timelines are merged chronologically by segment start time.
curl -s $TALKIES_URL/v1/audio/transcriptions \
-F "file=@interview-stereo.wav" \
-F "model=whisper-large-v3-turbo" \
-F "diarization=true" \
-F "response_format=verbose_json" | jq
What changes:
verbose_json— every segment/word gets"channel": "L"or"R". Segments re-numbered after merge.text/response_format=text— rebuilt as alternating turn lines:L: ...\nR: ...\n.... Consecutive same-channel segments collapsed into one line per turn.srt/vtt— each cue prefixed withL:/R:.
Caveats:
- Exactly 2 channels required. Mono → 400. >2 channels → 400.
- Latency ~2× the mono case (model runs sequentially on each channel).
- The technique is exact for true two-mic setups (interview rigs, podcast splits). It does NOT magically separate speakers from a single-mic recording that's been rendered to stereo.
Translation
Canary multitask models can translate speech → text in a non-source language. canary-1b-flash covers en↔de, en↔fr, en↔es. The task is baked into the model slug, not passed per-request — you add a translation-specific slug via custom models.json (see Customizing the model registry):
{
"models": {
"canary-1b-flash-de2en": {
"repo": "nvidia/canary-1b-flash",
"executor": "canary_multitask",
"default_source_lang": "de",
"default_target_lang": "en",
"default_task": "s2t_translation",
"languages": ["de"]
}
}
}
Then call it normally — text carries the English translation:
curl -s $TALKIES_URL/v1/audio/transcriptions \
-F "file=@german-clip.wav" \
-F "model=canary-1b-flash-de2en" | jq
canary-180m-flash is English-ASR-only — don't point a translation slug at it. canary-qwen-2.5b is English ASR only too.
Long Files + VAD Chunking
Audio longer than 30 s (TALKIES_VAD_CHUNK_THRESHOLD) gets sliced through Silero VAD into ≤28 s speech regions before being handed to the backend. Timestamps are re-assembled by offsetting each chunk's segment/word timings — you get one continuous segments list spanning the whole file.
No client-side change. Long files just work. Verify by checking duration in verbose_json.
Error Contract
| Status | Shape | When |
|---|---|---|
| 200 | per response_format |
success |
| 400 | {"detail": "..."} |
bad audio, mono+diarization, >2 ch+diarization, both/neither of file/file_path, invalid file_path, URL download failure (DNS, HTTP error, size exceeded, SSRF blocked) |
| 401 | {"detail": "..."} |
only when TALKIES_AUTH_TOKEN is set: missing/wrong bearer. Includes WWW-Authenticate: Bearer. |
| 404 | {"detail": "..."} |
unknown model slug, file_path references missing file, model-evict on an unloaded model, file op on a missing /v1/files path |
| 413 | {"detail": "..."} |
upload exceeded TALKIES_MAX_UPLOAD_BYTES (multipart file and PUT /v1/files/{path} only — not file_path URL) |
| 422 | {"detail": [...]} |
Pydantic validation (missing fields, wrong types) |
| 500 | {"detail": "..."} |
unhandled backend failure |
API — POST /v1/audio/speech (TTS)
JSON body (not multipart). Returns the encoded audio bytes in the body with the matching Content-Type — no JSON envelope.
Every call sends your input text (and, for voice-cloning slugs, the referenced voice sample) to whatever $TALKIES_URL points at — that data leaves your host. Point $TALKIES_URL only at a talkies instance you run or explicitly trust; prefer HTTPS when it's not localhost/LAN. Don't synthesize sensitive or confidential text through a server you don't control.
curl -s $TALKIES_URL/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"model": "kokoro-82m",
"input": "The quick brown fox jumps over the lazy dog.",
"voice": "af_heart",
"response_format": "mp3",
"speed": 1.0
}' \
--output fox.mp3
Request Body
| Field | Required | Default | Notes |
|---|---|---|---|
model |
yes | — | TTS model slug. Kokoro: kokoro-82m, kokoro-82m-nvidia. Qwen3-TTS: qwen3-tts-0.6b, qwen3-tts-1.7b (base/cloning), qwen3-tts-0.6b-custom, qwen3-tts-1.7b-custom (preset speakers), qwen3-tts-1.7b-design (voice from NL description). Unknown → 404. ASR slug → 400. |
input |
yes | — | Text to synthesize. Empty / whitespace-only → 400. No fixed length cap; for very long inputs split client-side. |
voice |
no | model default_voice |
Semantics shift per Qwen3 mode (see Qwen3-TTS Modes). Kokoro: voice name (default af_heart). Qwen3 base: path of a reference WAV (default alloy). Qwen3 custom_voice: one of the 9 preset speakers (default Vivian). Qwen3 voice_design: ignored — sentinel "design". Unknown → 400 with catalog listed. |
response_format |
no | mp3 |
mp3 / opus / aac / flac / wav / pcm. |
speed |
no | 1.0 |
Playback rate, Kokoro only. Clamped to [0.25, 4.0]. Ignored by every Qwen3-TTS slug (no speed control in Qwen3-TTS). |
instructions |
no | — | Free-form style prompt. Required for qwen3-tts-1.7b-design (the NL voice description; empty → 400). Honoured by Qwen3-TTS base mode and qwen3-tts-1.7b-custom (threaded as instruct). Dropped by qwen3-tts-0.6b-custom (0.6B CustomVoice checkpoint limitation — logs a WARNING) and both Kokoro slugs (no instruction input). Accepted on every slug for OpenAI parity. |
language |
no | model default_language (English) |
Non-OpenAI extra field (send via extra_body={"language": "..."} on official SDKs). Selects the spoken language for Qwen3 custom_voice / voice_design; base mode reads it from the voice's sibling .lang file. Silently ignored by Kokoro. |
temperature |
no | 0.9 |
Non-OpenAI extra, Qwen3-TTS only (extra_body). Sampler temperature, [0.0, 2.0]. Ignored by Kokoro. |
top_k |
no | 50 |
Non-OpenAI extra, Qwen3-TTS only. Top-k truncation, [1, 1000]. Ignored by Kokoro. |
top_p |
no | 1.0 |
Non-OpenAI extra, Qwen3-TTS only. Nucleus sampling, [0.0, 1.0]. Ignored by Kokoro. |
repetition_penalty |
no | 1.05 |
Non-OpenAI extra, Qwen3-TTS only. Penalizes codec-token repeats, [0.5, 2.0]. Ignored by Kokoro. |
max_new_tokens |
no | 2048 (model max) |
Non-OpenAI extra, Qwen3-TTS only. Codec-step cap, [1, 2048]. Ignored by Kokoro. |
do_sample |
no | true |
Non-OpenAI extra, Qwen3-TTS only. false = greedy decode. Ignored by Kokoro. |
Out-of-range sampling values → 422 (Pydantic validation).
Qwen3-TTS Modes
The Qwen3-TTS mode is implicit in the model slug — the OpenAI wire format stays pure (model / voice / instructions / input), with voice and instructions carrying mode-specific semantics. No new endpoints.
| Mode | Slugs | What voice means |
What instructions means |
|---|---|---|---|
| base (voice cloning) | qwen3-tts-0.6b, qwen3-tts-1.7b |
Path of a reference .wav under the voices dirs (.wav stripped) |
Optional style hint (passed as instruct) |
| custom_voice (preset speakers) | qwen3-tts-0.6b-custom, qwen3-tts-1.7b-custom |
One of 9 preset speaker names | Emotion / style cue — 1.7B honours it; 0.6B drops it (checkpoint limitation, logs a WARNING) |
| voice_design (NL description) | qwen3-tts-1.7b-design |
Ignored — sentinel "design" |
Required. NL description of the voice (e.g. "A warm, friendly young female voice"). Empty → 400. |
The 9 custom_voice preset speakers (also returned by GET /v1/audio/voices for the chosen slug): Vivian, Serena, Uncle_Fu, Dylan, Eric (Chinese), Ryan, Aiden (English), Ono_Anna (Japanese), Sohee (Korean).
Output Formats
response_format picks the encoder applied to Kokoro's raw 24 kHz mono PCM. ffmpeg does the conversion in-process; no temp files.
response_format |
Content-Type | Codec / container | Notes |
|---|---|---|---|
mp3 (default) |
audio/mpeg |
libmp3lame, 128 kbps CBR | Most universal. |
opus |
audio/ogg |
libopus, 64 kbps VBR, Ogg container | Best quality-per-byte for speech. |
aac |
audio/aac |
AAC-LC, 128 kbps, ADTS | iOS-friendly. |
flac |
audio/flac |
FLAC | Lossless. |
wav |
audio/wav |
PCM s16le, 24 kHz mono, RIFF header | Lossless, largest. |
pcm |
application/octet-stream |
Raw PCM s16le, 24 kHz mono — no container, no header | Real-time chaining. Caller must know sample rate / format. |
Streaming PCM (Qwen3-TTS)
qwen3-tts-* slugs stream response_format=pcm requests as chunked int16 PCM instead of buffering the whole utterance — first-audio latency drops from seconds to sub-second. Kokoro and every non-pcm format are unaffected (buffered as normal). Response carries an X-Sample-Rate header; chunk size (codec steps per yielded chunk) is tunable via TALKIES_QWEN3_STREAM_CHUNK_SIZE (default 8) — see references/setup.md.
curl -s $TALKIES_URL/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"model":"qwen3-tts-0.6b","input":"Streaming test.","response_format":"pcm"}' \
--output stream.pcm
Voices
curl -s $TALKIES_URL/v1/audio/voices | jq
Returns {"voices": [{"voice", "model", "default", "origin"}]}. The origin field is only present for engines that distinguish baked-in vs user-supplied voices (currently qwen3-tts-0.6b — "builtin" for image-baked samples, "custom" for /data/custom-voices/ mounts). Kokoro entries omit origin.
Kokoro voices encode <lang_code><gender>_<name>:
| Prefix | Language |
|---|---|
af_ / am_ |
American English (female / male) |
bf_ / bm_ |
British English (female / male) |
ef_ / em_ |
Spanish |
ff_ |
French |
hf_ / hm_ |
Hindi |
if_ / im_ |
Italian |
pf_ / pm_ |
Portuguese (Brazilian) |
41 voices ship in the image. Japanese (jf_* / jm_*) and Chinese (zf_* / zm_*) are filtered out because they need the optional misaki[ja] / misaki[zh] extras (MeCab + pypinyin chains).
Qwen3-TTS voices come from two on-disk dirs merged into one catalog:
/opt/talkies/qwen3-voices/— baked into the CUDA image. Ships three curated samples (alloy,echo,fable) so voice cloning works out-of-the-box.origin=builtin./data/custom-voices/— host-mounted via the data volume. Dropfoo/bar/me.wavand voicefoo/bar/meimmediately appears inGET /v1/audio/voices(catalog is rescanned per request — no restart).origin=custom.
Voice names are the wav's path relative to its parent dir with .wav stripped — nested subdirs are preserved. custom-voices/team-a/jane.wav → voice team-a/jane. Custom voices shadow builtin voices with the same name; dropping a custom-voices/alloy.wav overrides the builtin alloy sample (its origin flips to custom).
Optional sibling metadata next to each <name>.wav:
<name>.txt— reference transcript for the clip (ICL voice cloning works without it, but clone fidelity is noticeably better with a faithful transcript).<name>.lang— language label string (defaults toEnglish).
Path-traversal guard: hostile symlinks whose resolve() escapes the voices dir are skipped (the wav can't be used to read arbitrary host files as a voice prompt).
# Add a custom clone voice (server picks it up on next request — no restart).
mkdir -p ~/talkies-data/custom-voices/team-a
cp jane-reading.wav ~/talkies-data/custom-voices/team-a/jane.wav
echo "And the silken sad uncertain rustling of each purple curtain." \
> ~/talkies-data/custom-voices/team-a/jane.txt
# Use it.
curl -s $TALKIES_URL/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3-tts-0.6b",
"input": "Hello from a cloned voice.",
"voice": "team-a/jane",
"response_format": "wav"
}' \
--output cloned.wav
First synth is slow on Qwen3-TTS — the predictor + talker CUDA graphs are captured on first call (~30-60 s on a mid-range GPU). Subsequent generations are sub-second. The model and graphs stay resident until evicted by sibling load or the idle sweeper.
Error Contract (TTS)
| Status | When |
|---|---|
| 200 | success (audio bytes in body) |
| 400 | empty input, unknown voice, unsupported response_format, model isn't TTS (e.g. POSTing whisper-large-v3 here) |
| 401 | TALKIES_AUTH_TOKEN set, missing / wrong bearer |
| 404 | unknown model slug |
| 422 | Pydantic validation (missing required fields, wrong types) |
| 500 | unhandled ffmpeg or kokoro internal failure |
| 503 | TTS snapshot files missing under ${TALKIES_DATA_DIR}/models/<slug>/ (slug excluded from TALKIES_ENABLED_MODELS but still being called); or qwen3-tts-0.6b requested on a non-CUDA device (the backend hard-fails at load time) |
Resource-Management Endpoints (Ollama-Style)
talkies mirrors a subset of speaches / Ollama, so a LiteLLM proxy can drive both.
| Endpoint | Behavior |
|---|---|
GET /healthz |
Unauthenticated liveness. Returns {ok, device, models}. |
GET /v1/models |
OpenAI-style list of configured slugs. Each entry includes a modality field (asr or tts) so clients can filter. |
GET /api/ps |
Currently-loaded models with per-model idle_seconds. |
DELETE /api/ps/{model_id} |
Evict one model from memory. Slug can be URL-encoded (/ → %2F). 404 if not loaded. |
POST /unload |
Evict every loaded model. Returns the list actually unloaded. |
Model eviction (DELETE /api/ps/..., POST /unload) forces a cold-load for anyone mid-request, so only do it for explicit maintenance the user asked for (e.g. "free up VRAM"). Auth is only enforced when TALKIES_AUTH_TOKEN is set — require it on shared deployments and don't expose these routes on an unauthenticated network.
Behind these: an idle sweeper runs every TALKIES_SWEEPER_INTERVAL s (default 60) and unloads anything not used in TALKIES_MODEL_TTL s (default 600). Set TALKIES_MODEL_TTL=0 to disable.
There's also sibling eviction at request time — every transcribe or speech request evicts other loaded models so VRAM doesn't get split. ASR and TTS share the same pool; loading Kokoro evicts a resident Whisper and vice versa. One model resident at a time, per container. If you need two models simultaneously, run two containers.
# Which models are loaded right now.
curl -s $TALKIES_URL/api/ps | jq
# Free VRAM after a job — evict one model.
curl -s -X DELETE "$TALKIES_URL/api/ps/whisper-large-v3-turbo"
# Or evict everything.
curl -s -X POST $TALKIES_URL/unload | jq
Server-Side File Staging (/v1/files)
For repeated transcribes of the same file (different response_format, different model, iterating on params), stage the file once and reference it by path. Files land under ${TALKIES_DATA_DIR}/files/<path>.
Staged files persist until explicitly removed — nothing auto-expires them — and GET /v1/files enumerates every staged path to anyone who can reach the API. Don't stage sensitive/private media on a server without auth enabled (TALKIES_AUTH_TOKEN); clean up staged files when done.
Guardrail — this is a shared, unisolated bucket, not a private workspace. There's no per-caller ownership: any path any caller staged is listable and readable by any other caller with API access. An agent must:
- only read or delete paths it staged itself in the current, user-approved workflow;
- never call
GET /v1/filesto browse/enumerate what other callers have staged, and never delete a path it didn't create; - clean up what it staged once the workflow is done, since nothing expires automatically.
At the deployment level: require TALKIES_AUTH_TOKEN by default, treat per-caller isolation and retention limits as the operator's responsibility (talkies itself provides neither).
| Endpoint | Behavior |
|---|---|
GET /v1/files |
List every staged file. Returns {"files": [{"path", "size", "modified"}]}. Enumerable by anyone with API access — no per-file ownership/isolation. |
PUT /v1/files/{path} |
Upload raw bytes (--data-binary @local-file). Capped at TALKIES_MAX_UPLOAD_BYTES. Atomic write (.part → rename). |
GET /v1/files/{path} |
Streams file back. Content-Type guessed by extension. 404 if missing. |
DELETE /v1/files/{path} |
Removes a staged file and prunes empty parent dirs (404 if missing). Files don't self-expire, so call this to clean up when done. On a shared bucket every path is listable by any caller, so only remove paths you staged yourself. |
# Stage once.
curl -X PUT --data-binary @lecture.mp3 \
-H "Content-Type: audio/mpeg" \
$TALKIES_URL/v1/files/lectures/2026-03-15/lecture.mp3
# Reuse across multiple transcribe calls.
curl -s $TALKIES_URL/v1/audio/transcriptions \
-F "file_path=lectures/2026-03-15/lecture.mp3" \
-F "model=whisper-large-v3-turbo" \
-F "response_format=verbose_json" | jq
# Cleanup.
curl -X DELETE $TALKIES_URL/v1/files/lectures/2026-03-15/lecture.mp3
Path safety: null bytes, backslashes, . / .. segments and double slashes are rejected (400). Symlinks pointing outside the root are refused. Leading / is stripped — /foo/bar.mp3 and foo/bar.mp3 resolve identically.
URL file_path (Download + Cache)
file_path also accepts http:// / https:// URLs. First request downloads to ${TALKIES_DATA_DIR}/files/downloads/<sha256(url)[:16]>-<basename>, subsequent requests with the same URL hit the cache.
The download happens server-side, and the result is cached persistently on the talkies server's disk — not a transient client-side fetch. Anyone who can reach the API can later list/read that cached copy via GET /v1/files (see Server-Side File Staging above). Don't pass URLs to private/sensitive media unless the talkies server itself is trusted and access-controlled (TALKIES_AUTH_TOKEN); invalidate the cache entry when done.
# First call: downloads, transcribes off the cached copy.
curl -s $TALKIES_URL/v1/audio/transcriptions \
-F "file_path=https://example.com/podcasts/ep-042.mp3" \
-F "model=whisper-large-v3-turbo" | jq
# Second call: same URL → cache hit, no re-download.
curl -s $TALKIES_URL/v1/audio/transcriptions \
-F "file_path=https://example.com/podcasts/ep-042.mp3" \
-F "model=canary-1b-flash" \
-F "response_format=srt" > ep-042.srt
Downloads appear in GET /v1/files listings under downloads/. Invalidate a single cached URL by removing it from /v1/files/downloads/.
Constraints applied during download:
- Size capped by
TALKIES_MAX_DOWNLOAD_BYTES(default 1 GiB). - 5 redirect hops max; SSRF guard re-applied at every hop.
- 10 s connect, 300 s per-chunk read timeout.
- SSRF off by default. Set
TALKIES_BLOCK_PRIVATE_DOWNLOADS=trueto reject URLs whose hostname resolves to private/loopback/link-local/multicast/reserved IPs.
MCP Endpoint (/v1/mcp)
talkies exposes a Model Context Protocol server over Streamable HTTP at /v1/mcp. Same FastAPI process, same BACKENDS / REGISTRY, same auth middleware — a model loaded by the MCP transcribe tool is the same instance the HTTP endpoint sees.
MCP exposes the ASR surface only. TTS (/v1/audio/speech) is HTTP-only — generated audio bytes don't round-trip through JSON-RPC cleanly. list_models filters out TTS slugs so transcribe only ever sees ASR backends.
| Tool | What it does |
|---|---|
list_models |
Discover ASR slugs (TTS slugs are filtered out). Returns [{slug, executor, default_source_lang, default_target_lang, default_task, loaded}]. |
transcribe |
Run ASR on a file_path (URL or staged path). Args: model, language?, response_format? (json/verbose_json/text/srt/vtt), diarization?. JSON formats return a JSON-encoded string; text/srt/vtt return raw. |
list_files |
Same payload as GET /v1/files. |
put_file |
Upload to staging. Body is base64 (content_base64). Decoded size capped at TALKIES_MAX_UPLOAD_BYTES. For big files, prefer PUT /v1/files/{path} over HTTP — JSON-RPC + base64 chews token budget. |
get_file |
Read a staged file as base64. Same size cap. Same advice — for big bytes, hit GET /v1/files/{path} over HTTP. |
delete_file |
Remove a staged file, prune empty parents. |
The transport requires Accept: application/json, text/event-stream. Wire it into Claude Code:
claude mcp add --transport http talkies $TALKIES_URL/v1/mcp
With auth:
claude mcp add --transport http talkies $TALKIES_URL/v1/mcp \
--header "Authorization: Bearer $TALKIES_AUTH_TOKEN"
Note: the canonical mount path is /v1/mcp/ (trailing slash). Bare /v1/mcp is rewritten internally to /v1/mcp/ so clients that don't follow Starlette's 307 redirect work too.
Raw JSON-RPC
For debugging or non-MCP-aware callers, hit it as JSON-RPC over HTTP POST:
# tools/list
curl -s $TALKIES_URL/v1/mcp/ \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'
# tools/call
curl -s $TALKIES_URL/v1/mcp/ \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-d '{
"jsonrpc": "2.0", "id": 2, "method": "tools/call",
"params": {
"name": "transcribe",
"arguments": {
"file_path": "https://example.com/clip.mp3",
"model": "whisper-large-v3-turbo",
"response_format": "json"
}
}
}'
Bearer-Token Auth
If TALKIES_AUTH_TOKEN is set on the server, every route except /healthz and CORS preflight (OPTIONS) requires Authorization: Bearer <token>. Wrong/missing token returns 401 with WWW-Authenticate: Bearer. Compared with hmac.compare_digest (constant-time).
curl -H "Authorization: Bearer $TALKIES_AUTH_TOKEN" $TALKIES_URL/v1/models
Empty / unset token = wide open. For untrusted networks, combine the token with a reverse proxy doing TLS + rate limiting.
Typical Workflows
Quick one-off transcribe
curl -s $TALKIES_URL/v1/audio/transcriptions \
-F "file=@audio.mp3" \
-F "model=whisper-large-v3-turbo" | jq -r .text
Generate subtitles for a video
ffmpeg -i video.mp4 -vn -acodec libmp3lame audio.mp3
curl -s $TALKIES_URL/v1/audio/transcriptions \
-F "file=@audio.mp3" \
-F "model=whisper-large-v3" \
-F "response_format=srt" > video.srt
# burn in: ffmpeg -i video.mp4 -vf subtitles=video.srt -c:a copy video-subbed.mp4
Iterate on the same file with different settings
# Stage once.
curl -X PUT --data-binary @lecture.mp3 \
-H "Content-Type: audio/mpeg" \
$TALKIES_URL/v1/files/work/lecture.mp3
# Try different models / formats without re-uploading.
for fmt in json verbose_json srt; do
curl -s $TALKIES_URL/v1/audio/transcriptions \
-F "file_path=work/lecture.mp3" \
-F "model=whisper-large-v3-turbo" \
-F "response_format=$fmt" > "lecture.$fmt"
done
# Clean up the staged file when done (see the /v1/files reference above).
Diarized interview transcript
curl -s $TALKIES_URL/v1/audio/transcriptions \
-F "file=@interview-stereo.wav" \
-F "model=whisper-large-v3-turbo" \
-F "diarization=true" \
-F "response_format=text"
# stdout:
# L: hi how's it going
# R: not bad you
# L: cool man
Synthesize speech from text
# Default voice, MP3 output.
curl -s $TALKIES_URL/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"model":"kokoro-82m","input":"Greetings, human."}' \
--output greetings.mp3
# Pick a voice from GET /v1/audio/voices, choose a format.
curl -s $TALKIES_URL/v1/audio/voices | jq -r '.voices[].voice'
curl -s $TALKIES_URL/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"model": "kokoro-82m",
"input": "Buongiorno, mondo.",
"voice": "if_sara",
"response_format": "opus"
}' \
--output ciao.opus
Free VRAM after a job
curl -s -X POST $TALKIES_URL/unload | jq
Bulk transcribe from URLs
for url in $(cat urls.txt); do
curl -s $TALKIES_URL/v1/audio/transcriptions \
-F "file_path=$url" \
-F "model=whisper-large-v3-turbo" \
-F "response_format=text"
echo "---"
done
The first hit on each URL downloads + caches; re-running the loop is free.
For local files, replace the file_path form field with file=@path/to/audio
in the same request shape.
Tips
- Use
whisper-large-v3-turboas your default — it's the speed/quality sweet spot for general-purpose ASR. Switch towhisper-large-v3only when you need the last few % of accuracy on hard audio. - URL
file_pathover multipart upload — if the audio is already at a URL, send the URL. Saves bandwidth (the file isn't going up and then back down), gets cached server-side, no upload size cap. - Stage repeated files via
PUT /v1/files/{path}and call withfile_path=to avoid re-uploading on every retry/iteration. response_format=textfor the "just give me the string" case — nojq -r .textneeded, content-type istext/plain.- One active model family at a time — every transcribe request evicts other loaded models. Multiple live streams may share one pinned model up to
TALKIES_STREAM_MAX_CONNECTIONS; a request or stream for a different model receives a conflict until the streams end. Use two containers if you need concurrent models. POST /unloadafter a job — explicit eviction frees VRAM/RAM faster than waiting for the 10-min idle sweeper. Useful in CI / batch scripts.canary-qwen-2.5bhas no timestamps —verbose_json.segments/.wordscome back empty,srt/vttcollapse to one cue. Use a Whisper or Canary multitask slug if you need timing data.- Diarization requires true stereo — if your "stereo" file is the same mono signal copied to both channels, diarization won't separate speakers. The technique is exact for two-mic setups, useless otherwise.
- Long files just work — VAD chunking happens transparently. Don't pre-split. Send the whole file.
- ASR's
prompt/temperatureare ignored even though the request schema accepts them. TTS'sinstructionsis different — Kokoro ignores it, but Qwen3-TTS honors it inbasemode and*-custom(except the 0.6B checkpoint) and requires it for*-design. - Watch
/api/psto see what's resident. A request that hangs at "loading model" is doing the first cold load — subsequent calls are fast. - Customizing the model registry for translation slugs or to restrict the served set — see references/setup.md.
- Kokoro uses native voice names — no OpenAI aliases. Hit
GET /v1/audio/voicesonce to discover what's shipped; pass thevoicefield accordingly. The 41 voices cover en (US + UK), es, fr, hi, it, pt; ja/zh are filtered out. - Voice cloning is
qwen3-tts-0.6b— drop a.wav(10-30 s of clean speech is plenty) into/data/custom-voices/<anywhere>.wav. Optionally drop a sibling.txtwith a faithful transcript for higher clone fidelity. The voice appears inGET /v1/audio/voiceson the next request — no restart. CUDA required. - Qwen3-TTS first synth is slow — CUDA graph capture runs once after model load (~30-60 s). Subsequent synths are sub-second. If you're benchmarking, throw away the first call.
- Qwen3-TTS ignores
speed— the model has no playback-rate control. Pass it for OpenAI compat; nothing happens. Only Kokoro honorsspeed. - Both TTS engines emit 24 kHz mono PCM — Kokoro and Qwen3-TTS both output int16 24 kHz mono. ffmpeg re-encodes into your chosen
response_format;pcm(raw, no container) hands back that same rate directly — check theX-Sample-Rateresponse header on Qwen3-TTS streaming responses if you need it confirmed per-request. - TTS
response_format=pcmis for chaining — raw int16 mono PCM, no container, no header. Use it when piping into another encoder or a real-time playback path. Otherwise stick withmp3(default) oropusfor size. - TTS evicts loaded ASR and vice versa — they share the same one-model-resident pool. Synthesizing with Kokoro after a transcribe burst incurs Kokoro's cold load. Same applies to Qwen3-TTS (plus the CUDA-graph capture re-runs on cold reload).