talkies-x-5

内容来源：clawhub · 原始地址 · 查看安装指南

原始内容

name: talkies description: Self-hosted OpenAI-compatible speech service. /v1/audio/transcriptions fronts 12 open ASR models (Whisper, Parakeet, Nemotron-3.5-ASR, Canary, Sherpa-ONNX, Vosk); /v1/audio/transcriptions/stream accepts live PCM over WebSocket. /v1/audio/speech fronts 2 TTS engines / 3 backends — Kokoro-82M (41 baked voices, PyTorch + ONNX runtimes) and the CUDA-only Qwen3-TTS family (voice cloning, preset speakers, voice design). Stereo diarization, URL fetching, MCP endpoint, bearer auth. homepage: https://github.com/psyb0t/docker-talkies user-invocable: true permissions:

  • network: outbound HTTP to the configured TALKIES_URL; the talkies SERVER also performs its own outbound fetch when a URL is passed as file_path (server-side download, not client-side)
  • shell: documented setup/workflow examples invoke local curl / ffmpeg / docker
  • filesystem: reads/writes server-side staged files via the /v1/files endpoints (list/put/get/delete); no local filesystem access by this skill itself metadata: { "openclaw": { "emoji": "🎙️", "primaryEnv": "TALKIES_URL", "requires": { "bins": ["docker", "curl"] } } }

talkies

Self-hosted speech service — ASR and TTS, one container. OpenAI-compatible wire shape on both endpoints; point an OpenAI client at it, change the model slug, done.

ASR (POST /v1/audio/transcriptions): twelve bundled slugs — whisper-large-v3, whisper-large-v3-turbo, parakeet-tdt-0.6b-v3, nemotron-3.5-asr-0.6b, canary-180m-flash, canary-1b-flash, canary-qwen-2.5b, four selectable English Sherpa Zipformer variants, and vosk-small-en-us-0.15.

TTS (POST /v1/audio/speech): 2 engines / 3 backends across 7 slugs — kokoro-82m (PyTorch) and kokoro-82m-nvidia (ONNX/ORT) with 41 baked voices across en/es/fr/hi/it/pt, plus the CUDA-only Qwen3-TTS family: qwen3-tts-0.6b / qwen3-tts-1.7b (voice cloning from reference clips), qwen3-tts-0.6b-custom / qwen3-tts-1.7b-custom (9 preset speakers), qwen3-tts-1.7b-design (voice from an NL description). Discover voices via GET /v1/audio/voices.

Extras: live PCM ASR over WebSocket, stereo diarization on transcription, URL file_path fetching, server-side file staging, MCP endpoint with 6 ASR-side tools, optional bearer-token auth.

For installation, configuration, and container setup, see references/setup.md.

Security & safety

This skill is not low-risk to orchestrate blindly — it issues local shell commands (curl, ffmpeg, docker in the typical deployment/setup path) and outbound HTTP requests to whatever $TALKIES_URL points at:

  • Outbound HTTP to an operator-chosen host — data leaves your host. Every command in this skill, including TTS input text and voice-cloning reference samples, is sent to whatever $TALKIES_URL points at — a server the skill does not control or vet. Point it only at an instance you run or explicitly trust; prefer HTTPS.
  • file_path URL fetches happen server-side, not client-side. Passing a URL causes the talkies server to download it — see URL file_path (Download + Cache) below.
  • Staged files persist server-side until explicitly deleted, and are enumerable by anyone who can reach the API — see Server-Side File Staging below.
  • Shared file staging — staged files (/v1/files) live in one namespace with no per-caller ownership: any caller can see or remove any staged file. Only remove paths you staged in the current task, and treat file management as admin-only on a shared instance — see Server-Side File Staging below.
  • No auth by default — TALKIES_AUTH_TOKEN is opt-in; an unconfigured server is wide open to anyone who can reach the port. Require the token and least-privilege network exposure in any non-trusted environment.
  • Voice cloning requires consent. Only clone or synthesize a voice you have explicit authorization/consent to use — synthesized speech of a real person can be used for impersonation, fraud, or deception. See Qwen3-TTS Custom Voices.
  • Local shell execution. references/setup.md and workflow examples run docker, curl, and ffmpeg directly on the local host — review commands before running them against unfamiliar hosts/images.

When To Use

  • Transcribe audio files (any format ffmpeg decodes — WAV, MP3, M4A, FLAC, OGG, WebM, Opus, MP4 audio).
  • Generate SRT/VTT subtitles for video.
  • Transcribe podcasts, lectures, interviews, voicemails, calls.
  • Stream 16 kHz mono PCM from a microphone or decoded live-audio source through WS /v1/audio/transcriptions/stream.
  • Stereo two-mic recordings → per-speaker diarized output (L: / R: channel tagging).
  • German/French/Spanish ↔ English speech-to-text translation via Canary-1B-Flash.
  • Synthesize speech from text via Kokoro-82M — English (American + British), Spanish, French, Hindi, Italian, Portuguese.
  • Voice-clone speech via Qwen3-TTS-0.6B from a reference .wav you provide — drop into /data/custom-voices/, immediately appears under GET /v1/audio/voices with origin=custom. Only clone voices you're authorized to use, with the speaker's consent — see Qwen3-TTS Custom Voices.
  • Drop-in replacement for api.openai.com/v1/audio/transcriptions and api.openai.com/v1/audio/speech in existing client code.

When NOT To Use

  • OpenAI-compatible live ASR — /v1/audio/transcriptions remains request/response only. Use talkies-specific WS /v1/audio/transcriptions/stream for raw PCM. (TTS has one streaming exception: qwen3-tts-* + response_format=pcm streams chunked PCM — see Streaming PCM (Qwen3-TTS).)
  • Speaker identification from voice (only stereo-channel diarization is supported, not voice clustering).
  • Per-request prompt / temperature on /v1/audio/transcriptions — accepted for OpenAI compat, ignored. (instructions on /v1/audio/speech is different: Kokoro ignores it, but Qwen3-TTS honors it in most modes — see Request Body.)
  • Japanese / Chinese TTS — Kokoro upstream supports them but talkies filters those voices out (they need the misaki[ja] / misaki[zh] extras).
  • Kokoro on OpenAI aliases (alloy, echo, fable, onyx, nova, shimmer) — Kokoro exposes its native voice names only (af_*, bm_*, etc.). Map client-side. (Qwen3-TTS does ship alloy / echo / fable as builtin voice slugs, but they're voice-cloned samples, not OpenAI's voices — there's no audio compatibility.)
  • qwen3-tts-0.6b on CPU — voice cloning hard-fails without CUDA at load time. The faster_qwen3_tts upstream raises ValueError on non-CUDA devices; talkies surfaces this as a load failure on the first request.
  • qwen3-tts-0.6b speed parameter — Qwen3-TTS has no playback-rate control. Field is accepted for OpenAI compat but ignored (only Kokoro honors speed).
  • arm64 hosts — linux/amd64 only.

Setup

The container should already be running. Set the base URL:

export TALKIES_URL=http://localhost:8000

If the server has TALKIES_AUTH_TOKEN set, export it too:

export TALKIES_AUTH_TOKEN=<your-token>
# every request below needs: -H "Authorization: Bearer $TALKIES_AUTH_TOKEN"

Verify: curl $TALKIES_URL/healthz returns {"ok": true, "device": "...", "models": [...]}.

For install / configuration / env vars / CPU vs CUDA images / custom model registry, see references/setup.md.

Quick Start

# Discover what's available.
curl -s $TALKIES_URL/v1/models | jq

# Simplest transcribe — file upload, JSON response.
curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file=@audio.mp3" \
  -F "model=whisper-large-v3-turbo" | jq

# Same call, but the audio lives at a URL — talkies downloads + caches it.
# WARNING: the URL is fetched by the talkies SERVER, not this client, and the
# download is cached persistently on disk (see "URL file_path" below). Don't
# pass URLs to private/sensitive media unless you trust the server operator.
curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file_path=https://example.com/podcasts/ep-042.mp3" \
  -F "model=whisper-large-v3-turbo" | jq

# Full Whisper-shape JSON with per-segment + per-word timestamps.
curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file=@audio.mp3" \
  -F "model=whisper-large-v3-turbo" \
  -F "response_format=verbose_json" | jq

# SRT subtitles.
curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file=@lecture.mp3" \
  -F "model=whisper-large-v3" \
  -F "response_format=srt" > lecture.srt

# Discover TTS voices, then synthesize an MP3.
curl -s $TALKIES_URL/v1/audio/voices | jq
curl -s $TALKIES_URL/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
        "model": "kokoro-82m",
        "input": "Hello from talkies.",
        "voice": "af_heart",
        "response_format": "mp3"
      }' \
  --output hello.mp3

Live streaming ASR

WS /v1/audio/transcriptions/stream is talkies-specific rather than an OpenAI endpoint. It accepts headerless 16 kHz, mono, signed 16-bit little-endian PCM. Send one JSON start message, wait for ready, stream binary PCM frames, then send {"type":"end"} for final and stats (or {"type":"cancel"} to discard the stream). Use a configured ASR slug with streaming support; native sessions are available through parakeet.cpp, Sherpa-ONNX, and Vosk, while faster-whisper uses a bounded rolling window. When authentication is enabled, send the bearer token in the WebSocket upgrade header, never in the URL.

See the repository's docs/streaming.md for the exact start/event shapes, a Python client, limits, and custom Sherpa or Vosk model-registry entries.

Supported Models

ASR

Slug Family CPU CUDA Languages Strength
whisper-large-v3 faster-whisper yes yes 99 auto-detect best accuracy, slowest
whisper-large-v3-turbo faster-whisper yes yes 99 auto-detect sweet spot — fast, accurate
parakeet-tdt-0.6b-v3 NeMo TDT no yes English only very fast on GPU
nemotron-3.5-asr-0.6b parakeet.cpp / ggml (CPU inference) yes yes 40+ locales, auto-detect CPU-optimized multilingual; pin via language=
canary-180m-flash NeMo Canary yes yes English only (small) smallest, runs anywhere
canary-1b-flash NeMo Canary no yes en/de/fr/es + translation multilingual, translation
canary-qwen-2.5b NeMo SALM no yes English only best English accuracy (no timestamps)
sherpa-zipformer-en-left-64 Sherpa-ONNX Zipformer yes yes English only native live ASR, lower left context
sherpa-zipformer-en-left-128 Sherpa-ONNX Zipformer yes yes English only native live ASR, higher left context
sherpa-zipformer-en-int8-left-64 Sherpa-ONNX Zipformer INT8 yes yes English only smaller native live ASR variant
sherpa-zipformer-en-int8-left-128 Sherpa-ONNX Zipformer INT8 yes yes English only smaller native live ASR variant
vosk-small-en-us-0.15 Vosk yes yes English only native live ASR; CPU decoder

Pick by use case:

  • General-purpose: whisper-large-v3-turbo.
  • English-only, max accuracy on GPU: canary-qwen-2.5b (but no per-segment timestamps).
  • Translation EN↔DE/FR/ES: canary-1b-flash (requires custom model registry — see Translation).
  • Low-overhead live English ASR: one of the Sherpa Zipformer variants or vosk-small-en-us-0.15; Sherpa uses CUDA in the CUDA image, while Vosk remains CPU-decoded.

TTS

2 engines / 3 backends. Kokoro ships in two runtimes (kokoro-82m PyTorch, kokoro-82m-nvidia ONNX/ORT) — same weights, same voice catalog, same wire format. Qwen3-TTS ships 5 CUDA-only slugs across three modes (base cloning / custom_voice preset speakers / voice_design). Mode is implicit in the slug — see Qwen3-TTS Modes.

Slug Family Mode CPU CUDA Languages Voices
kokoro-82m Kokoro (PyTorch in-process, 24 kHz) — yes yes en (US + UK), es, fr, hi, it, pt 41 baked (discover via GET /v1/audio/voices)
kokoro-82m-nvidia Kokoro (ONNX via ORT, 24 kHz) — yes yes en (US + UK), es, fr, hi, it, pt 41 baked (same catalog as kokoro-82m)
qwen3-tts-0.6b Qwen3-TTS (24 kHz) base no yes 17 (en, zh, ja, ko, fr, de, es, it, pt, ru, vi, th, id, ar, tr, pl, nl) 3 builtin samples + any .wav under /data/custom-voices/
qwen3-tts-1.7b Qwen3-TTS (24 kHz) base no yes 10 (en, zh, ja, ko, fr, de, es, it, pt, ru) 3 builtin samples + any .wav under /data/custom-voices/
qwen3-tts-0.6b-custom Qwen3-TTS (24 kHz) custom_voice no yes en, zh, ja, ko 9 preset speakers (instructions dropped — 0.6B limitation)
qwen3-tts-1.7b-custom Qwen3-TTS (24 kHz) custom_voice no yes en, zh, ja, ko 9 preset speakers + emotion via instructions
qwen3-tts-1.7b-design Qwen3-TTS (24 kHz) voice_design no yes en, zh, ja, ko voice synthesized from NL description in instructions (required)

Pick by use case:

  • General-purpose multi-voice TTS: kokoro-82m — fast, 41 baked voices, runs on CPU. Use kokoro-82m-nvidia for the ONNX/ORT execution path (CUDA EP on the CUDA image, CPU EP otherwise).
  • Voice cloning from a reference clip: qwen3-tts-0.6b / qwen3-tts-1.7b — drop a .wav into /data/custom-voices/, immediately usable. CUDA required.
  • Preset speakers (no reference WAV): qwen3-tts-0.6b-custom / qwen3-tts-1.7b-custom — 9 baked speakers; the 1.7B honours instructions for emotion. CUDA required.
  • Invent a voice from a description: qwen3-tts-1.7b-design — the NL description goes in instructions. CUDA required.

canary-qwen-2.5b produces no segment/word timestamps — verbose_json.segments and .words come back empty, srt/vtt collapse to a single full-duration cue. Transcription itself is whole-file. Use a Whisper or Canary multitask slug if you need timing.

API — POST /v1/audio/transcriptions

Multipart form. Same field names as OpenAI's transcription endpoint where they overlap.

Request Fields

Field Required Default Notes
file one of file/file_path — Audio file. Capped at TALKIES_MAX_UPLOAD_BYTES (default 100 MB).
file_path one of file/file_path — Either a path under the staging area (/v1/files) or an http(s):// URL (downloaded + cached server-side). Not subject to the 100 MB upload cap; URL downloads capped by TALKIES_MAX_DOWNLOAD_BYTES (default 1 GiB).
model yes — One of the configured slugs (see GET /v1/models). Unknown → 404.
language no model default ISO-639-1 code. Whisper auto-detects when omitted; Canary uses its default_source_lang.
response_format no json json / text / verbose_json / srt / vtt.
timestamp_granularities[] no — Accepted for OpenAI compat; ignored — verbose_json always emits both segment + word.
prompt no — Accepted, ignored.
temperature no — Accepted, ignored.
diarization no false Stereo-channel diarization. Requires 2-channel input — mono returns 400.

Exactly one of file or file_path must be set — passing both or neither returns 400.

Response Formats

response_format Content-Type Shape
json (default) application/json {"text": "..."} — just the transcript.
text text/plain The transcript as plain text.
verbose_json application/json Full Whisper shape — task, language, duration, text, segments[], words[].
srt application/x-subrip SubRip subtitle file, one cue per VAD-segmented chunk.
vtt text/vtt WebVTT subtitle file, one cue per VAD-segmented chunk.

json shape:

{ "text": " full transcript as a single string" }

verbose_json shape — segments and words are always present (empty arrays for backends with no alignment output):

{
  "task": "transcribe",
  "language": "en",
  "duration": 6.42,
  "text": " full transcript",
  "segments": [{ "id": 0, "start": 0.0, "end": 2.31, "text": " ...", "tokens": [], "temperature": 0.0, "avg_logprob": null, "compression_ratio": null, "no_speech_prob": null }],
  "words": [{ "word": " the", "start": 0.0, "end": 0.12 }]
}

Whisper-only confidence fields (avg_logprob, compression_ratio, no_speech_prob) are emitted as null regardless of backend so clients reading them don't crash. tokens is always [].

Stereo Diarization

Pass diarization=true and upload a 2-channel file. Left channel = speaker L, right channel = speaker R. Each channel is transcribed independently, the two timelines are merged chronologically by segment start time.

curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file=@interview-stereo.wav" \
  -F "model=whisper-large-v3-turbo" \
  -F "diarization=true" \
  -F "response_format=verbose_json" | jq

What changes:

  • verbose_json — every segment/word gets "channel": "L" or "R". Segments re-numbered after merge.
  • text / response_format=text — rebuilt as alternating turn lines: L: ...\nR: ...\n.... Consecutive same-channel segments collapsed into one line per turn.
  • srt / vtt — each cue prefixed with L: / R:.

Caveats:

  • Exactly 2 channels required. Mono → 400. >2 channels → 400.
  • Latency ~2× the mono case (model runs sequentially on each channel).
  • The technique is exact for true two-mic setups (interview rigs, podcast splits). It does NOT magically separate speakers from a single-mic recording that's been rendered to stereo.

Translation

Canary multitask models can translate speech → text in a non-source language. canary-1b-flash covers en↔de, en↔fr, en↔es. The task is baked into the model slug, not passed per-request — you add a translation-specific slug via custom models.json (see Customizing the model registry):

{
  "models": {
    "canary-1b-flash-de2en": {
      "repo": "nvidia/canary-1b-flash",
      "executor": "canary_multitask",
      "default_source_lang": "de",
      "default_target_lang": "en",
      "default_task": "s2t_translation",
      "languages": ["de"]
    }
  }
}

Then call it normally — text carries the English translation:

curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file=@german-clip.wav" \
  -F "model=canary-1b-flash-de2en" | jq

canary-180m-flash is English-ASR-only — don't point a translation slug at it. canary-qwen-2.5b is English ASR only too.

Long Files + VAD Chunking

Audio longer than 30 s (TALKIES_VAD_CHUNK_THRESHOLD) gets sliced through Silero VAD into ≤28 s speech regions before being handed to the backend. Timestamps are re-assembled by offsetting each chunk's segment/word timings — you get one continuous segments list spanning the whole file.

No client-side change. Long files just work. Verify by checking duration in verbose_json.

Error Contract

Status Shape When
200 per response_format success
400 {"detail": "..."} bad audio, mono+diarization, >2 ch+diarization, both/neither of file/file_path, invalid file_path, URL download failure (DNS, HTTP error, size exceeded, SSRF blocked)
401 {"detail": "..."} only when TALKIES_AUTH_TOKEN is set: missing/wrong bearer. Includes WWW-Authenticate: Bearer.
404 {"detail": "..."} unknown model slug, file_path references missing file, model-evict on an unloaded model, file op on a missing /v1/files path
413 {"detail": "..."} upload exceeded TALKIES_MAX_UPLOAD_BYTES (multipart file and PUT /v1/files/{path} only — not file_path URL)
422 {"detail": [...]} Pydantic validation (missing fields, wrong types)
500 {"detail": "..."} unhandled backend failure

API — POST /v1/audio/speech (TTS)

JSON body (not multipart). Returns the encoded audio bytes in the body with the matching Content-Type — no JSON envelope.

Every call sends your input text (and, for voice-cloning slugs, the referenced voice sample) to whatever $TALKIES_URL points at — that data leaves your host. Point $TALKIES_URL only at a talkies instance you run or explicitly trust; prefer HTTPS when it's not localhost/LAN. Don't synthesize sensitive or confidential text through a server you don't control.

curl -s $TALKIES_URL/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
        "model": "kokoro-82m",
        "input": "The quick brown fox jumps over the lazy dog.",
        "voice": "af_heart",
        "response_format": "mp3",
        "speed": 1.0
      }' \
  --output fox.mp3

Request Body

Field Required Default Notes
model yes — TTS model slug. Kokoro: kokoro-82m, kokoro-82m-nvidia. Qwen3-TTS: qwen3-tts-0.6b, qwen3-tts-1.7b (base/cloning), qwen3-tts-0.6b-custom, qwen3-tts-1.7b-custom (preset speakers), qwen3-tts-1.7b-design (voice from NL description). Unknown → 404. ASR slug → 400.
input yes — Text to synthesize. Empty / whitespace-only → 400. No fixed length cap; for very long inputs split client-side.
voice no model default_voice Semantics shift per Qwen3 mode (see Qwen3-TTS Modes). Kokoro: voice name (default af_heart). Qwen3 base: path of a reference WAV (default alloy). Qwen3 custom_voice: one of the 9 preset speakers (default Vivian). Qwen3 voice_design: ignored — sentinel "design". Unknown → 400 with catalog listed.
response_format no mp3 mp3 / opus / aac / flac / wav / pcm.
speed no 1.0 Playback rate, Kokoro only. Clamped to [0.25, 4.0]. Ignored by every Qwen3-TTS slug (no speed control in Qwen3-TTS).
instructions no — Free-form style prompt. Required for qwen3-tts-1.7b-design (the NL voice description; empty → 400). Honoured by Qwen3-TTS base mode and qwen3-tts-1.7b-custom (threaded as instruct). Dropped by qwen3-tts-0.6b-custom (0.6B CustomVoice checkpoint limitation — logs a WARNING) and both Kokoro slugs (no instruction input). Accepted on every slug for OpenAI parity.
language no model default_language (English) Non-OpenAI extra field (send via extra_body={"language": "..."} on official SDKs). Selects the spoken language for Qwen3 custom_voice / voice_design; base mode reads it from the voice's sibling .lang file. Silently ignored by Kokoro.
temperature no 0.9 Non-OpenAI extra, Qwen3-TTS only (extra_body). Sampler temperature, [0.0, 2.0]. Ignored by Kokoro.
top_k no 50 Non-OpenAI extra, Qwen3-TTS only. Top-k truncation, [1, 1000]. Ignored by Kokoro.
top_p no 1.0 Non-OpenAI extra, Qwen3-TTS only. Nucleus sampling, [0.0, 1.0]. Ignored by Kokoro.
repetition_penalty no 1.05 Non-OpenAI extra, Qwen3-TTS only. Penalizes codec-token repeats, [0.5, 2.0]. Ignored by Kokoro.
max_new_tokens no 2048 (model max) Non-OpenAI extra, Qwen3-TTS only. Codec-step cap, [1, 2048]. Ignored by Kokoro.
do_sample no true Non-OpenAI extra, Qwen3-TTS only. false = greedy decode. Ignored by Kokoro.

Out-of-range sampling values → 422 (Pydantic validation).

Qwen3-TTS Modes

The Qwen3-TTS mode is implicit in the model slug — the OpenAI wire format stays pure (model / voice / instructions / input), with voice and instructions carrying mode-specific semantics. No new endpoints.

Mode Slugs What voice means What instructions means
base (voice cloning) qwen3-tts-0.6b, qwen3-tts-1.7b Path of a reference .wav under the voices dirs (.wav stripped) Optional style hint (passed as instruct)
custom_voice (preset speakers) qwen3-tts-0.6b-custom, qwen3-tts-1.7b-custom One of 9 preset speaker names Emotion / style cue — 1.7B honours it; 0.6B drops it (checkpoint limitation, logs a WARNING)
voice_design (NL description) qwen3-tts-1.7b-design Ignored — sentinel "design" Required. NL description of the voice (e.g. "A warm, friendly young female voice"). Empty → 400.

The 9 custom_voice preset speakers (also returned by GET /v1/audio/voices for the chosen slug): Vivian, Serena, Uncle_Fu, Dylan, Eric (Chinese), Ryan, Aiden (English), Ono_Anna (Japanese), Sohee (Korean).

Output Formats

response_format picks the encoder applied to Kokoro's raw 24 kHz mono PCM. ffmpeg does the conversion in-process; no temp files.

response_format Content-Type Codec / container Notes
mp3 (default) audio/mpeg libmp3lame, 128 kbps CBR Most universal.
opus audio/ogg libopus, 64 kbps VBR, Ogg container Best quality-per-byte for speech.
aac audio/aac AAC-LC, 128 kbps, ADTS iOS-friendly.
flac audio/flac FLAC Lossless.
wav audio/wav PCM s16le, 24 kHz mono, RIFF header Lossless, largest.
pcm application/octet-stream Raw PCM s16le, 24 kHz mono — no container, no header Real-time chaining. Caller must know sample rate / format.

Streaming PCM (Qwen3-TTS)

qwen3-tts-* slugs stream response_format=pcm requests as chunked int16 PCM instead of buffering the whole utterance — first-audio latency drops from seconds to sub-second. Kokoro and every non-pcm format are unaffected (buffered as normal). Response carries an X-Sample-Rate header; chunk size (codec steps per yielded chunk) is tunable via TALKIES_QWEN3_STREAM_CHUNK_SIZE (default 8) — see references/setup.md.

curl -s $TALKIES_URL/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-tts-0.6b","input":"Streaming test.","response_format":"pcm"}' \
  --output stream.pcm

Voices

curl -s $TALKIES_URL/v1/audio/voices | jq

Returns {"voices": [{"voice", "model", "default", "origin"}]}. The origin field is only present for engines that distinguish baked-in vs user-supplied voices (currently qwen3-tts-0.6b — "builtin" for image-baked samples, "custom" for /data/custom-voices/ mounts). Kokoro entries omit origin.

Kokoro voices encode <lang_code><gender>_<name>:

Prefix Language
af_ / am_ American English (female / male)
bf_ / bm_ British English (female / male)
ef_ / em_ Spanish
ff_ French
hf_ / hm_ Hindi
if_ / im_ Italian
pf_ / pm_ Portuguese (Brazilian)

41 voices ship in the image. Japanese (jf_* / jm_*) and Chinese (zf_* / zm_*) are filtered out because they need the optional misaki[ja] / misaki[zh] extras (MeCab + pypinyin chains).

Qwen3-TTS voices come from two on-disk dirs merged into one catalog:

  • /opt/talkies/qwen3-voices/ — baked into the CUDA image. Ships three curated samples (alloy, echo, fable) so voice cloning works out-of-the-box. origin=builtin.
  • /data/custom-voices/ — host-mounted via the data volume. Drop foo/bar/me.wav and voice foo/bar/me immediately appears in GET /v1/audio/voices (catalog is rescanned per request — no restart). origin=custom.

Voice names are the wav's path relative to its parent dir with .wav stripped — nested subdirs are preserved. custom-voices/team-a/jane.wav → voice team-a/jane. Custom voices shadow builtin voices with the same name; dropping a custom-voices/alloy.wav overrides the builtin alloy sample (its origin flips to custom).

Optional sibling metadata next to each <name>.wav:

  • <name>.txt — reference transcript for the clip (ICL voice cloning works without it, but clone fidelity is noticeably better with a faithful transcript).
  • <name>.lang — language label string (defaults to English).

Path-traversal guard: hostile symlinks whose resolve() escapes the voices dir are skipped (the wav can't be used to read arbitrary host files as a voice prompt).

# Add a custom clone voice (server picks it up on next request — no restart).
mkdir -p ~/talkies-data/custom-voices/team-a
cp jane-reading.wav ~/talkies-data/custom-voices/team-a/jane.wav
echo "And the silken sad uncertain rustling of each purple curtain." \
  > ~/talkies-data/custom-voices/team-a/jane.txt

# Use it.
curl -s $TALKIES_URL/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
        "model": "qwen3-tts-0.6b",
        "input": "Hello from a cloned voice.",
        "voice": "team-a/jane",
        "response_format": "wav"
      }' \
  --output cloned.wav

First synth is slow on Qwen3-TTS — the predictor + talker CUDA graphs are captured on first call (~30-60 s on a mid-range GPU). Subsequent generations are sub-second. The model and graphs stay resident until evicted by sibling load or the idle sweeper.

Error Contract (TTS)

Status When
200 success (audio bytes in body)
400 empty input, unknown voice, unsupported response_format, model isn't TTS (e.g. POSTing whisper-large-v3 here)
401 TALKIES_AUTH_TOKEN set, missing / wrong bearer
404 unknown model slug
422 Pydantic validation (missing required fields, wrong types)
500 unhandled ffmpeg or kokoro internal failure
503 TTS snapshot files missing under ${TALKIES_DATA_DIR}/models/<slug>/ (slug excluded from TALKIES_ENABLED_MODELS but still being called); or qwen3-tts-0.6b requested on a non-CUDA device (the backend hard-fails at load time)

Resource-Management Endpoints (Ollama-Style)

talkies mirrors a subset of speaches / Ollama, so a LiteLLM proxy can drive both.

Endpoint Behavior
GET /healthz Unauthenticated liveness. Returns {ok, device, models}.
GET /v1/models OpenAI-style list of configured slugs. Each entry includes a modality field (asr or tts) so clients can filter.
GET /api/ps Currently-loaded models with per-model idle_seconds.
DELETE /api/ps/{model_id} Evict one model from memory. Slug can be URL-encoded (/ → %2F). 404 if not loaded.
POST /unload Evict every loaded model. Returns the list actually unloaded.

Model eviction (DELETE /api/ps/..., POST /unload) forces a cold-load for anyone mid-request, so only do it for explicit maintenance the user asked for (e.g. "free up VRAM"). Auth is only enforced when TALKIES_AUTH_TOKEN is set — require it on shared deployments and don't expose these routes on an unauthenticated network.

Behind these: an idle sweeper runs every TALKIES_SWEEPER_INTERVAL s (default 60) and unloads anything not used in TALKIES_MODEL_TTL s (default 600). Set TALKIES_MODEL_TTL=0 to disable.

There's also sibling eviction at request time — every transcribe or speech request evicts other loaded models so VRAM doesn't get split. ASR and TTS share the same pool; loading Kokoro evicts a resident Whisper and vice versa. One model resident at a time, per container. If you need two models simultaneously, run two containers.

# Which models are loaded right now.
curl -s $TALKIES_URL/api/ps | jq

# Free VRAM after a job — evict one model.
curl -s -X DELETE "$TALKIES_URL/api/ps/whisper-large-v3-turbo"

# Or evict everything.
curl -s -X POST $TALKIES_URL/unload | jq

Server-Side File Staging (/v1/files)

For repeated transcribes of the same file (different response_format, different model, iterating on params), stage the file once and reference it by path. Files land under ${TALKIES_DATA_DIR}/files/<path>.

Staged files persist until explicitly removed — nothing auto-expires them — and GET /v1/files enumerates every staged path to anyone who can reach the API. Don't stage sensitive/private media on a server without auth enabled (TALKIES_AUTH_TOKEN); clean up staged files when done.

Guardrail — this is a shared, unisolated bucket, not a private workspace. There's no per-caller ownership: any path any caller staged is listable and readable by any other caller with API access. An agent must:

  • only read or delete paths it staged itself in the current, user-approved workflow;
  • never call GET /v1/files to browse/enumerate what other callers have staged, and never delete a path it didn't create;
  • clean up what it staged once the workflow is done, since nothing expires automatically.

At the deployment level: require TALKIES_AUTH_TOKEN by default, treat per-caller isolation and retention limits as the operator's responsibility (talkies itself provides neither).

Endpoint Behavior
GET /v1/files List every staged file. Returns {"files": [{"path", "size", "modified"}]}. Enumerable by anyone with API access — no per-file ownership/isolation.
PUT /v1/files/{path} Upload raw bytes (--data-binary @local-file). Capped at TALKIES_MAX_UPLOAD_BYTES. Atomic write (.part → rename).
GET /v1/files/{path} Streams file back. Content-Type guessed by extension. 404 if missing.
DELETE /v1/files/{path} Removes a staged file and prunes empty parent dirs (404 if missing). Files don't self-expire, so call this to clean up when done. On a shared bucket every path is listable by any caller, so only remove paths you staged yourself. 
# Stage once.
curl -X PUT --data-binary @lecture.mp3 \
  -H "Content-Type: audio/mpeg" \
  $TALKIES_URL/v1/files/lectures/2026-03-15/lecture.mp3

# Reuse across multiple transcribe calls.
curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file_path=lectures/2026-03-15/lecture.mp3" \
  -F "model=whisper-large-v3-turbo" \
  -F "response_format=verbose_json" | jq

# Cleanup.
curl -X DELETE $TALKIES_URL/v1/files/lectures/2026-03-15/lecture.mp3

Path safety: null bytes, backslashes, . / .. segments and double slashes are rejected (400). Symlinks pointing outside the root are refused. Leading / is stripped — /foo/bar.mp3 and foo/bar.mp3 resolve identically.

URL file_path (Download + Cache)

file_path also accepts http:// / https:// URLs. First request downloads to ${TALKIES_DATA_DIR}/files/downloads/<sha256(url)[:16]>-<basename>, subsequent requests with the same URL hit the cache.

The download happens server-side, and the result is cached persistently on the talkies server's disk — not a transient client-side fetch. Anyone who can reach the API can later list/read that cached copy via GET /v1/files (see Server-Side File Staging above). Don't pass URLs to private/sensitive media unless the talkies server itself is trusted and access-controlled (TALKIES_AUTH_TOKEN); invalidate the cache entry when done.

# First call: downloads, transcribes off the cached copy.
curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file_path=https://example.com/podcasts/ep-042.mp3" \
  -F "model=whisper-large-v3-turbo" | jq

# Second call: same URL → cache hit, no re-download.
curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file_path=https://example.com/podcasts/ep-042.mp3" \
  -F "model=canary-1b-flash" \
  -F "response_format=srt" > ep-042.srt

Downloads appear in GET /v1/files listings under downloads/. Invalidate a single cached URL by removing it from /v1/files/downloads/.

Constraints applied during download:

  • Size capped by TALKIES_MAX_DOWNLOAD_BYTES (default 1 GiB).
  • 5 redirect hops max; SSRF guard re-applied at every hop.
  • 10 s connect, 300 s per-chunk read timeout.
  • SSRF off by default. Set TALKIES_BLOCK_PRIVATE_DOWNLOADS=true to reject URLs whose hostname resolves to private/loopback/link-local/multicast/reserved IPs.

MCP Endpoint (/v1/mcp)

talkies exposes a Model Context Protocol server over Streamable HTTP at /v1/mcp. Same FastAPI process, same BACKENDS / REGISTRY, same auth middleware — a model loaded by the MCP transcribe tool is the same instance the HTTP endpoint sees.

MCP exposes the ASR surface only. TTS (/v1/audio/speech) is HTTP-only — generated audio bytes don't round-trip through JSON-RPC cleanly. list_models filters out TTS slugs so transcribe only ever sees ASR backends.

Tool What it does
list_models Discover ASR slugs (TTS slugs are filtered out). Returns [{slug, executor, default_source_lang, default_target_lang, default_task, loaded}].
transcribe Run ASR on a file_path (URL or staged path). Args: model, language?, response_format? (json/verbose_json/text/srt/vtt), diarization?. JSON formats return a JSON-encoded string; text/srt/vtt return raw.
list_files Same payload as GET /v1/files.
put_file Upload to staging. Body is base64 (content_base64). Decoded size capped at TALKIES_MAX_UPLOAD_BYTES. For big files, prefer PUT /v1/files/{path} over HTTP — JSON-RPC + base64 chews token budget.
get_file Read a staged file as base64. Same size cap. Same advice — for big bytes, hit GET /v1/files/{path} over HTTP.
delete_file Remove a staged file, prune empty parents.

The transport requires Accept: application/json, text/event-stream. Wire it into Claude Code:

claude mcp add --transport http talkies $TALKIES_URL/v1/mcp

With auth:

claude mcp add --transport http talkies $TALKIES_URL/v1/mcp \
  --header "Authorization: Bearer $TALKIES_AUTH_TOKEN"

Note: the canonical mount path is /v1/mcp/ (trailing slash). Bare /v1/mcp is rewritten internally to /v1/mcp/ so clients that don't follow Starlette's 307 redirect work too.

Raw JSON-RPC

For debugging or non-MCP-aware callers, hit it as JSON-RPC over HTTP POST:

# tools/list
curl -s $TALKIES_URL/v1/mcp/ \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'

# tools/call
curl -s $TALKIES_URL/v1/mcp/ \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{
    "jsonrpc": "2.0", "id": 2, "method": "tools/call",
    "params": {
      "name": "transcribe",
      "arguments": {
        "file_path": "https://example.com/clip.mp3",
        "model": "whisper-large-v3-turbo",
        "response_format": "json"
      }
    }
  }'

Bearer-Token Auth

If TALKIES_AUTH_TOKEN is set on the server, every route except /healthz and CORS preflight (OPTIONS) requires Authorization: Bearer <token>. Wrong/missing token returns 401 with WWW-Authenticate: Bearer. Compared with hmac.compare_digest (constant-time).

curl -H "Authorization: Bearer $TALKIES_AUTH_TOKEN" $TALKIES_URL/v1/models

Empty / unset token = wide open. For untrusted networks, combine the token with a reverse proxy doing TLS + rate limiting.

Typical Workflows

Quick one-off transcribe

curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file=@audio.mp3" \
  -F "model=whisper-large-v3-turbo" | jq -r .text

Generate subtitles for a video

ffmpeg -i video.mp4 -vn -acodec libmp3lame audio.mp3
curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file=@audio.mp3" \
  -F "model=whisper-large-v3" \
  -F "response_format=srt" > video.srt
# burn in:  ffmpeg -i video.mp4 -vf subtitles=video.srt -c:a copy video-subbed.mp4

Iterate on the same file with different settings

# Stage once.
curl -X PUT --data-binary @lecture.mp3 \
  -H "Content-Type: audio/mpeg" \
  $TALKIES_URL/v1/files/work/lecture.mp3

# Try different models / formats without re-uploading.
for fmt in json verbose_json srt; do
  curl -s $TALKIES_URL/v1/audio/transcriptions \
    -F "file_path=work/lecture.mp3" \
    -F "model=whisper-large-v3-turbo" \
    -F "response_format=$fmt" > "lecture.$fmt"
done

# Clean up the staged file when done (see the /v1/files reference above).

Diarized interview transcript

curl -s $TALKIES_URL/v1/audio/transcriptions \
  -F "file=@interview-stereo.wav" \
  -F "model=whisper-large-v3-turbo" \
  -F "diarization=true" \
  -F "response_format=text"
# stdout:
#   L: hi how's it going
#   R: not bad you
#   L: cool man

Synthesize speech from text

# Default voice, MP3 output.
curl -s $TALKIES_URL/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model":"kokoro-82m","input":"Greetings, human."}' \
  --output greetings.mp3

# Pick a voice from GET /v1/audio/voices, choose a format.
curl -s $TALKIES_URL/v1/audio/voices | jq -r '.voices[].voice'
curl -s $TALKIES_URL/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
        "model": "kokoro-82m",
        "input": "Buongiorno, mondo.",
        "voice": "if_sara",
        "response_format": "opus"
      }' \
  --output ciao.opus

Free VRAM after a job

curl -s -X POST $TALKIES_URL/unload | jq

Bulk transcribe from URLs

for url in $(cat urls.txt); do
  curl -s $TALKIES_URL/v1/audio/transcriptions \
    -F "file_path=$url" \
    -F "model=whisper-large-v3-turbo" \
    -F "response_format=text"
  echo "---"
done

The first hit on each URL downloads + caches; re-running the loop is free. For local files, replace the file_path form field with file=@path/to/audio in the same request shape.

Tips

  1. Use whisper-large-v3-turbo as your default — it's the speed/quality sweet spot for general-purpose ASR. Switch to whisper-large-v3 only when you need the last few % of accuracy on hard audio.
  2. URL file_path over multipart upload — if the audio is already at a URL, send the URL. Saves bandwidth (the file isn't going up and then back down), gets cached server-side, no upload size cap.
  3. Stage repeated files via PUT /v1/files/{path} and call with file_path= to avoid re-uploading on every retry/iteration.
  4. response_format=text for the "just give me the string" case — no jq -r .text needed, content-type is text/plain.
  5. One active model family at a time — every transcribe request evicts other loaded models. Multiple live streams may share one pinned model up to TALKIES_STREAM_MAX_CONNECTIONS; a request or stream for a different model receives a conflict until the streams end. Use two containers if you need concurrent models.
  6. POST /unload after a job — explicit eviction frees VRAM/RAM faster than waiting for the 10-min idle sweeper. Useful in CI / batch scripts.
  7. canary-qwen-2.5b has no timestamps — verbose_json.segments / .words come back empty, srt/vtt collapse to one cue. Use a Whisper or Canary multitask slug if you need timing data.
  8. Diarization requires true stereo — if your "stereo" file is the same mono signal copied to both channels, diarization won't separate speakers. The technique is exact for two-mic setups, useless otherwise.
  9. Long files just work — VAD chunking happens transparently. Don't pre-split. Send the whole file.
  10. ASR's prompt / temperature are ignored even though the request schema accepts them. TTS's instructions is different — Kokoro ignores it, but Qwen3-TTS honors it in base mode and *-custom (except the 0.6B checkpoint) and requires it for *-design.
  11. Watch /api/ps to see what's resident. A request that hangs at "loading model" is doing the first cold load — subsequent calls are fast.
  12. Customizing the model registry for translation slugs or to restrict the served set — see references/setup.md.
  13. Kokoro uses native voice names — no OpenAI aliases. Hit GET /v1/audio/voices once to discover what's shipped; pass the voice field accordingly. The 41 voices cover en (US + UK), es, fr, hi, it, pt; ja/zh are filtered out.
  14. Voice cloning is qwen3-tts-0.6b — drop a .wav (10-30 s of clean speech is plenty) into /data/custom-voices/<anywhere>.wav. Optionally drop a sibling .txt with a faithful transcript for higher clone fidelity. The voice appears in GET /v1/audio/voices on the next request — no restart. CUDA required.
  15. Qwen3-TTS first synth is slow — CUDA graph capture runs once after model load (~30-60 s). Subsequent synths are sub-second. If you're benchmarking, throw away the first call.
  16. Qwen3-TTS ignores speed — the model has no playback-rate control. Pass it for OpenAI compat; nothing happens. Only Kokoro honors speed.
  17. Both TTS engines emit 24 kHz mono PCM — Kokoro and Qwen3-TTS both output int16 24 kHz mono. ffmpeg re-encodes into your chosen response_format; pcm (raw, no container) hands back that same rate directly — check the X-Sample-Rate response header on Qwen3-TTS streaming responses if you need it confirmed per-request.
  18. TTS response_format=pcm is for chaining — raw int16 mono PCM, no container, no header. Use it when piping into another encoder or a real-time playback path. Otherwise stick with mp3 (default) or opus for size.
  19. TTS evicts loaded ASR and vice versa — they share the same one-model-resident pool. Synthesizing with Kokoro after a transcribe burst incurs Kokoro's cold load. Same applies to Qwen3-TTS (plus the CUDA-graph capture re-runs on cold reload).

本页内容由上游仓库自动镜像，如有疑问请访问原始仓库。