Local Voice Studio

Generate 1–5 voice options on a running server

For multiple options use POST /api/candidates with the normal generation fields and candidate_count (integer 1–5, default 2). Check health/capabilities first. Poll GET /api/candidates/{id}; the response contains ordered jobs, done_count and group status. Each completed job.result has its own audio_url/path/id; preview each and save the chosen result using POST /api/voices with output_id. The UI defaults to two and offers separate playback, selection, download and Save voice for every option. Reference/saved voices produce alternative takes of that speaker; design modes explore seeds and may change identity. Seeds increment from the base modulo 2**32, with other inputs retained. No guaranteed distinct identity or perceptual improvement is implied.

Groups and all child jobs are committed atomically in the project-local SQLite queue. Use a stable candidate idempotency_key to recover the same group after timeouts/restarts; changed requests with that key fail 409. Candidate and single-take key scopes are separate. Jobs run sequentially with ordinary/immediate queue ordering; one failure does not discard successful options. Monitor partial/error states and explicitly retry only intended failed children through their queue IDs; pending deletion/clear never cancels active work. View all options in History/Queue reopens a group. Existing POST /api/generate remains a single-take API; use /api/candidates for the count selector.

Connect and manage the service

Locate the checkout from the user's location or service.json with the expected name; do not assume an alias or this Mac's paths. Read README.md and API.md. Every backend defaults to http://localhost:5743/; honor an explicit PORT override. Read GET /health and /api/capabilities on the configured port; fetch /openapi.json for the authoritative schema. Preserve model mode, recordings, catalogs and active work.

First terminal setup asks for a storage folder; Enter selects the project, and an external absolute folder creates links and is remembered. AI/scripts should use make setup NONINTERACTIVE=1 or an explicit DATA_DIR to avoid a prompt. Use make setup only if needed; it installs pinned dependencies and verifies pinned model assets. Honor VENV, BOOTSTRAP_PYTHON, DATA_DIR, OUTPUT_DIR and PORT overrides consistently. Confirm external volumes are mounted before creating storage. On this Mac the serving project and venv are under ~/Documents/models/VoiceGen-Service; ignored storage.json keeps DATA_DIR and the shared GPU lock on the SSD. If present, read that file before setup and preserve all linked data directories. A fresh checkout needs no local settings file. storage.json.example is portable; never copy another machine’s local settings blindly. Serving is offline and must not download missing weights implicitly.

Use make serve for a retained foreground process, or make start for explicitly requested background operation. Reuse an existing matching instance. make stop verifies a managed idle process; do not kill an unrelated listener or interrupt a job. No automatic login service. Do not stop other model services to free memory without authorization.

The user's default after testing or maintenance is all VoiceGen servers stopped. Finish active test jobs, stop the verified managed backend and temporary UI previews, and verify listeners are gone. Do not automatically restore Vox or another previously running backend. Leave a server running only when the user explicitly requests it.

VoxCPM2 supports steps/guidance and emotion with designed or reference voices. Fetch capabilities before choosing controls. A common schema does not mean every backend accepts every setting. Unknown/unsupported settings fail. No speed control or streaming API is exposed. Reference identity/accent/emotion remain approximate; a seed is not a voice lock.

POST /api/generate with text plus description or a saved voice_id/reference_id. Keep passages short and comply with live token/media limits. Optional absolute output_dir refers to this server's filesystem. Include a stable idempotency_key for recoverable retries, poll GET /api/jobs/id, and keep capacity busy until the worker completes even if the client disconnects. No cancellation or progress percentage is promised. On timeout recover the job; do not automatically create duplicates. SQLite jobs resume after restart in original order and reconcile complete WAV/metadata artifacts before synthesis. Legacy JSON jobs without replay inputs stay interrupted errors. Inspect job/History before using a new key.

Return the verified WAV and JSON paths and local playback URL after success. Check actual output rate/duration, cold loading versus synthesis time, and truncation warnings. Do not claim perceptual quality was verified unless audio was actually auditioned. Local links are not phone-accessible; upload/export only when explicitly authorized for that file.

Save reusable recordings using POST /api/voices, with name, catalog description and normalized tags. Search/filter/page through GET /api/voices. PATCH updates metadata without changing audio/original design. Save exact excerpt transcripts where appropriate; long samples save only their first 30 seconds. Uploads retain original bytes plus bounded mono working copies. Never delete user recordings as cleanup.

Run make test for no-GPU validation and make smoke for one small real inference when requested/appropriate. Select Qwen with MODEL=qwen3 in this same VoiceGen-Service checkout; no sibling Qwen project or source synchronization is required. Stop the active backend using its MODEL/PORT before starting another: all backends share one library and one project-local SQLite queue. Preserve the former Qwen SSD data folders: imported history indexes still resolve their historical audio there. Consolidation backups of private source/configuration and SQLite state are under ignored project-local state/backups. Keep private audio, transcripts, metadata, indexes, jobs, caches and environments outside Git. Do not upload or publish anything without user authorization.

Expressiveness additions

Read EXTENSIONS.md and the live API 2.1 capabilities. Qwen may offer preset_voice and the experimental steering_emotion/steering_strength fields only when its CustomVoice checkpoint/pack is installed. Do not apply these vectors to Base or VoiceDesign. Accent is a literal prompt direction in supported modes, not a trained accent adapter. VoxCPM2 continuation requires a reference plus its exact transcript and uses prompt_audio/prompt_text; explicit reference mode remains available. Do not infer that a language LoRA or unrelated runtime extension can be loaded by MLX. Inspect applied settings and decode small real outputs after changes; do not claim perceptual improvement without listening.

Combined cloning and performance references

VoxCPM2 accepts clone_mode: "combined" with a saved voice_id or uploaded reference_id and an exact saved/override ref_text. This sends the same recording as both ref_audio (identity) and prompt_audio plus prompt_text (recorded performance). Continuation and combined modes reject new emotion/accent directions. A reference description is catalog context, not an applied instruction. The default clone_mode: "auto" resolves to combined only for a Vox reference with an exact transcript and blank emotion/accent directions; otherwise it resolves to reference. Explicit modes retain their behavior; existing queued effective modes stay frozen. Check modes.reference.clone_modes; Qwen and other unsupported backends reject combined mode. The full text is synthesized in one call, and its transcript counts toward the input budget.

For expressive variants, upload clean recordings of the same desired speaker, save each with a distinct performance name, mood tags and its exact spoken transcript, then select the variant for combined cloning. Preview the reference first: tags are organizational notes, not proof of accent quality. The selected recording supplies both identity and performance; a separate identity anchor is not exposed. Private recordings, transcripts and presets remain outside Git. Accent retention and delivery require listening; no accent guarantee is made.

Longer sentence pauses are not automatically aligned or inferred. Preserve raw full-line takes; do not synthesize tiny sentence fragments or treat energy minima as verified sentence boundaries. Reviewed boundary edits can add silence to reach a desired total gap while preserving voiced samples and longer existing gaps. No speed, temperature or accent-strength control is added for VoxCPM2.

Reviewed pause editing

Completed/history takes offer Adjust pauses. Listen, select the entire quiet gap with playback times or fields, add/remove reviewed spans, then create a separate version. Default target is a minimum total selected gap of 750ms; longer selected gaps are preserved. The CPU editor only inserts zeros inside explicitly reviewed quiet spans and preserves every source sample and the raw file. Quietness checks do not establish sentence boundaries, and unselected pauses are not automatically adjusted. The API route /api/artifacts/{id}/pauses and its exact limits/provenance/retry contract are documented in API.md. Model inference and GPU locks are not involved. Never invent reviewed offsets from energy minima or claim all sentence gaps were verified automatically.

Startup selection

Read BACKENDS.md. Select one MODEL at startup: voxcpm2, qwen3, chatterbox, indextts, moss, fish or higgs. Use the matching Python/runtime environment and lock. No MODEL preserves the original profile/layout. WEIGHTS_DIR reuses existing immutable model files independently of DATA_DIR. Keep source/env/native builds internal and model/cache/private data in configured external storage. Missing linked volumes fail without fallback downloads. A fresh make setup asks once for storage; Enter means project-local. DATA_DIR or NONINTERACTIVE=1 avoids prompts.

Do not run two JobManagers against one DATA_DIR. The lifetime owner lock protects recovery; the shared GPU lock covers all actual load/generation/unload work. Native MOSS/Higgs use one Metal subprocess per job and report model_loaded=false between jobs. Their timing includes cold load; do not compare it as warm synthesis. Preserve native logs for errors. Do not disable memory safeguards or stop unrelated listeners.

Chatterbox Original supports reference cloning/exaggeration/CFG with a fixed1000-token output ceiling, without prompt emotion/accent. Index2.5 supports reference cloning plus separate emotion recording or vector and duration/segment-silence controls, without installed text-to-emotion guidance. Both are experimental on24GB due to observed high MPS driver allocations and slow inference; allocator figures are not physical-RAM peak measurements. MOSS designs from descriptions, cannot clone references, and does not lock identity across lines. Fish/Higgs clone with reference+exact transcript and explicit inline emotion/prosody tags in text. Reject unsupported controls; do not silently rewrite supplied speech or pretend a supported language guarantees a Romanian accent.

Retain exact Bilibili/Fish/Boson research/noncommercial terms and output/attribution conditions as documented. No commercial-use confirmation is required for already authorized personal research use. Copying a venv is not a portable deployment: rebuild from a permanent internal interpreter and the pinned lock. Reuse SSD model paths and rebuild native source at its final internal location so deployment does not depend on an isolated worktree.

Interactive make setup lists all supported models; Enter selects the original profile, VoxCPM2 here. Interactive make serve gives selection numbers only to models with verified local assets and a compatible installed runtime, using lightweight checks without loading weights. Unavailable models are shown separately without numbers, each with a make setup MODEL= command, and cannot be selected. Enter selects the original profile when ready, otherwise the first ready model clearly marked as the menu default. No ready model lists unavailable models and fails with setup/remount guidance. Explicit MODEL and noninteractive input skip the menu; noninteractive serving preserves the explicit/configured model and fails if it is not ready. No implicit downloads or backend substitution. Use typed chatterbox/indextts option objects for new backend controls; flat aliases are deprecated and cannot be combined with their namespaces. Read model-scoped license notices and authoritative terms in capabilities/UI/startup output, including Higgs's conditional creator grant. Notices do not require acknowledgment.

Model license warnings

Restricted backends emit a prominent MODEL LICENSE WARNING at every server startup; managed make start also prints it to the terminal, and the server logs it. Use the model-scoped commercial_use_notice and license_url in capabilities for the verified conditions: Fish commercial model use, Higgs commercial model/output use outside its conditional Creator Use Grant, and IndexTTS organization-size thresholds. Do not display a blanket commercial-license requirement for permissive backends or uploaded reference recordings. The warning requires no acknowledgment and does not block serving.

Durable queue operation

Use the live schema for POST /api/queue (same 202 Job body as /api/generate, also /api/jobs), GET listing/detail, DELETE one pending entry or clear pending, and POST /{id}/retry for an explicitly intended failed-job retry. API generate_immediately defaults false; UI sends true by default. True means next available slot ahead of normal pending work, FIFO among equal priorities, never interrupting current inference. Only the startup backend executes; other pending backends are paused. Do not switch or load models automatically to drain them.

SQLite/journals and the worker owner lock live in ignored serving-project state/; media and legacy JSON remain on configured DATA_DIR storage. Startup migrates the legacy database consistently, retains an SSD backup and fails closed on collisions. Health reports queue_database. The existing shared GPU_LOCK_FILE is held through inference; contention waits without failing admitted jobs. A second data owner must not recover a live queue. Native descendants inherit the actual queue/data-owner and GPU descriptors; an orphaned native child keeps these reservations until it exits. Graceful stop finishes active work and retains pending work. Queue clearing removes pending entries only, retaining tombstones/retry keys and all completed artifacts/History/reference media. It never cancels active inference. Frozen effective settings/transcript plus a reference audio hash prevent silent substitutions after catalog edits; missing/changed bytes fail for review. IndexTTS's separate emotion reference is frozen/hashed too. No duplicate reference snapshot is stored.

A complete matching WAV plus sidecar is reconciled into a done job/index after restart. A crash before the sidecar can repeat synthesis under the original ID, so never promise exactly-once effects. Retry the same request/key after a timeout; errors require explicit retry and deleted keys remain deleted. Keep jobs, outputs and artifact indexes together in storage backups. No new downloads or inference are needed for queue tests; use an isolated SSD validation DATA_DIR and fake engines.

The serving repository is https://github.com/ashwinmahesh/VoiceGen-Service and its source checkout is named VoiceGen-Service. Backend IDs and existing SSD data/catalog paths are independent of that folder name. Preserve them when moving or updating source. Moved virtual environments/native builds require explicit path repair or pinned recreation and validation; do not assume they are relocatable.

Model-specific performance

Read capabilities and API.md before choosing a candidate. Preserve full submitted text and accurate reference transcripts. Fish delivery_tags and Higgs official enum controls render visible model-scoped prefixes; do not duplicate manual leading directives, invent accent tokens, inject SFX/words, or promise milliseconds from native tags. Reuse restores their typed options. Chatterbox Original offers optional .5/.3 deliberate and .7/.3 expressive candidates at temperature .8; its first6s speech-token/first10s decoder/full-waveform speaker embedding inputs differ. Index retains the original speaker anchor while optionally transferring emotion from another authorized recording or a named vector; alpha .3–.5 is experimental, and segment gaps are not sentence timing. Qwen Base ICL has no free-text acting instruction path and an effective repetition floor1.5; VoiceDesign/CustomVoice mean different identities. MOSS's audio temperature1.5 remains the baseline; a repeated seed does not lock identity across new text. No cross-model vectors, training, new LoRA or ineffective sampler controls are implied. Preserve raw samples and judge accents by listening, separately from expression and word accuracy.

Recommended model defaults, sources and evidence limits are in VOICE_QUALITY.md. Use Vox auto for transcribed reference performance unless explicit acting calls for reference mode; never replace frozen queued modes. Chatterbox Original exaggeration .3/CFG .5/temperature .8 is provisional; retain explicit overrides. Fish/Higgs start untagged, Index follows its speaker recording, and MOSS retains its specialized sampler. Apply recommended defaults resets the studio performance controls while keeping identity inputs, text and descriptions.

Punctuation pacing and UI pages

New Fish, IndexTTS, Chatterbox and Qwen reference/fixed-speaker requests resolve optional model-scoped punctuation_pauses=true, sentence_pause_ms=500 (0–3000) and comma_pause_ms=200 (0–1500). Values add silence between separate phrase calls; they are not measured total pauses. Full input limits remain checked, with a maximum of 32 phrases/600 seconds. The same reference, transcript and style settings are reused, but phrase context can change flow/identity. Zero skips that punctuation class; false restores whole-line synthesis. Manual Fish tags and Qwen VoiceDesign default to whole-line; explicit incompatible pacing is rejected. Existing queued settings remain frozen and old idempotency hashes compatible. MOSS uses the same controls as approximate total-pause instruction targets in a single native pass; never claim exact timing or separate phrase generation for MOSS. Vox/Higgs remain unchanged. Check effective..pacing for actual sample insertions or native guidance.

The main studio is /, Collection is /collection (History/Voice library), and Queue is a header popup modal on both pages. Voice/take/reuse actions navigate to the studio with the selected item. Pacing controls live under Advanced settings. Preserve queues and catalogs when changing UI layout.

Upload recordings directly to Collection

Use multipart POST /api/voices/upload with file and name, optional description/transcript, and repeated tags fields to import a recording and save a Collection voice in one request. HTTP201 returns a Voice; use its id as voice_id and preview its audio_url. Preserve the exact reference transcript or leave it empty if unknown. Limits remain 25 MiB, 1–30 seconds, 8–96 kHz, mono/stereo; original bytes are retained under references/originals and the working voice is normalized to mono16kHz. Uploads do not load/train a model or create a synthesis queue job. The UI action is Collection → Voice library → Upload voice, also available from the studio Saved voice tab. POST /api/references remains a reference import; POST /api/voices saves an already imported reference or generated take. All stored recordings stay private and on the configured data drive. Full-page README screenshots use only synthetic demo data.