Voice-Clone Recipe & Harness — Annex

A voice clone good enough to fool the family is one bad gate away from a clone that emits confident gibberish. We know, because we built both by accident. This is the bench behind The Franglais Turing Test. Read that page for the story. Read this one to rebuild the rig.

The three-gate scorer

One scorer, eval/score_voice.py, is the source of truth for every engine. A candidate differs from a rival only in which wav it produced. Scoring is identical, so cross-engine comparison stays apples-to-apples — the same discipline as the eval harness.

Gate	Measures	Method	Pass
Noise	Clean, not hiss	Per-frame spectral flatness geomean(power)/mean(power), silence-gated by RMS energy; fraction of voiced frames above a tonal threshold	dur < 15s and frac < 0.20
Intel	The right words	Parakeet STT :9000, bag-of-words overlap (multiset intersection / expected word count) vs expected text	overlap >= 0.70
Cosine	Sounds like Bert	resemblyzer cosine vs a real-Bert anchor (centroid of held-out real clips)	higher = closer; ceiling ~0.96

The intelligibility gate is what makes the harness trustworthy. Drop it and the rig lies to you. A noise-only or noise-plus-cosine harness green-lights speech-shaped gibberish: audio that is spectrally clean, carries enough timbre for a non-trivial cosine, yet transcribes as a degenerate loop. Identity without intelligibility is a hallucinated impostor.

Caution

Parakeet :9000 transiently false-zeroes the intel gate under rapid sequential load (empty-body responses), which would poison a whole sweep. The scorer retries three times with a short linear backoff and treats an empty body as a transient hiccup, not a silent wav. A flaky read is not a failing sample.

The Chatterbox T3 LoRA recipe

Chatterbox is a ~0.5B Llama-style T3 text-to-token backbone feeding a frozen S3Gen codec stage. The fine-tune is a LoRA on the T3 backbone only, S3Gen frozen. We learned the dials the hard way.

• Train against the transformer (model.t3.tfmr) with a manual teacher-forced cross-entropy, not the model’s own t3.loss() — which has an axis bug. Trainer: train_t3_lora.py.
• Inference: attach the adapter with PeftModel.from_pretrained onto t3.tfmr, then prepare_conditionals(dry_ref) and generate with the dry reference as the audio prompt.
• Inference scale is the gibberish dial. A strong adapter at alpha 32 / rank 16 (effective scale 2.0) saturates into gibberish. Sweep: scale 0.5 and 0.25 → 100% word-overlap; 1.0 → 64%; 2.0 → 0%. Canonical setting near 0.5 — a stronger adapter wants a lower inference scale.
• Reference acoustics dominate identity. The model copies the reference clip’s room. A dry, close-mic reference beats a stage clip; reference-room acoustics is a bigger lever than any engine swap.

Multilingual base

Chatterbox-Multilingual is the same identity on a 23-language base (t3_mtl23ls), with a French tokenizer and a required language_id parameter. We isolate it in its own venv, .venv-cbx-mtl. The text must be re-tokenized with the multilingual tokenizer; the S3 speech tokens carry over.

Caution

A silent trap on the multilingual venv: pip pulled setuptools 82, which dropped pkg_resources, which made the PerTh watermarker silently initialize to None — clips would have gone out unwatermarked with no error. Pin setuptools==80.10.2.

Apple-Silicon notes

Two machines, one flaky link between them. Here is where they bite you.

• Heavy generation and training run on the MacBook Pro over the TB5 link; the Mini holds the council. Run long fine-tunes off the flaky link (cloud/HF GPU, or tmux with frequent checkpoints).
• Pin mlx / mlx-audio exactly. The proven pair is mlx 0.31.1 / mlx-audio 0.4.2; a one-minor bump silently breaks synthesis. Treat a green /health as a false positive — verify with a real self-warm that emits non-zero audio bytes.
• The STT side is Parakeet (parakeet-tdt-0.6b-v3), which is multilingual and handles French; measured ~7.7 GB true footprint — the surprise memory hog. The TTS/STT stack is Python; the Rust sanctum-tts :8007 dispatcher is built but not yet deployed.

The Québécois campaign

This part is a 2026-07-04 addendum. The bilingual clone shipped English at Turing grade and French at “way too Parisian.” The judge was Bert’s native ear — the only accent instrument that matters in this haus. Fixing it produced three durable artifacts: a locked best-of-Chatterbox French recipe, a calibrated accent judge, and an F5-TTS fine-tune pipeline that measurably moves a base model’s accent prior. This is the sibling work to council fallback and voice.

The locked Chatterbox French recipe

Before pivoting engines, we measured the deployed recipe’s ceiling — about 24 scored candidates across every dial. The keeper lives in sanctum-bert-say.py:

Language	Model	cfg	Post
EN	cbx-bert-mlx-q8 (English base)	0.5	none — Turing-approved
FR	cbx-bert-mtl-mlx-q8 (multilingual base, language_id=fr)	0.55	no atempo, no chunking

FR cfg 0.55, up from 0.4, killed two birds. The “réalitéééé” stutter was a low-cfg sampling degeneracy, amplified by the old atempo 0.9 stretch. Higher cfg is naturally slower, so the atempo crutch got deleted outright.

The ceilings that forced the pivot were architectural. Prosody plateaus flat and robotic, around 60/100 across 19 candidates — a T3 trait no dial breaks. French accent is doubly blocked. The multilingual base carries a Parisian prior no reference overrides. The English base has no French phoneme inventory at all — it mangles “chu” into a glitch. Even a leading English word, “Hey,” primes the whole utterance anglophone. That is cold-start language bleed, and proper nouns (“Bertrand”, “Outremont”) hit hardest.

The calibrated forced-choice judge

The old absolute-score Gemini rubric rated a clip Bert heard as flatly Parisian at joual 95/100. So we made a rule, twice-burned: an LLM ear cannot score accent on an absolute scale. It can, however, pick which of two anchors a clip sits closer to.

f5-eval/accent_judge.py judges clip X against A — real Bert Québécois — and B — a France-French TTS anchor (macOS Eddy (French (France))). Three runs, majority vote, plus a resemblyzer identity cosine against a Bert-FR anchor. Before we trust it, it must pass a calibration set with known answers. That set includes the exact clip the old rubric blew. The judge now calls it Parisian, siding with the native ear. It also cites its phonetic markers — affrication, nasal quality, diphthongization, laxing — which turns it from a scoreboard into a debugging instrument.

Caution

Transcripts feed everything downstream — and Whisper launders joual: measured on 40 clips, Whisper preserved 0 joual markers where a verbatim-prompted Gemini heard 13 (“pis”→“puis”, “chu”→“je suis”, fillers dropped). F5’s ref_text must be exact or the ref-trim leaks reference audio into the output; training labels must be verbatim or the model never learns what written joual sounds like.

F5-TTS: moving the prior instead of fighting it

F5-TTS is the pivot because its failure is fixable. It is a flow-matching DiT, full fine-tune only. Zero-shot, the only public French base (RASPIAUDIO/F5-French-MixedSpeakers-reduced, LibriVox European French, v0 arch) scored Parisian 0/9 at confidence 100 — even with a perfect joual reference. The reference cannot beat the base prior. But the prior itself moves with training, which is exactly what Chatterbox never offered.

Campaign v1 trained on 886 French plus 95 franglais clips — 47 minutes, Whisper labels, LR 1e-5, 5280 updates. The accent climbed in textbook phonetic order. Affrication [t͡sy] at update 500. Diphthongization [lɑʊ̯] by 1000. Then vowel laxing, pis→[pɪs], and the first probe-level QUEBEC verdict by 1500-2000. It peaked at 3/9 québécois votes around updates 2000-3500, then overfit back down to 0/9. Native-ear check at the peak: “still parisian, but getting there.” Judge and ear agreed, a third time.

v2 attacks v1’s measured limits. Labels are now Gemini-verbatim, so the joual lives in the text. Joual-dense clips are oversampled ×2-3. The ~106 English-heavy “franglais” rows are dropped — only one genuinely code-switched clip exists in the whole corpus, and the diglossia strikes again. Training stops deliberately inside v1’s peak zone, around 2940 updates.

Apple-Silicon F5 traps, banked

• f5-tts-mlx is v1-arch-hardcoded (text_mask_padding=True, no pe_attn_head); a converted v0 French checkpoint loads clean and generates garbage. Torch worker until the DiT grows v0 switches. Its duration param is total (reference + generation) seconds.
• finetune_cli saves checkpoints inside the venv (files("f5_tts")/../../ckpts/), copies the --pretrain file there, and auto-resumes from model_last.pt — so a kill that lands on the every-100-update save truncates the zip and the relaunch dies on a miniz error. TERM between saves, never SIGKILL near a ×100 boundary.
• macOS default 256 fds × 16 dataloader workers respawning per epoch = EMFILE at the epoch boundary. ulimit -n 65536 in the launch line.
• Full checkpoints are 5.4 GB (model+EMA+optimizer), not the 1.3 GB EMA-only size — budget keep_last_n against real disk, and copy any peak checkpoint out before the pruner eats it (v1’s two best checkpoints died this way; v2’s eval loop preserves best-so-far automatically).

None of this is done. v2 is training inside the peak zone, the judge is watching, and somewhere in the haus Bert will play the next clip and tell us, in one word, whether it is still Paris. That verdict outranks every gate on this page. The rig exists only to get us close enough that his answer is worth waiting for.