Fine-tune an extraction model with zero labeled transcripts
Generate the structured record first — as relational data in Tonic Fabricate — and render the conversation from it. The training labels are correct by construction and verified by machine, so a small model on Fireworks can learn a frontier-grade extraction task without a single hand-labeled example.
The problem
Care-coordination calls, telehealth visits, and post-discharge check-ins all end the same way: someone turns a conversation into a structured encounter record. Automating that with a frontier model works — at frontier prices, on every single call. A small fine-tuned model would be radically cheaper, but fine-tuning needs labeled transcripts, and that's where the plan usually dies:
- Real transcripts are PHI — they can't leave the compliance boundary, and often can't be used at all.
- Hand-labeling is slow, expensive, and inconsistent exactly where it matters — negation ("no chest pain"), family history ("my mother had heart failure"), medication changes ("I stopped taking it last week").
- Model-generated labels inherit the teacher's mistakes, and you can't measure what you can't trust.
The approach: structure first, conversation second
Fabricate generates synthetic data schema-first. Instead of writing conversations and then labeling them, it authors the structured truth as relational tables — conditions, symptoms, medications, measurements, care-plan items, each with controlled vocabularies and per-fact directives (mention it explicitly, imply it, omit it, correct it mid-conversation, or plant it as an irrelevant distractor). The conversation is rendered from those rows. The extraction target is a deterministic projection of those rows.
No model ever writes a label. And because every fact carries an exact-substring evidence quote, label quality isn't asserted — it's machine-checked, fact by fact, before anything reaches training.
Verified, not just generated
"Correct by construction" still gets audited. Every batch passes four machine gates before anything reaches training: integrity (evidence quotes resolve exactly, controlled vocabularies hold, omitted facts stay omitted), parity (attribute densities and label distributions match between train and eval), realism (template-skeleton deduplication keeps dialogue genuinely conversational), and behavioral canaries (nested training tiers make any weak batch visible immediately). Generation quality is measured, never assumed.
One example, end to end
This is record t0801-clean from the published dataset's training split — a post-discharge check-in. Fabricate authored the facts first, with per-fact directives; the conversation was rendered to realize them. You can pull this exact record from the dataset and check every quote.
| Fact | Key attributes | Directive |
|---|---|---|
| naproxen 500 mg twice daily | status current · action decrease · adherence adherent | explicit — dose comes down; the patient hasn’t missed a day |
| ibuprofen 200 mg as needed | action continue · adherence missed_doses | explicit — stays as-is, adherence imperfect |
| amlodipine 10 mg once daily | action refill · adherence unknown | explicit — the patient genuinely can’t say; unknown stays unknown |
| fever | assertion absent | explicit — a negation probe |
| anemia | assertion suspected | explicit — raised, not diagnosed |
| carpal tunnel syndrome | experiencer other | distractor |
How you drive Fabricate
The generation contract is a natural-language prompt to Fabricate's agent — it builds the schema, authors the rows, renders the conversations, and self-checks, autonomously. An excerpt from the actual pilot contract:
Generate clinical-conversation scenarios as relational data, then render
each as free-prose dialogue.
SCENARIO FACTS — author rows in controlled-vocabulary tables (conditions,
symptoms, medications, measurements, care-plan items, escalations). Every
fact carries a directive: explicit | indirect | omitted | corrected — plus
a distractor flag for third-party facts that must appear in dialogue but
never in the target.
DIALOGUE — free prose, one conversation at a time: 8–40 utterances, varied
openings and registers, patient rambling and self-repairs, facts realized
out of order. Templated or slot-filled rendering is forbidden, audited by
skeleton deduplication (<30% of utterances may share a recurring skeleton
after normalizing numbers and clinical entities).
REALIZATION — every non-omitted fact is spoken at its authored specificity:
formal diagnosis terms said at least once, measurement values said aloud,
medication dose/unit/frequency/adherence ≥95% spoken, symptom severity and
onset ≈80%. Attributes the dialogue doesn't state stay NULL. Negated facts
≈10%, phrased as varied natural denials.
EVIDENCE — every non-omitted fact links to an exact-substring quote from
one utterance, written at insert time.
SELF-CHECK before export: evidence substring failures = 0, enum violations
= 0, omitted-fact leaks = 0; report skeleton-duplication share, attribute
densities, and coverage against targets.And the pipeline around it
# the label rule, in its entirety — no model ever writes a target
released = [f for f in facts if f.mention_style != "omitted" and not f.is_distractor]
# deterministic gates + variants + tiers + training
python validate.py clinical_transcripts.db # evidence, leaks, enums, split integrity
python noise.py clean.jsonl noisy.jsonl # seeded ASR variants, evidence-safe
python make_sft.py --pairs 250 --out tier250.jsonl
python train_tier.py tier250.jsonl --tier 250 # LoRA on Fireworks, ~15 minCookbook steps
First question: is fine-tuning even needed?
Before generating training data, the pipeline generates only an evaluation set (100 conversations plus 50 counterfactual minimal-pairs, rendered by a different model family than the training data) and runs a zero-shot ladder. Two gates decide whether the task is worth fine-tuning:
GATE G0 · TASK IS WELL-POSED — PASS
A frontier-class anchor (gpt-oss-120B) scores 0.95 schema-valid and 0.72 lenient slot-F1 on realistic dialogue with near-zero trap intrusion — the labels are consistent and the task is learnable, and genuinely hard.
GATE G1 · HEADROOM EXISTS — PASS
The small tunable student (Qwen3 4B) trails the anchor by 26 lenient F1 points zero-shot, with negation accuracy near zero. That gap is the fine-tuning opportunity.
Second question: how much synthetic data does it take?
This is the number teams actually need — not "does SFT help" but how many synthetic conversations buy how much accuracy, at what cost. The pipeline trains one LoRA per nested data tier (each tier a superset of the last) and evaluates all of them from a single multi-LoRA Fireworks deployment against the frozen eval set.
Two findings carry the chart. First, 300 verified synthetic conversations put the 4B student past the 120B anchor on every metric — including strict F1 and negation — and 900 stretch the lenient-F1 lead to nine points (0.818 vs 0.724), with ASR-noisy audio within two points of clean. Second, the curve localizes the remaining work: strict attribute precision — getting every dose, status, and assertion simultaneously right — hovers at frontier-model level for every contender, tuned or not. It's the hard half of the task, and attribute-dense training data, not model scale, is the lever
The curve is also a debugger
Because the tiers are nested, the learning curve doubles as a data-quality instrument: train on strictly more data with everything else held fixed, and any dip in the curve indicts the newest batch of data, never the model. Together with the pre-training gates, this keeps label quality measurable end to end — including for the eval itself, which passes the same audits (0% template skeletons, independently verified).
Try it yourself
