ASR Fine-Tuning
General speech recognition models are very good at general speech. They struggle with the words your work depends on: professional terminology, brand names, accents and speakers who switch languages mid-sentence. I fine-tune speech models so they recognise your vocabulary, measure the word error rate before and after on recordings the model never saw, and deploy them wherever you need them to run.
Why fine-tune an ASR model?
Baseline accuracy is strong. Niche requirements are where it breaks down.
Baseline word error rate (WER) is now robust across the leading speech recognition models. For everyday speech in a major language, most of them will do. The gaps appear with niche requirements: accents the model heard little of in training, code-switching between languages in the same sentence, and, most commonly, niche vocabulary.
I have done this for my own dictation: my open-source ASR fine-tunes are small variants trained for one need each. I bring the same process to professional and business use.
Niche vocabulary
Legal, medical, technical and industry terms, product and brand names, internal project names and acronyms. The words that matter most in your transcripts are the ones a general model gets wrong.
Accents
Speakers whose accent is under-represented in the base model's training data see higher error rates. A fine-tune trained on their speech closes much of that gap.
Code-switching
Speakers who move between languages mid-sentence, such as Spanish and English, Hindi and English, or English with Hebrew words dropped in, confuse models trained to expect one language at a time. What that looks like ↓
Demonstrated reductions in word error rate
My own fine-tunes, measured on recordings none of the models were trained on.
Wrong words per 100, stock Parakeet against my fine-tune, on 129 held-out recordings of English with Hebrew words mixed in. That is about 14 mistakes per 100 words down to about 5.
Out of every 100 Hebrew words in those recordings, the stock model spelled 20 correctly. The fine-tune spells 70 correctly.
Word error rate on English technical dictation, stock Parakeet against an English-only vocabulary fine-tune. Niche terms were the gap.
Word error rate on LibriSpeech, a standard English benchmark. The code-switching fine-tune gained its new skill without getting worse at ordinary English.
Base model: nvidia/parakeet-tdt-0.6b-v2. Code-switching figures are from the public test set, scored 2026-10-27 (hebrish.app). The others are normalised WER on the shared test set in the model catalogue, where the models, test sets and training data are published.
What code-switching does to speech recognition
Code-switching means moving between languages within one conversation, often within one sentence.
It is how bilingual people actually talk: a Spanish word in an English sales call, an English technical term in a Hindi meeting, a Hebrew word for a form or an office in an English email. Here is one sentence from my test set, with the Hebrew words in bold.
Yes, you can pay with a kartis ashrai, but if you pay mezuman there's a small discount.
Yes, you can pay with the Cartisa Shrai, but if you pay him as Uman, there's a small discount.
Yes, you can pay with a kartis ashrai, but if you pay mezuman there's a small discount.
kartis ashrai: credit card. mezuman: cash. Real output from both models on a recording neither was trained on.
Why models get it wrong
- They only know one language at a time. An English model has no word for mezuman, so it picks English words that sound close: "him as Uman". Multilingual models usually choose one language for the whole recording.
- The error spreads. Once the model has guessed wrong on the foreign word, the words around it suffer too. Here "a" became "the", and a word that was never said ("him") was added.
- The training data is mostly one language per recording. Mixed sentences are rare in the corpora speech models learn from, so the model has seldom heard the pattern.
Fine-tuning on mixed speech fixes this. The training set must also contain plain speech in the main language, so that accuracy on ordinary speech holds; that is the LibriSpeech figure above. The same method applies to any language pair your speakers mix.
Decent for everyone, perfect for nobody
How ASR models are trained explains where they fall short.
Speech recognition models are trained on very large corpora of transcribed speech. The training process averages across all of it, which is what makes the result good across a wide range of speakers and topics. It is also why the result is decent for everyone and perfect for nobody: specialist vocabulary is a tiny fraction of that data, and much of it is not there at all.
Professional vocabulary also keeps changing. New brand names appear, fields develop their own terms, and there are a great many niche fields: lawyers practising in Israel, architects using industry vocabulary, clinicians, engineers. A general model trained on last year's data cannot keep up with each of them.
Fine-tuning creates a variant of an ASR model in which your niche vocabulary has been trained in, to increase recognition accuracy for it. Depending on how far the vocabulary is from general speech, the improvement can be very large.
How it works
From choosing the base model to keeping the fine-tune current.
1. Choose the base model
The right base model is a function of your actual workload and the devices it needs to run on: a large model on a server, or a small one on a laptop or phone.
2. Create the training plan
Which vocabulary to target, how much data is needed, and how the fine-tune will be measured against the base model, using the same evals before and after.
3. Gather the data
Recordings from real speakers reading context-appropriate sentences, synthetic speech data, or a mix of both, built so the training sentences reflect how your vocabulary is actually used.
4. Train and evaluate
Fine-tuning the base model on your dataset and comparing word error rate on your vocabulary and on general speech, so accuracy on everything else does not quietly drop.
5. Deploy
Rolling the fine-tune out where it will be used: offline, behind an API, or both. See deployment options below.
6. Update periodically
Retraining as your vocabulary changes, with each new version checked against the previous one before it replaces it.
Deployment options
The same fine-tune can be deployed in several ways, to match how you actually use it.
Offline
Run on your own devices or servers with no network connection, for security-sensitive recordings that must never leave your environment.
Public-facing API
Hosted on a serverless inference service such as Replicate, for use from your applications or by your customers.
Internal proprietary API
Served as an authenticated API inside your own infrastructure, available to your team's tools and pipelines and nobody else.
Frequently asked questions
Can we upgrade our fine-tune later?
Yes. A fine-tune can be retrained with new vocabulary and new recordings, or rebuilt on a newer base model when a better one is released.
Can we specify our own vocabulary?
Yes. You define the terms. I work with you to create datasets of context-appropriate sentences, so the training data reflects your actual usage patterns rather than words in isolation.
Do we need recordings of our own staff?
Not necessarily. Real speakers give the best results for accents and code-switching; for vocabulary alone, synthetic speech data can carry much of the load. The training plan sets the mix.
Hiring me for ASR Fine-Tuning
A fine-tuned speech model for your vocabulary and your speakers, measured against the stock model on recordings it never trained on. My own fine-tunes took code-switched speech from 13.7% to 4.9% wrong words and technical dictation from 2.51% to 1.35% WER, with ordinary English unchanged.
Who it's for
- Teams whose transcripts keep getting the same terms wrong: product names, jargon, drug or part names
- Speakers with accents, or who switch languages mid-sentence, in any language pair
- Organisations whose recordings can't go to a cloud transcription API
- Developers building voice or dictation features on an open ASR model
What you get
- The fine-tuned model weights, yours to keep
- A word error rate report: base model vs fine-tune, on your vocabulary and on general speech
- The training dataset and the recipe that produced it, so it can be retrained
- A deployment: offline, an internal API or a hosted endpoint
- A retraining plan for when your vocabulary changes
What I need from you
- Your term list, or examples of what transcription currently gets wrong
- A sample of real recordings, if you have them (not always required)
- Where the model has to run: device, server or API
- One person who can check transcripts and answer vocabulary questions
How it runs
Typically 5–8 weeks, depending on how much data has to be gathered.
- Week 1
Scope and base model
The workload, the target terms and the devices it has to run on decide the base model.
- Week 1–2
Training plan and baseline
Which vocabulary to target, how much data, and the stock model scored on your test set before any training.
- Weeks 2–4
Data
Real recordings, synthetic speech or a mix, built from sentences that use your terms the way you use them.
- Weeks 3–6
Train and evaluate
Fine-tune, then compare WER on your vocabulary and on general speech so nothing else quietly gets worse.
- Weeks 5–8
Deploy and hand over
Roll it out where it will be used, with the weights, data and recipe handed over.
Training work has no public price list. Each model is scoped after a short call and quoted as a fixed price, paid against milestones.