ASR Fine-Tuning
General speech recognition models are very good at general speech. They struggle with the words your work depends on: professional terminology, brand names, accents and speakers who switch languages mid-sentence. I fine-tune ASR models so they recognise your vocabulary, and deploy them wherever you need them to run.
Why fine-tune an ASR model?
Baseline accuracy is strong. Niche requirements are where it breaks down.
Baseline word error rate (WER) is now robust across the leading speech recognition models. For everyday speech in a major language, most of them will do. The gaps appear with niche requirements: accents the model heard little of in training, code-switching between languages in the same sentence, and, most commonly, niche vocabulary.
I have done this for my own dictation: my open-source ASR fine-tunes are small variants trained for one need each. I bring the same process to professional and business use.
Niche vocabulary
Legal, medical, technical and industry terms, product and brand names, internal project names and acronyms. The words that matter most in your transcripts are the ones a general model gets wrong.
Accents
Speakers whose accent is under-represented in the base model's training data see higher error rates. A fine-tune trained on their speech closes much of that gap.
Code-switching
Speakers who move between languages mid-sentence, such as Hebrew and English in an Israeli workplace, confuse models trained to expect one language at a time.
Decent for everyone, perfect for nobody
How ASR models are trained explains where they fall short.
Speech recognition models are trained on very large corpora of transcribed speech. The training process averages across all of it, which is what makes the result good across a wide range of speakers and topics. It is also why the result is decent for everyone and perfect for nobody: specialist vocabulary is a tiny fraction of that data, and much of it is not there at all.
Professional vocabulary also keeps changing. New brand names appear, fields develop their own terms, and there are a great many niche fields: lawyers practising in Israel, architects using industry vocabulary, clinicians, engineers. A general model trained on last year's data cannot keep up with each of them.
Fine-tuning creates a variant of an ASR model in which your niche vocabulary has been trained in, to increase recognition accuracy for it. Depending on how far the vocabulary is from general speech, the improvement can be very large.
How it works
From choosing the base model to keeping the fine-tune current.
1. Choose the base model
The right base model is a function of your actual workload and the devices it needs to run on: a large model on a server, or a small one on a laptop or phone.
2. Create the training plan
Which vocabulary to target, how much data is needed, and how the fine-tune will be measured against the base model, using the same evals before and after.
3. Gather the data
Recordings from real speakers reading context-appropriate sentences, synthetic speech data, or a mix of both, built so the training sentences reflect how your vocabulary is actually used.
4. Train and evaluate
Fine-tuning the base model on your dataset and comparing word error rate on your vocabulary and on general speech, so accuracy on everything else does not quietly drop.
5. Deploy
Rolling the fine-tune out where it will be used: offline, behind an API, or both. See deployment options below.
6. Update periodically
Retraining as your vocabulary changes, with each new version checked against the previous one before it replaces it.
Deployment options
The same fine-tune can be deployed in several ways, to match how you actually use it.
Offline
Run on your own devices or servers with no network connection, for security-sensitive recordings that must never leave your environment.
Public-facing API
Hosted on a serverless inference service such as Replicate, for use from your applications or by your customers.
Internal proprietary API
Served as an authenticated API inside your own infrastructure, available to your team's tools and pipelines and nobody else.
Frequently asked questions
Can we upgrade our fine-tune later?
Yes. A fine-tune can be retrained with new vocabulary and new recordings, or rebuilt on a newer base model when a better one is released.
Can we specify our own vocabulary?
Yes. You define the terms. I work with you to create datasets of context-appropriate sentences, so the training data reflects your actual usage patterns rather than words in isolation.
Do we need recordings of our own staff?
Not necessarily. Real speakers give the best results for accents and code-switching; for vocabulary alone, synthetic speech data can carry much of the load. The training plan sets the mix.
Is your transcription getting the important words wrong?
Send me examples of the vocabulary and speakers involved. I'll tell you what a fine-tune could fix.