Benchmarks & Evals
There are plenty of public benchmarks. Even a great one tells you little unless it measures your task. I develop evals and benchmarks for the specific workloads your AI pipelines run, so you can compare open- and closed-weight models on what actually matters to you, and keep comparing them as new ones are released.
A stable frame of comparison
The models keep changing. Your yardstick shouldn't.
New models arrive every few weeks, each with a set of leaderboard results. Those results are measured on someone else's tasks. Whether the new release is better at classifying your tickets, extracting your contract terms or following your output schema is a separate question, and the only way to answer it is to test it on that work.
An internal eval gives you a fixed frame of comparison that stays put while the world of generative AI models moves at a breakneck pace underneath you. When a new model is released, you run it through the same tests as the one in production and decide on evidence whether to switch.
What I do
From defining what "good" means for a task to running the tests and reporting the results.
Internal eval development
Building evals and benchmarks for your own tasks: the test set, the scoring method and the scripts to run it, owned by you and repeatable by your team.
Grading criteria
Working with your organisation to define the criteria models are graded on: objective technical measures such as accuracy and schema conformance, and subjective ones such as tone or usefulness, scored consistently.
Evaluating models for your pipelines
Setting up evals for the closed-source models behind your AI pipelines, and recommending which model to use for each stage based on the results rather than the marketing.
Open- and closed-weight comparisons
Benchmarks that put frontier APIs, open-weight models and your own custom models through the same tests for a specific internal task.
Upgrade decisions on new releases
When a new model comes out, re-running your evals to see whether it is worth upgrading to, what would change, and what it would cost.
Evaluation as an external service
For a defined workload, I run the evals and report on the results for you, as an independent party, on a one-off basis or each time a relevant model is released.
How an evaluation engagement runs
Scoped to one workload at a time.
1. Define the workload
Which task, which inputs, and what a correct or acceptable output looks like. Representative examples come from your own data.
2. Agree the criteria
The technical and subjective criteria, how each is scored, and how much each one weighs in the overall result.
3. Run and report
The candidate models run through the eval, with a written report on how each performed, where they failed, and what I recommend.
Hiring me for Benchmarks & Evals
An eval built on your workload, so you can compare models on what your pipeline actually does and rerun the same test whenever a new model is released.
Who it's for
- Teams choosing between models for a pipeline step
- Teams deciding whether to upgrade when a new model comes out
- Anyone about to fine-tune who needs a before-and-after measure
- Teams who want model evaluation run as an external service
What you get
- An eval set built from your own data
- Written grading criteria and weights, agreed with your team
- A runnable eval harness you keep
- A report on each candidate model: scores, failures and a recommendation
- Re-runs on new model releases, if you want them
What I need from you
- Representative examples from your workload
- The models you are considering, and API access if they are closed
- People who can say what a good output looks like
- Any constraints: cost, latency, where the model can run
How it runs
Typically 2–4 weeks per workload.
- Week 1
Define the workload
Task, inputs and what a correct or acceptable output looks like.
- Weeks 1–2
Agree the criteria
Technical and subjective criteria, how each is scored and weighted.
- Weeks 2–4
Run and report
Run candidate models and write up results and a recommendation.
For a fixed-fee model benchmark, credited in full to a fine-tune signed within 60 days. Repeatable eval suites are scoped after a short call.