Private benchmarks
Your job, your cases, your reviewers’ standard. We build the benchmark, run every model worth testing on it, and hand it over.
Which model can do this job?
A public score says how a model does on someone else’s work. The answer for yours comes from a test built out of it.
Most teams still choose a model without:
A test built from their own cases
A written standard for a correct answer
Scores they can check line by line
Cost per task next to the score
A re-run when a new model ships
The files to run it themselves
How it works
Four steps, a fixed scope, and a benchmark that stays yours.
- 1
Collect the cases
You send 50 to 150 real examples of the job, with the answers your team accepted. We can anonymise them first.
- 2
Write the rubric
Each case becomes a task. Your reviewers agree what a correct answer contains, line by line, and where your bar sits.
- 3
Run every model
Every model worth testing runs each task three times. Lines are checked by code where they can be, and by a judge model, checked against a person, where they cannot.
- 4
Hand over, then re-run
You get the report and the files. When a model worth testing ships, we run it again and send one page: switch or stay.
What you keep
A finished benchmark, not a seat on a dashboard.
The benchmark files
Tasks, rubric and grader as plain files. Run them in your own pipeline, with any model.
A ranked report
Score, interval, cost and time per task for each model, with every miss shown next to its reply.
A plain answer
Which model to use for the job, or a clear statement that none is ready yet.
Re-runs on release
One page each time a model worth testing ships: switch or stay, and why.