// tmb scoreboard / decision
Which model routes the request.
Before a router picks a model, a small decision model reads the request and names the skill it needs. It runs on every request, so it has to be right and fast.
- Models evaluated
- 14
- Qualified for production
- 8
- Requests per model
- 208
- 18 skills to choose from
- Graded by
- Code, no judge
- data 2026-11-03
What “qualified” means
A qualified decision model can route production traffic. It needs all of these:
- Within 2 points of the reference on the main set and on a holdout set measured apart.
- Never wrong while sure: no wrong skill given with a confidence of 0.9 or more, because the router would trust it.
- Answers before the router gives up (15 s); after that, the router falls back to a default skill.
- Tagged slow above 1 s at p95: still usable, but every request pays that wait.
Ranking
End-to-end accuracy: the skill the router finally uses, low-confidence answers falling back to the default. Ties go to the holdout, then to the faster model. The reference is shown in place, out of competition.
#Model AccuracyHoldout p95Median
- Sub:Snap Reference 100.0%
1 vela3-turbo-bf16 Qualified Slow 100.0%
2 d3-ultra-bf16 Qualified Slow 100.0%
3 d3-ultra-q8h16 Qualified Slow 100.0%
4 ~typesafe/jev-latest Qualified 99.4%
5 d3-flash Qualified Slow 98.8%
6 vela3-flash-bf16 Qualified Slow 98.8%
7 d3-mini Qualified Slow 98.1%
8 vela3-mini-bf16 Qualified Slow 98.1%
9 d1-3b more than 2 pts below the referenceholdout more than 2 pts below the reference 91.9%
10 d3-lite more than 2 pts below the referenceholdout more than 2 pts below the reference 89.4%
11 vela3-lite-bf16 more than 2 pts below the referenceholdout more than 2 pts below the reference 89.4%
12 d3-nano more than 2 pts below the referenceholdout more than 2 pts below the referencewrong while sure 88.7%
13 typesafe/jev-latest more than 2 pts below the referenceholdout more than 2 pts below the referenceslower than the router waits n/a
How it is measured
The exact question the production router asks, sent to each model at its provider, one call per request. Provider errors are counted apart and never scored as a wrong decision.