// tmb scoreboard / methodology

How we test models.

What we measure, how we grade it, and where the limits are. The tasks themselves stay private.

Back to the scoreboard

From a task to a score

  1. 01

    Task

    A real professional request, identical for every model.

  2. 02

    Answer

    One answer per model, kept verbatim.

  3. 03

    Criteria

    The judge scores each criterion of a written grid.

  4. 04

    Totals

    The harness sums the points. The judge never does.

  5. 05

    Skills

    17 skills, each fed by one or more tasks.

  6. 06

    Categories

    5 categories, then the overall score.

What the scoreboard measures

42 tasks across five families, one per category. Each one is a request a professional would actually send, not a quiz with a single right token.

Agentic

10 tasks

English

Choosing and sequencing tool calls to complete an operational request.

Coding

10 tasks

French instructions

Python, general code, debugging, React, Swift and refactoring, graded on correctness and craft.

Planning

4 tasks

French

Breaking down a project, writing a hand-off spec, making judgment calls under constraints.

Reasoning

10 tasks

French

Hard problems under constraints: a single valid solution, chained calculations, audits with traps that look right.

Technical

8 tasks

French

Legal and compliance analysis, reasoning and calculation, business and creative writing.

An English track of 7 translated tasks (coding, reasoning, technical) runs alongside. It is not part of the published score yet.

Running a task

  • Every model receives the same instructions. Tool definitions are sent whenever a task involves tools.
  • Each model answers once per task, with the sampling settings we set for that model or that task.
  • Before a run, a short control task with a known answer checks the setup for every combination of settings the run will use. If it fails, the run does not start.
  • Answers are checked for harness artifacts (a provider refusing tools, a truncated or leaked reasoning block). Repeated artifacts stop the run, so a broken setup never counts as a bad answer.
  • The answer is stored verbatim. Nothing is edited before grading.

Grading

  • Each task has a written grid: weighted criteria, each with described levels.
  • The reference judge, Panel (moyenne des juges), scores criterion by criterion and flags factual errors. It never computes a total.
  • The harness sums the points, capped per criterion. Arithmetic is code, not model output.
  • A category is one bench: its score is the points obtained over the points available on every graded task of that bench. The overall score averages the categories, each weighing the same, and is shown when at least four of five are covered. Skills are read across benches from the same grades.

We test the judge too

A score is only as good as its judge. We regularly re-grade the same answers with other judges and measure how far they drift from the reference, on the overall ranking and skill by skill.

Second judge, GLM 5.3

0.94

Rank correlation of the overall score with the reference over 92 models. Average gap of 2.2 points.

Criterion by criterion

0.85

Agreement on 3,649 individual grades. Close on rankings, looser on single criteria: one task never decides a rank.

Rejected judges

out

Candidates that diverged too far from the reference were dropped rather than averaged in.

Cost and speed

Cost per task is what each model actually cost us on our tasks, reasoning included, at the provider price in force when it ran. It is measured, not estimated, and it is the same figure as in the bench ranking. Local models run on our own hardware and are not priced. Speed is the median decode rate we measured, in tokens per second.

Out of competition

Reference models are frontier systems we run as calibration points. They appear in the ranking for context. They never get a rank and are never recommended.

Agentic: work done alone

A separate scoreboard, graded by code with no judge. It does not enter the overall score above: a strong agent is not always a strong writer, and the reverse. See the agentic scoreboard.

The task

18 tasks, easy to hard, each targeting one capability: tool use, planning, recovery after a failure, respecting constraints, resisting prompt injection, and stopping at the right time, which includes asking when the request is ambiguous. The model gets a real Linux sandbox with the network cut, a shell, file tools and up to 30 turns. Each task runs twice.

The check

The host checks the result in a fresh container, against hidden expected values, without ever running the model's code as a judge of itself. Three tasks hide a secret behind injected instructions; reading or sending it is caught in the whole trajectory. Every task is attacked by a cheating probe before it enters the bench.

The score

Models are ranked by clean success: a task solved while leaking the secret or breaking a constraint counts as a failure. Six public Terminal-Bench tasks run as an external control, never scored, so that our own tasks cannot flatter a model. Qualified means no leak, no broken constraint, 85% clean success and 50% on the control.

Decision: routing a request

A router first asks a small decision model which skill a request needs. This bench measures those models, graded by code. See the decision scoreboard.

The requests

208 requests in French, written like real ones, each labelled with the skill it needs: a main set of 160 and a holdout of 48 measured apart. The model chooses among 18 skills, two of which are never the right answer.

The call

The exact question the production router asks, built by the same code, sent to each model at its own provider. Below the confidence threshold the router falls back to a default skill, and so does the score.

The score

End-to-end accuracy, then holdout, then latency. An online decision model is the reference. Qualified means within 2 points of it on both sets, never wrong while sure, and fast enough for the router. Above 1 s it is tagged slow.

Known limits

Language

The agentic family is in English, the other four in French. An English track of 7 translated tasks runs alongside, not scored yet.

One reference judge

Any judge has preferences. We track them through calibration, and keep frontier models out of competition so the judge never ranks its own family.

Blind grading, but not invisible

The judge sees the grid and the answer, never the model's name. A model that names itself in its answer can still be recognised.

Saturation

Above roughly 97 points, differences are smaller than the judge's variance. That is why ties within 1 point are shown as ties.

One answer per task

A single sample per task keeps costs sane, but a model can have a good or bad day on any one task. Averages over many tasks absorb most of it.

Speed depends on hardware

Local speeds are measured on our own machines and quantizations. On different hardware they will differ.

What we keep private, and why

Prompts, grids and answers stay private. Once published, tasks end up in training data and a benchmark stops measuring anything but memory. Keeping them private is what lets the same tasks stay meaningful over time. If you want a model evaluated, get in touch.

Protocol v2. Data 2026-11-18.