Does Your Agent Need a Coding Model?
The model that wins at coding loses at running an agent. Not by a hair — by a clear, repeatable margin, on a benchmark built to isolate exactly the difference.
That single inversion is the whole story. Four open-weight models landed within the same window, every one of them wearing the word agentic somewhere on the box. Three are Qwen 3.5 fine-tunes pitched for agents. Google ships Gemma 4 and calls it an agent runner. We built a benchmark to test the claim on its own terms — not “can it write the diff,” but “does it behave like an agent” — and the result splits the field along a fault line the leaderboards never show.
Whether you’re standing up Hermes, Pi, Omnigent, or whatever harness you favor, the model underneath isn’t doing the job a pure coding model does. It looks similar enough that the distinction gets ignored. It shouldn’t be.
What a coder optimizes, and what an agent optimizes
A coding model is graded on one thing: is the diff correct, inside a harness that hands it a reproducible repo and runs the tests. Terminal-Bench, SWE-Bench, NL2Repo — all of them measure outcome in a well-marked environment. They are good benchmarks. They measure the wrong thing for an agent.
An agent lives in the open. It decides when the task is finished, refuses work it wasn’t asked to do, notices when a tool lied to it, and — least glamorous, most decisive — actually emits a final answer instead of dissolving into its own reasoning. None of that is diff correctness. None of it shows up on a coding leaderboard.
So we built TMB Agentic: ten scenarios, A01 through A10, one-shot, temperature zero, scored on a grid out of 500. No tools execute. The model is handed a realistic dev situation — repo tree, error context, session state — and asked to respond. What we score is cognition: does the model know how a good agent should behave? Stop criterion (A05). Scope discipline (A04). Catching a tool that reports success while writing to the wrong file (A10). Reactive investigation — run the failing command before reading code blind (A01). Multi-step coherence across files (A06).
These are agent virtues. A model can be brilliant at all of them and still be useless if it never finishes a sentence.
The inversion, in one table
Here is the field. The two columns that matter sit side by side: how each model does on a coding harness, and how it does on agent behavior.
| Model | Optimized for | SWE-Bench Verified | TMB Agentic /500 | Completion |
|---|---|---|---|---|
| Gemma-4-31B | agent runner | 52 | 421 (84%) | 10/10 |
| Ornith-1.0-35B | agentic coding | 75.6 | 404 (81%) | 10/10 |
| Gemma-4-26B | agent runner | — | 396 (79%) | 10/10 |
| Nex-N2-mini | agentic (Qwen FT) | — | 354 (71%) | 9/10 |
| Qwen-AW-35B | agentic (Qwen FT) | — | ~200 (44%) | 5/10 |
Read the top two rows again. Ornith-1.0 is a self-improving family trained specifically for agentic coding, post-trained on Gemma 4 and Qwen 3.5, and on the coding harnesses it doesn’t just beat Gemma — it laps it. 75.6 against 52 on SWE-Bench Verified. 64.2 against 42.1 on Terminal-Bench. By every coding measure its makers publish, Ornith is the better engineer by a wide margin.
Then you ask it to behave like an agent, and it lands second to a model it crushed at coding.
That gap — same model, opposite ranking depending on which question you ask — is the entire argument. Coding skill and agent behavior are not the same capability, and optimizing hard for one buys you nothing automatic in the other. Ornith’s RL sharpened the diff. It didn’t sharpen the decision of when to stop, or the instinct to run before reading. Those came from the base, not the fine-tune. The specialized training that makes it a coding champion is, on agent ground, invisible.
Where the “agentic” fine-tunes break
If the agentic label were doing real work, the two Qwen fine-tunes built explicitly for agents — Nex-N2-mini and Qwen-AgentWorld — would sit at the top. They sit at the bottom, and they get there by two different routes that are both worth seeing.
Qwen-AgentWorld is a 3B-active MoE trained to plan and reason before acting. It learned the first half and not the second. On half the tests it produces a hundred lines of Thinking Process: and never transitions to an answer — generation runs out mid-thought. The reasoning, where you can read it, is correct: it spots the decoy file, diagnoses the falsy-zero bug, gets the schema right. It just never says so. On the four tests where it does finish, it scores 92.5% — right at Gemma’s level. The cognition is there. It arrives only half the time.
Nex-N2-mini has the opposite tell: its internal monologue leaks straight into the output stream. Every answer opens with a paragraph of “We need answer to user. Need infer task. We don’t have actual tool access. Hmm…” before it gets to the point. On short, bounded tasks — A04, A05, A09, A10 — it scores 47-48, neck and neck with the top of the field. On A01, the one test demanding a long ordered sequence, it spends its entire token budget deliberating whether it’s allowed to include the fix’s code, and produces no list at all. Zero.
This is where completion rate stops being a footnote. A model whose reasoning is excellent but whose answer arrives one time in two is, in production, a model you cannot deploy without retry logic wrapped around every call. The leaderboards never measure this because in a coding harness the loop runs until it resolves. An agent doesn’t get that luxury — it has a turn, and either it produces something usable or it doesn’t.
The lesson sits plainly across both: training a model to reason before acting is not the same as training it to finish acting. The agentic fine-tune optimized the deliberation and forgot the exit.
The ceiling nobody clears
Lest this read as a Gemma advertisement: every model in the panel fails A06, the multi-step coherence test, at 8 to 10 out of 50. All of them. Gemma included.
The failure is identical each time. Given a function-signature migration across three files, the model confuses a call site — result = get_orders(db, user_id) — with a signature definition, and tries to “fix” a declaration that isn’t in the context. From there the error cascades: it invents a file, skips the real target, propagates nothing. This is the exact skill that separates “runs autonomously” from “needs a human at every step.” At the 26-35B scale, nobody in this panel has it. That looks like a size limit, not a fine-tune gap — and it’s the test that should govern whether you let any of these models touch a multi-file refactor unsupervised. None of them yet, is the honest answer.
So — does your agent need a coding model?
On this evidence, no. It needs a model that behaves like an agent, and the two are not interchangeable.
If your agent’s job is to drive a terminal coding CLI — OpenHands, OpenClaw, a Cline-style loop — then yes, Ornith’s coding edge is real and you want it, with a human watching the multi-file steps. But if the job is general agent work — routing, tool orchestration, knowing when a task is done and when a tool is lying — the coding crown is the wrong signal. Gemma 4-31B wins that ground, and it wins it because Google built it to, with answers three to ten lines long and no reasoning overhead to parse. Ornith’s general base carries it to a strong second despite a fine-tune that points the other way. And the two models that wear agentic most explicitly finish last, undone not by reasoning but by the inability to stop reasoning.
The word on the box tells you what the makers optimized. It does not tell you what they got. Benchmark the behavior you actually need, not the one the label advertises — they are, more often than the label admits, two different models.
Sophie, The Monocle Bear