Baboons
& AI Agents
The AI coding agent you choose matters more than the AI model inside it. We measured it — three times.
Independent benchmarks by ESL: the same tasks were given to two coding agents, Amp and Claude Code. From NP-complete quantum-compilation problems to a deliberately silly baboon-shooting browser game, the better-engineered agent finished faster, cost far less, and matched or beat correctness.
A frontier model (Claude, GPT, Gemini…) is the engine. The agent that wraps it — the harness — is the transmission, the steering and the driver: how it plans, reads your code, runs tools and checks its own work. Give two agents the identical model and task, and the better harness wins on time and dollars while matching quality. The Kaggle “New SDLC with Vibe Coding” whitepaper argues the harness carries the overwhelming share of the outcome — our results are consistent with that.
matches the 2026 academic definition
A May 2026 survey from UIUC × Meta × Stanford (Code as Agent Harness, arXiv:2605.18747) defines the harness almost word-for-word the same way: “the software layer that surrounds an LLM with tools, APIs, sandboxes, memory, validators, permission boundaries, execution loops, and feedback channels, thereby turning a stateless model into a functional agent.” The car metaphor above is not marketing spin — it is the field’s emerging consensus.
Hard quantum-compilation tasks in C++, including an NP-complete qubit-routing problem. Amp (fast “rush” mode) vs Claude Code (max effort), scored by hidden tests.
Result: Amp 39× cheaper, 24× faster, perfect 1.000 correctness.
See the quantum results → (slide deck — arrow keys / swipe)
“How do I know the grader is right?” We formally verified the benchmark’s checker: 12 machine-checked theorems and 247 automated proofs — the scoring is mathematically sound.
Result: the scores above are backed by machine-checked proof.
See the proofs → (slide deck — arrow keys / swipe)
Both agents got the identical prompt and the identical model and built the same silly browser shooter. The only variable left is the harness — and you can play both results.
Result: Amp 2.1× faster, 2.6× cheaper. You judge the games.
Play the face-off →In May 2026, a 66-page survey “Code as Agent Harness” (arXiv:2605.18747, UIUC × Meta × Stanford) formalized the exact thesis of this site: the harness — not the base model — is the bottleneck of agent autonomy. Its open problem #1 is “harness-level evaluation and oracle adequacy,” the complaint that almost every public benchmark (SWE-bench, OSWorld, AndroidWorld, MLE-bench…) scores only final task success and thereby “conflates the capabilities of the base model, the quality of the harness, the reliability of tools, the informativeness of feedback, and the difficulty of the environment.” The survey lists the six dimensions a real harness benchmark should measure. Here they are — with the ESL result that already hits each one.
The survey also names the Plan–Execute–Verify (PEV) loop as the control structure that separates a good harness from a brittle one: planning forms a contract over the intended change, execution runs inside a sandboxed and permissioned environment, and verification uses deterministic sensors — tests, linters, fuzzers, type checkers — to decide whether to accept, revise, or escalate. That is precisely the loop our two experiments stress-test, and precisely where Amp’s harness beat Claude Code’s on the same model. A harness that skips Verify is a harness that confuses motion with progress.
One more idea the survey makes explicit and ESL leans into: the harness is no longer just deployment infrastructure — it is becoming a distillation surface. Cursor trains on its own usage traces; OpenAI’s GPT-5.1-Codex-Max is trained on long-horizon coding interactions that mirror the Codex harness loop. The implication for buyers: the agent you pick today is also the data flywheel that shapes the model you get tomorrow. Measuring the harness now is how you avoid locking into a worse flywheel later.
After the quantum benchmark, we asked Claude Code running Fable 5 at maximum effort what it thought about its loss on the QMAP (qubit-routing) task — where Amp, running in its fast “rush” mode, scored a perfect 1.000 against Claude’s 0.889, at 62× lower cost. Its answer:
We replied: “Check again — QMAP is not a simple problem.”
Qubit routing is NP-complete — one of the hardest classes of problems in computer science. The agent in fast mode had quietly solved it, cheaper and faster, while the max-effort setup scored lower and then misjudged the task’s difficulty. That is the whole thesis in one exchange.
If two agents on the same model differ by 2–40× in cost and speed, then picking an AI coding agent by model name alone is leaving money and time on the table. The right way to choose is to benchmark the agents on your own tasks — and verify the results, not just eyeball them.
That is what ESL does: independent benchmark design, formal-verification pilots (Lean 4 / CBMC) and agentic-AI tooling strategy — evidence, not hype.
Daniel Liezrowice designs and runs independent benchmarks of AI coding agents — measuring what actually moves speed, cost and correctness in the SDLC, and separating harness effects from model effects.
ESL helps teams adopt agentic AI with evidence, not hype: benchmark design, formal-verification pilots (Lean 4 / CBMC) and tooling strategy.