Agent-Harness Benchmark · Quantum-Compiler C++ Kernels · mid-2026
The Harness
Beats the Model
The #1 model lost — then admitted the task it failed
wasn't basic at all.
Quantum circuit routingMagic-state schedulingPulse-level control · GRAPETensor-network contraction
vs Claude
Amp · rush  vs  Claude Code + Fable 5 · --effort max  ·  4 from-scratch quantum-computing optimization problems in C++ · warm ×3 · identical hidden tests
The result
A fast, cheap agent beat the world's #1 model.
Correctness
4/4
Claude solved 3/4 at max effort
Total cost
$1.93
vs ~$75 — 39× cheaper
Speed / iter
49s
vs 1,181s — 24× faster
The cheaper agent didn't just win on price — it won the one task that needed real algorithmic depth, exactly where "more effort" was supposed to help.
On the record
Even the model agreed.
Claude Fable 5
"A fast, cheap agent is the right tool here — max-effort Fable is massive overkill."
Me
"Are you sure QMAP is a basic problem? Check again."
Claude Fable 5
"QMAP is not a simple problem — qubit routing is NP-hard… That makes the failure worse, not better."
The model that lost explained, in its own words, why losing mattered. (Live exchange paraphrased; the concession is written verbatim in Fable's own post-run review.)
The thesis
The model is the engine.
The harness is the car.
Agent = Model + Harness

An LLM just predicts text. The harness is everything around it — how it reads your repo, which tools it calls, how it manages the context window, how it runs tests, reads errors, and, crucially, knows when to stop.

Harness > Model.
The analogy, in full
Analogy: a car and its engine is like a coding agent and its LLM. The engine (the LLM) sits at the center and powers every surrounding system/capability; a shared foundation supports both.
Agent = Model + Harness
The model is ~10% of what makes an agent work.
10%
Model
90%
Harness
Industry framing — Kaggle, The New SDLC with Vibe Coding (Fig. 7). Our benchmark is independent evidence consistent with it; we did not measure 90/10 ourselves.
Anatomy of a harness
The car, the road & the traffic laws.
Cloud Infrastructure
managed runtimes · deploy config · service & scaling
Developer Interface
CLI / IDE · session & memory · eval & testing · observability
Framework
instructions & rules · tools & MCP · orchestration · guardrails & hooks
Core
LLM (~10%)
Rush vs Fable Max differ almost entirely in these outer rings — not the engine.
The contenders
Two products you'd actually choose.
rush mode
Amp's fast, low-cost tier. Listed as running GPT-5.5 on Amp's site at the time of testing.
Claude Code + Fable 5
--effort max — maximum reasoning effort. #1 on SWE-bench Verified (~95%) at test time.
Model attribution is time-scoped: each label reflects the vendor's product as advertised in mid-2026.
Methodology
Same tasks. Same tests. Same loop.
  • 8 header-only C++/Python tasks — from-scratch, no starter code.
  • Hidden weighted target tests compiled with g++ (MSYS2 15.2.0) — reward = weighted fraction passed.
  • Warm 3-iteration loop — each iteration improves the previous (no cold restart); only the prior reward scalar is fed back.
  • Baselines — oracle (all pass) + zero (empty) verified before every run.
Fair comparison = the 4 tasks where both agents ran to completion. Claude's other 4 were blocked by a monthly spend limit (billing, not capability) and excluded.
How we measure
Seven axes, not one.
CS · completion score
Mean reward across tasks (weighted fraction of hidden targets passed). 1.000 = everything solved.
RR · resolved rate
Fraction of tasks fully solved (reward = 1.000). Stricter than CS — no partial credit.
ES · efficiency score
Rewards fewer lines of code for the same result. Less code → higher ES.
LOC
Net non-blank lines in the produced solution (include/ for C++, src/ for Python).
time · cost
Agent wall-clock per iteration and provider-reported USD, logged every run.
asm
Assembly badge — flags any .s/.S/.asm or inline asm. Both agents were allowed to use it.
Correctness is table stakes. The interesting signal lives in cost, speed, and lines-of-code for the same passing result.
The pipeline
Automated, objective, cheat-resistant.
Task
instruction.md + hidden targets
→
Agent
rush / Fable max writes solution in a clean workdir
→
Compile + Grade
g++ runs weighted target tests
→
Capture
reward · cost · time · LOC · asm
→
Report
per-task tables + charts
Agents never see the target tests — only whether the previous attempt scored higher. That isolates harness discipline from memorized boilerplate.
The problems
We tested the kernels, not the coursework.

Each grand-challenge quantum-compiler system (15k–200k LOC in the wild) was distilled to its hardest algorithmic kernel and wrapped as a header-only C++ task graded by hidden tests — objective and cheat-resistant, while preserving the real depth: graph search, heuristics, scheduling, numerical optimization.

Difficulty ratings (9/10, 10/10, 11/10) come from an independent problem taxonomy — not from us.
The four quantum optimization kernels — routing · scheduling · control · contraction — are detailed next. A JSON parser (jsonmini) served as a normal-software control; the head-to-head cost/quality numbers use the tasks both agents completed.
Quantum optimization · 1 / 4
Qubit routing  cpp-qmap
NP-complete Difficulty 9/10 A* · SABRE · token swapping
GivenA logical circuit (CNOTs) + a hardware coupling graph with limited connectivity (~20 qubits).
MinimizeInserted SWAP gates, circuit depth and total gate count needed to make every 2-qubit gate hardware-legal.
Why hardEvery extra SWAP adds error & depth on noisy hardware. The SWAP-selection sub-problem generalizes token swapping, itself NP-hard (Bonnet et al.) and conjectured PSPACE.
Key references Siraichi, Santos, Collange, Pereira — Qubit Allocation, CGO 2018 · Li, Ding, Xie — Tackling the Qubit Mapping Problem for NISQ-Era Devices (SABRE), ASPLOS 2019 (proved NP-complete) · Cowtan et al. — On the Qubit Routing Problem (t|ket⟩), TQC 2019.
Verify against: Qiskit SABRE · TKET · Cirq routing.
q0 q1 q2 q3 q4 q5 CNOT q0·q5 (not adjacent) ⇄ SWAP
Non-adjacent 2-qubit gates force SWAP chains along the graph.
Quantum optimization · 2 / 4
Magic-state scheduling  cpp-ftsched
NP-hard Difficulty 8–10/10 lattice surgery · RCPSP-like
GivenA Clifford+T circuit as a DAG + magic-state factory throughput and surface-code patch layout.
MinimizeMakespan / space-time volume under factory-rate and dependency constraints.
Why hardT gates dominate fault-tolerant cost; scheduling their magic-state consumption is like resource-constrained project scheduling (NP-hard) on a QEC substrate.
Key references Bravyi & Kitaev — Universal QC with ideal Clifford gates and noisy ancillas (magic states), PRA 2005 · Herr, Nori, Devitt — Optimization of lattice surgery is NP-hard, npj QI 2017 · Litinski — A Game of Surface Codes, Quantum 2019 · Fowler et al. — Surface codes, PRA 2012.
T1 T2 T3 stall factory emits magic states at rate r makespan
Under-provisioned factories stall the T-gate schedule.
Quantum optimization · 3 / 4
Pulse-level control · GRAPE  cpp-pulse
Non-convex · high-dim optimal control Difficulty 11/10 bosonic · gradient ascent
GivenDrift + control Hamiltonians, a target unitary/gate, and N discrete time slices.
MinimizeInfidelity (reach ≥99.9%), pulse count, duration and accumulated error.
Why hardNot an NP decision problem — a continuous, high-dimensional, non-convex optimal-control landscape. No global-optimum guarantee once amplitudes are bounded / piecewise-constant; controllability must hold.
Key references Khaneja, Reiss, Kehlet, Schulte-Herbrüggen, Glaser — Optimal Control of Coupled Spin Dynamics by Gradient Ascent (GRAPE), J. Magn. Reson. 2005 · Glaser et al. — Training Schrödinger’s cat: quantum optimal control, EPJ D 2015 · Ma, Puri, Schoelkopf, Devoret, Girvin, Jiang — Quantum control of bosonic modes, 2021.
fidelity → 99.9% time (N slices) u_x(t) u_y(t)
Piecewise-constant controls tuned by gradient ascent to hit target fidelity.
Quantum optimization · 4 / 4
Tensor-network contraction order  cpp-tncon
NP-hard treewidth-bound exp blow-up
GivenA tensor network: a graph of tensors sharing bond indices (bond dim χ).
MinimizeFLOPs and peak memory of the full contraction — both set by the order of pairwise contractions.
Why hardOptimal order maps to minimizing the line-graph treewidth (Markov & Shi); computing treewidth is NP-complete, and cost is exponential in treewidth. This is how classical machines simulate/benchmark quantum circuits (e.g. Sycamore).
Key references Markov & Shi — Simulating Quantum Computation by Contracting Tensor Networks, SIAM J. Comput. 2008 · Dumitrescu et al. — Benchmarking treewidth (optimal sequence NP-complete), PLoS ONE 2018 · Gray & Kourtis — Hyper-optimized tensor network contraction, Quantum 2021.
Note: this kernel is not part of the head-to-head cost table (control-run only).
A B C D E χχχ order A cost ~ χ³ order B cost ~ χ⁶ (×10⁶)
Same network, different order → cost differs by orders of magnitude.
Why these tasks
They separate agents, not just models.
  • Large search spaces → graph algorithms & heuristics
  • Priority queues, cost functions → algorithmic judgment
  • DAG scheduling under constraints → correctness bookkeeping
  • Numerical optimization → cold-start robustness
  • Hidden edge cases → test / verify loops
These are the exact places where context management, tool design, and knowing-when-to-stop decide the outcome.
Scoreboard · 4-task intersection
Warm ×3 · reward 1.000 = fully solved
TaskAmp finalAmp $Claude finalClaude $Cost ×
cpp-qmap · routing1.000$0.460.889$28.7462×
cpp-pulse · GRAPE1.000$0.421.000$13.96+TO33×
cpp-ftsched · magic-state1.000$0.571.000$12.5322×
cpp-jsonmini · JSON1.000$0.481.000$20.1742×
ALL 41.000$1.930.972$75.4039×
~39× cheaper & ~24× faster per iteration, at equal-or-better correctness. (~$86 if the one timed-out cold run is estimated in.)
Cost & speed
Two orders of magnitude apart.
Total cost — 4 tasks
Amp rush
$1.93
Fable max
$75.40
Avg time / iteration
Amp rush
49s
Fable max
1,181s
39× cheaper · 24× faster — for equal-or-better results.
Correctness · reward per task
One task decided it: qmap.
qmap · Amp
1.000
qmap · Claude
0.889
pulse · both
1.000
ftsched · both
1.000
jsonmini · both
1.000
Claude — max effort, 3 tries, ~$29 — never passed target_08_final_mapping. Amp solved the whole task cold in 91s for $0.13.
Efficiency · net lines of code (final iter)
Less code, same correctness.
qmap
150
qmap
460
pulse
41
pulse
468
jsonmini
261
jsonmini
432
Shared 4 tasks: Amp 558 LOC vs Claude 1,466 (~2.6× less). All 8: Amp totals 890 LOC at ES 1.000. ftsched: both converged to the same 106 LOC.
Warm-iteration progression · mean reward
1.0 0.5 0 iter 1 iter 2 iter 3 Amp Fable
pulse (Amp): it1 0.235 → it2 1.000 → it3 1.000 — the harness learned from its own test feedback. Fable held flat; Amp dipped on the cold start, then climbed past it.
The full run · all 8 tasks
Twice the work, a fraction of the cost.
Amp rush
8 / 8 · $4.59
CS 1.000 · RR 1.000 · ES 1.000 · 52s mean. Finished every task.
Claude Fable 5 · effort max
4 / 8 · $75.40
4 solved; the other 4 stopped when Claude Code hit its own monthly spend limit — billing, not capability.
Reward by config × task  ·  all 8
ftsched
jsonmini
pulse
qmap
strutil
tncon
logfmt
textkit
Amp
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Claude
1.00
1.00
1.00
0.89
blk
blk
blk
blk
blk = billing-blocked (monthly spend cap) — not a capability failure. Green = solved, lime = partial (qmap 0.89).
The full run · dashboard
Completion, resolved, efficiency — and speed vs quality.
CS · RR · ES  (all 8 tasks)
Amp CS
1.000
Amp RR
1.000
Amp ES
1.000
Cl CS
0.486
Cl RR
0.375
Cl ES
0.211
Claude's aggregate is dragged down by its 4 billing-blocked tasks. On the 4 it actually ran: CS 0.972 · RR 0.75.
Speed vs quality · all 8  (top-left = ideal)
1.0 0.5 0 0s 650s 1300s mean run time — left / faster is better Amp · 52s · 1.000 Fable · 1,181s · 0.486
The dog that didn't bark
Assembly was allowed. Neither agent used it.

The prompt explicitly permitted dropping to assembly where suitable — and required it to be flagged if used. Across all 8 optimization-heavy tasks (SWAP routing, GRAPE pulse control, fault-tolerant scheduling, tensor contraction), the asm badge never lit up: zero .s/.S/.asm files and no inline asm from either agent.

Amp rush
0 assembly
Header-only, high-level C++.
Claude Fable 5
0 assembly
Header-only, high-level C++.
These tasks were won and lost on algorithms, harness discipline, and iteration — not micro-optimization. Modern agents default to readable, high-level code, and here it was enough.
Interpretation
Why the cheap agent won.
  • The cost gap is not a warm-mode artifact — it's already ~33× on the cold first iteration alone.
  • "More effort" spent ~40× the money refactoring already-working code and over-engineering a search heuristic…
  • …while never fixing a simple mapping bug that Amp's harness caught and closed on its first cold run.
The scaffolding — not the model —
did most of the work.
The one honest Claude win
Max effort buys cold-start robustness.

On pulse (genuinely novel numerics), Amp failed cold (0.235) and needed iteration 2 to recover. Claude solved it cold on the first try.

The honest, quotable takeaway: max effort buys cold-start robustness on novel problems — at ~33× the cost.
We report this because a benchmark you can trust names the places it lost, too.
The field agrees
This isn't a one-off.
90 / 10
harness vs model — Kaggle, New SDLC
6×
perf gap from harness alone — Meta-Harness (Stanford/MIT)
~40×
cost swing by scaffold — Sentient, ICML 2026
52.7 > 52.0
weak model + strong scaffold beats strong model — Confucius CCA
10–20 pts
same model, harness-only swing — Scale / Digital Applied
3.8→12.5%
ACI without new weights — SWE-agent, NeurIPS 2024
"The models are converging.
The scaffolding is diverging."
Limitations — we go first
Honest caveats.
  • n = 4 — a small, hard set; one task (qmap) decided the correctness delta.
  • Not effort-matched — Amp's fast tier vs Fable's most expensive knob.
  • Claude's other 4 tasks were spend-limit blocked, not failed.
  • USD conflates model price with token efficiency.
  • METR: harness gains are real but task-dependent — sometimes a coin flip.
Rebuttal: we didn't benchmark a model — we benchmarked two products you'd actually choose. And the loser's own write-up agrees qmap is the real signal.
What to do Monday
Buy the harness, not the leaderboard.
  • Stop shopping for the biggest model.
  • Evaluate the harness on your tasks — context management, tool design, test/verify loops, cost discipline.
  • The leaderboard number is the least useful number on the page.
Run the candidates on your code.
Now the research agrees - arXiv:2605.18747
A 2026 survey names what this benchmark
has been measuring all along.

A May 2026 UIUC x Meta x Stanford survey, Code as Agent Harness, formalizes the exact thesis of this deck: the harness - not the base model - is the bottleneck of agent autonomy. Its open problem #1 is "harness-level evaluation and oracle adequacy": public benchmarks (SWE-bench, OSWorld, MLE-bench) score only final task success and "conflate the capabilities of the base model, the quality of the harness, the reliability of tools, the informativeness of feedback, and the difficulty of the environment." The six dimensions the survey says a real harness benchmark should measure - and the ESL result that already hits each one:

01 - Trajectory efficiency
39x / 24x
cheaper & faster - tokens, tool calls, wall-clock
02 - Verification strength
12 + 247
Lean 4 theorems + CBMC proofs behind the grader
03 - Recovery ability
1.000 vs 0.889
on NP-complete QMAP - self-repair where the other stalls
04 - State consistency
controlled
baboons: identical model + prompt - only the harness varies
05 - Replayability
playable
both games ship as the artifact - re-run and judge yourself
06 - Safety / governance
formal-verify
"human-in-the-loop safety as harness state" - auditable
"The bottleneck of autonomy is not only the reasoning ability of the base model, but also the reliability of the system that connects model outputs to long-horizon actions and persistent states." - Sec 1, Code as Agent Harness
Survey: arxiv.org/pdf/2605.18747. Also names the Plan-Execute-Verify loop this deck stress-tests, and the harness-as-distillation-surface trend (Cursor RL, GPT-5.1-Codex-Max) - measuring the harness now is how you avoid locking into a worse flywheel.
vs
A #1 model is only as good as
the harness you run it in.
Local agent-harness benchmark · C++ · full per-task data available · mid-2026
ESL logoPrepared by Daniel Liezrowice · linkedin.com/in/liezrowice · ESL — AI SDLC consultants · www.eswlab.com
1 / 27
← Swipe to navigate →