The Harness Is the King

Amp vs Claude Code on Meta Superintelligence Labs' SWE-sweep bug benchmark, same model on both arms, plus a Parasoft C++test control

4/60 vs 2/60

Reference bugs resolved. Identical model (claude-opus-5-5) in both arms. Only the agent harness differs.

6.7%

Amp's score on this 60-bug subsample. The best official SWE-sweep leaderboard entry: 4.7% at $7,230 per full run.

Daniel Liezrowice CTO, ESL - Engineering Software Lab

Embedded software consultancy, Parasoft and Perforce partner in Israel for over 20 years. We benchmark agents with the same evidence discipline we use for safety-critical code.

linkedin.com/in/liezrowice

What is SWE-sweep?

Meta Superintelligence Labs: can agents autonomously find and fix bugs in large codebases?

• 100 real repositories, 4,100 confirmed bugs. Built from issue–pull request pairs; each repo snapshot is a commit where the maximum number of bugs coexist.
• Open-ended task. The agent gets the repository at a fixed base commit and one broad instruction: find and fix as many bugs as you can. No issue text, no filenames, no line ranges, no hints.
• Brutal scoring. Every candidate fix runs against hidden fail-to-pass tests from the upstream PR, plus the repository's whole existing test suite. One regression zeroes the repo's reward.
• ICLR 2026 preprint by Lieret, Ma, Wei, Yang, Press and colleagues at Meta Superintelligence Labs, Harvard, UW and Stanford.

How a bug is scored

Two test sets per bug, and a gate that punishes collateral damage

{
  "reward": 0.2,            # fraction of the 5 reference bugs resolved
  "n_resolved": 1.0,        # all hidden fail-to-pass tests pass
  "visible_ok": 1.0,        # the repo's original test suite still passes
  "n_regressions": 0.0,     # any value > 0 zeroes the whole reward
  "agent_patch_applied": 1.0,
  "agent_build_ok": 1.0,
  "grader_failed": 0.0
}

• A bug counts only if every hidden test for it passes and the untouched upstream test suite still passes.
• The agent is explicitly told to avoid regressions. Breaking existing behavior on one line can erase a correct fix elsewhere in the same repo.
• This is why a plausible-looking 400-line patch can score exactly 0.

What the authors say about harnesses

From the official SWE-sweep FAQ: does the scaffold matter?

• The benchmark's own authors found harness choice does not separate the big scaffolds at their resolution: Claude Code, Codex and mini-swe-agent land in the same band.
• That is exactly the question we set out to test at ESL, but head to head, same model, production harnesses, with the SWE-sweep verifier as the referee.
• And note rank 5: Opus 5 (xhigh) in the mini-swe-agent harness scores 1.3%, while the same Opus-class model in Amp's harness scored multiples of that in our runs.

Academic context: the harness debate

This benchmark is one data point in a 2026 research conversation

Citation for SWE-sweep: Lieret et al., 2026. SWE-sweep: Can Agents Autonomously Find and Fix Bugs? Preprint. Full BibTeX on swesweep.com.

The question: same model, different harness

If two production agent harnesses run the identical LLM, does the harness decide the outcome?

You are given a repository that contains
bugs. Find as many bugs as you can and
fix them. Output your changes as a
unified diff.

You will not be told how many bugs
there are, where they are, or what
kind of bugs they are.

The harness we built around both agents

Cheating is impossible by construction: the agents never see the tests

# timeout 3600s -> exit 124 means the 1h cap expired; still a complete measurement
cd "$REPO" && timeout 3600 "$AGENT_CLI" < prompt.md > session.log
git diff > "patches/${AGENT}__${REPO}.diff"
echo "[$AGENT/$REPO] exit=$? duration=${SECONDS}s diff_lines=$(wc -l < ...)"

# per bug: run hidden fail-to-pass tests + the repo's original suite
run_verifier "$TASK_DIR" --patch "patches/${AGENT}__${REPO}.diff"
cat .../verifier/reward.json   # n_resolved, n_regressions, visible_ok

• Patches, session logs and reward JSONs are kept for every one of the 24 agent-repo runs (plus the earlier cost-tier run).
• exit=1 rows on the results page are Claude Code hitting its own plan spend cap; those patches were still graded by the verifier.

Where this lands on the leaderboard

Our subsample scores mapped next to the official full-run entries (directional: 60 bugs, not 4,100)

• Each resolved bug is worth 1.67 points on our 60-bug subsample, so 4-vs-2 is a real gap but not statistical proof. It is directional, and it matches the prior quantum-kernel benchmark (slide 16).
• Our runs used a 1-hour cap per repo; official entries ran the full 100-repo benchmark. The comparison is a placement, not a record claim.
• Opus-class model + mini-swe-agent officially scores 1.3%. The same model class in Amp's harness scored 5× that here.

What the agents got right: click (Python)

Amp, 80-line patch, scored 1/5: two Python 3 API bugs in the bundled examples

--- a/examples/aliases/aliases.py
+++ b/examples/aliases/aliases.py
@@ -27,7 +27,7 @@ class Config:
         parser.add_section("aliases")
         for key, value in self.aliases.items():
             parser.set("aliases", key, value)
-        with open(filename, "wb") as file:
+        with open(filename, "w") as file:      # configparser.write needs text mode
             parser.write(file)

--- a/examples/imagepipe/imagepipe.py
+++ b/examples/imagepipe/imagepipe.py
@@ -89,7 +89,7 @@ def open_cmd(images):
             if image == "-":
-                img = Image.open(click.get_binary_stdin())
+                img = Image.open(click.get_binary_stream("stdin"))

• Small, surgical, exactly at the failing lines. The hidden tests passed and the original suite stayed green.
• This is the harness discipline pattern we see across all four Amp wins: locate, minimally fix, stop.

Claude Code's wins: tokio (Rust)

A real, minimal, correct fix. The harness found it; the model wrote it.

--- a/tokio-macros/src/entry.rs
+++ b/tokio-macros/src/entry.rs
@@ -303,7 +303,7 @@ fn build_config(
-                    "flavor" | "worker_threads" | "start_paused" => {
+                    "flavor" | "worker_threads" | "start_paused" | "crate" => {
                         format!("The `{}` attribute requires an argument.", name)

--- a/tokio-stream/src/stream_map.rs
+++ b/tokio-stream/src/stream_map.rs
@@ -578,15 +578,15 @@ where
-        let mut ret = (0, Some(0));
+        let mut ret: (usize, Option) = (0, Some(0));
             for (_, stream) in &self.entries {
-            ret.0 += hint.0;
+            ret.0 = ret.0.saturating_add(hint.0);

• Two latent bugs in one patch: the macro's requires-argument list was missing "crate", and size_hint could overflow on huge stream maps.
• 553 diff lines total, scored 1/5 on the hidden tests with a clean regression gate.

Claude Code's wins: rxjava (Java)

The original code called onError(null) recursively, re-entering the null branch forever

--- a/src/main/java/io/reactivex/internal/operators/flowable/FlowableCreate.java
+++ b/src/main/java/io/reactivex/internal/operators/flowable/FlowableCreate.java
@@ -619,7 +619,7 @@ public final class FlowableCreate extends Flowable {
             if (e == null) {
-                onError(new NullPointerException(
+                e = new NullPointerException(      # assign, do not recurse with null
                     "onError called with null. Null values are generally not allowed in 2.x operators and sources."));
             }
             error = e;

• The bug: onError(e) with e == null builds a NullPointerException and passes it back through onError(null), recursing instead of storing it.
• The fix assigns the exception instead of recursing. One line, judged by hidden behavioral tests. This is the exact bug class where the Parasoft control found zero coverage (slide 19).

What the agents failed: z3 (C++), 399 lines, scored 0

Amp burned the full hour, produced a plausible 399-line patch, and every hidden test still failed

--- a/src/api/api_ast.cpp
+++ b/src/api/api_ast.cpp
@@ -1326,6 +1326,7 @@ extern "C" {
             case OP_SEQ_INDEX: return Z3_OP_SEQ_INDEX;
+            case OP_SEQ_LAST_INDEX: return Z3_OP_SEQ_LAST_INDEX;
             case OP_SEQ_TO_RE: return Z3_OP_SEQ_TO_RE;
@@ -1345,7 +1346,7 @@ extern "C" {
-            case _OP_REGEXP_FULL_CHAR: return Z3_OP_RE_FULL_SET;
+            case _OP_REGEXP_FULL_CHAR: return Z3_OP_RE_FULL_CHAR_SET;

• exit code 124: the 1-hour cap expired mid-exploration. The bug surface (string theory solver internals) needed longer-horizon search than either arm got.
• Claude Code wrote 1,006 diff lines on z3 and also scored 0. On these behavioral solver bugs, more output is not more resolution.
• Failure mode: exploring a 100k-line C++ codebase without a falsifiable hypothesis per edit. The hidden tests never got a chance to pass.

The regression gate: roslyn (C#), zeroed

One resolved bug, one broken existing test, reward 0.0

--- a/src/Compilers/CSharp/Portable/Lowering/LocalRewriter/
+++ b/LocalRewriter.DecisionDagRewriter.cs
@@ -924,8 +924,8 @@ private void EnsureStringHashFunction(...)
                     StringPatternInput.String => ...SynthesizedStringHashFunctionName,
-                    StringPatternInput.SpanChar => ...SynthesizedReadOnlySpanHashFunctionName,
-                    StringPatternInput.ReadOnlySpanChar => ...SynthesizedSpanHashFunctionName,
+                    StringPatternInput.SpanChar => ...SynthesizedSpanHashFunctionName,
+                    ...

{ "reward": 0.0, "n_resolved": 0.0, "visible_ok": 0.0, "n_regressions": 1.0 }

• SWE-sweep scores the patch, not the intention. A fix that passes hidden tests but breaks the existing suite earns nothing.
• The same gate zeroed a Claude Code roslyn diff with 22 regressions in our earlier cost-tier run.
• Lesson for production agent use: always run the agent inside a regression-gated loop. Both harnesses needed the fence.

Before this: the quantum-compiler kernels said the same thing

Our earlier head-to-head on 4 from-scratch C++ quantum-optimization problems, graded by hidden tests

• A fast, cheap agent beat the era's #1 model on the one task that needed real algorithmic depth, at 39× lower cost and 24× faster per iteration.
• Different benchmark, different model class, same conclusion: the loop around the model decides.

The Parasoft C++test control

Before crediting agents, we asked: can classic static analysis find these bugs at all?

• Tool: Parasoft C++test Professional 2025.2.0, licensed, run on 6 C/C++ SWE-sweep repos: libexpat, jerryscript, hdf5, proj, z3, tarantool.
• Five configurations: Recommended Rules, Flow Analysis Standard, Flow Analysis Aggressive, MISRA C 2025, MISRA C++ 2023.
• ~500,000 findings in total across the configurations.
• Ground truth: the SWE-sweep oracle patches. A finding "hits" a bug only if its line falls on a base-side line that the upstream fix actually changed.

Parasoft result: 0 to 0.3% overlap with the real bugs

Findings landing on lines the upstream fix changed

• Wherever we can compute a fair baseline, the hit rate sits at or below the random density of changed lines in the file. The apparent jerryscript overlap is explained by line density, not detection.
• MISRA "hits" are style-rule violations that coincidentally touch changed regions; the MISRA configs flag 79,364 findings in tarantool alone.

Why static analysis is not relevant to SWE-sweep

And why it is still not the loser of this story

Why the harness is the king

The same weights, two different products, two different outcomes

• The model is the engine. The harness is the car. Nobody commutes to work on an engine.
• In our SWE-sweep runs the deciding harness behaviors were: repo-wide context management, when to stop exploring and edit, patch-size discipline, and running the visible test suite before declaring victory.
• The models are converging. The scaffolding is diverging. That is where evaluators, buyers and builders should look.

The winner: Amp

On this benchmark, with the model held constant, the harness with the better loop won. Twice.

• 2× the resolved bugs on identical weights: 4/60 (6.7%) vs Claude Code's 2/60 (3.3%), same model, same prompt, same cap, same verifier.
• Above the state of the art: 6.7% on our subsample vs 4.7% for the best official full-run entry, which cost $7,230.
• Cleaner cost profile: Amp completed every session inside the cap and produced minimal, targeted diffs; Claude Code needed larger patches for fewer wins and lost three sessions to its own spend limit.
• Consistent with our prior study: on the quantum-compiler kernels, Amp solved 4/4 where max-effort Claude Code stopped at 3/4, at 39× lower cost.
• The compliment that matters: Amp's context handling, tool execution and verification loop turn the same engine into measurably more working software. The harness is the product, and Amp built a better one.

Honest limitations

A benchmark you can trust names its own boundaries first

• n = 60, one seed. One bug equals 1.67 points, so 4-vs-2 is directional evidence, not a statistical proof. It aligns with two independent studies pointing the same way.
• Subsample: 5 frozen reference bugs per repo, not the full 4,100. Our percentages are placement estimates against official full runs.
• Quota endings: three Claude Code sessions ended at the plan spend cap. Their patches were graded by the real verifier, but those sessions had less effective exploration time.
• Costs are estimates from measured wall time, not vendor invoices; token-level attribution is not exposed by either CLI.
• Both harnesses are good. Claude Code resolved bugs Amp missed (tokio). This is a ranking on one benchmark slice, not a verdict for all workloads. Run candidates on your own code.

What to do Monday

Buy the harness, not the leaderboard

• Stop shopping for the biggest model. The same model scored 1.3%, 3.3% or 6.7% depending on the loop you put around it.
• Evaluate harnesses on your repos: context management, tool execution, verify loops, spend discipline, regression fences.
• Keep static analysis for the rule book and compliance; keep agents for behavioral bugs; keep hidden tests as the only referee.
• Budget the plan limit, not just the tokens: quota interruptions cost us five Claude batch attempts before we stabilized the protocol.

ESL - Engineering Software Lab

Embedded software consultancy · optimizing and migrating embedded projects · AI-augmented SDLC with evidence

Daniel Liezrowice CTO, ESL

Benchmarks, migration proofs and security assessments with executable evidence. Parasoft and Perforce partner in Israel for over 20 years.

linkedin.com/in/liezrowice · eswlab.com