mini-swe_sweep: Amp vs Claude Code

Repository-level bug-fixing benchmark on real SWE-sweep bugs · hidden-test Docker verifier per bug · last updated: generated from run data after the final batch · page auto-refreshes every 5 min

Final run-2 status (Oct 21, 2026): all 24 agent-repo sessions are complete. Rows marked quota-end ended when Claude Code hit its own spend cap, but the patch each session had produced was fully evaluated by the SWE-sweep verifier and is scored here. Claude Code used its logged-in subscription despite an ANTHROPIC_API_KEY being set; exit code 1 marks those endings, not invalid scores.

Run 2 — same model on both arms: claude-opus-5-5 (both arms)

Identical prompt, identical repos at frozen upstream releases, identical 60-minute cap per agent-repo. Score = hidden reference-bug tests passed via the SWE-sweep verifier; a visible-test regression zeroes a repo's reward.

Repo	Language	Amp (default mode, claude-opus-5-5) score / wall time / patch size / est. cost	Claude Code (headless) score / wall time / patch size / est. cost	Data status
libexpat	C	0/5 21m11s, 37 diff lines, est $1-$2	0/5 5m01s, 15 diff lines, est $0-$1	valid
click	Python	1/5 16m07s, 80 diff lines, est $1-$2	0/5 3m09s, 30 diff lines, est $0-$0	valid
gson	Java	1/5 19m49s, 291 diff lines, est $1-$2	0/5 8m10s, 53 diff lines, est $1-$1	valid
roslyn	C#	0/5 6m26s, 256 diff lines, est $0-$1	0/5 24m44s, 256 diff lines, est $2-$3	+1 reg
z3	C++	0/5 58m34s, 399 diff lines, est $4-$7	0/5 32m43s, 1006 diff lines, est $2-$4	valid
tokio	Rust	0/5 15m19s, 58 diff lines, est $1-$2	1/5 23m58s, 553 diff lines, est $2-$3	valid
hdf5	C	0/5 19m37s, 61 diff lines, est $1-$2	0/5 14m11s, 70 diff lines, est $1-$2	valid
proj	C++	1/5 57m52s, 329 diff lines, est $4-$7	0/5 12m05s, 147 diff lines, est $1-$1	quota-end
black	Python	0/5 57m52s, 34 diff lines, est $4-$7	0/5 12m28s, 13 diff lines, est $1-$1	quota-end
rxjava	Java	1/5 57m52s, 152 diff lines, est $4-$7	1/5 39m40s, 190 diff lines, est $3-$5	quota-end
fsharp	C#	0/5 56m56s, 747 diff lines, est $4-$7	0/5 13m08s, 57 diff lines, est $1-$2	+1 reg
syn	Rust	0/5 26m42s, 118 diff lines, est $2-$3	0/5 12m47s, 45 diff lines, est $1-$1	valid
Totals		4/60, wall 6.9h, est $28-$48	2/60, wall 3.4h, est $13-$24

Final: Amp 4/60 (6.7%) vs Claude Code 2/60 (3.3%) on the identical model. Amp resolved bugs in 4 of 12 repos (click, gson, proj, rxjava); Claude Code in 2 (tokio, rxjava). For context, the best official SWE-sweep leaderboard entry is 4.7%.

Run 1 — cost-tier comparison (not model-equal)

Amp in low mode (budget model tier) vs Claude Code on Opus defaults. Shows the cost/quality frontier; cannot isolate the agent harness.

Repo	Language	Amp low mode score / wall time / est. cost	Claude Code (Opus defaults) score / wall time / est. cost
libexpat	C/C++	2/5 11m02s, est $0-$0	1/5 25m17s, est $2-$3
click	Python	0/5 12m32s, est $0-$0	2/5 3m58s, est $0-$0
gson	Java	0/5 12m59s, est $0-$0	0/5 10m46s, est $1-$1
roslyn	C#	0/5 58m18s, est $0-$1	0/5 +22 reg 58m13s, est $4-$7
Totals		2/20, wall 1.6h, est $1-$2	3/20, wall 1.6h, est $7-$11

Cost estimates: how they were computed

Per-session wall time is exact (measured by the harness, shown per row). Token counts are not exposed by either CLI, so cost is estimated from wall time.
Opus-class sessions (both arms of run 2, and Claude Code in run 1): estimated at 4-7 dollars per hour of wall time. Assumes ~3M input tokens/hour (mostly cache reads billed near 10%), ~40k output tokens/hour, at roughly $5 per M input and $25 per M output. Treat as order-of-magnitude.
Amp low mode (run 1 only): estimated at 0.5-1.5 dollars per hour (budget-tier model, cheaper per token).
Run 2 estimate total: Amp arm $28-$48 + Claude arm $13-$24 (Claude figure would roughly double once the 6 truncated repos are rerun with full caps). Run 1 estimate total: Amp low $1-$2 + Claude $7-$11.
Amp subscription plans bill separately from metered API; if sessions run inside a subscription the marginal cost can be zero, but the metered-equivalent estimate above is what the work would cost on pay-per-token.

Observations

These behavioral bugs are hard for both agents: the best full-run score on any repo is 1/5 (20%), consistent with SWE-sweep difficulty.
Same model on both arms: Amp's harness resolved 4 bugs vs Claude Code's 2 across the 12 repos, and finished all 12 sessions inside the cap; several Claude Code sessions ended at the plan's spend cap (their patches were still evaluated).
Amp sessions typically used the time budget fully; Claude Code sessions in this benchmark either finished early with small diffs or died to quota. Session durations per row above.
Operational lesson: quota interruptions complicated 5 of 6 Claude batch attempts before the API-key rerun. For agent benchmarks, budget the plan limit, not just the tokens.

Artifacts and provenance

Harness, patches, session logs, and per-run verifier outputs: github.com/zuwasi/mini-swe-sweep. Companion study: Parasoft C++test vs SWE-sweep reference bugs (0–0.3% line-level detection of the same bugs; agents are complementary). Verifier: SWE-sweep task definitions with hidden-bug scoring.