QuickSWE Dashboard

Generated 2026-04-09 13:50:35  |  100 tasks  |  9 run(s) per agent

🏆 Notable Finding: Amp Catches a Bug in the Benchmark Itself

During testing of task_032 (SQL Query Engine), Amp identified and refused to comply with an incorrect test assertion. The test expected len(result) == 2 for a WHERE amount > 200 query against orders with amounts [250, 150, 300, 450, 200]. The mathematically correct answer is 3 (amounts 250, 300, and 450 all satisfy > 200).

Instead of blindly producing code to pass the wrong test, Amp analyzed the data, performed the arithmetic, and correctly flagged the discrepancy in its output — consistently across all 3 runs.

What each agent did:
Agent Behavior on incorrect test Result
Amp Analyzed the data, identified the assertion was wrong (expected 2, correct answer is 3), and refused to produce incorrect code 0% pass — correctly rejected bad test
Claude Code Brute-forced a solution that sometimes matched the wrong assertion by coincidence 67% pass — passed a test that was wrong

Verdict: A higher pass rate on a flawed test is not a win — it means the agent lacks the reasoning to question incorrect specifications. In production, this is the difference between an agent that introduces subtle bugs and one that flags them. Reasoning integrity > blind compliance.

🐍 Python — 50 Tasks

50
Tasks
100.0%
Amp-deep3 Resolve Rate
99.0%
Claude Resolve Rate
1.5x
Amp-deep3 Speed Advantage
76.0s
Amp-deep3 Avg Time
112.5s
Claude Avg Time
🏆 Amp-deep3
Winner

📊 Overall Resolution Rate

Percentage of tasks each agent successfully resolved across all runs.

⚡ Speed Comparison — Top 15 Largest Gaps

Actual completion times with biggest speed differences.

🚀 Speed by Difficulty Tier (avg seconds)

Average time per task at each difficulty level.

📂 Resolution by Category

Performance across bug fixes, features, and refactoring.

📈 Resolution by Difficulty

Performance scaling with task difficulty.

🛡️ Regression Rate

Percentage of tasks where the agent broke existing tests. Lower is better.

🥧 Amp-deep3 — Outcome Breakdown

🥧 Claude — Outcome Breakdown

🕸️ Multi-Dimensional Comparison

⏱️ Average Time per Task

📉 Consistency Across Runs

🔍 Per-Task Comparison

🗺️ Head-to-Head Results Matrix

TaskCategory Amp-deep3 ResolveTime Claude ResolveTime
🟢 task_001Bug Fix100%56.2s100%44.1s
🟢 task_002Bug Fix100%59.5s100%62.8s
🟢 task_003Bug Fix100%63.1s100%42.4s
🟢 task_004Feature100%62.3s100%39.8s
🟢 task_005Bug Fix100%58.4s100%41.4s
🟢 task_006Bug Fix100%56.4s100%50.9s
🟢 task_007Bug Fix100%50.7s100%50.6s
🟢 task_008Bug Fix100%50.4s100%40.7s
🟢 task_009Feature100%82.0s100%44.6s
🟢 task_010Bug Fix100%53.7s100%47.3s
🟡 task_011Bug Fix100%123.9s100%240.5s
🟡 task_012Bug Fix100%89.7s100%52.4s
🟡 task_013Bug Fix100%53.2s100%42.7s
🟡 task_014Feature100%77.9s100%49.2s
🟡 task_015Bug Fix100%75.0s100%47.1s
🟡 task_016Bug Fix100%60.0s100%45.5s
🟡 task_017Refactoring100%59.4s100%41.5s
🟡 task_018Bug Fix100%62.9s100%45.1s
🟡 task_019Bug Fix100%62.7s100%44.1s
🟡 task_020Bug Fix100%53.4s100%45.2s
🟠 task_021Bug Fix100%59.4s100%51.6s
🟠 task_022Bug Fix100%472.5s50%2904.4s
🟠 task_023Bug Fix100%56.9s100%55.2s
🟠 task_024Bug Fix100%57.5s100%63.4s
🟠 task_025Bug Fix100%74.8s100%55.2s
🟠 task_026Bug Fix100%52.0s100%44.1s
🟠 task_027Bug Fix100%117.2s100%91.6s
🟠 task_028Bug Fix100%50.7s100%36.9s
🟠 task_029Bug Fix100%71.1s100%75.1s
🟠 task_030Bug Fix100%54.6s100%42.4s
🟠 task_031Bug Fix100%59.7s100%48.1s
🟠 task_032Bug Fix100%69.2s100%50.7s
🟠 task_033Bug Fix100%63.5s100%46.5s
🟠 task_034Bug Fix100%51.8s100%45.2s
🟠 task_035Bug Fix100%58.1s100%48.7s
🔴 task_036Bug Fix100%62.4s100%49.3s
🔴 task_037Bug Fix100%55.3s100%48.1s
🔴 task_038Bug Fix100%111.1s100%69.0s
🔴 task_039Bug Fix100%53.6s100%38.6s
🔴 task_040Bug Fix100%56.6s100%49.8s
🔴 task_041Bug Fix100%56.7s100%46.8s
🔴 task_042Bug Fix100%63.3s100%48.3s
🔴 task_043Bug Fix100%58.0s100%41.5s
🔴 task_044Bug Fix100%161.1s100%139.8s
🔴 task_045Bug Fix100%66.6s100%48.0s
🔴 task_046Bug Fix100%70.3s100%53.1s
🔴 task_047Bug Fix100%66.8s100%54.8s
🔴 task_048Bug Fix100%61.3s100%54.8s
🔴 task_049Bug Fix100%86.4s100%56.8s
🔴 task_050Bug Fix100%88.7s100%49.4s

Green = ≥80% resolved with <20% regressions. Blue = ≥50%. Yellow = partial. Red = not resolved.

⚙️ C/C++ — 25 Tasks

25
Tasks
94.7%
Amp-deep3 Resolve Rate
49.0%
Claude Resolve Rate
0.7x
Amp-deep3 Speed Advantage
114.8s
Amp-deep3 Avg Time
76.9s
Claude Avg Time
🏆 Amp-deep3
Winner

📊 Overall Resolution Rate

Percentage of tasks each agent successfully resolved across all runs.

⚡ Speed Comparison — Top 15 Largest Gaps

Actual completion times with biggest speed differences.

🚀 Speed by Difficulty Tier (avg seconds)

Average time per task at each difficulty level.

🗺️ Head-to-Head Results Matrix

TaskCategory Amp-deep3 ResolveTime Claude ResolveTime
🟢 task_051Bug Fix100%65.1s100%54.5s
🟢 task_052Bug Fix100%69.0s100%81.1s
🟢 task_053Bug Fix100%60.1s100%33.9s
🟢 task_054Bug Fix100%59.6s100%42.3s
🟢 task_055Bug Fix100%75.2s100%72.4s
🟡 task_056Bug Fix33%212.8s0%163.1s
🟡 task_057Bug Fix100%64.6s100%48.2s
🟡 task_058Bug Fix67%238.9s25%300.3s
🟡 task_059Bug Fix100%62.3s100%45.4s
🟡 task_060Bug Fix100%86.9s25%116.2s
🟠 task_061Bug Fix100%59.5s100%53.5s
🟠 task_062Bug Fix100%140.3s100%81.9s
🟠 task_063Bug Fix100%91.9s75%64.6s
🟠 task_064Bug Fix100%86.5s25%61.3s
🟠 task_065Bug Fix100%103.7s50%152.6s
🟠 task_066Bug Fix100%95.7s25%42.8s
🟠 task_067Bug Fix100%72.0s25%23.9s
🟠 task_068Bug Fix100%94.8s25%26.4s
🔴 task_069Bug Fix100%110.4s25%31.8s
🔴 task_070Bug Fix83%220.2s0%19.4s
🔴 task_071Bug Fix100%152.9s25%222.7s
🔴 task_072Feature83%227.2s0%169.1s
🔴 task_073Bug Fix100%105.0s0%8.7s
🔴 task_074Refactoring100%127.2s0%2.9s
🔴 task_075Bug Fix100%188.1s0%2.9s

Green = ≥80% resolved with <20% regressions. Blue = ≥50%. Yellow = partial. Red = not resolved.

🟢 CUDA — 25 Tasks

25
Tasks
72.0%
Amp-deep3 Resolve Rate
56.0%
Claude Resolve Rate
0.5x
Amp-deep3 Speed Advantage
86.1s
Amp-deep3 Avg Time
42.8s
Claude Avg Time
🏆 Amp-deep3
Winner

📊 Overall Resolution Rate

Percentage of tasks each agent successfully resolved across all runs.

⚡ Speed Comparison — Top 15 Largest Gaps

Actual completion times with biggest speed differences.

🚀 Speed by Difficulty Tier (avg seconds)

Average time per task at each difficulty level.

🗺️ Head-to-Head Results Matrix

TaskCategory Amp-deep3 ResolveTime Claude ResolveTime
🟢 task_076Bug Fix100%71.9s100%22.5s
🟢 task_077Bug Fix0%0.0s0%0.0s
🟢 task_078Bug Fix100%69.8s100%20.9s
🟢 task_079Bug Fix100%78.2s100%16.9s
🟢 task_080Bug Fix100%101.2s100%17.6s
🟡 task_081Bug Fix100%89.5s0%20.4s
🟡 task_082Bug Fix100%66.9s100%19.9s
🟡 task_083Bug Fix100%104.0s100%446.6s
🟡 task_084Bug Fix100%107.2s100%54.5s
🟡 task_085Bug Fix100%76.6s100%29.0s
🟠 task_086Bug Fix0%0.0s0%0.0s
🟠 task_087Bug Fix100%136.7s0%27.3s
🟠 task_088Refactoring100%87.9s100%19.1s
🟠 task_089Bug Fix0%0.0s0%0.0s
🟠 task_090Bug Fix100%140.3s100%71.4s
🟠 task_091Bug Fix0%122.3s0%21.4s
🟠 task_092Bug Fix0%0.0s0%0.0s
🟠 task_093Bug Fix0%0.0s0%0.0s
🔴 task_094Bug Fix100%272.8s0%30.2s
🔴 task_095Bug Fix100%96.6s100%32.0s
🔴 task_096Bug Fix100%197.5s0%24.7s
🔴 task_097Bug Fix100%115.2s100%100.3s
🔴 task_098Bug Fix0%0.0s0%0.0s
🔴 task_099Bug Fix100%106.0s100%50.7s
🔴 task_100Bug Fix100%112.6s100%45.5s

Green = ≥80% resolved with <20% regressions. Blue = ≥50%. Yellow = partial. Red = not resolved.