FIRST SLIDE

Ampcode

vs

C

Claude Code

TECHNICAL PROCUREMENT EVALUATION

AI Coding Agent Head-to-Head

Controlled benchmark on Kubernetes v1.34.5 · commit ebfac057

📋 What Was Tested?

We compared Ampcode and Claude Code on a real Kubernetes engineering task, not a toy coding prompt.
Both tools had to trace a cross-system workflow, implement a non-trivial code change, and prove correctness with validation.
Claude Code showed broader technical completeness in its strongest single run.
Ampcode showed better cold-start repeatability, faster execution, and stronger operational consistency.
Under engineering-only weighting, the result was nearly tied.
Under procurement-style weighting, Ampcode was the clear winner.
Current recommendation: choose Ampcode, while confirming with one additional benchmark on internal code.

27,579

files in the test repo

12

runs completed — all successful

0

human interventions

Evaluated by Daniel Liezrowice · LinkedIn

ESL
Independent Evaluation

Branches: ampcode claude-code

Why Are We Doing This?

🎯 The Decision

We are evaluating which AI coding agent to invest in for enterprise-wide adoption. This is a high-value procurement decision backed by reproducible, auditable evidence — not marketing demos.

💰 What's at Stake

Annual licensing cost across engineering teams
Developer productivity multiplier
Ability to work autonomously on real codebases
Trust — can we rely on the agent's output?

🛡️ Guiding Principle

We do not claim one tool "can't" do something the other can. We measure which agent performs better, faster, cheaper, and more autonomously under identical controlled conditions.

What Are We Comparing?

CANDIDATE A

Ampcode

Sourcegraph-backed code intelligence
Librarian (cross-repo understanding)
Native multi-tool orchestration
Deep monorepo navigation

CANDIDATE B

Claude Code (Opus 4.6)

Anthropic's agentic coding tool
Terminal + file access
MCP-extensible (optional augmentation)
Strong general reasoning

📌 Same Task, Same Repo, Same Rules

Both agents receive the identical prompt on the same Kubernetes v1.34.5 codebase (commit ebfac057). No coaching, no hints, no style guidance. They must navigate 27,579 files and produce working code autonomously.

The Benchmark Task

Deliberately chosen to stress large-codebase reasoning, not trivial single-file edits.

Kubernetes Dynamic Resource Allocation — Lifecycle Trace & Patch

Requirement	What the Agent Must Do
Trace	Map the complete DRA lifecycle: API types → scheduler → kubelet → driver
Identify	Find every file, struct, interface, validation, mutation, and serialization point
Patch	Add a new `allocationTraceID` field and propagate it through the full lifecycle
Build	Ensure the patch compiles and does not break existing tests
Report	Document assumptions, files changed, rationale, limitations, and validation commands actually run

This task spans 4+ packages, requires cross-package dependency tracing, and exposes semantic navigation weaknesses.

How We Test — Three Modes

🔒

Mode A: Baseline

Local tools only.
No code intelligence.
No MCP servers.

Tests raw navigation ability in a massive codebase.

🔧

Mode B: Augmented

Vendor-supported tools enabled.
Amp + Sourcegraph.
Claude + MCP.

Tests realistic enterprise usage.

🤖

Mode C: Autonomy

Max 2 human hints.
Must self-recover.
No coaching.

Tests independence & self-correction.

All 12 runs complete (Mode A + Test 2). Mode A: 6 runs. Test 2 Repeatability: 6 runs. Ampcode demonstrated 3 true cold-start repeats (1.0–1.6 min). Claude Code's Test 2 had only 1 true cold start; runs 002/003 reused context. Both agents achieved 100% success rate, 0 human interventions.

Independent Judge Scores (Test 1 + Test 2 Combined)

Revised by GPT-5.4 independent judge across 8 categories including Repeatability. Scale: 0–10.

Category	Ampcode	Claude Code	Δ
Technical correctness & completeness	8.1	9.3	+1.2
Patch quality	8.5	9.2	+0.7
Validation discipline	8.7	9.3	+0.6
Repeatability	9.9	6.8	+3.1
Autonomy	10.0	10.0	TIE
Time efficiency	9.8	6.8	+3.0
Auditability & management readiness	9.0	9.2	+0.2
Cost efficiency	9.0	7.8	+1.2

Updated with Test 2 repeatability evidence. Claude leads in 3 technical categories. Ampcode leads in repeatability (+3.1), time (+3.0), and cost (+1.2). Both achieve perfect autonomy.

Head-to-Head: Run Facts

Metric	Ampcode	Claude Code (Opus 4.6)
Mode	A — Baseline	A — Baseline
Human Interventions	0	0
Task Completed	✅ Yes	✅ Yes
Files Changed	6	14 (6 shared + 8 additional)
Lines Changed	+31 / −1	+80 / −1
Build Status	✅ All pass	✅ All pass
Test Suites Run	4 suites	9 suites
Test Status	✅ All pass	✅ All pass
go vet Run	No	Yes (5 packages)
Lifecycle Steps Traced	5 major phases	14 detailed steps
Mutation Points Found	11	13 (+ indirect)

Patch Scope Comparison

Ampcode — 6 Files

v1 types.go — field added
Internal types.go — field added
v1 conversion — roundtrip
Scheduler bindClaim — sets UUID
Kubelet manager — logs trace ID
Scheduler test — ignore field

Strategy: Minimal, focused patch. Core v1 + internal + scheduler + kubelet.

Claude Code — 14 Files

v1 types.go — field added
Internal types.go — field added
v1 conversion — roundtrip
Scheduler bindClaim — sets UUID
Kubelet manager — logs trace ID
Scheduler test — ignore field
+ v1beta2 types.go
+ v1beta2 conversion
+ v1 generated.proto
+ v1beta2 generated.proto
+ Validation (256-char max)
+ Kubelet state checkpoint
+ ClaimInfo propagation
+ Controller logging

Strategy: Broader propagation. Cross-version, proto, validation, checkpoint, controller.

The Answer Depends on What You Optimize For

Same raw scores, two legitimate weighting models — two different winners.

🔬 Engineering-Focused Model

Correctness 30% · Patch 18% · Validation 18% · Repeatability 12% · Autonomy 10% · Time 7% · Audit 3% · Cost 2%

8.84

Claude Code

vs

8.85

Ampcode

Effective tie — Amp +0.01

Claude's broader patch completeness is now offset by Ampcode's repeatability and cold-start speed advantage.

💰 Procurement-Focused Model

Correctness 18% · Cost 15% · Repeatability 15% · Time 12% · Validation 12% · Autonomy 10% · Patch 10% · Audit 8%

9.08

Ampcode

vs

8.45

Claude Code

Ampcode wins by 0.63

Ampcode's repeatability, speed, and cost efficiency create a decisive procurement advantage after Test 2.

Key Findings — Mode A Complete

Claude Code — Broader Technical Scope

Broader lifecycle coverage (v1beta2, proto, checkpoint, validation)
14-file patch — technically more complete per run
9 test suites including controller + all API versions
⚠️ Test 2: Only 1 of 3 runs was a true cold start
Engineering-weighted score: 8.84

Ampcode — Stronger Repeatability & Operational Value

✅ Test 2: 3 true cold-start runs, 3 identical patches
Fastest execution: 1.0–1.6 min per cold-start run
Repeatability score: 9.9/10
Lower cost per task (fewer tokens consumed)
Procurement-weighted score: 9.08

🤝 Shared Strengths

Both agents achieved 100% success rate, 0 human interventions, perfect autonomy (10/10). Both self-recovered from test failures. Neither fabricated file paths, commands, or success claims. Engineering model is an effective tie. Procurement model shows Ampcode ahead by 0.63 points.

Capability Radar — All 8 Dimensions

Each axis represents a scoring category. Larger area = stronger overall profile.

Ampcode's shape extends further on Repeatability, Time, and Cost. Claude's extends on Correctness, Patch, and Validation.

Score Comparison — Side-by-Side Bars

Blue = Ampcode · Pink = Claude Code. Longer bar = higher score.

Correctness

8.1

9.3 ★

Patch Quality

8.5

9.2 ★

Validation

8.7

9.3 ★

Repeatability

9.9 ★

6.8

Autonomy

10.0

Time Efficiency

9.8 ★

6.8

Auditability

9.0

9.2 ★

Cost Efficiency

9.0 ★

7.8

4 categories won by Ampcode

3 categories won by Claude

1 tie

Test 2 Cold-Start Execution Timeline

Each bar represents one run. Green = true cold start. Red outline = warm context (not independent).

3 / 3

Ampcode true cold starts

1 / 3

Claude true cold starts

40×

Ampcode faster (avg cold-start)

Cost Efficiency & Projected ROI Impact

Test 2 cold-start cost per run · Time-to-value · Projected annual savings at scale.

Ampcode

Claude Code

5.5×

Cost Advantage

Ampcode costs ~82% less per cold-start task

⏱️ Time-to-Value

Ampcode avg cold-start

1.2 min

Claude cold-start (Run 001)

64 min

📈 Projected Annual Impact

Tasks per developer per week	~10
Developers using the tool	50
Annual tasks	26,000
Cost delta per task	~$4.55
Projected annual savings	~$118K

Procurement Model — Weight Contribution Waterfall

Each bar shows how much each weighted category contributes to the final score. Taller cumulative stack = higher total.

📂 Full Evidence on GitHub — Explore Everything

Every patch, log, report, and protocol is publicly available. Nothing hidden — full transparency.

`main`

Interactive presentation
Management comparison
Test protocols
All 12 run artifacts
Central run ledger
Original K8s source files

`ampcode`

Ampcode's 6-file patch applied
Focused: core v1 + internal
Scheduler + kubelet changes
+23 lines, −1 line
Identical across 3 cold starts

View Diff →

`claude-code`

Claude's 14-file patch applied
Broader: v1 + v1beta2 + proto
Validation + checkpoint + controller
+62 lines, −2 lines
Only 1 true cold start

View Diff →

🔍 What You Can Learn From This Repo

Compare the diffs — see exactly which files each agent touched and why
Read session logs — full transcripts of how each agent navigated 27K files
Inspect patches — byte-level proof of repeatability (or lack thereof)

Review test evidence — build logs, test pass/fail, go vet results
Reproduce the evaluation — protocol docs explain every step
Audit cost data — token counts and cost estimates per run

Cost & Budget Tracking

Two-layer accounting ensures transparency for finance and engineering leadership.

Layer 1: Vendor-Reported

Dashboard / billing exports
Token counts (input + output)
Model usage per run
Billed cost in USD

Layer 2: Local Run Ledger

Independent per-run CSV log
Wall-clock time
Human intervention count
Infrastructure cost overlay

📊 Decision Metrics (Not Just License Price)

$/task

Cost per successful task

hrs/mo

Engineering hours saved

%

Autonomous completion rate

What's Next — Full Evaluation Plan

📋 Remaining Runs

Agent	Mode A	Mode B	Mode C
Ampcode	3 of 3 ✅	0 of 3	0 of 3
Claude Code	3 of 3 ✅	0 of 3	0 of 3

Total: 12–18 runs needed for a defensible comparison.

📦 Additional Benchmarks Needed

PUBLIC Multi-repository open-source task
INTERNAL Real task from our codebase
ROUTINE Small maintenance task (overhead test)

Do not decide based on a single benchmark.

🔐 Recommendation Rule

Do not select a vendor based on one dramatic success or failure. Recommend only after:
✓ 6–12 controlled runs ✓ At least 1 internal task ✓ Cost reconciliation ✓ Engineering + finance review

LAST SLIDE

Executive Summary

12 runs completed (Mode A + Test 2). 100% success. Zero human interventions.
Engineering model: effective tie — Claude's broader patch vs Ampcode's repeatability. (8.84 vs 8.85)
Procurement model: Ampcode wins — repeatability, speed, and cost create a decisive advantage. (9.08 vs 8.45)
For procurement-weighted decisions, Ampcode has the stronger case.

Model	Claude	Amp	Winner
Engineering	8.84	8.85	Tie (Amp +0.01)
Procurement	8.45	9.08	Amp +0.63

📌 Recommended Next Steps

Give Claude Code a fair rerun with 3 truly independent cold starts
Run at least one more benchmark pack (multi-repo or internal task)
Complete Mode B (augmented) and Mode C (autonomy) runs
Reconcile vendor billing against local run ledger

Ampcode Diff → Claude Diff →

All evidence archived — reports, diffs, logs, cost data, and modified Kubernetes source on separate branches.