Independent blind evaluation
Droid as a Judge

Droid as a Judge

APWeb Blind Security Bake-off

Amp

Ultra · $8.24

10 min · 6.80

Claude Code

High · $110

~16h · 5.66

ArduPilot APWeb @ 50b6b7ac · 3,157 lines of C · 2026-08-07

AmpClaudeESL
← Swipe to navigate →
01 · Target

The Target — Overview

What it is

APWeb is ArduPilot's embedded HTTP/WebSocket server: configuration UI, file transfer, telemetry and firmware-facing workflows in 3,157 lines of C.

Where it runs

Linuxhost build
FreeRTOSflight hardware

Technologies

CHTTP/1.1WebSocketMAVLinkmultipart uploadtemplatesfilesystem

Why this target?

A compact, security-relevant, clean target with zero answer-key. Both contestants started blind from the same commit.

02 · Surface

Architecture & Attack Surface

Browser / API client
        │ HTTP + WebSocket
        ▼
 ┌──────────────── APWeb ────────────────┐
 │ parser → router → handlers            │
 │   │         │       ├─ multipart      │
 │   │         │       ├─ templates      │
 │   │         │       └─ firmware/files │
 │   └─ WebSocket ↔ MAVLink global state │
 └──────────────┬────────────────────────┘
                ├─ Linux filesystem
                └─ FreeRTOS / flight controller

⚠ No authentication

Network-reachable parsing and privileged file/firmware operations share one trust boundary. Memory safety, path handling and state integrity all matter.

03 · Contestant A

The Tools — Amp

Amp

What was tested

  • Focused source-code review
  • AI-assisted security analysis
  • Exploit-path reasoning
  • Local proof-of-concept validation

How it ran

  • Ultra reasoning mode
  • $8.24 total
  • 10m 08s wall time
  • 0 subagents
  • Blind, cold-start review

Result: 7 register classes covered, 12 delivered findings, 4 confirmed vulnerabilities — reliable and economical, but narrow.

04 · Contestant B

The Tools — Claude Code

What was tested

  • Multi-agent source review
  • claude-security plugin
  • Broad file-by-file fan-out
  • Cross-agent synthesis

How it ran

  • High reasoning
  • $110 total
  • 565 subagents launched
  • 495 errored
  • Account limit reached
  • ~16 hours, blind

Result: 21 register classes covered and substantially greater depth, offset by severe orchestration failure and cost.

05 · Protocol

Methodology

Setup

  • Same commit pin: 50b6b7ac
  • Cold-start, blind runs
  • No answer-key or cross-contamination
  • Contestant outputs untouched

What the judge did

  • Normalized claims into vulnerability classes
  • Reproduced or inspected disputed evidence
  • Separated platform and confidence scope
  • Applied one weighted rubric
CriterionWeightCriterionWeight
Coverage25%Calibration15%
Evidence12%Reliability15%
Scope honesty10%Cost13%
Triage5%Operational safety5%
06 · Register

Findings Register — Part 1

7Amp classes
21Claude classes
#Vulnerability classScopeAmpClaude
1unescape() OOB read/writeboth✓ + PoC✓
2Multipart unsigned underflowboth✓ under-rated✓ Critical
3NULL-deref, no Content-Typeboth✓✓
4grab_line() 1-byte OOB writeboth✗✓ High
5Template parser use-after-freeboth✗✓ High
6grab_line() realloc NULL derefboth✓✗
7atoi Content-Lengthboth✓✓
8MAVLink off-by-one (latent)both✓ mis-FP✓
9MAVLink char-array over-readboth✗✓ suspected
10Unbounded recursion → stack DoSboth✗✓
11MAVLink global state data raceboth✗✓
12uint16_t vs int argcboth✗✓ suspected
07 · Register

Findings Register — Part 2

#Vulnerability classScopeAmpClaude
13msg memory leakboth✗✓
14(*len) -= 2 boundary desyncboth✗✓
15Wildcard CORSboth✗✓
16Path traversal fs/FreeRTOSdetected, dropped✓
17Arbitrary file write (upload)FreeRTOS✗✓
18get_ssid stack overflowFreeRTOS✗✓ unverified
19Firmware MD5-only (no sig)FreeRTOS✗✓ unverified
20WiFi password disclosedFreeRTOSpartial✓
21WiFi password loggedFreeRTOS✗✓
22Hardcoded HERE Maps credsclientfalse neg✓ Low
23path[strlen-1] off-by-ones ×3FreeRTOS✓✗
24Output injection / XSSFreeRTOSdetected, droppedcandidates

The union is 24

Neither tool is the answer-key. Their overlap raises confidence; their unique findings make the combined review stronger.

08 · Adjudication

Judge's Rulings

APWEB-19: UAF confirmed

Template parser lifetime crosses invalidation. Claude's High finding stands.

APWEB-03: Critical → Medium

Unsigned underflow exists, but practical impact depends on word size and allocation behavior.

grab_line: two bugs

The one-byte OOB write and realloc NULL dereference are distinct classes. Each tool found one.

HERE Maps: Amp false negative

Hardcoded client credentials were present; dismissal was not supported by the code context.

09 · Economics

Cost Analysis

$8.24Amp · Ultra · 10m08s
$110Claude · ~16h

Thread usage

4,965,331 total tokens

4,910,680 input · 54,651 output · 85% cached

40 requests · 2 models

Model breakdown

Fable 5: 38 requests · $8.237

GPT-5.6 Luna: 2 requests · $0.002

Total both tools: $118.24 · Amp is ~13× cheaper

Amp unitCalculationCost
Delivered finding$8.24 / 12$0.69
Confirmed vulnerability$8.24 / 4$2.06
Register class covered$8.24 / 7$1.18
Accounting A: direct contestant spend, consistently measured.
You can't disown cost: failed agents still consumed budget and shaped reliability.
10 · Weighted result

The Scorecard

Coverage Calibration Evidence Reliability Scope honesty Cost Triage Op. safety■ Amp■ Claude
CoverageCalibrationEvidenceReliabilityHonestyCostTriageSafetyTotal
Amp3.26.85.09.29.09.48.29.36.80
Claude7.57.55.03.07.53.07.52.05.66
11 · Decision

The Verdict

Winner: Amp

6.80 vs 5.66

Amp ran in Ultra mode — its highest reasoning tier.

Amp wins reliability & cost

Clean completion, evidence discipline and ~$0.69 per delivered finding.

Claude wins depth

21 of 24 classes and the strongest multi-hop discoveries.

Weight-sensitive

Coverage-first buyers may rationally prefer Claude despite the official score.

The union is the result

Together they expose the most complete actionable register.

12 · Action

Recommendations

For the security team

  • Run both: complementary discovery is material.
  • Use Amp for repeatable cadence at $8.24/run.
  • Use Claude for periodic depth scans, with unstable fan-out disabled.
  • Fix order: confirmed memory safety → writes/traversal → secrets → hardening.

Before round 3

  • Run Claude twice to measure variance.
  • Use structured subagent/task fan-out and broader review coverage. Ultra mode was already used — the gap isn't reasoning depth, it's analysis breadth. Fan out across files with cross-check agents to reach multi-hop bugs.
  • Restore local PoC execution and preserve evidence artifacts.
  • Use two targets to reduce target-specific bias.
13 · Credits

About / Credits

AmpClaudeESL
Daniel Liezrowice

Daniel Liezrowice

CEO & Co-Founder, ESL

in LinkedIn · GitHub

Droid as a Judge · APWeb bake-off · 2026-08-07

Contestant outputs untouched