Wazatator watches Claude Code use your SKILL.md: every prompt, tool call, and file. Then it scores the run, so you know whether a skill edit helped instead of guessing.
Presented by ESL (Engineering Software Lab), a member of Anthropic's Claude Partner Network.
Claude might answer from general knowledge and never invoke your skill. Nothing tells you.
"Explain this code" with no snippet: does the skill look for the file, or ask the user to paste it?
Without repeatable tasks and graders, every change is a vibe check.
| Spectator | Wazatator |
|---|---|
| Watches the game | Watches the full Claude run: transcript, tools, files |
| Sees every move | Records each tool call (Skill, Read, Write, MCP) |
| Judges the play | Graders and LLM judges give pass/fail and a score |
| Never plays | Doesn't change your skill; it measures it |
CLI: wazatator, a fork of Microsoft Waza (MIT) with a new Claude Code executor.
Plugin: /wazatator:new-eval, /wazatator:eval, /wazatator:quality inside Claude Code.
Graders: text, regex, files, tool calls, skill invocation, token budget, trigger precision, and LLM-as-judge.
Output: results.json, a terminal report, model comparison, and a CI exit code.
AI agents now plan, write, review and test code. The skills that steer them deserve the same discipline: a spec, tests, review, and a release gate. Wazatator covers the Test and Review stages for agent skills.
Findings loop back: Review → fix the SKILL.md in Code → re-run in Test → compare.
.claude/skills/ so Claude discovers it nativelyclaude -p --output-format stream-json with personal plugins and MCP turned offargs := []string{"-p", "--output-format", "stream-json", "--verbose", "--permission-mode", "bypassPermissions", // results depend on the skill under test, not on this machine's setup "--setting-sources", "project,local", "--strict-mcp-config"} if req.SessionID != "" { args = append(args, "--resume", req.SessionID) } // follow-ups and judges if len(req.Tools) > 0 { bridge, _ := startClaudeToolBridge(req.Tools) } // judge tools over local MCP
name: code-explainer-eval skill: code-explainer config: executor: claude-cli # run through Claude Code model: sonnet trials_per_task: 3 # repeat to catch flakiness timeout_seconds: 300 inject_skill_body: false # Claude must find the skill itself graders: # apply to every task - type: text name: no_errors config: regex_not_match: ["(?i)fatal error|crashed"] tasks: - "tasks/*.yaml"
executor: claude-cli: real Claude Code sessions, the same way your users run the skill.
trials_per_task: models vary from run to run. Three trials show whether a pass is reliable.
inject_skill_body: false: tests real discovery. If the skill's description is weak, Claude won't pick it, and the eval shows that.
Exit codes: 0 = pass, 1 = test failure, 2 = config error, ready for CI.
id: explain-python-recursion-001 inputs: prompt: "Explain this code to me" # deliberately vague files: - path: factorial.py expected: output_contains: ["recursive", "factorial"] graders: - name: used-skill type: skill_invocation config: required_skills: [code-explainer] mode: any_order - name: judge type: prompt # LLM-as-judge config: model: haiku continue_session: true # judge reads the real conversation prompt: |- If the answer explains the recursion and base case, call set_waza_grade_pass, otherwise set_waza_grade_fail.
Rules
output_contains, regex, JSON schema, file contents and diffs: fast, free and deterministic.
Behavior
skill_invocation, tool-call sequences, token and turn budgets: did Claude do the right things?
Judgment
prompt graders: a second Claude session reviews quality that keywords can't capture, and gives its reason.
skill: pirate-greeter should_trigger_prompts: - prompt: "Hello!" - prompt: "Say hello to my team" - prompt: "Good morning, how about a greeting?" should_not_trigger_prompts: # near-misses - prompt: "Translate this paragraph into Spanish" - prompt: "Tell me a pirate joke about coffee" - prompt: "Proofread this email for typos"
TRIGGER ACCURACY Accuracy: 100.0% Precision: 100.0% Recall: 100.0% F1: 100.0% TP: 4 FP: 0 FN: 0 TN: 5
Precision catches a skill that grabs requests it shouldn't (the pirate joke). Recall catches a skill Claude never picks. Each trigger run stops as soon as the skill fires, which saves tokens.
"Explain this code to me" got "I don't see any code in your message yet", even though factorial.py was right there.
Judge: "fails to greet Priya… asks for clarification about who the person is." The name was in newcomer.txt.
Trigger recall 0%: with the skill pasted into the prompt, Claude answered without ever calling its Skill tool.
A read-only agent (tools: ["read"]) was asked to write a file. Only Read was available; any other call fails the run.
Mocked GitHub MCP server: Claude called mcp__github__list_issues and reported issue #363. Wrong arguments fail loudly.
5 tasks + judges + 9 trigger prompts: 72 s, 2,685 output tokens, about $0.31 on Sonnet.
+ ## Finding the Code + If the user refers to "this code" but no snippet is in + the message, do not ask for the code right away: + 1. Use code from the conversation or attached files. + 2. Otherwise list the working directory. If there is + exactly one source file, read and explain it, and + say which file you explained. + 3. Ask only when there is no candidate or several + equally likely ones.
| Task | Before | After (3 trials) |
|---|---|---|
| "Explain this code to me" | ✗ asked for code | ✓ 3/3 |
| "Explain the code in factorial.py" | ✓ | ✓ 3/3 |
tools: Skill → Glob → Glob → Read output: "I explained `factorial.py`, since it's the only source file in the working directory. ..."
The fix went into the skill, not the test, so real users benefit too.
Target: parasoft-static-analysis-misra-c-2023. Same eval, run alone and then with every other skill on this machine loaded via --skill-library, including its near-twin parasoft-static-analysis.
| Metric | Skill alone | With 48 other skills |
|---|---|---|
| Trigger recall | 50% | 25% |
| Trigger precision | 100% | 100% |
| MISRA prompts missed | 2 of 4 | 3 of 4 |
| MISRA review task (scale.c) | ✓ passed | ✓ passed |
Claude Code, Sonnet, one run per prompt, 3 Oct 2026. Treat single runs as signals, not verdicts.
Lost when crowded
"Check this C function against MISRA C:2023…" used the skill alone, but not once the library was installed.
Never attracted
Rule questions ("signed/unsigned conversions", "switch without default") were answered from Claude's memory in both runs. The description doesn't claim rule-knowledge questions.
Correct routing
AUTOSAR C++14 went to the sibling parasoft-static-analysis; libFuzzer went to libfuzzer. No false triggers.
Action: sharpen the description to claim MISRA C:2023 rule questions and to say how it differs from parasoft-static-analysis, then re-run both modes. Result on the next slide.
- description: "Analyzes C/C++ code using Parasoft C/C++test - static analysis rules combined with MISRA C:2023 standard. - Covers MISRA-C:2023, CERT-C, AUTOSAR, CWE, ... Use when - reviewing code for MISRA C:2023 compliance ..." + description: "MISRA C:2023 expert with the matching Parasoft + C/C++test checkers. Use for ANY request that names MISRA + C:2023: what a rule requires, whether a construct violates + it (goto, switch without default, signed/unsigned or + narrowing conversions), rule numbers such as Rule 10.4 ... + Prefer this over parasoft-static-analysis whenever MISRA + C:2023 is named; use parasoft-static-analysis for AUTOSAR, + CERT, or older MISRA editions."
Same 9 trigger prompts and review task, Claude Code, Sonnet, one run per prompt, 3 Oct 2026.
| Trigger recall | Before | After |
|---|---|---|
| Skill alone | 50% | 100% |
| With 48 other skills | 25% | 100% |
| Precision (both modes) | 100% | 100% |
| MISRA review task | ✓ | ✓ |
The old description listed standards; the new one claims the questions users actually ask and draws the line against its near-twin. AUTOSAR still routes to parasoft-static-analysis and libFuzzer to libfuzzer: the fix didn't steal anyone else's prompts.
"Our study follows prior work by directly injecting active skills into the agent prompt. This setup does not evaluate skill retrieval or triggering, which becomes important as the number of available skills grows."
Tang et al., Google Research and Virginia Tech, Aug 2026.
In Claude Code a skill has to win the routing decision against every other installed skill before it can help. That's exactly what --skill-library measures: our MISRA skill fell to 25% recall in a 49-skill library, then reached 100% after one description change.
| WikiSkill method | Now in Wazatator v0.3.0 |
|---|---|
| Paired bootstrap, 1,000 resamples, p < 0.05 | Significance in --baseline and compare |
| Cross-model transfer table, negative transfer | Skill transfer matrix per --model |
| Gating and rollback on validation score | gate + keep-or-revert loop in the plugin |
skill-impact.md history tracker | --impact-log skill-impact.md |
| Train / validation / test splits | Held-out tasks: --tags '!holdout' |
| 5 failing + 3 passing traces, 15k chars each | Failure diagnosis in /wazatator:eval |
| "When to Apply / When NOT to Apply"; 45 to 143-line skills | check: boundary sections, skill-too-long, low-level-steps |
# 1. tune on visible tasks, log every attempt wazatator run eval.yaml --model haiku --tags '!holdout' \ -o new.json --impact-log skill-impact.md # 2. keep the SKILL.md edit only if this exits 0 wazatator gate --baseline old.json --current new.json pass rate: 33.3% -> 100.0% outcome: PASS (exit 0) wazatator compare old.json new.json Significance: +50.0 pp, 95% CI [+0, +100], p = 0.15 -> not significant: add tasks or trials # 3. final run: holdout included, three Claude models wazatator run eval.yaml --baseline --model haiku --model sonnet --model opus
pirate-greeter demo skill, Claude Code. Steps 1-2: 2 trials per task, 4 Oct 2026. Step 3: one run, 4 tasks, 3 trials per task, 5 Oct 2026; cost is what Claude Code reported for the 12 with-skill task runs.
| Model | With | Without | Δ pp (95% CI) | p | Cost |
|---|---|---|---|---|---|
| haiku | 66.7% | 25.0% | +41.7 [+0, +83] | 0.12 | $0.26 |
| sonnet | 75.0% | 25.0% | +50.0 [+0, +100] | 0.12 | $0.39 |
| opus | 100% | 25.0% | +75.0 [+25, +100] | 0.01 * | $0.74 |
Gate passed
A description fix took Haiku from 33% to 100% on the tuning tasks. Logged and kept.
Holdout failed
"Say hi to this person" failed on Haiku and Sonnet: the skill fired, but they never read newcomer.txt to find the name. Only Opus did. Tuning alone would have shipped that gap.
Which model?
Only Opus is significant (p = 0.01), at 2.8× Haiku's cost. Fix the skill to read the file and re-run before paying for Opus; Wazatator shows the trade-off instead of guessing.
Replay of an actual run on 2 Oct 2026 (Sonnet runs the tasks, Haiku judges). Output is abridged; timings are compressed.
/wazatator:new-eval pirate-greeter # writes eval.yaml, 5 tasks, fixtures, trigger tests /wazatator:eval evals/pirate-greeter/eval.yaml # runs it, explains failures, proposes SKILL.md fixes, # keeps a fix only if the gate passes, logs it, checks holdout /wazatator:quality skills/code-explainer # static check + LLM review of the SKILL.md itself
Tested live: Claude Code ran the plugin, which ran Wazatator, which ran its own Claude Code sessions.
/wazatator:quality on code-explainer: 3.8 / 5"Primary weaknesses are fuzzy scope boundaries around code complexity/type and ambiguous guidance about when to invoke versus use file tools directly."
| Capability | claude plugin eval | Wazatator |
|---|---|---|
| What it tests | A plugin; the suite lives in the plugin's evals/ | Any SKILL.md, wherever it lives; no packaging |
| Cheapest Claude model that does the job | One model per run; compare by hand | ✓ Haiku, Sonnet, Opus with and without the skill in one run: does it make Haiku good enough? |
| Score with vs. without the skill | ✓ | ✓ --baseline |
| Collisions with your other skills | – | ✓ --skill-library, prompts stolen per skill |
| Trigger precision / recall / F1 | Per-case "skill fired" check | ✓ should / should-not suites, confusion matrix |
| Custom-code graders | – "no custom-code graders" | ✓ code, program, JSON schema, file diff |
| Lint the SKILL.md itself | – | ✓ check (token budget, compliance), quality (LLM rubric) |
| Prompt-injection / adversarial packs | – | ✓ adversarial |
| Skill transfer / negative transfer | – | ✓ per-model with/without matrix, flags models the skill hurts |
| Statistical significance | Mean Δ only | ✓ paired bootstrap: 95% CI and p-value |
| Regression gate vs. last version | Absolute score threshold | ✓ gate vs. previous results, --impact-log, held-out tasks |
| Built in, zero install | ✓ | – CLI + Claude Code plugin |
claude plugin eval tells you whether your plugin helps Claude. Wazatator tells you whether your skill is well written, triggers precisely, survives your other skills and attacks, and which Claude model it needs. Use both.
Lance Martin's /claude-api prompt-audit (also /checkup prompt-audit) scans CLAUDE.md, AGENTS.md, skills and prompts for six anti-patterns newer models no longer need: verification rituals, emphasis boosters, mandatory scaffolds, stale few-shot examples, contradictory rules, and dated configuration.
| prompt-audit | Wazatator | |
|---|---|---|
| Method | Static review of the instruction text | Runs real Claude Code sessions and grades them |
| Scope | CLAUDE.md, AGENTS.md, skills, agents, prompts | Skills and .agent.md agents |
| Finds | Wording that may hurt performance | Behavior that does: missed triggers, collisions, wrong answers, policy violations, cost |
| Proves a change helped | – advice | ✓ before/after, --baseline, compare |
| Overlap | Static review only: wazatator quality also scores an anti_patterns dimension | |
prompt-audit says "delete this rule". It can't tell you whether deleting it breaks the skill; Wazatator measures exactly that. The MISRA skill's text looked fine, yet only running it showed 25% recall in a crowded library.
1. Run prompt-audit and apply its suggestions.
2. Re-run wazatator run --baseline and --skill-library.
3. Keep the change only if the scores hold or improve.
claude-skillify (0xMH, MIT) ports Anthropic's internal /skillify: at the end of a session it analyzes the conversation, interviews you, and generates a SKILL.md with steps and success criteria. It doesn't run or test the skill it writes.
| Skillify | Wazatator | |
|---|---|---|
| Job | Author: turn a workflow you just did into a skill | Test: measure how a skill behaves, then improve it |
| Input | The current conversation + your answers | Any SKILL.md + an eval suite of realistic tasks |
| Output | A new SKILL.md | Scores, failures with reasons, gated fixes, skill-impact.md |
| Does the skill fire? Collisions? | – | ✓ trigger precision / recall, --skill-library |
| Better than no skill? On every model? | – | ✓ --baseline, significance, transfer matrix |
| Overlap | Small: wazatator new skill scaffolds an empty template; it doesn't mine your session | |
A skill generated from one session encodes what worked once, for one model, often as detailed steps. That's where WikiSkill saw negative transfer, and where trigger collisions hide. Skillify gives you the draft; Wazatator tells you whether to trust it.
1. /skillify at the end of the session.
2. /wazatator:new-eval: its per-step success criteria make good graders.
3. /wazatator:eval with --baseline and --skill-library.
4. Keep the skill, and every later edit, only if the scores hold.
tw93/Waza (MIT, 7k+ stars) is a pack of eight skills: /think, /ui, /check, /hunt, /write, /learn, /read, /health. Same root word, different projects: Wazatator is a fork of Microsoft Waza, an eval framework, and has no relation to tw93's skill pack.
| tw93/Waza | Wazatator | |
|---|---|---|
| What it is | A skill pack: things Claude does | A test harness: measures what a skill does |
| Output | Plans, reviews, root-cause debugging, polished prose | Pass rates, trigger precision / recall, collisions, with vs. without skill, significance |
| Evidence its skills help | "Refined through 300+ sessions"; repo tests | That evidence is the product: --baseline, transfer matrix, gate |
| Overlap | Small: /health audits an agent's setup statically; wazatator check / quality review one SKILL.md statically | |
Waza's rule, "state the outcome, the red lines, and how it's verified, then step back", matches WikiSkill's finding: general procedures transfer across models, while low-level workarounds can hurt them. Wazatator's low-level-steps check warns about exactly that.
--baseline: does /hunt beat plain Claude?--skill-library: do /check and /health, or /read and /learn, compete for the same prompts?--model × 3: which Claude model each skill needs.
| Security benefit | How an eval expresses it |
|---|---|
| Prompt injection resistance | Adversarial task: malicious instructions inside an input fixture. Graders fail the run if the agent disregards the skill's rules. |
| Tool usage restrictions | Name the prohibited commands and tools in the graders; pass/fail is computed from the recorded tool calls. |
| Sensitive data protection | Output and file validation: fail the run if generated outputs contain credentials or other restricted information. |
| Security regression detection | --baseline over security tasks: repeat them after any skill, model, or agent configuration change, and gate in CI. |
| Investigation evidence | Transcripts, tool calls, and generated files from every run, to understand an unsafe action instead of guessing. |
These are uses of the documented capabilities (adversarial testing, execution recording, behavior and output graders, CI integration) that you configure per skill; they are not security checks that run out of the box. Source: security assessment of github.com/zuwasi/wazatator, 13 Oct 2026.
A security eval needs three things from you: the security requirement, the adversarial or representative input, and an observable pass/fail rule. Example: hide instructions to disclose a synthetic secret inside an untrusted document, then grade the output and tool calls for prohibited behavior.
Not a runtime enforcement system, not a vulnerability scanner, and not a substitute for sandboxing and access controls. Recording an unsafe action or failing an evaluation does not necessarily prevent it.
With appropriate isolation, restricted permissions, and synthetic sensitive data. Passing a suite is evidence for the scenarios tested: it does not prove resistance to every attack or guarantee the same behavior in a different execution environment.
Recommended positioning: "An AI agent skill evaluation framework supporting security testing and behavioral assurance."
irm https://raw.githubusercontent.com/zuwasi/wazatator/main/install.ps1 | iex
curl -fsSL https://raw.githubusercontent.com/zuwasi/wazatator/main/install.sh | bash
/plugin marketplace add zuwasi/wazatator-plugin /plugin install wazatator@wazatator
Claude Code installed and logged in. Every task and judge is a real session on your account, so keep trials_per_task: 1 while iterating.
MCP servers and hermetic MCP mocks, .agent.md tool policies, follow-up turns, reasoning effort, CI gating, and side-by-side model comparison with wazatator compare.
CLI and releases: github.com/zuwasi/wazatator
Claude Code plugin: github.com/zuwasi/wazatator-plugin
Based on Microsoft Waza (MIT License). Wazatator adds the Claude Code executor and plugin. It is not affiliated with or endorsed by Microsoft.
Since 2005, ESL has helped Israeli software teams adopt and get the most out of leading development tools: code analysis, testing, version control, compliance, and AI coding agents. As a member of Anthropic's Claude Partner Network, we help teams adopt Claude and Claude Code, and build open tools like Wazatator to make AI agents measurable.