Droid as the Locate-and-Judge Component

How SkillSpector + Droid implement the two-stage security architecture from the arXiv paper

Detecting Malicious Agent Skills in the Wild using Attention

← Swipe to navigate →

The Problem: Single-Stage Scanning Has a Recall Ceiling

SkillSpector is a fast, regex-based skill scanner. It has high precision — when it flags something, it's usually right. But it has a fundamental limitation.

0.929

Precision (good)

0.200

Recall (misses 80%)

0/7

Hidden malicious skills detected

Regex patterns can't catch what they don't explicitly match. Hidden malicious skills — obfuscated payloads, encoded instructions, disguised attacks — sail right through.

The arXiv Paper Benchmark

Researchers tested production skill scanners against 65 confirmed malicious skills. The results:

System	True Positives	False Positives	False Negatives	Precision	Recall
Locate-and-Judge	62	7	3	0.899	0.954
SkillSpector	13	1	52	0.929	0.200
Cisco Skill Scanner	10	0	55	1.000	0.154

The paper's insight: two stages beat one. A fast locator flags candidates, a smart LLM judge evaluates them.

The Two-Stage Architecture

Stage 1: LOCATE

Find suspicious regions

SkillSpector (regex)
+ Droid Deep Scan

→

Stage 2: JUDGE

Evaluate each finding

Droid (LLM)
Real risk or FP?

→

REPORT

Verified findings +
knowledge base

Learning loop
improves over time

Droid plays the Judge role — and in Deep Scan mode, it plays both roles for skills that passed regex.

Stage 1: Locate — Two Modes

Regex Locate

SkillSpector's engine scans for known malicious patterns: prompt injection phrases, eval() calls, network exfiltration, dangerous imports.

Fast High precision Misses hidden threats

Droid Deep Scan Locate

For skills that passed regex, Droid reads the full content and hunts for obfuscated payloads, encoded instructions, disguised attacks.

Catches hidden threats Slower Uses past FN indicators

Deep Scan includes known False Negative indicators from the knowledge base — patterns that regex missed before but were confirmed malicious.

Stage 2: Judge — Droid Verdicts

For each flagged finding, Droid reads the code, the rule explanation, and the context, then decides:

REAL RISK

Droid confirms the finding is a genuine security threat. Provides the fix, shows corrected code, rates severity.

FALSE POSITIVE

Droid explains why it's not a real risk. The case is recorded to the knowledge base with the safe indicator. Auto-suppressed next time.

FALSE NEGATIVE

Droid finds a real risk that regex missed. The indicator is recorded so future Deep Scans hunt for it.

The Learning Loop

Every Droid verdict is recorded and fed back into future prompts. The system gets smarter with each scan.

⚡

Scan

Droid Judges

📚

Record Verdict

🔄

Feed Back

FP cases → auto-suppressed → fewer false alarms
FN indicators → Deep Scan hunts for them → higher recall

The Knowledge Base

Two persistent files preserve every verdict:

droid_knowledge.md

Human-readable append-only log. Every FP and FN verdict with timestamps, explanations, and indicators.

droid_knowledge.json

Machine-readable index. Used to build prompt sections with relevant past cases.

suppressions.yaml

Auto-populated when an FP is recorded. Same rule+path won't be flagged again.

Prompt Injection

Future Droid prompts include past FP cases as context. Droid reasons by analogy: "same safe indicator = likely FP."

The Full Pipeline

⚡ SkillSpector regex scan
   ▼ flags known malicious patterns
   ◀▶ findings list

🤖 Droid verifies each finding (Judge)
   ▼ reads code + rule + past FP context
   ◀▶ REAL RISK → fix + severity
          FALSE POSITIVE → record to KB + suppress

🔍 Droid Deep Scans clean skills (Locate + Judge)
   ▼ reads full content + past FN indicators
   ◀▶ finds hidden malicious patterns regex missed
          FALSE NEGATIVE → record indicator to KB

📚 Knowledge Base grows
   ▼ next scan: smarter prompts, fewer FPs, higher recall
   🔄 repeat

Paper vs. Our Implementation

Aspect	Paper (Locate-and-Judge)	Our Implementation
Stage 1: Locate	Attention-based neural network	SkillSpector regex + Droid Deep Scan
Stage 2: Judge	DeepSeek (LLM)	Droid (LLM, local)
Training data	Skill-Inject corpus (lab)	Real-world verdicts (production)
Learning	Static model	Learning loop — KB grows
F1 Score	0.925 (fixed)	Improves over time
Hidden skills recall	0.83	Droid Deep Scan + FN indicators
Infrastructure	ML training pipeline	Droid on your machine

SkillSpector Dashboard — Command Center

SkillSpector performs the fast lexical Locate pass; Droid performs the contextual Judge pass. The Scan tab is where operators triage findings.

Actions menu — copy issues, open in preferred editor at the exact line, Verify with Droid (single issue judge), Analyze checked issues in Droid (batch judge)
False Positives menu — mark FPs (with confirmation), view the suppression list; approved FPs are annotated in source by Droid
Knowledge menu — record Droid verdicts, Deep Scan clean skills to hunt hidden false negatives, view the knowledge base
Reports menu — export dashboard HTML reports
8-column findings table — Check, Severity, Rule, Location, Confidence, Pattern, Category, FP; clicking a rule code (e.g. OH1) opens an explanation page with examples
Details panel — full rule explanation, remediation guidance, and the matched code snippet for the selected issue

Droid Auto-Marks False Positives in Source

When you confirm a False Positive, Droid does not just suppress it in the dashboard — after approval, the workflow asks Droid to annotate the actual source file using that file's comment convention, so the verdict is permanently documented in the code.

FP annotation per language

# SkillSpector-FP: OH1 - input sanitized via whitelist
# Python / shell / YAML comment

// SkillSpector-FP: OH1 - input sanitized via whitelist
// JavaScript / C / Java comment

<!-- SkillSpector-FP: OH1 - safe -->
<!-- Markdown / HTML comment -->

What happens after approval

Droid reads the file and inserts the FP comment at the flagged line
The case is recorded in droid_knowledge.md with the safe indicator
The rule + path is added to suppressions.yaml
Future scans skip this finding automatically
Future Droid prompts receive this case as context for similar findings

A backup of the original file is saved before Droid makes any change, and the file is retested and re-verified.

Droid as Judge — Terminal View

The embedded terminal runs Droid directly inside the dashboard. Droid evaluates located findings, suggests new scanner rules, and confirms clean files.

Droid evaluates each finding — explains the risk, suggests mitigations, rates severity
Suggests new scanner rules — e.g. PI2 - Untrusted-Content-As-Instructions, INST1 - System Python Protection Bypass
Confirms clean files — lists skills that passed both regex and Droid Deep Scan
Real risk or FP? — Droid's verdict is recorded to the knowledge base for future scans and prompts

Configuration — The Learning Loop Toggle

Everything is configurable: agent discovery, preferred editor, Droid path, engine updates, and the knowledge feedback loop.

Auto-detect agents — finds Amp, Claude Code, Factory/Droid skill locations automatically
Preferred editor — auto-detects VS Code, allows manual override; clicking a finding opens the file at the exact line
Droid path — auto-detected or manual; Droid runs in the embedded terminal
Feed past Droid knowledge into future prompts — the learning-loop toggle. When enabled, Droid receives past FP/FN cases as context
Engine auto-update — checks for upstream SkillSpector engine updates daily and auto-applies them
Save / Load config — persists all settings to a portable config.json

Knowledge Transfer: Droid → SkillSpector

Droid's verdicts do not disappear after the terminal session. They flow into persistent knowledge bases that make both Droid and SkillSpector smarter.

Droid Analyzes

Reads the finding, the code, the context

→

Droid Verdict

REAL RISK, FALSE POSITIVE, or FALSE NEGATIVE

→

Knowledge Base

Recorded with explanation + indicator

If FALSE POSITIVE

droid_knowledge.md — case logged with safe indicator
suppressions.yaml — auto-suppressed, won't re-flag
Source code annotated with FP comment
Future prompts: "same safe indicator = likely FP"

If FALSE NEGATIVE

droid_knowledge.md — missed indicator logged
droid_knowledge.json — machine-readable index
Future Deep Scans: "hunt for this indicator"
Droid suggests new regex rules to catch it

The knowledge base is the bridge: Droid's intelligence becomes SkillSpector's permanent memory.

Locate-and-Judge, Built on What You Have

No engine rewrite. No ML infrastructure. Just SkillSpector + Droid + a knowledge base that learns.

Stage 1

SkillSpector + Deep Scan

Stage 2

Droid Judge

∞

Learning loop

The more you scan, the smarter it gets.