Literal interpretation
Ambiguity, implied conditions, and contradictory criteria can change the answer.
JEV 1.13 · AN APPLICATION-LEVEL EXPERIMENT← Swipe to navigate →
01 / Know the boundary
TypeSafe documentation · Jev 1.13Structured answers.
Not guaranteed correctness.
The limitations tell us where the model should stop and application code should take over.
Numbers and dates
Counting, arithmetic, and date comparisons belong in deterministic code.
Multiple reasoning steps
Indirection and multi-hop questions are less reliable. Keep decisions narrow.
Context and injection
Irrelevant information distracts. Adversarial text can still steer decisions.
No universal threshold
Question formats can disagree. Test each task, model, and decision policy.
Selection, not generation
Use a generative model for explanations, prose, and code.
A response can satisfy the schema and still send the finding to the wrong reviewer.
02 / Filter by component
SYNTHETIC EXAMPLELess noise.
Keep the inconvenient evidence.
Target: demo-server. The question is which review queue should receive the finding, not whether it is safe.
Before: all available context
Target descriptionA debug endpoint is enabled in deployment settings.
demo-server · fixture:deploymentenable_debug_endpoint=true. Inspect deployment settings.
unrelated-lib · other lockfileA dependency version needs investigation.
BackgroundUnrelated release notes and surrounding report text.
After: component-specific context
Keep the full descriptionDo not rewrite away ambiguity or misleading claims.
Keep every matching evidence itemPreserve source references and contradictions for the target.
Remove unrelated contextExact component identity, not a keyword or sentiment filter.
The caller must map evidence to the right component.
Filtering is not sanitization. Relevant cross-component evidence must be attached to the target or reviewed manually.
03 / Follow one finding
SCRIPTED ILLUSTRATION · NO LIVE AIA review gate helps.
It does not prove correctness.
1 · Finding
2 · Context
3 · Decision gate
4 · Outcome
04 / Count errors and deferred work
SYNTHETIC · SCRIPTED MODEL · NOT JEV PERFORMANCEFewer misroutes is only half the story.
8 invented cases. 7 labeled important. The demo always picks the dependency queue with probability 0.97.
Try 0.98: all findings are deferred, not solved.
Correct automatic routeWrong automatic routeDeferred for review
Missing evidence gates defer 3 important cases at 0.90. Misleading text and conflicting evidence still cause wrong routes. This is not a measured Jev improvement.
05 / The AI SDLC boundary
Let the model propose the next move.
Let evidence establish what actually happened.
275 passing tests
46 new experiment checks plus 229 existing adapter tests. Filtering, review policy, metrics, and API contract mocks. These are software checks, not 275 Jev accuracy trials.
Real-model evaluation
No live Jev calls or historical reviewed cases in this experiment. Calibrate on one set, evaluate on a held-out set, then measure repeatability and review workload.
No finding is suppressed, closed, or declared harmless. No model weights or SDK semantics were changed.
Daniel Liezrowice · ESL · AI SDLCIndependent experiment · 20 September 2026