Misrouted
support ticket
CASE / 0011% error probability × $5 loss
$0.05
01 / The example
Same probability.
Different action.
A 1% chance of being wrong is not a decision policy.
The consequence of being wrong changes everything.
Misrouted
CASE / 001Wrongful
account closure
CASE / 0021% error probability × $10,000 loss
$100
Illustrative costs, not measured Jev performance. Assumes validated error probabilities, zero loss when correct, and error-free review.
*Economic preference only. Hard rules and mandatory approvals still apply.
← Swipe to navigate → or use the arrow keys
02 / The decision lab
Move the probability.
Watch the policy change.
Same model estimate. Two consequences. A separate layer decides what happens next.
These switches model preconditions. Checking a box here does not perform authorization or validate a model.
Ticket / $5 if wrong
$0.05
expected error loss
Account closure / $10,000 if wrong
$100.00
expected error loss
Automate only if p(error) × cost(wrong) < cost(review)
Simplified expected-cost comparison: correct actions cost $0; review is error-free and has no delay cost. Equality goes to REVIEW. Real review has residual errors and delay costs; real policies may forbid automation regardless of the economics.
03 / The implementation
Make abstention
part of the contract.
A small, model-independent policy. No model gets to grant itself permission.
decision_policy.py / Python
def decide(p_error, wrong_cost, review_cost, *,
policy_allows, calibration_valid):
if not policy_allows:
return "BLOCK"
if not calibration_valid or p_error is None:
return "REVIEW"
if not 0 <= p_error <= 1:
raise ValueError("p_error must be between 0 and 1")
expected_loss = p_error * wrong_cost
return "AUTO" if expected_loss < review_cost else "REVIEW"
print(decide(0.01, 5, 2,
policy_allows=True, calibration_valid=True))
print(decide(0.01, 10_000, 2,
policy_allows=True, calibration_valid=True))Educational policy, not a Jev SDK integration. Assumes finite non-negative cost inputs. Production systems also need input validation, audit records, and explicit handling of API failures.
04 / Two technical distinctions
A valid answer
can still be the wrong action.
01 / JEV CONFIDENCE
A concentrated distribution
is not permission.
TypeSafe describes confidence as a statistic derived from the probability distribution of Choice and Score answers. It is not an independent correctness check. Noul answers do not carry this field.
Validate calibration on your own workload. Do not read confidence = 0.99 as “99% safe to execute.”
02 / STRUCTURED LLM OUTPUTS
Typed output is not
exclusive to Jev.
Modern LLM APIs support schema-constrained structured outputs. Applications still need to handle refusals and incomplete responses. A schema-valid answer can contain a semantic mistake.
Compare measured decision quality, calibration, latency, and cost, not “typed versus untyped” alone.
https://developers.openai.com/api/docs/guides/structured-outputs ↗Measure how much work you can automate
at an acceptable error cost.
Source documentation reviewed 22 September 2026. These are design considerations, not comparative model benchmark results.
05 / The takeaway, in one image
99% correct
is not
a policy.
Typed output is not proof of correctness.
Confidence is not permission.
Keep an explicit abstain path.
Illustrative economics with stated assumptions.
Generated illustration, not evidence of model performance.
06 / ESL AI SDLC
The model estimates.
The policy authorizes.
Put the distinction into your AI software lifecycle:
define the rubric, validate the estimates, enforce the policy, monitor the outcomes.
Start with one decision.
Which action in your agent needs a cost-aware policy, not just a confidence threshold?
Let’s talk AI SDLC ↗Daniel Liezrowice
ESL · Engineering Software Lab
daniel.l@eswlab.com+972 9 8855803https://eswlab.com/ ↗ESL AI SDLC field notes · September 2026 · Educational example
Explore more ESL presentations