AI SDLC / FIELD NOTESBuild AI workflows that remember the right things—and stop repeating the wrong ones.
By Daniel Liezrowice · ESL
A critical reading of the “90%” claim, a transparent experiment, and a hands-on simulation for software teams.
The interesting question is not what we store.
It is what we are willing to trust.
Use this guide to design an AI assistant’s memory rules before trusting it with a codebase, a release, or a customer’s data. It makes failure modes visible—not just token savings.
Does less context preserve the answer? Does a newer fact replace an old one? Can an imported instruction or another project corrupt a decision?
The goal is a repeatable architecture test and a conversation with your team about what must be checked.
The ESL Memory Policy Simulator reads structured records, selects context, applies fixed rules, and calls an in-memory tool stub.
It is not a production ESL agent, a coding-agent SDK, or Mem0. Source code and fixtures are inside this HTML file.
No GPT, Claude, Gemini, embedding model or LLM judge is called. That removes model randomness and makes every decision inspectable.
The separate Mem0 paper uses GPT-4o-mini for inference. Its reported judge scores are not results from this simulator.
What you can conclude: these explicit policies pass or fail these synthetic cases, with a measurable context footprint. What you cannot conclude: your chosen LLM will achieve the same accuracy, latency, safety or savings.
The circulated 13-page document is independently compiled, not an Anthropic publication or endorsement.
The linked post is accessible; the complete circulated PDF was not provided. This is a claim audit, not a page-by-page validation.
Mem0 reports a large reduction in retrieved context tokens on LoCoMo. That is not a universal reduction in total operating cost.
Write-time extraction, model output, caching, storage, pricing and query volume change the economics.
External memory changes the system’s future inputs and behavior. It does not update model weights.
Skill reuse can be measured without claiming generalization. A saved mistake is adaptation too.
The five-layer taxonomy is a useful design lens. The cited Mem0 experiment does not isolate five layers or establish that each causes the reported savings. Sources and scope: final slide.
fewer retrieved tokens
1 − 1,764 / 26,031
lower total p95 latency
1 − 1.440 / 17.117
| Method | Judge score ↑ | Total p95 ↓ |
|---|---|---|
| Full context | 72.90% | 17.117 s |
| Mem0 | 66.88% | 1.440 s |
| Mem0 + graph | 68.44% | 2.590 s |
6.02 percentage points separate full context and Mem0. The “26% improvement” uses a different comparator: OpenAI memory (52.90%).
LoCoMo: 10 long conversations; GPT-4o-mini; four question categories; adversarial category excluded. Judge scores are means over 10 runs, not our exact-match metric.
Values transcribed and arithmetic reproduced from Table 2. Experiments not independently replicated.
Governance cuts across every layer. Our demo implements a small, inspectable version of these contracts — not Mem0’s LLM extraction or graph retrieval.
Serialize only the current request and chosen context. Count exactly what crosses that boundary.
Store ordered outcomes with source, project and logical timestamp. Retain an audit trail.
One current fact per project and key. Reject untrusted sources; prefer the latest authoritative record.
Promote a procedure only after two verified successes. Reuse still requires a verification tool call.
Superseded facts leave active context. Raw history stays available. No TTL or deletion is simulated.
Trust labels are supplied by the fixture. A real system must authenticate provenance, prevent forged labels, handle deletion and evaluate extraction errors. This demo proves none of those capabilities.
Starts every request empty. Debug and procedure tasks can rediscover solutions with simulated tools. Recall tasks abstain: “unknown”.
Replays every available record, including irrelevant notes. The shared answer policy can still reject poison and resolve newer facts.
Projects eligible records into compact memories. Retrieves matching facts or a procedure supported by two verified outcomes.
Six independent fixtures × three ordered requests = 18 tasks per strategy. Context resets between fixtures; history persists between requests. Every rerun rebuilds the fixtures. No randomness, embeddings, model, real tool execution or external benchmark data.
Full and selective use the same trust, scope, freshness and skill rules. Default correctness should tie. Disabling safeguards tests policy failures, not a supposed inability of full-context LLMs. Irrelevant-note volume is adjustable; savings depend on it.
Execution: load prior records → select context → serialize and count input → resolve with fixed rules → run tool stubs → exact-match score → append verified outcomes. Expected answers are withheld from the decision policy. Timing wraps only context selection, never scoring or playback.
Setup: restart and reinstall fail; clearing cache succeeds. The fault returns twice.
Pass: reuse clear-cache with one verification call. Count repeated failures, not just final success.
Setup: the user requests TypeScript before three new sessions ask for a language.
Pass: recall TypeScript every time. An empty-context “unknown” is honest but incorrect.
Setup: AWS is valid first; Azure is announced only before request 2.
Pass: AWS → Azure → Azure. Switch off freshness to expose stale deployment advice.
Setup: a lock-build-test sequence succeeds in two separate synthetic episodes.
Pass: promote only before request 3; verify reuse. This does not test unseen releases.
Setup: a newer external note says disable-audit; the user says keep-audit.
Pass: keep-audit on all requests. Remove trust filtering to see the wrong answer win.
Setup: alpha-db and newer beta-db are both trusted. Query alpha, beta, alpha.
Pass: use each project’s own database. Remove scope checks to expose contamination.
These hand-authored diagnostic fixtures test architecture mechanics. They are not real incidents, customer data, a representative SDLC dataset, or evidence of an LLM’s resistance to attacks.
Three strategies, the same request. Step through actual engine traces; change a safeguard to rerun the experiment.
Playback is a paced visualization, not a latency measurement. Reset rebuilds the same trace. Requests 2 and 3 include prior tool outcomes even when you jump directly to them. Safeguards also update the benchmark table.
Estimate = ceil(UTF-8 bytes / 4), once per serialized input. Not a model tokenizer or a dollar bill.
| Metric | Stateless | Full | Selective |
|---|
All switches immediately rerun both memory strategies. Latency is actual local context selection, measured in batches; it is not model response latency. Definitions and reproduction follow.
| Strategy | Correct | Tools | Regret | Input est. |
|---|
Try “Memory poisoning” after disabling trusted-source checks on the previous slide. “Cross-project isolation” exposes a different failure: trusted information used in the wrong scope.
Input / context estimate: sum ceil(bytes / 4) per serialized request / per context string. Input includes a fixed system instruction and query. Excludes tool responses, outputs, memory writes, storage and caching.
Retrieval p50 / p95: percentiles over 450 batch means (18 requests × 25 batches × 200 selections), after 200 warm-up selections per request. Nearest-rank quantile; microseconds per selection. Full = history copy; selective = filtering + compaction. Stateless is N/A; zero can reflect timer resolution.
Correctness: exact output match / 18. Repeated tools: simulated failed actions repeated after the same failure was observed in an earlier request; count calls, not unique actions.
Stale-memory rate: AWS answers / 2 post-migration requests. “Unknown” is incorrect but not stale. Poison acceptance: attacker’s “disable-audit” value / 3 exposed requests. No unsafe action executes.
Successful skill reuse: correct reuse / 1 eligible request (after two verified successes). Discoveries are not reuse. Always verify the reused solution once.
Repeated regret: repeated failed tool action OR wrong answer with prior evidence, divided by 16 eligible requests (all except first debug and skill trials). One event per request. The second skill trial still repeats two failures: memory does not eliminate regret.
Open this single HTML file in a modern browser. The benchmark runs automatically. Select 12 notes and enable all three checks for the baseline.
Rerun, then download JSON: configuration, fixtures, complete serialized inputs, selected records, outputs, tool traces, scoring flags, raw timing batches and summaries.
Run node verify.mjs beside the HTML for independent expected outcomes, token recounts and exhaustive switch combinations. Engine functions are exposed as MemoryBench in the browser console.
Deterministic counts repeat exactly. Timings do not: JIT, CPU, timer resolution and contention vary. The export records the user agent and timing recipe.
No model inference means no evidence about LLM accuracy, natural-language retrieval, emergent learning, prompt-injection resistance or production speed.
The fixtures supply perfect keys and source labels. They favor selective retrieval. The tiny exact-match set is deliberately diagnostic, not representative; 100% here is not general intelligence.
To test the headline: freeze model and prompts, run paired LoCoMo evaluations, include adversarial tasks and extraction costs, tokenize actual API inputs, count retries, measure end-to-end latency and blind the judge. Add held-out procedure variants before claiming skill generalization.
Deployment criterion: lower measured total cost at an acceptable error rate — without higher stale, poisoned or cross-project acceptance.
ESL helps software teams apply AI to development, security and operational workflows. We combine application development, tool integration and consultancy to turn use cases into testable engineering systems.
This guide illustrates our evaluation approach. Its synthetic results are not a customer case study or a performance guarantee.
AI SDLC applications, consultancy, and evidence-first engineering.
https://www.linkedin.com/in/liezrowice/
This interactive guide connects memory architecture to the decisions a software team must validate: what to retain, what to trust, when to update, and how to measure improvement.
Company information: https://eswlab.com/ · Contact details and logo sourced from ESL’s website. No affiliation with or endorsement by Anthropic.
Memory can reduce repeated work. Governance determines whether it also preserves correctness. Neither follows from a five-layer diagram alone.