Skip to content

Context Graphs Are Not a Future Feature — They're What Run 7's Cryptography Failure Demanded

journey/22 named the concept and filed it as direction. journey/38.5 shows what happens when the concept goes unimplemented and the corpus grows past the point where the underlying signal can support it.

This entry is the step-back. What does the cryptography case actually demonstrate about the shape of memory in an agentic system?

The diagnosis from 38.5, in one sentence

A tip was retrieved during a successful attempt. The tip's advice was ignored. The success was caused by something else. The retrieval system recorded outcome_value = 1 for that tip because the rule passed. The tip accumulated a "helpful" reputation it hadn't earned. Five runs later, an obedient-to-context Worker followed the same tip. The rule failed.

The memory system stored an outcome. It didn't store the path to the outcome.

Why this is the context-graph thesis, restated

In journey/22 the framing was theoretical, drawn from Foundation Capital and the Mem0/Graphiti work: "the knowledge about why a decision was made may be more valuable than the output itself." The pull-quote at the time was about NIST audit logs and traceability for high-consequence agentic systems. It read as a forward-looking property.

The cryptography case is the same thesis in present tense. The tip-retrieval row in Postgres records:

(tip_id, attempt_id, rank, similarity_score, outcome_value, outcome_confidence)

It does not record: did the attempt's behavior reflect this tip's content. That single missing column is the entire gap. The system tracks what happened (rule passed/failed) and what was retrieved (these five tips at these ranks). It doesn't track what caused what. Without that signal, the credit-assignment rule "tip was present + rule passed → tip is helpful" looks reasonable but is mechanically false when the rule passed despite the tip.

This is the knowledge graph vs. context graph distinction in concrete form:

  • Knowledge graph stores facts: rule X exists, tip Y exists, retrieval Z happened.
  • Context graph stores causation: the Worker's script in attempt A did follow the advice in tip Y at semantic match 0.74, and did not follow tip Z at semantic match 0.08; the Evaluator returned PASS at confidence 1.0; therefore credit assigns to tip Y disproportionately.

V2's stig.tips + stig.tip_retrievals schema is a knowledge graph. The cryptography failure is what happens when you ask a knowledge graph to do context-graph work — credit assignment requires causal links the schema doesn't carry.

What's actually changing tomorrow

The architectural change is small in code and large in meaning. We're adding two columns to tip_retrievals:

ALTER TABLE stig.tip_retrievals
  ADD COLUMN tip_followed_llm BOOLEAN,
  ADD COLUMN tip_followed_emb DOUBLE PRECISION,
  ADD COLUMN tip_followed_computed_at TIMESTAMPTZ;

Two columns, both populated by the dream pass at end-of-run by reading the attempt's fix_script from the JSONL alongside the tip's text:

  • tip_followed_llm: an LLM judge reads tip + script and answers "did the script follow this tip's advice?" as a structured yes/no.
  • tip_followed_emb: cosine similarity between the tip text and the fix_script after sentence-transformers encoding. A continuous score the ranker can use directly.

These are deliberately both deterministic on a given input — the LLM judge uses temperature 0 and a fixed prompt; the embedding model is loaded once and is deterministic per-input. Two independent signals scoring the same thing. If they agree, we have high confidence. If they disagree, we have a flag for review.

Crucially: the signal lives at the per-retrieval row. Not on the tip. Not on the attempt. On the intersection — the specific instance of "tip Y was retrieved into the prompt for attempt A." That's the unit at which causation lives. A single tip can be followed in one retrieval and ignored in another, and the score has to reflect that.

This is the context-graph shape Foundation Capital described, expressed in PostgreSQL:

  • Vertex set: tips, attempts, rules, runs (knowledge-graph layer, already present).
  • Edge attributes: per-retrieval causal attribution (context-graph layer, this is what's being added).

The edge attributes are what the dream pass and retrieval ranker query against. A tip with high outcome but low follow-rate gets less weight in future retrievals than a tip with high outcome AND high follow-rate, because the former is a lucky neighbor and the latter is a causal contributor.

Why the LLM judge is load-bearing here

Memory feedback feedback_non_determinism_has_cost says: not everything should be an LLM; deterministic code belongs where correctness must be trustable. The judge is an LLM call. Two reasons it earns its place anyway:

  1. It's not in the critical path. The judge runs in the dream pass at end-of-run, not during retrieval ranking and not during prompt assembly. A wrong judgment slows down credit assignment by some noise floor; it never breaks the live loop. The deterministic-where-correctness-matters principle is about hot-path decisions. This is a cold-path scoring task.
  2. The alternative — pure regex/keyword matching — is too brittle. "Use update-crypto-policies" vs. "use system-wide crypto-policies via update-crypto-policies --set FIPS" should match. Keyword overlap might catch it, but only if you've enumerated the right keywords. The Worker writes natural-language reasoning before its bash blocks; the tip writes natural-language advice. Comparing them at the semantic level needs a model that understands "the script cat > ssh_config is replacing the file, which contradicts the tip's advice to use update-crypto-policies instead of editing the file."

The embedding cosine is the deterministic-where-correctness-matters leg of the same signal. We get both. We can compare them. If embedding alone is sufficient over time, we drop the LLM judge later.

The architectural pattern this generalizes to

What the cryptography case demonstrates isn't unique to STIG and isn't unique to bash scripts. The general pattern:

An agentic system that ingests its own past actions as context must record the causal contribution of each context element, not just the outcome they were present for, or it will accumulate plausible-but-wrong context that survives precisely because it's correlated with success without being responsible for it.

In other domains the cause-vs-correlation problem looks different:

  • A code-review agent learns from past PR comments. A comment was made on a PR; the PR shipped successfully. Did the comment's suggestion get applied? Without tracking the diff between the comment's recommendation and the actual change merged, the comment looks helpful regardless.
  • A customer-support agent retrieves past resolved tickets. A ticket got resolved; the playbook step at index 3 was retrieved as a relevant past case. Did the agent use playbook step 3, or step 5 from a different retrieval? The ticket's outcome credits all retrievals equally without that signal.
  • A research-paper-summarization agent surfaces related papers. The user accepted the summary. Did the related-papers context shape the summary, or was it ignored? Without per-retrieval contribution scoring, the related-papers retriever can't distinguish "useful pointer" from "noise the user tolerated."

These are all forms of the same gap: credit-assignment in retrieval-augmented agentic systems requires causal attribution at the retrieval row, not at the outcome aggregate. The cryptography case is just the version of it we can name with specific tips and specific scripts.

What this means for the whitepaper

The "memory architecture" section of the whitepaper currently leans on the dream pass, the per-tip eviction policy, and the V2 schema as the story. After Run 8 the story has a different shape:

  • Before this finding: "We built a tip-retrieval system with outcome tracking."
  • After this finding: "We discovered the outcome-only signal accumulates plausible-but-wrong tips. The fix is two columns capturing causal attribution. This is the practical form of the context-graph thesis."

The second framing is more interesting both as engineering and as architecture. It's also closer to where the literature is going — Mem0, A-MEM, Graphiti, the NIST AI-agent standards. The trillion-dollar thesis from journey/22 isn't future-tense once you have the per-retrieval causal column.

The memory system stored an outcome. It didn't store the path to the outcome. That one missing column is the whole context-graph thesis in concrete form.

What lands tomorrow and what's still open

Landing in Run 8's pre-flight:

  • Schema additions (tip_followed_llm, tip_followed_emb, tip_followed_computed_at).
  • Dream pass extensions: LLM judge stage + embedding cosine stage.
  • Retrieval ranker uses the new columns in its weighted average.
  • A per-skill prompts/tip_follow_judge.md (STIG ships; CVE adopts the same pattern when next run).
  • An architecture/02 entry capturing the pattern at the architecture-layer reference level, not chronologically.

Open for follow-up:

  • DEF-03 (per-tip dream-pass credit) becomes simpler with tip_followed in place — the dream pass can compute per-tip helpful-when-followed rates instead of category-level signal.
  • DEF-28 (probably): cross-skill validation that the context-graph pattern works the same way for CVE. STIG's tips are bash scripts; CVE's are dnf advisory selections — a different kind of "did you follow the advice."
  • DEF-29 (probably): does the same architectural treatment apply to episodic memory (the chain of reflections) and not just to tips? Reflections in journey/26 are stored similarly to tips and probably have the same lucky-neighbor problem.

The whole arc from journey/22 (the context-graph spark, April 12) to journey/38.6 (the cryptography case, May 21) is 39 days. The first entry filed it as "long-term direction." The second one says "tomorrow morning."