The Cryptography Regression Was Misleading Tips, Not Drafter Randomness¶
journey/38 named the regression and named two hypotheses. The triage found a third one neither had anticipated. The cryptography category went 83% → 29% not because the drafter changed token sampling and not because the snapshot diverged, but because the tip retrieval pipeline served five misleading tips that the corpus had marked as helpful for an artifactual reason — Run 6 succeeded by ignoring them.
This entry is the postmortem. The next entry (journey/38.6) is the architectural read.
The rule that regressed¶
xccdf_org.ssgproject.content_rule_harden_sshd_macs_openssh_conf_crypto_policy — "Configure SSH Client to Use FIPS 140-2 Validated MACs: openssh.config." Same rule, same starting snapshot, same Postgres tip pool, same OpenSCAP evaluator:
- Run 6: remediated on attempt 2.
- Run 7: escalated after 3 attempts.
The Worker scripts side-by-side:
# Run 6, attempt 2 (succeeded)
cp /etc/ssh/ssh_config /etc/ssh/ssh_config.bak.$(date +%s)
cat <<EOF > /etc/ssh/ssh_config
# STIG Remediation: Configure SSH Client to Use FIPS 140-2 Validated MACs
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,...
EOF
# Run 7, attempt 1 (failed)
cp /etc/ssh/ssh_config /etc/ssh/ssh_config.bak.$(date +%s)
sed -i '/^MACs /d' /etc/ssh/ssh_config
echo "MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,..." >> /etc/ssh/ssh_config
The difference looks cosmetic. It isn't.
man ssh_config:
For each parameter, the first obtained value will be used.
Run 6's cat <<EOF > wiped the file, eliminating any Include /etc/crypto-policies/back-ends/openssh.config directive that would otherwise have set MACs to a non-FIPS value earlier in the file. The new MACs line had no competitor and won.
Run 7's echo >> appended to the bottom of the file. Whatever the Include directive resolved to earlier in the file beat the appended line. Same MACs value, different position, different outcome.
That's the entire mechanism.
The five tips that were waiting¶
Both runs retrieved the same top-5 tips for this rule, in identical rank order, from stig.tip_retrievals:
| Rank | Tip text (excerpt) |
|---|---|
| 1 | "Attempt 6 used @system-crypto-policy alias in ssh_config; failed rule check. Learned that the scanner likely requires explicit MAC values..." |
| 2 | "Attempted appending MACs to ssh_config; failed because the rule likely targets sshd_config or requires system-wide crypto-policy activation." |
| 3 | "Attempted updating ssh_config; failed because rule likely targets sshd_config or system-wide crypto-policies. Use update-crypto-policies instead." |
| 4 | "Attempt 3 failed by editing ssh_config; learned that crypto-policy rules require system-wide policy updates via update-crypto-policies, not manual client config." |
| 5 | "Use update-crypto-policies --set FIPS to satisfy SSH crypto-policy rules instead of manually editing ssh_config." |
Every single one says some variant of "don't edit ssh_config, use update-crypto-policies." And every single one was wrong. The successful Run 6 script edited ssh_config directly (by overwriting), which is exactly what these tips advised against.
The retrieval log shows the outcome each retrieval got assigned:
- Run 6 retrievals (these five tips during the successful attempt):
outcome_value = 1for all five. Marked helpful because the rule passed. - Run 7 retrievals (same five tips during the failed attempts):
outcome_value = 0. Marked unhelpful because the rule failed.
Run 6's "helpful" label was a credit-assignment artifact. The tips were retrieved into the prompt. The Worker ignored their literal advice and overwrote ssh_config. The rule passed because of the overwrite. The retrieval system, having no causal signal, attributed the success to whatever was in the prompt — including the tips that said the opposite.
What MTP actually did¶
journey/38's "drafter randomness" hypothesis isn't wrong but it isn't the cause. The right framing: MTP made the Worker more obedient to in-context advice.
The drafter is a separate model trained on the same corpus as the verifier. When the Worker's prompt contains five tips all saying "use update-crypto-policies," the drafter speculates token sequences that match the tips' surface form — they look like the kind of tokens a good completion would have given that context. The verifier accepts those speculations at the 73% rate journey/38 measured.
Pre-MTP Gemma (Run 6) generated more independently from the in-context signals. The pre-MTP Worker on this rule chose cat <<EOF > despite the tips pointing at update-crypto-policies. That divergence was load-bearing — it was the entire reason Run 6 worked.
Post-MTP Gemma (Run 7) is more conditioned on the prompt. It read the same tips, found them coherent, drafted token sequences consistent with them, and the verifier accepted. The Worker followed the literal advice. The literal advice was wrong. The rule failed.
The cryptography corpus had five years' worth of tips all advising the same wrong thing, and the existing retrieval pipeline had no way to know they were wrong because the rule passed every time anyway. MTP didn't introduce the problem. It exposed it.
The other regressed categories probably share this pattern¶
Run 7's per-category regressions were:
| Category | Run 6 | Run 7 | Delta |
|---|---|---|---|
| cryptography | 83% | 29% | −54pp |
| logging | 82% | 64% | −18pp |
| filesystem | 86% | 75% | −11pp |
| audit | 35% | 27% | −8pp |
| ssh | 72% | 67% | −5pp |
These are the categories with the longest tip-corpus history — categories Ralph has worked through across six prior runs. They're the categories with the most accumulated plausible-but-misleading tips, by construction. A tip survives only if outcome_value averaged across retrievals is positive. A misleading tip survives if the rule happens to pass for other reasons whenever it's retrieved.
The categories that didn't regress (kernel +3pp, authentication +1pp, package-management flat) are categories where the fix is mechanical and unambiguous. The Worker's script and the tip's advice converge by default. There's no room for the tips to be wrong about how a package install should be done.
This is testable. Run 8 should show: rules in regressed categories had higher follow rate of retrieved tips than rules in stable categories. We don't have the data yet because we don't record tip-follow signal — see journey/38.6 for the architectural fix that puts that record in place.
What we shipped in journey/38 that this revises¶
journey/38 closed with two open hypotheses (drafter randomness, snapshot non-determinism) and stated Run 8 would narrow them. Both can be retired:
- Drafter randomness — partially right: MTP did change Worker behavior. But the change was specifically toward closer adherence to in-context signals, not random variance. The Worker doesn't roll different dice; it weights the prompt differently.
- Snapshot non-determinism — not the cause. The shared rule (
harden_sshd_macs_openssh_conf_crypto_policy) existed in both runs with the same starting state. The difference was downstream of the Worker's script choice, not upstream of the rule's pickability.
The actual cause is the third one neither hypothesis named: the memory pipeline has no signal for "tip's advice was followed vs. ignored," so misleading tips can accumulate utility scores by being lucky neighbors of successful outcomes.
Run 6 succeeded in spite of its retrieval, not because of it. The corpus had no way to know the difference.
The fix lives in the next entry: a two-dimensional outcome signal that separates result quality from causal attribution. Once the tip-follow signal is in place, the dream pass can downweight tips that have high apparent utility but low follow-rate — the exact signature these five crypto tips have.
Bets going into Run 8¶
- The cryptography rules will still fail in Run 8 if we leave the tips unchanged. The Worker is still post-MTP; the tips still advise wrong things; the structural fix is the new signal.
- After the dream pass at end of Run 8 (now scoring
tip_followedvia LLM judge + embedding cosine, see journey/38.6), the five crypto tips will accumulate lowtip_followedrates when their advice was bypassed in successful attempts, and they'll get downweighted in Run 9's retrievals. - Run 9's cryptography category should partially recover. Not to 83% — that was a fluke benefiting from a corpus state we can't reproduce — but well above 29%. Mid-50s would be a satisfying result.
Related¶
journey/38— the production datapoint that left this open.journey/22— the original memory pivot, where decision-provenance vs. decision-output was first named.journey/38.6— the architectural step-back: what this case says about context-graph memory.adr/0016— the V2 memory stack the failure is inside.deferred.mdDEF-27 — the formal entry for the tip-causal-signal work.