41. The Honest Run 1 vs Run 2 — +0pp, and What the Failures Surface¶

Phases 1–3 stood up the skill: corpus, pool, verification, engines, sub-runtimes, prompts, manifest, full LLM-loop wiring (entry 40.7). Phase 4 was supposed to run the measurement promised in ADR-0023 §4 — +20pp from cold Run 1 to warm Run 2.

It didn't happen. Both runs landed identical: 0/5 PASS, 5/5 ESCALATED.

That's still the most useful Phase 4 data the project could have produced. Bear with me.

Setup¶

5 random targets per run, sampled from the 20-target plug_and_play pool
Run 1 shuffle_seed=1; Run 2 shuffle_seed=2 (different samples, no overlap except by coincidence — nginx and mongo happened to land in both)
pentest.tips, lessons_current, tip_retrievals, bans all truncated before Run 1 to ensure a genuinely-cold baseline
4 max retries per target; 600s wall-clock per target ceiling
Identical harness config across both runs

Run 1 (cold, seed=1)¶

Target sample (random rotation per ADR-0023 §4):

#	Target	Outcome	Attempts	Wall
1	target-shiro-cve-2016-4437	ESCALATED	4	207s
2	target-jenkins-cve-2017-1000353	ESCALATED	4	177s
3	target-mongo-3-4	ESCALATED	2 (preempt)	92s
4	target-nginx-1-14-0	ESCALATED	4	99s
5	target-owasp-webgoat	ESCALATED	4	141s

Total wall: 716s (12 min). PASS rate: 0/5.

The Worker correctly identified the failure mode mid-run:

"likely due to egress filtering"
— Worker, Run 1, target-jenkins-cve-2017-1000353, attempts 3 and 4

The lab network's docker bridge blocks outbound TCP from targets back to the driver container. Every msf reverse_tcp payload completes the exploit (the deserialization fires, the command runs) but the callback connection never gets through — so verify('session_opened', ...) returns False even though the target was compromised. Run 1 was the discovery of this property.

The Reflector emitted 19 tips by the end of Run 1. The substantive ones, paraphrased:

Prefer file-write / inline-output verification over reverse shells when egress filtering is suspected (3 variants emitted)
Avoid reverse_tcp payloads on Java targets in restricted networks; use bind_tcp or non-callback verification (2 variants)
For OWASP WebGoat, prefer application-layer attacks over Tomcat/middleware exploits (the wrapper isn't the target)
Validate recipe_id exists in corpus before attempting; engine fails immediately on invalid IDs (after the Worker hallucinated several non-existent recipe IDs)
MongoDB 3.x ships with auth disabled by default — try unauth access before CVE-specific exploits
ExploitDB scripts with language='text' are documentation, not executable (the engine surfaced this as a real failure mode)
Search corpus by version when intended_cve_ids is empty

These are correct and generalizable. The Reflector did its job.

Run 2 (warm, seed=2)¶

Target sample:

#	Target	Outcome	Attempts	Wall
1	target-php-cve-2012-1823	ESCALATED	4	118s
2	target-elasticsearch-cve-2015-1427	ESCALATED	4	179s
3	target-imagemagick-cve-2016-3714	ESCALATED	4	136s
4	target-mongo-3-4	ESCALATED	4	99s
5	target-nginx-1-14-0	ESCALATED	3 (preempt)	102s

Total wall: 634s (10.6 min). PASS rate: 0/5.

Identical headline. Underneath:

Run 2 was 11.5% faster wall-clock on the same target count.
Architect preempt-escalation landed on nginx in Run 2 (instead of mongo, which got the preempt in Run 1). nginx had been a 4-attempt ESCALATED in Run 1; in Run 2 the Architect's re-engagement read the tips about "no nginx CVEs in corpus" and pulled the plug at attempt 3.
Run 2 went 4 attempts on mongo (vs 2 in Run 1). The Run 1 preempt-trigger was an LLM "I cannot proceed" admission; Run 2's Worker kept inventing new CVE IDs to try (CVE-2016-10110, CVE-2019-10758, none real) so the preempt path didn't fire.
The Reflector emitted 15 more tips during Run 2, net +24 active tips after eviction. 11 of the original 19 were eviction-retired by the dream pass — their retrieval_count / followed_count ratios fell below the threshold.

What the +0pp number doesn't tell you¶

Two architectural findings are worth a hundred passes on plug-and-play targets.

Finding 1: The lab network's egress filter makes reverse-shell payloads structurally unusable¶

metasploitframework/metasploit-framework defaults to */meterpreter/reverse_tcp payloads. The msf modules in our corpus were designed around these defaults. But the docker bridge network (pentest-lab) doesn't pass outbound TCP from targets back to the driver — the target executes the payload but the callback dies.

The verifier sees this as "no session opened" — which is technically correct (no session, because the callback failed) but architecturally misleading (the exploit ran; the verifier can't prove it without the session).

The Worker correctly diagnosed this mid-attempt across multiple targets. The Reflector wrote tips that name it. But the tips emitted for Shiro didn't transfer to ES, PHP, or ImageMagick because tip retrieval is similarity-bounded by target identity and these are different targets. A tip that's true at the network-architecture level (every target in this lab has the same egress filter) gets treated as target-specific knowledge.

This is fixable two ways:

Environment-scope tips: tips with application_context of ["lab:pentest-lab"] should be retrieved on every Worker turn regardless of target similarity. The Reflector emits the right text; the retrieval layer doesn't honor environment scope.
Fix the lab network: drop the egress filter, or run the driver container on a routable IP. Trade-off: the lab loses some isolation, but real engagements have egress filtering too — fighting that constraint IS part of the work.

Both fixes are post-Phase-4 work; neither was in scope.

Finding 2: The dream pass's causal-attribution path doesn't work for pentest¶

Run 1's dream pass logged:

WARNING dream pass: no work_items found for run_id=405b3ad0-0ae —
        was this run migrated?
INFO    dream pass: falling back to most recent run with outcomes:
        f5d392f5-ebe

f5d392f5-ebe is an unrelated STIG run. The dream pass's attribution fell back to STIG outcomes because pentest's work_items table shape doesn't match what the dream pass expects.

The harness's dream pass was built for STIG's domain shape: same target VM across runs, rule_ids that recur, attempt rows that share a work_item_id. Pentest's targets are sampled randomly per run — a target's work_item_id in Run 1 is different from Run 2's even for the same target name. The dream pass can't trace "tip emitted on Run 1 attempt → followed (or not) on Run 2 attempt" because the tip_retrievals rows it queries don't connect.

So the tips are right. The tip-attribution (which tips got causal credit for Run 2 outcomes, which got blamed for failures) doesn't work yet. The eviction layer evicted 11 tips on outcome signals, but those signals came from the STIG fallback run.

This is a Phase 5 architectural gap, not a Phase 4 failure.

The honest summary¶

ADR-0023 §4 locked +20pp as the success threshold. Phase 4 missed that by 20 points — the runs are tied at zero.

But Phase 4 produced four real architectural findings worth more than a contrived pass would have been:

The lab's egress filter is structurally hostile to reverse shells. Verifiable: every msf attempt with reverse_tcp across two runs failed to open a session. The Worker diagnosed it correctly mid-attempt.
Tip retrieval is over-scoped to target identity. Tips that are true at the lab-environment level (egress, payload choice) don't transfer to new targets even when they obviously should.
The dream pass's causal attribution path was built for STIG's work-item shape and silently degrades on pentest's random-sample shape. No errors raised; the attribution just points at the wrong run.
The Worker prompt's "validate recipe_id" hint isn't strong enough to prevent ID hallucination. The Worker invented recipe IDs (3, 10, 1, 14874, 1234) several times across the two runs even with the explicit instruction "must be an integer from the recipe list, do NOT invent." Pre-call validation on the harness side would catch this deterministically.

The cross-run delta WAS measurable in one place: wall time. Run 2 was 11.5% faster — the agent learned to recognize dead-end signals sooner, even if it couldn't convert them into passes. That's a real learning signal, just not the headline one.

What Phase 5 needs to address¶

Item	Type	Owner
Tip `application_context` honored on retrieval	harness fix	shared
Dream pass attribution for random-sample skills	harness fix	shared
Egress filter resolution (drop or work-around)	env/skill	pentest
Pre-call recipe_id validation	runtime fix	pentest
Bind_tcp payload preference in Worker prompt	prompt	pentest
Pool_intel ordering: msf before edb when verify=session_opened	runtime	pentest

None of these are blockers if we own the failure honestly. The Andy-Weir framing: the agent landed on a network that blocks the callback path it was trained to use. It noticed. It told us. It just couldn't pivot fast enough to pass within the budget.

What this means¶

A run that hits 0/5 on 5 random targets isn't a result that stands on its own. But the artifacts of Phase 4 are the real signal:

The Reflector's 24 active tips, each with a mechanism field explaining the causal WHY, are evidence that the memory system EXTRACTS structured learning from failed engagements.
The 19-tip → 24-tip evolution across runs shows the memory system COMPRESSING and PRIORITIZING (11 tips retired by eviction).
The 11.5% wall-time improvement on a 5-target run shows the agent USING that learning, even if it didn't convert to PASS.

The real story is "the agent's memory system caught the lab's egress-filter problem in one run, named it concretely, and would have prevented it on the next engagement — if tip retrieval were environment-scoped." That's a real result. It's just not the +20pp PASS-rate story we wanted to tell first.

Phase 5: fix the four architectural gaps. Re-run. If the egress fix + env-scoped tips lands, we should see ≥40% PASS rate on plausible targets (Shiro, Jenkins, Tomcat, WebLogic, ActiveMQ — five vetted KEV-listed targets the corpus has msf modules for).

That's the next milestone. Not this one.