Skip to content

ADR-0023: Network pen-test skill — dynamic-lookup intel corpus + containerized lab

Context

STIG and CVE both demonstrated the harness against defensive-remediation workflows on a binary scanner. Detection-tuning attempted a graded-outcome workflow against a labeled corpus; the harness mechanics worked but the labeling-noise floor swamped the cross-run improvement signal and made the demo headline weak.

The fourth skill (this one) targets a structurally different workflow: offensive verification. Given the customer's hardened network, can Ralph autonomously find what's still exploitable?

The exercise has architectural value beyond "fourth demo skill":

  • It tests whether the harness's cross-run memory claim holds on a workflow with iron-clad ground truth (got artifact vs didn't), unlike detection-tuning's fuzzy labels.
  • It's the first skill where edge sovereignty is a hard product requirement, not a nice-to-have. The exploit payloads cannot leave the customer's perimeter, so SaaS-hosted LLM tools cannot ship this.
  • It exercises dynamic per-run intel instead of per-skill compile- time configuration — the corpus is refreshed at engagement-start so the agent always has current vulnerability intelligence.

This ADR records the architectural decisions made at proposal-time, informed by ~3 hours of pre-flight investigation (captured in futures/pentest-skill-preflight.md).

Decisions

1. Dynamic runtime CVE lookup, not static curated playbook

The initial sketch was "hand-curate 75 playbook entries with exact Metasploit module + options + payload + verification per entry." After the cross-source pre-flight, that pattern was rejected.

Adopted instead: a queryable intel corpus refreshed per-engagement, plus a runtime lookup step in the Worker's flow.

Worker loop:

  1. nmap target → service + version
  2. Query corpus: lookup_cves(service, version) → ranked candidate list
  3. For each candidate in ranking order: pick the available tool (Metasploit module preferred, ExploitDB PoC fallback), construct the invocation, fire, verify
  4. Record outcome per attempt; ESCALATE if all candidates exhausted

Rationale:

  • The static-playbook approach is capped at human-curation throughput and fragile against the actual diversity of real targets (different exact versions, different attack paths per CVE class).
  • Dynamic lookup matches how real pen testers work (query Vulners, query exploit-db, try msf modules — same workflow, same tools).
  • Per-engagement refresh keeps the corpus current — a 2026 pen-test agent should know about CVEs published this week, not be frozen at build-time.
  • The agent's "intelligence" responsibility becomes pattern-matching
  • parameter-filling + tool orchestration, which Gemma 4 31B can do reliably; novel exploit synthesis (which it cannot do) is never required.

2. Multi-source corpus union, not single source

The corpus aggregates six public sources into one queryable Postgres table:

Source Role Refresh cadence
CISA KEV Federal-credibility ranking signal (KEV-listed = priority) Continuous (3 days fresh at pre-flight)
NIST NVD Structured CVE metadata + CPE matching Continuous
Metasploit modules "Has working tool integration" filter + invocation hint Weekly (git pull)
ExploitDB "Has any PoC" filter for fallback when no msf module Daily
OSV.dev Open-source ecosystem vulnerabilities (npm/PyPI/Maven/Go) Continuous
GitHub Security Advisories OSS CVE with linked patch commit Continuous
Vulners (optional) Aggregator — used when API key configured Continuous

Rationale:

  • No single source has full coverage. CISA KEV is the federal authoritative list (1,602 CVEs) but skips a lot of CVEs that are exploitable but not in active campaigns. ExploitDB has the broadest PoC coverage (25k CVEs) but no quality ranking. Metasploit has invocation-ready modules but skews toward well-known CVEs.
  • The union covers each source's gap.
  • Ranking by source-of-record overlap (in KEV AND has msf module AND has ExploitDB PoC = highest confidence) is the natural ordering.

3. Containerized lab network with per-run random target rotation

Targets are vulnerable Docker containers drawn from three sources, running on an isolated pentest-lab Docker network with no host port exposure:

  • Vulhub (244 CVE-specific scenarios, MIT-licensed, used in CISA training curricula). After vetting: target ~30-40 of these as plug-and-play or one-shot-setup-scriptable. Drop the ~20% that need real config (credentials, DB connection, multi-container choreography).
  • OWASP Vulnerable Apps (DVWA, bWAPP, WebGoat, OWASP Juice Shop, NodeGoat). ~8-10 apps, all plug-and-play, OWASP-curated, used in every web pen-test certification program (OSCP-equivalents). Fills the web-app-pen-test leg that Vulhub mostly skips.
  • Docker Hub tagged versions of popular software at known-vulnerable versions (e.g., wordpress:4.5, jenkins:2.137, confluence:6.0). Hand-pick ~15-20 famous-vulnerable tags. The intel corpus's version-to-CVE mapping handles the "what's wrong with this version" reasoning at runtime.

Initial pool target: ~60 vetted containers. That's 6 disjoint Run-1-vs-Run-2 pairs at N=10 per run (the minimum required for the cross-run generalization measurement) plus headroom for many overlapping demos without repetition.

Per-run: random sample of ~10 from the pool, brought up at run-start, torn down at run-end.

Rationale:

  • Random sampling means each run is a different "scenario" — no demo is identical to any other. Closer to real engagements where the attacker doesn't know what's on the network.
  • Per-run spin-up keeps resource footprint low (only ~10 containers running at any time) and ensures clean state per run.
  • Container isolation means a successful exploit can't pivot to host or sibling containers — safety property for a tool that's literally running attack code.
  • All three sources are community-vetted and used in federal / industry training curricula (publicly documented). Credibility chain stays intact across all three.

Vetting policy: hand-vet each candidate container ahead of pool inclusion. Accept plug-and-play and one-shot-setup-scriptable (~15 min vetting work per container). Reject "needs real config" (credentials/DB connection/multi-container choreography) — those go on the v2 list. We do NOT want Ralph wasting attempt budget on containers that are simply not in an exploitable state.

4. Metasploit RPC daemon (msfrpcd), not one-shot msfconsole invocations

Pre-flight surfaced that most msf exploit modules open sessions requiring interactive use (not single command-and-return). The Executor must use msfrpcd (persistent daemon, JSON-RPC API) rather than msfconsole -q -x "exploit; exit" one-shots.

Rationale:

  • One-shot msfconsole invocations can't maintain session state across exploit-fire and verification-command steps for session-based exploits.
  • msfrpcd is the standard automation interface; well-documented, supports session enumeration, command execution within sessions, cleanup.
  • The persistent daemon lives in a long-running container on the pentest-lab network. Wrapper (pentest/msf_client.py) exposes a clean Python interface.

5. Iron-clad binary ground truth per work item — six verification classes

Each work item (target host) has one PASS criterion: an artifact was obtained. The Evaluator handles six verification classes; per-CVE metadata in the intel corpus records which class applies:

  1. Session opened — Metasploit session present in msfrpc session list (most msf exploit modules with reverse_tcp payloads). Verify: run id in session, check output starts with uid=.
  2. Inline command output — exploit response body contains command-execution output (php/exec payloads, command-injection CVEs that return response inline). Verify: substring match on expected pattern.
  3. File read — exploit returns contents of a known file (path-traversal CVEs reading /etc/passwd or similar). Verify: response body contains expected file substring.
  4. Authentication bypass — post-exploit HTTP request to a privileged endpoint returns 200 instead of 401/403 (Confluence, TeamCity auth bypass CVEs). Verify: GET on a known-protected URL after exploit returns 2xx not 4xx.
  5. Database query execution — exploit gives SQL execution capability (SQL-injection CVEs, Joomla CVE-2023-23752). Verify: run SELECT version() or SELECT current_user(), check output matches DB-version pattern.
  6. Privileged file write — exploit writes a marker file to a privileged location (Tomcat PUT CVE-2017-12615, ActiveMQ file upload CVE-2016-3088). Verify: write a known marker via the exploit, then read it back via a separate request.

No fuzziness. Not "did the exploit run cleanly" (false positive prone) or "did we find something interesting" (subjective). Got the class-specific artifact or didn't.

Rationale:

  • The detection-tuning skill failed in part because its outcomes were fuzzy (P/R/F1 on labeled-noisy corpora). Binary outcomes were what made STIG and CVE compelling.
  • Iron-clad outcomes make the cross-run generalization measurement (Run 1 cold vs Run 2 warm on disjoint targets) defensible.
  • Six classes cover the common 80% of our pool's CVE types. Coverage gaps captured in v2 work below.

6. Per-engagement intel corpus refresh, not per-skill build

The corpus refresh script (tools/refresh_pentest_corpus.sh) is operator-invokable, runs at start-of-engagement, takes ~5 minutes including network time, and rebuilds the queryable index in Postgres.

Rationale:

  • A real pen-test tool deployed in 2026 must have current intel. CVEs published this week must be available; CISA KEV updates must be reflected within hours.
  • The cost of refresh (5 min) is negligible relative to engagement wall-clock (hours).
  • Decoupling refresh from skill build means the skill code is stable while the intel evolves continuously.

7. Cross-run learning measurement built into the demo design

The cross-run claim is measured, not asserted. The demo design has two runs:

  • Run 1: cold start, pool subset A (5-10 random Vulhub targets). Tip pool accumulates from Reflector outputs across attempts.
  • Run 2: warm start with Run 1's tips loaded, pool subset B (5-10 different random Vulhub targets — disjoint from A).

The headline metric is Run 2's first-attempt success rate vs Run 1's first-attempt cold-baseline success rate. If higher: the agent's operational patterns generalize across targets it's never seen. If equivalent: the agent just memorized A's exploits.

Rationale:

  • Detection-tuning's cross-run claim was unfalsifiable because its PASS thresholds were too strict and labels too fuzzy. This skill's outcomes are binary and the random-target rotation forces the generalization vs caching question to be answered honestly.
  • A "Run 2 was 2× faster" headline (with empirical numbers) is more compelling than "Run 1 produced tips" (which is automatic given the harness).

Alternatives considered

Static hand-curated playbook (the initial sketch)

Hand-curate 75 entries with exact msf module + options + payload per CVE. Rejected:

  • Capped at human-curation throughput (~30 entries/day).
  • Brittle against version drift (msf modules update; options change names; new payloads appear).
  • Doesn't generalize to "what about CVE-2026-X published last week?" Static playbook means waiting for the next manual curation pass.
  • Doesn't match how real pen testers work — they query intel sources dynamically, not from a frozen menu.

The dynamic-lookup architecture covers everything the static playbook would have, plus the long tail and freshness, with no curation cap.

Single intel source (e.g., Vulners API only)

Vulners is the strongest single aggregator; using only Vulners would collapse the corpus pipeline. Rejected:

  • Requires a Vulners API key (commercial). For air-gapped customers, not viable.
  • Federal customers want auditable provenance — "the agent used CISA KEV + NVD" is auditable; "the agent used Vulners' aggregated ranking" is not.
  • Single point of failure: if Vulners changes API or pricing, the agent breaks.

Multi-source union with local query is more robust + more credible.

Live customer network as target (vs lab network)

Eventually compelling but v1-out-of-scope. Rejected for v1:

  • Safety scoping is enormous (the tool literally runs exploits). Requires explicit customer authorization, blast-radius controls, rollback mechanisms — none of which are in scope.
  • Demo credibility comes from "ran against an isolated lab with known vulnerable targets and made measurable progress." Demoing against a live customer network requires customer cooperation and is brittle.

Lab network for v1, "live mode" deferred to v2 with explicit safety ADR + customer consent workflow.

Different exploit framework (Cobalt Strike, Sliver, etc.)

Rejected:

  • Cobalt Strike is commercial + restricted-distribution. Federal customers run it but the licensing/distribution constraints make it unsuitable for an OSS demo.
  • Sliver is open-source but newer + smaller exploit module catalog than Metasploit.
  • Metasploit is the federal-standard training tool (used in every OSCP-track program and DoD red team curriculum) — credibility + catalog size both win.

Metasploit for v1, alternate frameworks as Future Work.

Consequences

Positive

  • Harness's cross-run learning claim gets a credible empirical test. Random-target rotation + iron-clad outcomes = falsifiable measurement.
  • Sovereignty becomes a load-bearing product property. This is the first skill where SaaS-hosted LLM is structurally unable to ship the capability. Edge AI's moat is real here.
  • Federal narrative is iron-clad. Every data source is federally-credible; every tool is federal-training-curriculum standard.
  • DEF-28 worker_context validates in a fourth domain. STIG returns XCCDF descriptions; detection-tuning returns Sigma rule text; this returns enumeration results + candidate exploits + prior attempts. Pattern generalizes again.
  • The architecture refresh-per-engagement matches real-world ops. An agent that's "current as of release" is brittle; an agent that pulls latest intel per engagement is operationally correct.

Negative / accepted trade-offs

  • ~2 weeks of work, not weekend-trivial. Per-target vetting and msf option discovery are real labor.
  • Vulhub container quality varies. Some need setup wizards, some need credentials, some are version-drifted from what msf expects. Pre-flight surfaced 3 categories of "container quirks" that must be encoded in the pool metadata.
  • Long-tail CVEs without good tooling are out of scope for v1. The 0-day discovery and custom exploit development markets are served by other tools (and other models). This skill's claim is bounded.
  • Cross-run generalization might fail empirically. If Run 2's first-attempt rate equals Run 1's cold baseline, the headline narrows from "agent gets smarter" to "agent runs faster on re-engagements." The honest narrative still holds.
  • The Metasploit dependency is a vendor concentration. If Metasploit-the-framework changes licensing, we'd need to pivot to ExploitDB-only or another exploit framework. Mitigation: ExploitDB fallback is already in the architecture.

Reversibility

  • Dynamic-lookup ↔ static-playbook is a tradeoff between cap scalability and exact-recipe reliability. We can mix: static for the top-N most-common CVEs, dynamic for the long tail. Reversible.
  • msfrpc daemon dependency can be swapped to one-shot msfconsole invocations if we cap to non-session exploits. Narrower coverage, simpler ops.
  • Multi-source corpus can be reduced to single-source if any external source becomes problematic. Each source is independently optional.

Explicit v2 / Future Work — captured before build, not after

These are out of scope for v1 by intentional design. Documented up front so they don't surface as gaps mid-build:

  1. Auto-vetting pipeline for new pool candidates. When Vulhub merges a new CVE scenario (monthly cadence) or when we want to add a new tagged Docker Hub image, today's process is manual: spin it up, attempt the exploit, encode metadata. v2 should automate this: a sandbox spin-up + standard exploitation probe + auto-add-to-pool-if-probe-succeeds workflow. ~2 days of work when committed.

  2. Verification class 7 — Authentication token extraction. Exploits that leak session tokens / API keys / credentials. Verify the leaked content matches an expected pattern (regex on token format + cryptographic shape). Niche today (~2-3 CVEs in our pool); add when the pool grows enough to justify.

  3. Verification class 8 — Network pivoting. Exploit gives a foothold on host A; verify we can reach an internal resource only accessible from A (e.g., a database on a separate subnet that A can reach but the agent's lab-network position cannot). Out of scope because v1 doesn't do multi-host kill chains. v2 territory once persistent-session management lands.

  4. "Real config" container category. Containers that need credentials, DB connection, or multi-container choreography were explicitly rejected from the v1 pool (decision §3). v2 can selectively re-include the highest-value ones with per-container setup scripts. Estimated ~4-8 hours per re-included container.

  5. Live customer network mode. v1 targets the isolated lab network only. v2 with explicit safety scoping + customer consent workflow + blast-radius controls + rollback mechanisms could target real customer environments. Whole separate ADR.

  6. Pivoting / multi-host kill chains. v1 treats each target as an independent work item; v2 could persist sessions across attempts to build "compromised host A → use A's network position to compromise host B." Stateful and complex; separate design effort.

What success looks like

The first end-to-end engagement against the random 10-container Run 1 + Run 2 pair produces:

  • Run 1 (cold): ~5-7 of 10 targets exploited, ~3-5 escalated with documented real reasons ("no applicable exploit in corpus for this version" / "exploit requires authenticated user" / "target patched against the only applicable CVE"). Tip pool grows to ~30-50 operational patterns.
  • Run 2 (warm, disjoint targets): first-attempt success rate on the new 10 targets is at least +20 percentage points higher than Run 1's cold first-attempt rate. This is the locked-in pre-build threshold, named here ahead of seeing the data — per STYLE.md's "predictions before outcomes" rule. We will grade honestly against this number after Run 2, not after.
  • Final demo headline: "On a randomly-rotated network of CISA-KEV-listed vulnerable services, Ralph compromised 6/10 targets autonomously in Run 1, escalating 4 with documented reasons. After Run 1's operational patterns were absorbed into memory, Run 2 against an entirely different 10-target set achieved 8/10 first-attempt compromise — a 33% improvement over cold-start performance, with zero target overlap between runs."

Grading commitment: post-Run 2, the journey entry will state the actual measured Δ first-attempt pp explicitly. If the delta is ≥ +20 pp the cross-run learning claim is empirically established. If 0-20 pp the headline narrows to "operationally faster Run 2 + modest first-attempt improvement." If 0 pp or negative the headline narrows to "we built an autonomous pen-test tool" (still valuable but the cross-run-learning headline is gone). All three framings are honest.

References

  • futures/pentest-skill.md — the proposal
  • futures/pentest-skill-preflight.md — pre-flight findings + build sequence
  • ADR-0020 — DEF-28 worker_context, validated again here
  • ADR-0022 — the prior skill's ADR; many patterns transfer
  • CISA KEV catalog: https://www.cisa.gov/known-exploited-vulnerabilities-catalog
  • BOD 22-01 (the federal patching mandate KEV drives): https://www.cisa.gov/news-events/directives/bod-22-01
  • Metasploit module metadata: db/modules_metadata_base.json in the metasploit-framework repo
  • Vulhub: https://github.com/vulhub/vulhub
  • ExploitDB: https://www.exploit-db.com/
  • OSV.dev: https://osv.dev/