Autonomous Network Penetration Testing — Fourth Skill¶

What this entry is¶

A captured proposal for the fourth skill on the Ralph harness, surfaced 2026-05-25 after detection-tuning shipped (commits beb1c9b→8be9d59) and was set aside as a customer demo (the labeled-corpus approach plateaued at low PASS rates; cross-run improvement existed but didn't constitute a compelling story).

This skill addresses a different audience need: federal red teams running engagements in air-gapped or low-bandwidth environments who can't use a SaaS-hosted autonomous pen-test tool.

The architectural fit for the Ralph harness:

Patient grinding: real engagements grind through dozens-to-hundreds of technique attempts; humans give up; Ralph doesn't.
Cross-run memory: every engagement against the customer's network teaches the agent about their environment specifically.
Sovereignty: the entire offensive workflow runs on the customer's hardware. The exploit payloads never leave their VLAN. This is the product Anthropic-hosted models cannot ship.
Token-as-input economics: the limiting cost is wall-clock + LLM inference, both of which are operator-controlled at the edge.

The proposition¶

Build an autonomous pen-test skill targeting a network of containerized vulnerable services. Each run:

Work item: one live host on the lab network.
Worker job: enumerate the host (nmap), identify services + versions, query the local intel corpus (CISA KEV ∪ NVD ∪ Metasploit ∪ ExploitDB ∪ OSV ∪ GHSA), pick the highest-leverage applicable exploit, fire it via the appropriate tool (Metasploit module preferred, ExploitDB PoC fallback, manual construction last resort), verify success.
Evaluator: binary — got an artifact (msf session, shell command output containing expected substring, file read) or didn't.
Architect: pick which host to attack next, escalate when techniques exhaust without success.
Reflector: pattern-match across attempts ("Apache 2.4.x banner → always check OptionsBleed before path traversals", "msf's php_cgi module needs URL to end in .php").
Cross-run memory: operational patterns about tool usage, payload selection per target class, fingerprint→CVE mappings.

PASS criterion: artifact obtained (session opened OR command output matches expected pattern). Iron-clad binary outcome — same property STIG has, missing in detection-tuning.

Why this fits the harness — five-interface mapping¶

Harness mechanism	What it provides for pen-testing
Reflexion loop	Try exploit → check for artifact → reflector explains failure → tighten approach or pivot
Cross-run memory + DEF-27 attribution	Operational patterns accumulate: "for Tomcat hosts, try default creds before exploits", "msf option names are HttpUsername not USERNAME for ActiveMQ module"
`EvaluatorMetadata`	`signal_type=binary` (got shell or not), `expected_confidence=high`, `min_retrievals_before_eviction=3` (deterministic enough to evict bad tips fast)
Architect re-engagement	After N attempts on one host without progress → ESCALATE with reason ("target patched", "requires authenticated user we don't have", "exploit needs interactive shell our environment can't sustain")
DEF-28 `worker_context`	Returns `{enumeration_results, applicable_cves, ranked_exploit_candidates, prior_attempts_against_this_target_class}` — gives the Worker the concrete pen-test intel for THIS host
Snapshot/revert	Trivial — `docker compose down/up` to reset a target between attempts. Container-level, fast, deterministic.
Per-family batching	N/A for v1; could later batch "all the SMB targets" or "all the Java middleware targets"

Zero harness changes. Pure skill-side work, like detection-tuning was.

The architectural pivot from the first sketch¶

The first sketch was "hand-curate a static playbook of 75 CVEs with exact msf module + options + payload + verification per entry." That's fragile, narrow (capped at human-curation throughput), and unrealistic (real pen testers don't have pre-curated playbooks — they look stuff up on the fly).

The right architecture is two-tier:

Tier 1 — Intel corpus (refreshed per engagement)¶

A unified queryable index built from public sources, refreshed at the start of each engagement so the agent always has current intel:

CISA KEV — federal "actively exploited" catalog. Used for ranking (KEV-listed entries get priority). 1,602 entries as of 2026-05-22.
NIST NVD — every published CVE with structured CPE matching. 250K+ entries.
Metasploit Framework — module metadata extracted from db/modules_metadata_base.json. 2,653 exploit modules covering 2,497 CVEs.
ExploitDB — public PoC archive. ~25,000 CVEs with at least one PoC, expanding the exploit-availability layer beyond just msf.
OSV.dev — Google-run open-source vulnerability DB. 100K+ entries across language ecosystems (npm, PyPI, Maven, Go).
GitHub Security Advisories — every CVE in OSS with the patch commit linked. Free API.
Vulners (optional, requires API key) — commercial aggregator that pre-joins everything.

Refresh pattern: a tools/refresh_pentest_corpus.sh script runs at start-of-engagement, pulls latest data, rebuilds the unified queryable index in Postgres detection_pentest schema. ~5 minutes including network time.

Tier 2 — Dynamic CVE lookup at runtime¶

Instead of "pick from a 75-entry static menu," the Worker's loop is:

1. nmap target → service + version
2. query intel corpus: "what CVEs affect <service> <version>?"
   → ranked candidate list:
     a. KEV-flagged + has msf module + has ransomware association → tier 1
     b. KEV-flagged + has msf module → tier 2
     c. KEV-flagged + ExploitDB PoC only → tier 3
     d. NVD-only + has msf module → tier 4
     e. NVD-only + ExploitDB PoC only → tier 5
3. for each candidate in ranking order (capped at N attempts):
     pick exploit tool based on availability
     construct invocation (msf options OR adapt ExploitDB script)
     fire it
     verify (session opened OR command output matches)
     if PASS: record success + move on
     else: record failure mode + try next
4. if no candidate succeeded after N attempts: ESCALATE

The agent's Worker prompt is told the ranking, the tools available, and the verification patterns. Pattern-matching + parameter-filling + tool-call construction. Reliably within Gemma 4 31B's competence.

What ISN'T in the agent: novel exploit development, gadget-chain construction, custom payload writing. Those are 0-day territory and outside the model's capability + outside our claim.

Target lab — staging architecture¶

Per-run target rotation against a pool of pre-built vulnerable Docker containers:

Pool source: Vulhub (244 CVE-specific scenarios) + custom Dockerfiles for additional vulnerable software versions (e.g., old tagged versions of common services pulled from Docker Hub).
Pool size goal: 30-50 vetted plug-and-play containers across diverse vendors and CVE classes (cred dumping, web RCE, deserialization, auth bypass, file upload, command injection, etc.).
Per-run selection: random sample of ~10 from the pool, brought up on an isolated pentest-lab Docker network, no host port exposure.
Per-engagement refresh: when new Vulhub PRs add containers, those get pulled and added to the pool automatically.

Vulhub container quirks learned during pre-flight:

Some containers need setup before exploit works. Drupalgeddon2 (CVE-2018-7600) requires the Drupal install wizard to be completed; until then every request 302s to /core/install.php. The container metadata in our curated pool MUST encode prerequisite_setup: <script> per entry.
msf option names vary per module. ActiveMQ module uses HttpUsername/HttpPassword not USERNAME/PASSWORD. The Worker needs msfconsole -q -x "info <module>" output (or pre-extracted metadata) to know exact option names.
Payload compatibility matters. ActiveMQ module rejects cmd/unix/generic; needs a Java- or Linux-target-specific payload. Per-CVE compatible payload list is part of the playbook entry.

These are real complications — captured here so they don't surface as mid-build surprises.

The cross-run intelligence story (the actual demo value)¶

The architectural claim that matters: the tip pool accumulated over many engagements is composed of generalizable operational patterns, not just memorized exploits.

Hypothetical demo arc:

Run 1 against random pool subset A (10 containers across various vendors). Cold start. Agent gets shell on ~5, escalates ~5 with real reasons. Tip pool grows to ~30 operational patterns ("for Tomcat default creds always try first", "msf modules in /multi/http/ tend to need RPORT explicitly set", "java/jsp_shell_reverse_tcp works against most servlet containers").

Run 2 against random pool subset B (10 different containers, some same vendors). Tip pool from A is loaded. Agent's first-attempt success rate on B's containers measurably higher than A's was — not because exploits are memorized (different CVEs, different containers) but because operational patterns transfer.

Run 3 against random pool subset C. Same measurement. Tip pool grows. First-attempt success ramps further.

This is the cross-run learning architectural claim — measurable, falsifiable, with iron-clad ground truth (got shell or didn't).

Crucially, random-target-per-run forces the measurement to be honest. We can't get pure caching (Run 2's targets aren't the same as Run 1's). Either the agent's patterns generalize, or the measurement shows they don't and the demo fails on its own merits — same falsifiability principle detection-tuning needed but didn't have (because its labels were too fuzzy).

Why this skill specifically uses all four "construct" properties¶

Earlier skills exploited 2-3 of these. This skill needs all four simultaneously:

Property	How this skill needs it
Patient grinding	A typical engagement tries 5-20 techniques per host across dozens of hosts. Humans give up after 4 hours.
Cross-run memory	Each engagement against the customer's network teaches the agent the customer's environment specifically.
Sovereignty	No customer in a classified or even regulated environment runs autonomous exploitation tooling via cloud LLM. The exploit payloads MUST stay on-prem. This is the moat.
Token-as-input	A real pen tester at $200/hr can spend $2400 of labor in a 12-hour engagement. Ralph spends $20 of inference for the same wall-clock. The economics are decisive.

Open questions a practitioner should answer before building¶

Captured honestly so the build doesn't pretend they're solved:

What's the realistic pool size after vetting? Pre-flight cross- reference found 75 entries at the strict intersection of CISA KEV ∩ Vulhub ∩ (Metasploit ∪ ExploitDB). After hand-vetting which of these are actually plug-and-play (no setup wizard prerequisite) and have reliable msf modules, the demoable subset is likely 30-50.
Is the dynamic-lookup approach actually faster than static curation? The dynamic approach removes upfront curation cost but adds runtime complexity (the agent has to handle "I queried, got a candidate list, need to construct invocation from scratch"). Pre-flight Path B showed the construction step is non-trivial (option naming, payload compatibility). May need a hybrid: a static "exact-recipe" playbook for the top-N most common CVEs + dynamic lookup for the long tail.
How do we handle session-based exploits in the loop? Most msf exploits open a session that requires interactive use (not a single command-and-return). Two options:
msfrpc-based execution where msf stays alive across the loop and manages session state — the production-correct answer
One-shot payloads only (php/exec, cmd/unix/reverse_bash with a listener container) — simpler but narrower coverage
What's the right verification mechanism per CVE class? Some exploits return command output inline (php/exec on Drupal-style), some drop a webshell requiring a second request, some open an interactive session. The playbook entries need per-class verification handlers.
What's the right ranking signal? CISA KEV first is obvious. After that: by msf-availability? By CVE year (more recent → more likely patched)? By ransomware association? By rank field msf assigns modules (Excellent vs Average vs Manual)? Probably a weighted score — tuning is its own exercise.
Cross-run generalization vs memorization — how do we measure? The random-target-rotation forces this test: if Run 2's first-attempt success on disjoint targets beats Run 1's, that's real generalization; if not, the agent only memorized specific exploits. Need to commit to this measurement upfront.

Implementation sketch (post-pre-flight)¶

File tree:

skills/network-pentest/
├── skill.yaml                    # name, version, runtime block (manifest-driven)
├── runtime.py                    # NetworkPentestSkillRuntime + sub-runtimes
├── prompts/
│   ├── architect.md
│   ├── worker.md
│   ├── auditor.md
│   ├── reflector.md
│   └── tip_follow_judge.md
└── lab/
    ├── targets-pool.yaml         # the curated 30-50 plug-and-play container list
    └── compose-templates/        # per-target docker-compose snippets

gemma_forge/harness/tools/pentest/
├── corpus.py                     # intel corpus loader + query interface
├── corpus_refresh.py             # per-engagement refresh script (callable)
├── msf_client.py                 # msfrpc wrapper
├── exploitdb_runner.py           # download + execute ExploitDB PoCs
├── lab_network.py                # docker network + target spin-up/down
└── enumeration.py                # nmap output parsing + service identification

tools/
└── refresh_pentest_corpus.sh     # operator-invokable corpus refresh

migrations/pentest/
└── 0001..N.sql                   # forge_pentest schema (tips, work_items, etc.)

Sub-runtime mapping:

PentestWorkQueue — scans the live lab network with nmap, yields one WorkItem per discovered host.
PentestExecutor — invokes msfrpc (preferred) or ExploitDB script (fallback) with parameters from the Worker's tool call.
PentestEvaluator — verifies success via the playbook's verification field; returns EvalResult(passed, failure_mode, signals={tool_output, candidate_cve, payload_used}).
PentestCheckpoint — docker compose down/up the target between attempts (deterministic reset).
PentestSkillRuntime — bundles them, implements worker_context() returning {nmap_output, applicable_cves, ranked_exploit_candidates, prior_attempts_against_this_target_class}.

Failure mode taxonomy:

EXPLOIT_FAILED — exploit fired but didn't produce expected artifact (target patched, version mismatch, wrong payload variant).
NO_APPLICABLE_EXPLOIT — target's services have no entries in our intel corpus.
TARGET_UNREACHABLE — nmap couldn't reach target, container unhealthy.
EXPLOIT_REQUIRES_INTERACTION — exploit needs interactive shell that the v1 loop can't sustain (escalate to "manual review needed").
EXPLOIT_REQUIRES_PRECONDITIONS — target needs auth credentials, specific input file, or other state we don't have.

What this would prove¶

If it works:

The harness handles offensive workflows. STIG + CVE + detection- tuning all deal with "improve the defender's state." Pen-testing is "model an attacker against the defender." Different mental model, same harness.
Real cross-run intelligence on a clean ground-truth signal. Detection-tuning's labels were too fuzzy to measure transfer honestly; got-shell-or-not is iron-clad. If Run 2 first-attempt success on disjoint targets > Run 1's, the transfer claim is empirically established.
Sovereignty value is the moat. This is the first skill where the "edge AI" framing isn't ornamental — it's mandatory. The product only exists because Ralph runs on the customer's hardware.

What this would cost¶

Pre-flight: DONE (4 hours, captured in pentest-skill-preflight.md).
Intel corpus pipeline + queryable index: ~3 days.
Target pool curation (30-50 vetted containers): ~2 days of hand-vetting per-container quirks.
Skill code (5 sub-runtimes + msfrpc wrapper + lab network management): ~3 days.
First end-to-end run + measurement of Run 1 → Run 2 transfer: ~1 day.

Total: ~2 weeks, honestly priced. Not a weekend.

What this would NOT do¶

Discover 0-days. The agent only attempts published, exploit-tooled CVEs. If the customer's network is fully patched against everything in CISA KEV + has no exposed vulnerable services, the agent finds nothing. That's the correct result for a fully-hardened environment.
Construct novel exploits. Out of Gemma 4 31B's competency window.
Run in unrestricted "live customer network" mode. v1 targets the isolated lab. Future "deploy against your real environment" mode is v2 territory and needs extensive safety scoping.
Replace a human red teamer doing engagement scoping, customer briefing, or post-engagement report writing. v1 is the exploitation phase only.

Caveats and honest unknowns¶

Vulhub container quality varies. Some are pristine and plug-and- play; others have undocumented setup steps or version drift from what msf modules expect. Vetting effort is real.
Metasploit option naming is inconsistent. Pre-flight saw HttpUsername vs USERNAME confusion. The corpus needs to record exact module option vocabulary, not assume.
Cross-run generalization is the demo's load-bearing claim. If Run 2's first-attempt success rate doesn't beat Run 1's cold baseline, the demo fails — and that's by design (random target rotation forces this honest test). We should be prepared for that outcome and have a fallback narrative if it happens.
The demo will probably escalate ~30-40% of targets. That's honest pen-test behavior, not failure. A demo where Ralph compromises 10/10 looks rigged; 5-7/10 with documented escalation reasons looks like an actual red team result.

adr/0022 — the prior skill's ADR; this skill validates the same DEF-28 worker_context pattern in a third domain.
futures/detection-tuning-skill — the prior skill, set aside after the labeled-corpus approach plateaued.
futures/pentest-skill-preflight — the operational companion: pre-flight findings + build sequence.
adr/0023-network-pentest-skill-architecture.md (planned) — the architectural decisions captured at build-start.