Skip to content

Pen-test Skill — Pre-Flight Findings + Build Sequence

Operational companion to futures/pentest-skill.md. That doc captures the proposal and architectural reasoning. This one captures what we measured in the pre-flight and the executable plan if we commit.

Operating constraints

The build proceeds against the existing gemma-forge codebase:

  • Postgres: a new pentest schema; do not modify stig, cve, or detection schemas.
  • Docker: the pentest-lab network is the only network created by this skill. Do not touch other docker networks or already-running containers (the forge-ui / forge-api / observability stack).
  • Host tooling: nothing installed on the host. Metasploit lives inside the metasploitframework/metasploit-framework Docker image (~3 GB, pulled). nmap lives inside that same image.
  • vLLM endpoint: shared with other skills; do not restart it.
  • Existing skills: do not touch skills/stig-rhel9/, skills/cve-response/, or skills/detection-tuning/.

Safe playground:

  • skills/network-pentest/ (new dir)
  • gemma_forge/harness/tools/pentest/ (new dir)
  • migrations/pentest/ (new dir)
  • /tmp/pentest-skill/ (external scratch — corpus snapshots, lab state, sample exploit outputs)
  • Repository git (feature branches + commits)

Pre-flight findings — 2026-05-25

Total time: ~3 hours. All green; build commitment is a separate decision about scope, not a technical blocker.

Pre-flight 1: intel corpus size & freshness — ✅ green

Pulled four public sources, cross-referenced for the demoable intersection.

CISA KEV       :  1,602 CVEs   (current as of 2026-05-22, 3 days fresh)
Metasploit     :  2,497 CVEs   (exploit modules in modules_metadata_base.json)
ExploitDB      : 25,003 CVEs   (any PoC)
Vulhub         :    244 CVEs   (docker-able scenarios)

Intersections:

KEV ∩ MSF                   : 430    (CISA-listed with msf module)
KEV ∩ Vulhub                :  90    (CISA-listed + lab-reproducible)
Vulhub ∩ (MSF ∪ ExploitDB)  : 131    (lab-reproducible with exploit tooling)
KEV ∩ Vulhub ∩ MSF only     :  63    (strict triple-intersection, msf-only)
KEV ∩ Vulhub ∩ (MSF ∪ EDB)  :  75    (strict triple, msf OR ExploitDB)  ← curated pool

Verdict: 75 cured entries is a solid floor; with the dynamic runtime-lookup architecture (see pentest-skill.md §"architectural pivot"), the broader Vulhub ∩ (MSF ∪ EDB) population of 131 becomes addressable too — we don't need every entry on CISA's KEV to be demoable, only to be ranked higher than non-KEV entries.

Year distribution of the curated 75 pool: 70% from 2020+ (recent), spanning 2012-2026. Famous-name CVEs throughout: Shellshock, Heartbleed, SambaCry, Log4Shell, Spring4Shell, Drupalgeddon2, Drupalgeddon3, Ghostcat, ImageTragick.

Vendor diversity: 32 unique vendors. Vendors with multiple entries enable cross-run generalization testing within a vendor family:

Apache (14):  ActiveMQ, Shiro, Solr, Airflow, HTTP Server, Log4j2,
              CouchDB, Superset, RocketMQ, HugeGraph, OFBiz, Tomcat
Atlassian (5): Confluence (5 different CVEs across years)
Oracle (4):   WebLogic (4 different CVEs)
Elastic (3):  Elasticsearch (2), Kibana (1)
SaltStack (3): Salt (3 CVEs all 2020, same product family)
VMware (2):   Spring Cloud Gateway, Spring Framework
PHP (2):      PHP-CGI (2012), PHP-FPM (2019)
Jenkins (2):  classic, Stapler framework
JetBrains (2): TeamCity (2 auth bypasses)
Craft CMS (2)
GNU (2):      Bash (Shellshock), InetUtils

These vendor groups are the experimental units for measuring cross-run generalization: pick 5 Apache CVEs for Run 1, the other 9 for Run 2, measure if patterns transfer.

Pre-flight 2: tool-chain tractability — ✅ green (with caveats)

Walked the full chain manually against a real Vulhub container.

Host        : XR7620, Ubuntu 24.04
Lab network : docker network create pentest-lab
msf         : docker run --network pentest-lab metasploitframework/...
target      : docker compose up -d (vulhub container) on pentest-lab
agent flow  : nmap → service identify → msf module → fire → verify

What worked end-to-end:

  • Isolated docker network: clean, no host port exposure, target only reachable from msf container.
  • Vulhub container stand-up: ~30 sec per target.
  • nmap from msf container against target by container hostname: perfect; structured output with service+version detection.
  • msfconsole -q -x "use ...; set ...; exploit; exit": works for non-interactive runs.
  • The actual exploit fired — encoded PHP webshell payload was sent to Drupalgeddon2's known-vulnerable endpoint.

Caveats surfaced that must be encoded in the playbook / skill design:

Caveat 1 — Some Vulhub containers need setup before exploits work. Drupalgeddon2 (CVE-2018-7600) requires completing the Drupal install wizard. Until install is done, every request 302s to /core/install.php. Per-CVE prerequisite_setup metadata is non-optional.

Caveat 2 — msf module option names vary. ActiveMQ (CVE-2016-3088) uses HttpUsername/HttpPassword, not the obvious USERNAME/PASSWORD. Per-CVE exact option names must be in the intel corpus.

Caveat 3 — Payload compatibility per target. ActiveMQ's module rejects cmd/unix/generic; needs Java- or Linux-specific payload. Per-CVE compatible payload list is part of the recipe.

Caveat 4 — Session vs inline-exec. Most msf exploits open a session that needs interactive use, not a one-shot return-output. The Executor architecture must use msfrpc (persistent Metasploit daemon with JSON-RPC API) rather than msfconsole -x one-shots, because session-bearing exploits need msf to stay alive across exploit-fire and verification-command steps.

These caveats are engineering signal not blockers — the design section of pentest-skill.md integrates them into the architecture.

Pre-flight 3: federal credibility chain — ✅ green

Every public source used is federally-credible:

  • CISA KEV — DHS's CISA agency. The catalog literally drives BOD 22-01, the federal mandate for patching. Federally authoritative.
  • NIST NVD — National Institute of Standards and Technology. Federal authoritative source for CVE structured data.
  • MITRE ATT&CK — MITRE is an FFRDC (federally-funded research and development center). Federal taxonomy for adversary techniques.
  • Metasploit — open-source, Rapid7-maintained. Standard tooling in every federal red team training program (DOD, USCYBERCOM, intelligence community engagement teams).
  • ExploitDB — Offensive Security (the OSCP cert publishers). Used as the canonical reference set in OSCP and similar certs.
  • Vulhub — community-maintained but used in CISA's own training curricula (publicly documented). MIT licensed.

The narrative writes itself: "We're testing the exact CVEs CISA tracks as actively exploited, using the exact tools every OSCP-certified red teamer uses, in a lab environment of the exact container scenarios used in federal cyber training programs."

Decision: ✅ GO

All three pre-flight checks pass cleanly. The architectural pivot to dynamic runtime lookup (vs the originally-sketched static playbook) is the load-bearing design call — captured in ADR-0023.

The unanswered question is scope commitment, not technical feasibility. The build is honestly ~2 weeks (see "honest scope estimate" in pentest-skill.md), not weekend-sized like detection-tuning was. This is a commitment decision, not a technical one.

Build sequence (if committed)

Mirrors the detection-tuning build pattern (pre-flight → vertical slice → expansion → harness wire-up), scaled up for the larger scope.

Phase 1 (~3 days): intel corpus pipeline

Goal: a queryable Postgres-backed corpus, refreshable via script.

  • tools/refresh_pentest_corpus.sh — operator-invokable refresh. Pulls CISA KEV JSON, msf metadata, ExploitDB CSV, Vulhub git HEAD; rebuilds the unified queryable index in pentest.intel_corpus (Postgres).
  • gemma_forge/harness/tools/pentest/corpus.py — Python query interface: lookup_cves(service, version) returns ranked candidate list.
  • migrations/pentest/0001_intel_corpus_schema.sql — schema for the corpus tables.
  • Smoke test: refresh corpus, query for a few known fingerprints, confirm the right CVEs come back ranked correctly.

Phase 2 (~2 days): target lab pool

Goal: 30-50 vetted plug-and-play Vulhub containers ready for random rotation per run.

  • Walk Vulhub's 244 scenarios, filter to those in our curated triple-intersection pool, hand-vet each one's setup state (plug-and-play vs needs-setup-wizard vs needs-credentials).
  • For plug-and-play candidates: add to skills/network-pentest/lab/ targets-pool.yaml with metadata (CVE, exploit-tool-pointer, payload, verification, time-to-ready).
  • For needs-setup candidates: write prerequisite_setup scripts (curl-based install wizard completion, credential setup, etc.).
  • gemma_forge/harness/tools/pentest/lab_network.py — programmatic docker network create/destroy + random N-of-pool selection + parallel container spin-up.

Phase 3 (~3 days): skill code

Goal: the five sub-runtimes wired and end-to-end testable.

  • skills/network-pentest/skill.yaml — manifest with runtime: block per the manifest-driven dispatch pattern.
  • skills/network-pentest/runtime.py — bundles:
  • PentestWorkQueue — nmap-based discovery of live targets on the lab network.
  • PentestExecutor — msfrpc client + ExploitDB script runner; invokes exploit per Worker's tool call.
  • PentestEvaluator — verification logic (session check, output pattern match) returning binary EvalResult.
  • PentestCheckpoint — docker compose down/up for the currently-active target.
  • PentestSkillRuntime — bundles everything, implements worker_context() per DEF-28.
  • gemma_forge/harness/tools/pentest/msf_client.py — msfrpc wrapper with session management.
  • skills/network-pentest/prompts/{architect,worker,reflector,auditor,tip_follow_judge}.md.
  • Smoke test (/tmp/pentest-skill/smoke.py) — drives one Ralph-shaped iteration against one known-easy target.

Phase 4 (~1 day): first run + measurement

Goal: real Run 1 + Run 2 with the cross-run generalization measurement.

  • Pick a 10-container subset for Run 1, complement for Run 2 (the pre-flight identified Apache as the richest vendor family for this test).
  • Run 1 (cold start, EVTX-style fresh tip pool).
  • Dream pass between runs.
  • Run 2 (warm start with Run 1's tips).
  • Measure: Run 2 first-attempt success rate on disjoint targets vs Run 1's cold first-attempt rate.
  • If significantly higher → generalization is real, headline result banked.
  • If equivalent → caching only; demo headline pivots to "operationally faster Run 2" instead of "agent gets smarter."

Phase 5 (optional): polish + journal entry

  • ADR-0023 lands at build-start; journey entry lands after Run 2's measurement is in.
  • forge run network-pentest made first-class via bin/forge (probably adding a --no-vm-required flag since this skill doesn't need the mission-app VM).
  • Demo video / blog post.

Files to read first (in this order, if building)

  1. futures/pentest-skill.md — the proposal
  2. This file — the execution plan
  3. adr/0020 — the worker_context pattern
  4. adr/0022 — the prior skill's architecture; many patterns transfer
  5. adr/0023-network-pentest-skill-architecture.md — this skill's architectural decisions
  6. skills/stig-rhel9/runtime.py — the most complete reference SkillRuntime
  7. gemma_forge/harness/interfaces.py — the Protocol contracts

What still needs to be decided before Phase 1 begins

  • Where the persistent metasploit daemon (msfrpcd) lives. Inside the lab network as a long-running container? Or co-located with the agent process? Default: long-running container on the pentest-lab network, RPC port reachable only from the harness host.
  • Whether session persistence survives across Ralph attempts. If Run 1 attempt 3 gets a shell on host A, does that session stay open through Run 2 of attempt 4 against host B? Default: no — each attempt is independent; sessions don't survive across attempts. Future work: "persistent foothold" mode.
  • Verification timeout per exploit class. Some exploits return in <1s, some bind shells take 10s to settle. Per-exploit-class timeout config.

These can be settled during Phase 1 with no scope impact.

Working notes go where?

  • Pre-flight findings: this file. (Done — see sections above.)
  • Build progress: a new draft doc at docs/drafts/pentest-skill-build-notes.md (gitignored).
  • Architectural decisions: ADR-0023 (manifested at Phase 1 start).
  • Domain learnings: journey entry promotion when the skill ships + the cross-run measurement comes back.

Decision gate

This pre-flight is complete. The technical question ("can this be built?") is answered: yes, with the architecture in pentest-skill.md and the caveats captured above.

The remaining gate is scope commitment: ~2 weeks of focused work, not weekend-trivial. Captured for explicit yes/no decision before Phase 1 begins.