Five bands: TRIGGER → GENERATE → TARGET → EVALUATE & LEARN → OBSERVE. Feedback edges (purple, dashed) close the loop.
#1
Discovering vulnerabilities automatically
The system finds attack surface and chooses where to probe — no human picks the test cases.
SHIPPED
Orchestrator queues attacks across all 18 (category, subcategory) cells from the threat model. Scheduled probes + background traffic keep coverage warm. Heatmap shows where tests have landed.
NEXT
Coverage-gap scorer ranks under-tested cells highest each cycle. Event-driven trigger (target login / file ingest / LLM call → webhook → focused probe) so discovery reacts to live target activity.
CUT
A separate “is this harmful?” pre-screen LLM. The Judge already evaluates harm on responses; a second classifier on candidates duplicates work without adding signal.
#2
Generating adversarial attacks dynamically
Attacks aren't a fixed library. The red team adapts as the target adapts.
SHIPPED
Red Team uses local LLM (Nemotron) to mutate seed prompts per (category, subcategory). Probe + Run Battery actuators kick off campaigns from the dashboard.
NEXT
Adaptive mutator — consumes the last N target responses for a cell and prompts the LLM with “given these refusals/leaks, propose a novel variant.” This is literally the PRD's “adapting as attackers adapt.” Phase B.
CUT
Static jailbreak corpus. We seed from one but never depend on it; the live mutator is the source of truth.
#3
Measuring attack success and coverage over time
Every attack outcome is observable, attributable, and aggregatable.
SHIPPED
attack_attempts + verdicts + findings tables. coverage_cells aggregates pass rate per (cat, subcat) over 7d. cost_ledger + llm_calls give per-attack cost / tokens. 9-column trace step table per attack.
NEXT
Surface success-rate trend per cell (sparkline) inside the heatmap, and a rolling per-day verdict mix in the StatusBar.
CUT
Time-series Cost Burndown + Severity Over Time charts. Replaced with snapshot panes (Recent Traces + Open Findings) — at demo scale the time-axis read as noise.
#4
Converting successful exploits into repeatable evals
A confirmed vulnerability becomes a permanent test — but only if it's actually reproducible.
SHIPPED
Findings table holds every confirmed exploit with full reproducer payload (campaign_id, prompt, target response, judge rationale).
NEXT
Reproducer: re-fires the exploit ≥2 more times. Eval Promoter: promotes to suite only if judge_confidence ≥ 0.85 AND reproduces ≥ 2/3. Flaky exploits get tagged, not promoted. Phase B.
CUT
Auto-promotion of every success. Without the reproduce-gate the eval suite gets polluted with one-off flukes.
#5
Validating that fixes actually work
A “fix” isn't a fix until the variants also fail.
SHIPPED
patch_advisor drafts mitigation suggestions and queues them at /queue/patch_review for human apply. Per-finding reproducer payload makes manual re-test trivial.
NEXT
Fix Validator triple: on patch apply, run (a) the exact eval, (b) N mutated variants of the same seed, (c) the full (cat, subcat) category. All three pass = fix validated. Any fail = patch rejected. Phase B.
CUT
Single-test validation. Re-running the exact prompt only proves the fix handles that prompt; mutations are what catch over-fitted patches.
#6
Preventing regressions as the system evolves
The eval bank grows with every confirmed exploit and gates future changes — without becoming a runtime sink.
SHIPPED
Findings + battery_report endpoint give the canonical historical record. /v1/batteries/snapshot lets a battery be re-played on demand.
NEXT
Tiered Eval Suite: smoke (small, every commit), nightly (full), weekly (slow/expensive multi-turn). Retirement policy: evals retire when the target subsystem they probe is rewritten. Phase B → CI integration in Phase D.
CUT
Unbounded eval growth. An ever-growing list becomes a CI bottleneck and noise sink; tiered cadence + retirement keeps it healthy.
#7
Documenting vulnerabilities professionally
A CISO can read the report without asking us to explain it.
SHIPPED
Private Postgres findings table (the secure source of truth). /v1/batteries/report emits a campaign-level markdown summary. Plain-English Judge rationale (THIS HAPPENED / EXPECTED / FAILED BECAUSE / LOGGED AS) on every finding.
NEXT
CVE-style per-finding markdown: reproducer steps, impact statement, suggested mitigation, affected components, severity rubric. Surface inside the dashboard's Open Findings drawer. Phase B.
CUT
Public bug tracker / GitHub Issues integration. The findings DB is intentionally private — vulnerabilities surface to the CISO, not the world.
#8
Improving visibility into behavior under adversarial pressure
You can tell at a glance how hard the system is being pushed and where it's starting to crack.
SHIPPED
Coverage Heatmap (test surface), Recent Traces (live attack stream), Open Findings (current exploit list), per-node click-through for raw I/O, full 9-column step table per trace.
NEXT
Two new visibility surfaces in Phase B: (a) Pressure Gauge — attacks/min × rolling success rate as one number, the live stress meter; (b) Defense-Narrowing Diff — subcategories green 24h ago but red now, the live regression flag. Skip a latency-delta gauge until we control target metrics.
CUT
Heatmap-only visibility. Heatmap shows where we tested; it doesn't show how hard the target is sweating right now.