Deterministic Detection Benchmarks

DefenseClaw is designed to block proved effects, not merely suspicious words. We evaluated that design across public malicious-command, PII, tool-injection, and honest agent-trajectory datasets.

Headline validation results

The exact-proof command result has only nine positives. Its 100% recall point estimate therefore has a 95% Wilson lower bound of 70.09%. The observed 0/7,908 FPR has a 95% Wilson upper bound of 0.04855%. We publish both facts because a perfect point estimate should not be mistaken for broad coverage.

Does it interrupt benign work?

We separately ran honest agent and terminal trajectories. No evaluated benign call was blocked.

LinuxArena is the clearest remaining noise problem. Most findings are legitimate secret-like values used against local services. They are allow-only, but they can still create alert fatigue. Destination- and policy-aware suppression is the next improvement area.

What changed

The detector now uses four layers:

1. ActionFacts normalize a call into operation, resource, scope, sensitivity, result status, control flow, and value-minimized identity.
2. Bounded CEL and regex selectors cheaply identify candidates. A selector is not sufficient authority to block.
3. Code-owned semantic validators parse argv, SQL, Windows, Kubernetes, artifact lineage, and mutation boundaries when exact meaning matters.
4. Bounded chain proofs join only successful events in one authenticated session, using exact identities and a fixed 30-minute/eight-predecessor limit.

Failed, denied, cancelled, cross-session, replayed, unresolved, identity-mismatched, or mutated sequences cannot complete a proof.

Read the deterministic detection reference for the complete rule-family inventory, all 34 exact rule IDs, every bounded chain and enforcement posture, YARA coverage, and protection-pack status.

Multi-step attacks

MonitoringBench provided 2,442 environment-successful attacks with exact tool arguments. We found 41 closed sudoers proofs across fixed development, validation, and test splits. That is a proof-yield count, not a claim of 41/2,442 recall: most trajectory calls are setup, discovery, or dual-use actions.

The authored reverse-shell-artifact-to-persistence fixture is the first new fully blockable multi-step proof. Every profile scored 4 TP, 10 TN, 0 FP, and 0 FN and blocked all four complete proofs. Hard negatives include mismatched artifacts, intervening mutation, failed steps, and incomplete chains.

Several public contextual datasets remain difficult:

We expose these misses rather than converting nearby suspicious words into false-positive-prone rules.

How labels were produced

Public source labels were normalized, deduplicated, and group-split with fixed seed 741983. Optional GPT-OSS batch inference proposed action types and malicious, benign, contextual, or unresolved labels. No human review was used.

A deterministic finalizer then accepted a malicious positive only when the literal call proved a standalone effect or a complete bounded sequence proved every required role. Missing identity, unknown result, dynamic operands, conditional execution, intervening mutation, or unresolved policy context forced an abstention. GPT-OSS was never used at runtime.

Reproduce it

The public harness, source revisions, schemas, normalizers, label finalizers, fixtures, and machine-readable scorecard live in the repository's benchmarks/ directory. The harness downloads source data from its original public location and never executes dataset content.

go test ./benchmarks/...

go run ./benchmarks/cmd/defenseclaw-benchmark run \
  --corpus benchmarks/fixtures/smoke.jsonl \
  --dataset-lock benchmarks/datasets.lock.json \
  --profiles default,permissive,strict \
  --gate \
  --output outputs/benchmarks/smoke

go run ./benchmarks/cmd/defenseclaw-benchmark verify \
  --output outputs/benchmarks/smoke

For complete dataset commands, per-split matrices, initial-state results, labeling methodology, every rule family, and replication steps, read benchmarks/DETERMINISTIC-DETECTION-BENCHMARKS.md.