Splunk

DefenseClaw ships a single command — defenseclaw setup splunk — with three independent pipelines you can mix and match:

You can run more than one. In config v8, HEC and O11y become independent entries under observability.destinations, so adding one never replaces another.

Local mode (--logs)

The fastest path to a working Splunk demo. One command brings up Splunk Free 9.x in Docker, installs the bundled defenseclaw_local_mode app (with five dashboards), and points the gateway's HEC destination at it.

defenseclaw setup splunk --logs --accept-splunk-license

What happens:

Open Splunk at http://localhost:8000. The bundled app shows up as DefenseClaw (local mode). Generated HEC and bootstrap values are sensitive; do not reveal them in routine diagnostics or support output.

Bundled dashboards

Five dashboards land under Demo Dashboards in the app navigation. They're the same ones you'd build by hand against a real production deployment, just pointed at the local index for a self-contained demo.

Executive Agent Watch Overview

The CISO-friendly landing page: high-attention sessions, blocked actions, scanner heat, HITL backlog.

AI Discovery Inventory

What agents are running, which models they call, and which scanners have evaluated them. Useful as the "first thing the SOC sees" panel.

Connector Activity

Per-connector throughput, error rate, and policy hit ratios over time. Helps catch a connector that suddenly starts triggering rules.

On multi-connector installs, this is the fastest dashboard for comparing hook volume, errors, and policy hits across the active roster.

Policy Decisions

Every guardrail verdict, broken down by rule, severity, and outcome. The "did this actually fire?" view when you tune a rule pack.

Findings and HITL

Open scanner findings plus HITL request/decision history. This is an observability view, not a resumable approval queue; live approval happens only on connector events with a native ask surface.

What the dashboards query

All five views read from a shared set of SPL macros under bundles/splunk_local_bridge/splunk/apps/defenseclaw_local_mode/default/macros.conf, scoped to index=defenseclaw_local. The macros normalize the gateway's JSON envelope so the dashboards don't have to repeat field extractions.

The v8 Splunk adapter wraps the already-selected, already-redacted canonical log projection and derives compatibility aliases only from those immutable bytes. A removed field never reappears through a raw fallback. The canonical bucket, family/event name, action, source, connector, severity, trace/span IDs, and structured body remain available for new SPL, while documented v7 aliases remain available only during their compatibility window.

Useful starting queries:

index=defenseclaw_local sourcetype=defenseclaw:json bucket="agent.lifecycle" connector="codex"
| stats count by event_name, action, outcome
| sort -count

index=defenseclaw_local sourcetype=defenseclaw:json bucket="enforcement.action" action="block"
| stats count by connector, source, severity
| sort -count

index=defenseclaw_local sourcetype=defenseclaw:json bucket="agent.lifecycle"
| timechart span=5m count by connector

Enterprise mode (--enterprise)

Send events to an existing Splunk deployment. No Docker, no app install — just configure the logs-only HEC destination.

defenseclaw setup splunk --enterprise \
  --hec-endpoint https://splunk.example.com:8088 \
  --hec-token "$DEFENSECLAW_SPLUNK_HEC_TOKEN" \
  --index defenseclaw \
  --sourcetype _json

The CLI runs a one-shot probe by default — it sends a test event and confirms HEC returns 200 before persisting the config. If your HEC is behind a VPN, pass --skip-test.

Field mapping in your Splunk

The events arrive as v8 canonical log projections inside the HEC adapter envelope described in Observability. For Splunk Cloud or Splunk Enterprise without the bundled app, you'll typically want:

[defenseclaw:json]
KV_MODE = json
TIME_PREFIX = "ts":"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%QZ
SHOULD_LINEMERGE = false

LOOKUP-decision = decision_lookup decision
EXTRACT-trace = trace_id="(?<trace_id>[0-9a-f]+)"

We don't ship a transforms-only TA today; the local-mode app stays Splunk-Free-friendly so you can copy fields out as needed.

Splunk Observability Cloud (--o11y)

For organizations standardised on Splunk O11y instead of Splunk Enterprise, the gateway's OTel exporter sends to the OTLP HTTP ingest endpoint:

defenseclaw setup splunk --o11y \
  --realm us1 \
  --access-token "$SPLUNK_ACCESS_TOKEN" \
  --traces --metrics --logs

This writes an OTLP entry under observability.destinations (same pipeline as setup local-observability, pointed at https://ingest.us1.signalfx.com). With no explicit send/routes, general OTLP selects all logs, traces, metrics, and buckets, unredacted. Toggle individual signals with --no-traces / --no-metrics / --no-logs if your entitlement or data policy is narrower, and set a redaction profile when the realm must not receive full content.

Disable

defenseclaw setup splunk --disable --logs
defenseclaw setup splunk --disable --enterprise
defenseclaw setup splunk --disable --o11y

--disable disables the matching named destination (and stops the local container if you ran --logs). It does not touch other destinations.

Verify

defenseclaw doctor

defenseclaw observability plan
defenseclaw setup observability list

doctor confirms destination health without exposing credentials or response bodies. The plan shows the exact selected buckets/signals and effective profile. Local SQLite remains the durable log history. Configure a separate kind: jsonl destination only when a file stream is also required; it is not an implicit mirror of the Splunk projection.

Troubleshooting

See also

• Local observability stack — the OTel + Grafana sibling
• Defaults — what the gateway emits before you tune anything
• Reference → Configuration — the v8 destination schema