Splunk

DefenseClaw ships a single command — defenseclaw setup splunk — with three independent pipelines you can mix and match:

You can run more than one. They write to different blocks in ~/.defenseclaw/config.yaml, so adding one never undoes another.

Local mode (--logs)

The fastest path to a working Splunk demo. One command brings up Splunk Free 9.x in Docker, installs the bundled defenseclaw_local_mode app (with five dashboards), and points the gateway's HEC sink at it.

defenseclaw setup splunk --logs --accept-splunk-license

What happens:

Open Splunk at http://localhost:8000 (default credentials are surfaced by defenseclaw setup splunk --show-credentials). The bundled app shows up as DefenseClaw (local mode).

Bundled dashboards

Five dashboards land under Demo Dashboards in the app navigation. They're the same ones you'd build by hand against a real production deployment, just pointed at the local index for a self-contained demo.

Executive Agent Watch Overview

The CISO-friendly landing page: high-attention sessions, blocked actions, scanner heat, HITL backlog.

AI Discovery Inventory

What agents are running, which models they call, and which scanners have evaluated them. Useful as the "first thing the SOC sees" panel.

Connector Activity

Per-connector throughput, error rate, and policy hit ratios over time. Helps catch a connector that suddenly starts triggering rules.

Policy Decisions

Every guardrail verdict, broken down by rule, severity, and outcome. The "did this actually fire?" view when you tune a rule pack.

Findings and HITL

Open scanner findings + the operator approval queue. The day-to-day operator surface.

What the dashboards query

All five views read from a shared set of SPL macros under bundles/splunk_local_bridge/splunk/apps/defenseclaw_local_mode/default/macros.conf, scoped to index=defenseclaw_local. The macros normalize the gateway's JSON envelope so the dashboards don't have to repeat field extractions.

Useful starting queries:

index=defenseclaw_local sourcetype=defenseclaw:json kind="guardrail.verdict" decision="deny"
| stats count by rule, severity, agent
| sort -count

index=defenseclaw_local sourcetype=defenseclaw:json kind="judge.call"
| stats sum(tokens.in) AS in, sum(tokens.out) AS out, count by model

index=defenseclaw_local sourcetype=defenseclaw:json kind="hitl.prompt"
| stats latest(_time) AS asked_at, count by actor.operator, agent

Enterprise mode (--enterprise)

Send events to an existing Splunk deployment. No Docker, no app install — just configure the HEC sink.

defenseclaw setup splunk --enterprise \
  --hec-endpoint https://splunk.example.com:8088 \
  --hec-token "$DEFENSECLAW_SPLUNK_HEC_TOKEN" \
  --index defenseclaw \
  --sourcetype _json

The CLI runs a one-shot probe by default — it sends a test event and confirms HEC returns 200 before persisting the config. If your HEC is behind a VPN, pass --skip-test.

Field mapping in your Splunk

The events arrive as JSON with the envelope described in Observability. For Splunk Cloud or Splunk Enterprise without the bundled app, you'll typically want:

[defenseclaw:json]
KV_MODE = json
TIME_PREFIX = "ts":"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%QZ
SHOULD_LINEMERGE = false

LOOKUP-decision = decision_lookup decision
EXTRACT-trace = trace_id="(?<trace_id>[0-9a-f]+)"

We don't ship a transforms-only TA today; the local-mode app stays Splunk-Free-friendly so you can copy fields out as needed.

Splunk Observability Cloud (--o11y)

For organizations standardised on Splunk O11y instead of Splunk Enterprise, the gateway's OTel exporter sends to the OTLP HTTP ingest endpoint:

defenseclaw setup splunk --o11y \
  --realm us1 \
  --access-token "$SPLUNK_ACCESS_TOKEN" \
  --traces --metrics --logs

This writes the otel: block in config.yaml (same as setup local-observability, but pointed at https://ingest.us1.signalfx.com). Toggle individual signals with --no-traces / --no-metrics / --no-logs if you don't pay for all three.

Disable

defenseclaw setup splunk --disable --logs
defenseclaw setup splunk --disable --enterprise
defenseclaw setup splunk --disable --o11y

--disable removes the matching block from config.yaml (and stops the local container if you ran --logs). It does not touch other sinks.

Verify

defenseclaw doctor

tail -f ~/.defenseclaw/gateway.jsonl | jq 'select(.kind == "guardrail.verdict")'

doctor confirms the HEC endpoint is reachable and the token is valid. The JSONL tail mirrors what's being shipped to Splunk (the gateway writes the same payload to local JSONL and remote sinks) — useful when you suspect a sink is dropping events.

Troubleshooting

See also

• Local observability stack — the OTel + Grafana sibling
• Defaults — what the gateway emits before you tune anything
• Reference → Configuration — every audit-sink schema