Splunk
defenseclaw setup splunk configures three independent pipelines — a local Splunk in Docker for demos, remote Splunk Enterprise through HEC, and Splunk Observability Cloud through OTLP.
defenseclaw setup splunk has three independent modes. Each adds its own destination, so you can run more than one.
Each Splunk destination keeps its HEC token under its own reference in ~/.defenseclaw/.env, so setting up one never changes another's token. Remote setup takes the token only from --hec-token, DEFENSECLAW_SETUP_SPLUNK_HEC_TOKEN or the prompt, never from a token saved for another destination.
A destination is local Splunk only if local setup created it: setup records the container it started in that destination. Every other Splunk destination is remote, whatever its address, including a remote Splunk reached through an SSH tunnel on 127.0.0.1. setup splunk --disable --logs disables only the recorded local destination, and --disable --enterprise disables the others.
A local destination from DefenseClaw 1.0.3 or earlier has no record, so it counts as remote and keeps working unchanged. If local setup finds a destination it did not record under the name it uses, or one that uses the local token reference, it stops before starting the container and names the collision. Remove an older local destination with defenseclaw setup observability remove <name> --yes and run local setup again. Give a remote destination its own token reference by setting it up again with an independent token. Until local setup has recorded a destination, setup splunk --disable --logs leaves the local container running while any other Splunk destination is enabled, and names each one with the setup observability disable <name> command to run first.
| Mode | Use it for | What it sends |
|---|---|---|
--logs | A local demo | Logs to a Splunk container in Docker, with the bundled app |
--enterprise | Your Splunk deployment | Logs through HEC |
--o11y | Splunk Observability Cloud | Traces and metrics over OTLP (logs optional) |
Before you start
- For
--logs: Docker running on a Linux x86-64 engine, and ports 8000 and 8088 free. On Windows, use Docker Desktop with Linux containers. The bundledsplunk/splunk:10.2.0image is published forlinux/amd64; use--enterprisewith an existing Splunk server from another architecture. - For
--enterprise: a HEC URL, a HEC token, and an index created ahead of time. - For
--o11y: your realm (for exampleus1) and an access token.
The local Splunk bridge works only with an x86_64 Docker engine. On an ARM64 Docker engine, use --enterprise with an existing Splunk HTTP Event Collector (HEC) endpoint. See Known limits.
Flags
defenseclaw setup splunk has three modes (--logs, --enterprise, --o11y) and one subcommand, dashboards. Without flags it runs a wizard. --non-interactive uses flags instead of prompts.
| Flag | Applies to | Purpose |
|---|---|---|
--logs | Local | Start Splunk in Docker, in Free mode, with HEC and the bundled app |
--accept-splunk-license | Local | Acknowledge the Splunk General Terms. Without it the CLI prompts. |
--s3-export | Local | Enable local Splunk and also start the optional S3 exporter sidecar |
--s3-bucket TEXT | Local | Bucket for --s3-export. Also read from the S3_BUCKET environment variable. |
--s3-prefix TEXT | Local | Prefix for --s3-export. Default agentwatch/defenseclaw. |
--aws-region TEXT | Local | Region for --s3-export. Default us-west-2. |
--show-credentials | Local | Print the generated HEC token and the runtime-only bootstrap secret. Use it only in a private terminal. |
--refresh-bundle / --no-refresh-bundle | Local | Refresh ~/.defenseclaw/splunk-bridge/ from the installed bundle before starting, keeping your secrets in env/.env. On by default. A running stack is stopped, refreshed and restarted. |
--enterprise | Remote HEC | Send logs to your own Splunk through HEC. No Docker. |
--hec-endpoint TEXT | Remote HEC | The full HEC URL, used exactly as given |
--hec-token TEXT | Remote HEC | The HEC token. A value on the command line is visible to other local users in ps, so prefer DEFENSECLAW_SETUP_SPLUNK_HEC_TOKEN. |
--skip-test | Remote HEC | Save without the live HEC probe. Without it, setup sends one probe before it saves anything; a failed probe saves nothing (exit 1). Setup records each probe through your gateway: while it is stopped, the probe does not run, and setup saves the destination untested and prints a warning. |
--index TEXT | Local, remote HEC | HEC index. Default defenseclaw_local for local, defenseclaw for remote. Create a remote index ahead of time. |
--source TEXT | Local, remote HEC | HEC source. Default defenseclaw. |
--sourcetype TEXT | Local, remote HEC | HEC sourcetype. Default defenseclaw:json for local, _json for remote. |
--o11y | Observability Cloud | Send OTLP to Splunk Observability Cloud |
--realm TEXT | Observability Cloud | Your realm, for example us1, us0 or eu0 |
--access-token TEXT | Observability Cloud | The access token. Prefer DEFENSECLAW_SETUP_SPLUNK_ACCESS_TOKEN. |
--app-name TEXT | Observability Cloud | The OTel service name and the destination name. Default defenseclaw. |
--traces / --no-traces | Observability Cloud | Trace export |
--metrics / --no-metrics | Observability Cloud | Metric export |
--logs-export / --no-logs-export | Observability Cloud | Log export. Off unless you pass --logs-export. |
--disable | All | Disable the destination of the mode you name |
Local mode (--logs)
-
Start it.
defenseclaw setup splunk --logs --accept-splunk-licenseSetup exits nonzero if Docker is missing, its daemon is stopped, a required port is busy, or the image registry cannot resolve the pinned tag. Fix the reported prerequisite and run the command again.
Setup starts Splunk in Free mode, installs the DefenseClaw Local Mode app, and adds a
splunk_hecdestination for indexdefenseclaw_local. Open http://localhost:8000. In Splunk Free mode Splunk Web opens as admin with no login. With any other license, log in as admin;defenseclaw setup splunk --show-credentialsshows the password, the HEC token and the container's runtime secret. They stay hidden otherwise.Setup reports ready only after the installed app, Web UI and HEC are available. If startup fails, repair the reported local stack problem and run the setup command again.
What local mode does
defenseclaw setup splunk --logs --accept-splunk-license runs the Compose project defenseclaw-splunk-local from ~/.defenseclaw/splunk-bridge/. In order:
- License. Without
--accept-splunk-licensethe CLI prompts. Use--non-interactivein CI. - Docker preflight. Docker must be running, and ports 8000 (Splunk Web) and 8088 (HEC) must be free. A leftover local Splunk from an interrupted setup does not count as a conflict: setup restarts it.
- Container. Starts the stock
splunk/splunk:10.2.0image in Free mode, mounts the bundleddefenseclaw_local_modeapp, and waits for Splunk Web and HEC to answer. - Destination. Writes a logs-capable
splunk_hecdestination tohttps://127.0.0.1:8088/services/collector/event(HEC over TLS), indexdefenseclaw_local, sourcetypedefenseclaw:json. The private-network opt-in applies only to this local destination. The token is kept underDEFENSECLAW_LOCAL_SPLUNK_HEC_TOKENin~/.defenseclaw/.env. DefenseClaw sends the token and audit batches only to this account's own Splunk listener on127.0.0.1:8088. A listener owned by another account gets nothing, and the readiness probe sends no token. An upgrade from 1.0.2 or 1.0.3 records a local sink that an earlier setup created, so the same rule covers it.
Splunk Web is at http://127.0.0.1:8000, and the app appears as DefenseClaw Local Mode. The HEC token and the bootstrap secret stay hidden unless you pass --show-credentials; keep them out of diagnostics and support output.
The app has its dashboards, the shared SPL macros in default/macros.conf and the field extraction in default/props.conf.
S3 export
--s3-export starts a second container, splunk-s3-exporter, next to Splunk. It needs --s3-bucket (or S3_BUCKET). The prefix defaults to agentwatch/defenseclaw and the region to us-west-2:
defenseclaw setup splunk --s3-export --s3-bucket my-bucket --accept-splunk-licenseBundled dashboards
| Menu | Views |
|---|---|
| Investigate | Audit and security, runs and sessions, queue and runtime health |
| Observe | Gateway health, gateway logs, model usage and cost |
| Demo Dashboards | Agent Tokenomics Overview, Executive Agent Watch Overview, AI Discovery Inventory, AI Runtime Planes, Connector Activity, Policy Decisions, Findings and HITL |
| Operate | Alerts and saved searches, search and drill-down |
The saved searches have static syntax and field checks but have not been run against a live Splunk. Validate them in your deployment before you rely on them.
AI Runtime Planes on Linux with Tetragon
On Linux with Tetragon, AI Runtime Planes adds the kernel-controls fleet table and your Tetragon events through the defenseclaw_demo_kernel_denials and defenseclaw_demo_kernel_events macros. Splunk counts distinct findings where Grafana counts activity records.
What the dashboards query
Local Splunk installs the DefenseClaw Local Mode app. Its macros in default/macros.conf read index=defenseclaw_local and normalize the HEC envelope. props.conf sets KV_MODE = json, so do not add | spath.
Starting searches:
index=defenseclaw_local sourcetype=defenseclaw:json bucket="agent.lifecycle" connector="codex"
| stats count by event_name, action, outcome
| sort -countindex=defenseclaw_local sourcetype=defenseclaw:json bucket="enforcement.action" action="block"
| stats count by connector, source, severity
| sort -countEnterprise mode (--enterprise)
-
Put the token in the environment, then configure the destination.
read -rs DEFENSECLAW_SETUP_SPLUNK_HEC_TOKEN && export DEFENSECLAW_SETUP_SPLUNK_HEC_TOKEN defenseclaw setup splunk --enterprise --non-interactive \ --hec-endpoint https://splunk.example.com:8088/services/collector/event \ --index defenseclaw \ --sourcetype _jsonSplunk Enterprise configured (HEC) Live HEC probe skipped. Config saved to ~/.defenseclaw/config.yaml Splunk Enterprise (HEC): Status: enabled Destination: splunk-enterprise-splunk-example-com HEC: https://splunk.example.com:8088/services/collector/event Buckets: compliance.activity, security.finding, guardrail.evaluation, … Redaction: unredacted (none)The sample is from a run with
--skip-test. Without--skip-test, setup sends one probe to HEC before it saves anything. If the host name does not resolve, the connection or TLS handshake fails, or HEC does not answer 200, setup stops with exit 1 and saves nothing.--skip-testsaves the destination without the probe. If the gateway is not running, setup cannot record the probe, so it saves the destination untested and says so; start the gateway, then rundefenseclaw observability destination test <name>. Use--skip-testbehind a VPN. -
Use sourcetype
_json, ordefenseclaw:jsonif you also install aprops.confstanza withKV_MODE = jsonandSHOULD_LINEMERGE = false. Do not add| spathto searches on these sourcetypes; it doubles multivalue fields.
For a collector on a private network, use defenseclaw setup observability add splunk-enterprise --endpoint <url> --allow-private-networks.
Field mapping in your Splunk
The events arrive as DefenseClaw's canonical log projection inside the HEC adapter envelope. For Splunk Cloud or Splunk Enterprise without the bundled app, copy the stanza the app ships:
[defenseclaw:json]
KV_MODE = json
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 30
TRUNCATE = 0The app's own default/props.conf sets KV_MODE = json, SHOULD_LINEMERGE = false and TRUNCATE = 0 the same way for every sourcetype its macros search: otel:log, otel:metric, otel:trace, openclaw:gateway:json and openclaw:diagnostics:json.
Splunk then extracts each field once at search time. Do not add | spath to a search on these sourcetypes: it extracts every field a second time and doubles multivalue fields and counts. DefenseClaw ships no separate add-on for Splunk Cloud or Enterprise; copy the stanza and any macros you need from the app.
With sourcetype _json (the remote default) Splunk extracts the fields on its own, and you need no props.conf.
Splunk Observability Cloud (--o11y)
-
Put the access token in the environment, then configure the destination.
read -rs DEFENSECLAW_SETUP_SPLUNK_ACCESS_TOKEN && export DEFENSECLAW_SETUP_SPLUNK_ACCESS_TOKEN defenseclaw setup splunk --o11y --non-interactive --realm us1 --traces --metrics --logs-exportTraces and metrics are on by default;
--logs-exportadds logs. Do not pass--logshere, because it starts the local Splunk container. To create the Splunk Observability dashboards and detectors, rundefenseclaw setup splunk dashboards apply.
--o11y writes an OTLP HTTP destination under observability.destinations, pointed at ingest.<realm>.observability.splunkcloud.com. Traces and metrics are on by default and logs are off. The destination is named after --app-name, which is also the OTel service name. Apply a redaction profile when the realm must not receive full content.
Do not pass --logs with --o11y: in setup splunk, --logs starts the local Splunk container.
defenseclaw setup splunk dashboards has plan, apply and destroy. They use the bundled Terraform, keep its working files in ~/.defenseclaw/splunk-o11y-dashboards, and update the same Observability objects when you rerun them from the same state path.
Splunk Observability dashboards and detectors
defenseclaw setup splunk dashboards apply creates eight Splunk Observability dashboards and detectors from the bundled Terraform. Two parts cover sandboxes, from metrics only:
- Security and Policy adds Egress blocks by source, split into
goandts(guardrail egress checks),openshellanddc-egress-proxy. - Sandboxes shows active sandboxes, phase changes, and egress by source and decision, filtered by harness connector. Sandbox names are never metric dimensions.
- Detector
DefenseClaw - Sandbox egress blocks sustained(warning) fires when blocked sandbox egress stays above 30 a minute for 10 minutes. The Prometheus ruleDefenseClawSandboxEgressBlocksSustainedhas the same threshold. - Detector
DefenseClaw - Sandbox errors sustained(warning) fires when changes into the error phase stay above 0.2 a minute for 10 minutes. The Prometheus rule isDefenseClawSandboxErrorsSustained.
Disable
defenseclaw setup splunk --disable --logs
defenseclaw setup splunk --disable --enterprise
defenseclaw setup splunk --disable --o11yEach command disables its own destination and leaves the others alone. --logs also stops the local container once the relevant destinations are disabled. If a destination could not be disabled, the container keeps running and the command names the destination to resolve. An enabled destination that local setup did not record also keeps the container running with --logs (see above).
Verify
defenseclaw setup observability list
defenseclaw observability destination test <destination-name> --write-probe--write-probe sends one marked record. A plain test can pass even when HEC rejects events. In Splunk, search index=defenseclaw sourcetype=_json (local mode: index=defenseclaw_local).
Troubleshooting
| Symptom | Cause and fix |
|---|---|
| Port 8000 or 8088 is in use | Another program holds it. Stop it, or run defenseclaw setup splunk --disable --logs, free the port and run setup again. There is no port flag. |
Setup stops because the splunk container keeps restarting | Splunk exits during its own setup. Read why with docker logs defenseclaw-splunk-local-splunk-1, fix what it names, and run defenseclaw setup splunk --logs again. That also refreshes ~/.defenseclaw/splunk-bridge/ from the bundle. |
observability destination test passes but no events arrive | A handshake can pass while HEC rejects events. Run defenseclaw observability destination test <name> --write-probe. |
--write-probe fails with code 7 | The index is missing or the token cannot write to it. Create the index or correct --index, then probe again. |
| Remote HEC returns 403 | The token is wrong, or HEC is not enabled on that endpoint. Check outside DefenseClaw: curl -k $HEC_ENDPOINT -H "Authorization: Splunk $TOKEN" -d '{"event":"test"}'. --skip-test skips the same probe that setup runs. |
| Events arrive without extracted fields | The sourcetype is not _json and has no KV_MODE = json stanza. Use _json, or install the props.conf stanza above for defenseclaw:json. |
| Observability Cloud shows traces but no logs | Logs are off unless you passed --logs-export, and log ingest is a separate entitlement on the realm. Run again with --no-logs-export if you have only traces and metrics. |