Splunk

defenseclaw setup splunk configures three independent pipelines — a local Splunk in Docker for demos, remote Splunk Enterprise through HEC, and Splunk Observability Cloud through OTLP.

defenseclaw setup splunk has three independent modes. Each adds its own destination, so you can run more than one.

Each Splunk destination keeps its HEC token under its own reference in ~/.defenseclaw/.env, so setting up one never changes another's token. Remote setup takes the token only from --hec-token, DEFENSECLAW_SETUP_SPLUNK_HEC_TOKEN or the prompt, never from a token saved for another destination.

A destination is local Splunk only if local setup created it: setup records the container it started in that destination. Every other Splunk destination is remote, whatever its address, including a remote Splunk reached through an SSH tunnel on 127.0.0.1. setup splunk --disable --logs disables only the recorded local destination, and --disable --enterprise disables the others.

A local destination from DefenseClaw 1.0.3 or earlier has no record, so it counts as remote and keeps working unchanged. If local setup finds a destination it did not record under the name it uses, or one that uses the local token reference, it stops before starting the container and names the collision. Remove an older local destination with defenseclaw setup observability remove <name> --yes and run local setup again. Give a remote destination its own token reference by setting it up again with an independent token. Until local setup has recorded a destination, setup splunk --disable --logs leaves the local container running while any other Splunk destination is enabled, and names each one with the setup observability disable <name> command to run first.

ModeUse it forWhat it sends
--logsA local demoLogs to a Splunk container in Docker, with the bundled app
--enterpriseYour Splunk deploymentLogs through HEC
--o11ySplunk Observability CloudTraces and metrics over OTLP (logs optional)

Before you start

  • For --logs: Docker running on a Linux x86-64 engine, and ports 8000 and 8088 free. On Windows, use Docker Desktop with Linux containers. The bundled splunk/splunk:10.2.0 image is published for linux/amd64; use --enterprise with an existing Splunk server from another architecture.
  • For --enterprise: a HEC URL, a HEC token, and an index created ahead of time.
  • For --o11y: your realm (for example us1) and an access token.

The local Splunk bridge works only with an x86_64 Docker engine. On an ARM64 Docker engine, use --enterprise with an existing Splunk HTTP Event Collector (HEC) endpoint. See Known limits.

Flags

defenseclaw setup splunk has three modes (--logs, --enterprise, --o11y) and one subcommand, dashboards. Without flags it runs a wizard. --non-interactive uses flags instead of prompts.

FlagApplies toPurpose
--logsLocalStart Splunk in Docker, in Free mode, with HEC and the bundled app
--accept-splunk-licenseLocalAcknowledge the Splunk General Terms. Without it the CLI prompts.
--s3-exportLocalEnable local Splunk and also start the optional S3 exporter sidecar
--s3-bucket TEXTLocalBucket for --s3-export. Also read from the S3_BUCKET environment variable.
--s3-prefix TEXTLocalPrefix for --s3-export. Default agentwatch/defenseclaw.
--aws-region TEXTLocalRegion for --s3-export. Default us-west-2.
--show-credentialsLocalPrint the generated HEC token and the runtime-only bootstrap secret. Use it only in a private terminal.
--refresh-bundle / --no-refresh-bundleLocalRefresh ~/.defenseclaw/splunk-bridge/ from the installed bundle before starting, keeping your secrets in env/.env. On by default. A running stack is stopped, refreshed and restarted.
--enterpriseRemote HECSend logs to your own Splunk through HEC. No Docker.
--hec-endpoint TEXTRemote HECThe full HEC URL, used exactly as given
--hec-token TEXTRemote HECThe HEC token. A value on the command line is visible to other local users in ps, so prefer DEFENSECLAW_SETUP_SPLUNK_HEC_TOKEN.
--skip-testRemote HECSave without the live HEC probe. Without it, setup sends one probe before it saves anything; a failed probe saves nothing (exit 1). Setup records each probe through your gateway: while it is stopped, the probe does not run, and setup saves the destination untested and prints a warning.
--index TEXTLocal, remote HECHEC index. Default defenseclaw_local for local, defenseclaw for remote. Create a remote index ahead of time.
--source TEXTLocal, remote HECHEC source. Default defenseclaw.
--sourcetype TEXTLocal, remote HECHEC sourcetype. Default defenseclaw:json for local, _json for remote.
--o11yObservability CloudSend OTLP to Splunk Observability Cloud
--realm TEXTObservability CloudYour realm, for example us1, us0 or eu0
--access-token TEXTObservability CloudThe access token. Prefer DEFENSECLAW_SETUP_SPLUNK_ACCESS_TOKEN.
--app-name TEXTObservability CloudThe OTel service name and the destination name. Default defenseclaw.
--traces / --no-tracesObservability CloudTrace export
--metrics / --no-metricsObservability CloudMetric export
--logs-export / --no-logs-exportObservability CloudLog export. Off unless you pass --logs-export.
--disableAllDisable the destination of the mode you name

Local mode (--logs)

  1. Start it.

    defenseclaw setup splunk --logs --accept-splunk-license

    Setup exits nonzero if Docker is missing, its daemon is stopped, a required port is busy, or the image registry cannot resolve the pinned tag. Fix the reported prerequisite and run the command again.

    Setup starts Splunk in Free mode, installs the DefenseClaw Local Mode app, and adds a splunk_hec destination for index defenseclaw_local. Open http://localhost:8000. In Splunk Free mode Splunk Web opens as admin with no login. With any other license, log in as admin; defenseclaw setup splunk --show-credentials shows the password, the HEC token and the container's runtime secret. They stay hidden otherwise.

    Setup reports ready only after the installed app, Web UI and HEC are available. If startup fails, repair the reported local stack problem and run the setup command again.

What local mode does

defenseclaw setup splunk --logs --accept-splunk-license runs the Compose project defenseclaw-splunk-local from ~/.defenseclaw/splunk-bridge/. In order:

  1. License. Without --accept-splunk-license the CLI prompts. Use --non-interactive in CI.
  2. Docker preflight. Docker must be running, and ports 8000 (Splunk Web) and 8088 (HEC) must be free. A leftover local Splunk from an interrupted setup does not count as a conflict: setup restarts it.
  3. Container. Starts the stock splunk/splunk:10.2.0 image in Free mode, mounts the bundled defenseclaw_local_mode app, and waits for Splunk Web and HEC to answer.
  4. Destination. Writes a logs-capable splunk_hec destination to https://127.0.0.1:8088/services/collector/event (HEC over TLS), index defenseclaw_local, sourcetype defenseclaw:json. The private-network opt-in applies only to this local destination. The token is kept under DEFENSECLAW_LOCAL_SPLUNK_HEC_TOKEN in ~/.defenseclaw/.env. DefenseClaw sends the token and audit batches only to this account's own Splunk listener on 127.0.0.1:8088. A listener owned by another account gets nothing, and the readiness probe sends no token. An upgrade from 1.0.2 or 1.0.3 records a local sink that an earlier setup created, so the same rule covers it.

Splunk Web is at http://127.0.0.1:8000, and the app appears as DefenseClaw Local Mode. The HEC token and the bootstrap secret stay hidden unless you pass --show-credentials; keep them out of diagnostics and support output.

The app has its dashboards, the shared SPL macros in default/macros.conf and the field extraction in default/props.conf.

S3 export

--s3-export starts a second container, splunk-s3-exporter, next to Splunk. It needs --s3-bucket (or S3_BUCKET). The prefix defaults to agentwatch/defenseclaw and the region to us-west-2:

defenseclaw setup splunk --s3-export --s3-bucket my-bucket --accept-splunk-license

Bundled dashboards

MenuViews
InvestigateAudit and security, runs and sessions, queue and runtime health
ObserveGateway health, gateway logs, model usage and cost
Demo DashboardsAgent Tokenomics Overview, Executive Agent Watch Overview, AI Discovery Inventory, AI Runtime Planes, Connector Activity, Policy Decisions, Findings and HITL
OperateAlerts and saved searches, search and drill-down

The saved searches have static syntax and field checks but have not been run against a live Splunk. Validate them in your deployment before you rely on them.

AI Runtime Planes on Linux with Tetragon

On Linux with Tetragon, AI Runtime Planes adds the kernel-controls fleet table and your Tetragon events through the defenseclaw_demo_kernel_denials and defenseclaw_demo_kernel_events macros. Splunk counts distinct findings where Grafana counts activity records.

What the dashboards query

Local Splunk installs the DefenseClaw Local Mode app. Its macros in default/macros.conf read index=defenseclaw_local and normalize the HEC envelope. props.conf sets KV_MODE = json, so do not add | spath.

Starting searches:

index=defenseclaw_local sourcetype=defenseclaw:json bucket="agent.lifecycle" connector="codex"
| stats count by event_name, action, outcome
| sort -count
index=defenseclaw_local sourcetype=defenseclaw:json bucket="enforcement.action" action="block"
| stats count by connector, source, severity
| sort -count

Enterprise mode (--enterprise)

  1. Put the token in the environment, then configure the destination.

    read -rs DEFENSECLAW_SETUP_SPLUNK_HEC_TOKEN && export DEFENSECLAW_SETUP_SPLUNK_HEC_TOKEN
    
    defenseclaw setup splunk --enterprise --non-interactive \
      --hec-endpoint https://splunk.example.com:8088/services/collector/event \
      --index defenseclaw \
      --sourcetype _json
      Splunk Enterprise configured (HEC)
      Live HEC probe skipped.
      Config saved to ~/.defenseclaw/config.yaml
    
      Splunk Enterprise (HEC):
        Status:      enabled
        Destination: splunk-enterprise-splunk-example-com
        HEC:         https://splunk.example.com:8088/services/collector/event
        Buckets:     compliance.activity, security.finding, guardrail.evaluation, …
        Redaction:   unredacted (none)

    The sample is from a run with --skip-test. Without --skip-test, setup sends one probe to HEC before it saves anything. If the host name does not resolve, the connection or TLS handshake fails, or HEC does not answer 200, setup stops with exit 1 and saves nothing. --skip-test saves the destination without the probe. If the gateway is not running, setup cannot record the probe, so it saves the destination untested and says so; start the gateway, then run defenseclaw observability destination test <name>. Use --skip-test behind a VPN.

  2. Use sourcetype _json, or defenseclaw:json if you also install a props.conf stanza with KV_MODE = json and SHOULD_LINEMERGE = false. Do not add | spath to searches on these sourcetypes; it doubles multivalue fields.

For a collector on a private network, use defenseclaw setup observability add splunk-enterprise --endpoint <url> --allow-private-networks.

Field mapping in your Splunk

The events arrive as DefenseClaw's canonical log projection inside the HEC adapter envelope. For Splunk Cloud or Splunk Enterprise without the bundled app, copy the stanza the app ships:

props.conf
[defenseclaw:json]
KV_MODE = json
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 30
TRUNCATE = 0

The app's own default/props.conf sets KV_MODE = json, SHOULD_LINEMERGE = false and TRUNCATE = 0 the same way for every sourcetype its macros search: otel:log, otel:metric, otel:trace, openclaw:gateway:json and openclaw:diagnostics:json.

Splunk then extracts each field once at search time. Do not add | spath to a search on these sourcetypes: it extracts every field a second time and doubles multivalue fields and counts. DefenseClaw ships no separate add-on for Splunk Cloud or Enterprise; copy the stanza and any macros you need from the app.

With sourcetype _json (the remote default) Splunk extracts the fields on its own, and you need no props.conf.

Splunk Observability Cloud (--o11y)

  1. Put the access token in the environment, then configure the destination.

    read -rs DEFENSECLAW_SETUP_SPLUNK_ACCESS_TOKEN && export DEFENSECLAW_SETUP_SPLUNK_ACCESS_TOKEN
    
    defenseclaw setup splunk --o11y --non-interactive --realm us1 --traces --metrics --logs-export

    Traces and metrics are on by default; --logs-export adds logs. Do not pass --logs here, because it starts the local Splunk container. To create the Splunk Observability dashboards and detectors, run defenseclaw setup splunk dashboards apply.

--o11y writes an OTLP HTTP destination under observability.destinations, pointed at ingest.<realm>.observability.splunkcloud.com. Traces and metrics are on by default and logs are off. The destination is named after --app-name, which is also the OTel service name. Apply a redaction profile when the realm must not receive full content.

Do not pass --logs with --o11y: in setup splunk, --logs starts the local Splunk container.

defenseclaw setup splunk dashboards has plan, apply and destroy. They use the bundled Terraform, keep its working files in ~/.defenseclaw/splunk-o11y-dashboards, and update the same Observability objects when you rerun them from the same state path.

Splunk Observability dashboards and detectors

defenseclaw setup splunk dashboards apply creates eight Splunk Observability dashboards and detectors from the bundled Terraform. Two parts cover sandboxes, from metrics only:

  • Security and Policy adds Egress blocks by source, split into go and ts (guardrail egress checks), openshell and dc-egress-proxy.
  • Sandboxes shows active sandboxes, phase changes, and egress by source and decision, filtered by harness connector. Sandbox names are never metric dimensions.
  • Detector DefenseClaw - Sandbox egress blocks sustained (warning) fires when blocked sandbox egress stays above 30 a minute for 10 minutes. The Prometheus rule DefenseClawSandboxEgressBlocksSustained has the same threshold.
  • Detector DefenseClaw - Sandbox errors sustained (warning) fires when changes into the error phase stay above 0.2 a minute for 10 minutes. The Prometheus rule is DefenseClawSandboxErrorsSustained.

Disable

defenseclaw setup splunk --disable --logs
defenseclaw setup splunk --disable --enterprise
defenseclaw setup splunk --disable --o11y

Each command disables its own destination and leaves the others alone. --logs also stops the local container once the relevant destinations are disabled. If a destination could not be disabled, the container keeps running and the command names the destination to resolve. An enabled destination that local setup did not record also keeps the container running with --logs (see above).

Verify

defenseclaw setup observability list
defenseclaw observability destination test <destination-name> --write-probe

--write-probe sends one marked record. A plain test can pass even when HEC rejects events. In Splunk, search index=defenseclaw sourcetype=_json (local mode: index=defenseclaw_local).

Troubleshooting

SymptomCause and fix
Port 8000 or 8088 is in useAnother program holds it. Stop it, or run defenseclaw setup splunk --disable --logs, free the port and run setup again. There is no port flag.
Setup stops because the splunk container keeps restartingSplunk exits during its own setup. Read why with docker logs defenseclaw-splunk-local-splunk-1, fix what it names, and run defenseclaw setup splunk --logs again. That also refreshes ~/.defenseclaw/splunk-bridge/ from the bundle.
observability destination test passes but no events arriveA handshake can pass while HEC rejects events. Run defenseclaw observability destination test <name> --write-probe.
--write-probe fails with code 7The index is missing or the token cannot write to it. Create the index or correct --index, then probe again.
Remote HEC returns 403The token is wrong, or HEC is not enabled on that endpoint. Check outside DefenseClaw: curl -k $HEC_ENDPOINT -H "Authorization: Splunk $TOKEN" -d '{"event":"test"}'. --skip-test skips the same probe that setup runs.
Events arrive without extracted fieldsThe sourcetype is not _json and has no KV_MODE = json stanza. Use _json, or install the props.conf stanza above for defenseclaw:json.
Observability Cloud shows traces but no logsLogs are off unless you passed --logs-export, and log ingest is a separate entitlement on the realm. Run again with --no-logs-export if you have only traces and metrics.

See also