Observability

Observability

Every prompt, tool call, scan finding, and HITL decision lands in your SIEM and your dashboards. DefenseClaw ships local Grafana/Loki/Tempo and local Splunk stacks you can stand up with one command.

The DefenseClaw gateway is observable by design. Every decision — admission, guardrail verdict, judge call, scanner finding, HITL approval — is structured, correlated by trace_id/span_id, and emitted on three independent rails:

Guided example · No delivery to unconfigured services is implied

Follow one decision across every telemetry rail

The homepage enforcement event keeps shared correlation dimensions as it fans out to configured sinks.

Deterministic
audit:  sqlite: enabled  jsonl: enabledexporters:  otlp: configured_only  splunk: configured_only  webhooks: configured_onlyaudit:  sqlite: enabled  jsonl: enabledexporters:  otlp: configured_only  splunk: configured_only  webhooks: configured_only
DecisionCorrelate and export
Reason

Shared dimensions preserve decision context

Action

Fan out to configured sinks

04

ExportConfigured OTLP, Splunk, and webhook sinks receive the same dimensions.

Step 4 / 4
What DefenseClaw did — and did not do

What it did

  • Reuse the homepage event for narrative continuity
  • Show the supported local and external telemetry rails

What it did not do

  • Invent dashboard metrics
  • Claim delivery to services that have not been configured
  • Display production data

What you just saw

The same synthetic Cursor enforcement decision used on the homepage kept its trace, span, session, connector, decision, rule, and severity dimensions across local evidence and configured export rails. The example does not imply delivery to an OTLP, Splunk, or webhook destination that an operator has not configured.

OTLP
HEC JSON
JSONL
Control planeDefenseClaw Gatewayadmission · guardrail ·judge · scan decisions
ConnectorOpenTelemetryCollector
ConnectorSplunk HEC
Evidence storeLocal JSONL~/.defenseclaw/gateway.jsonl
Evidence storeLoki · Tempo ·Prometheus
Evidence storeSplunk app
Evidence storeGrafana
OperatorDefenseClaw TUI
Three independent sinks fanned out below the gateway. Failure on one rail does not affect the others.

You can use any combination of those — they're independent sinks, not exclusive backends. Most teams start with one of the bundled local stacks while they're building policy, then add their production SIEM as a second sink without rewriting anything.

The two bundled stacks

What lands in observability

Every event has the same envelope so you can correlate across rails:

{
  "ts": "2026-05-08T12:00:01.234Z",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "connector": "codex",
  "agent": "claudecode",
  "session_id": "...",
  "kind": "guardrail.verdict",
  "decision": "deny",
  "rule": "secrets/aws_access_key",
  "severity": "high",
  "actor": { "operator": "vineeth", "host": "..." },
  "payload": { "matched": "AKIA****", "tool": "fs.write" }
}

The connector field is the multi-connector dimension: it names which hook connector (codex, claudecode, antigravity, …) the event came from. One gateway can serve several connectors at once, so every rail carries it — see Filter by connector below.

The kind field is your top-level taxonomy:

kindEmitted when
admission.scanSkill or MCP scanner finishes a scan
admission.decisionWatcher applies an OPA verdict
guardrail.verdictA regex / LLM judge / inspector resolves
judge.callThe LLM judge is invoked (with model + token usage)
hitl.promptOperator is asked for approval
hitl.decisionOperator answered allow/deny
tool.call / tool.resultThe agent invoked an MCP/native tool
proxy.request / proxy.responseGateway-proxied LLM call
gateway.lifecycleStart / reload / shutdown

You don't pick the schema in setup — you pick the destinations. Process-wide traces, metrics, and logs use named otel.destinations[]; audit-only forwarding uses audit_sinks[]. Both live in ~/.defenseclaw/config.yaml, and the bundled commands maintain them without storing credential values there.

Manage destinations safely

Destination names are identities: a new name adds a route, while an existing name updates only that route. Inspect and preview before writing:

defenseclaw setup observability list
defenseclaw setup observability list --json
defenseclaw setup observability add otlp --name production-tempo \
  --endpoint collector.example.com:4317 --signals traces --dry-run

The list distinguishes otel from audit_sinks and shows kind, enabled state, signals, preset, and endpoint. The TUI Overview's full-width Observability Destinations · Runtime panel shows the runtime-loaded version of the same inventory, including process/global/connector scope, explicit connector suppression, and Galileo schema eligibility plus acknowledged, rejected, and failed delivery. It never shows headers or credential values. Open 0 Setup → Observability / Galileo to add, update, enable, disable, test, or remove destinations.

Audit contracts

gateway.jsonl is still the live structured runtime log. audit_events is the SQLite audit store used by the TUI, exports, and audit sinks. The details column is intentionally backward-compatible text; new machine-readable audit payloads live in structured_json and export as structured.

Connector hook audit rows use schemas/hook-audit-envelope.json (schema: "defenseclaw.hook.v1"). During migration, the same hook envelope is also embedded in details_json= inside details so existing Splunk SPL and grep workflows keep working. New consumers should read structured.

Filter by connector

When one gateway enforces guardrail policy for several hook connectors at once (see Multi-connector), every rail carries a connector dimension so you can slice telemetry per connector:

RailWhere the connector livesExample filter
OTel metricsconnector metric labeldefenseclaw_connector_hook_invocations_total{connector="codex"}
OTel spans / logsdefenseclaw.connector.source attributefilter spans on defenseclaw.connector.source = "codex"
Audit rows / OTLP-ingest audit rowstop-level connector field, plus structured.connector on hook rowsconnector="codex"
Splunk HECtop-level connector, and structured.connector on hook eventsstructured.connector="codex"

The OTel resource attribute defenseclaw.claw.mode reports multi when more than one connector is active, so you can also tell a fan-out gateway apart from a single-connector one. Grafana's Connectors (Overview) and Connector Detail boards are built on the connector metric label, and the Guardrail Evaluations board is connector-filterable. See Local observability stack and Splunk.

Reading multi-connector charts

On a multi-connector gateway, totals show the whole active roster. Use the connector label or field to split Codex, Claude Code, Hermes, OpenCode, OmniGent, and other hook peers; the same dimension appears in Grafana, Splunk, JSONL, and the TUI.

Pick a starting point

The fastest way to see what DefenseClaw is doing. One command brings up Grafana on :3000 with pre-built dashboards.

defenseclaw setup local-observability up

Local observability stack

You already have Splunk in the org. Point HEC at the gateway and you're done — no extra infra to operate.

defenseclaw setup splunk --enterprise \
  --hec-endpoint https://splunk.example.com:8088 \
  --hec-token "$DEFENSECLAW_SPLUNK_HEC_TOKEN"

Splunk integration

Run the local Grafana stack for engineering teams and forward the same events to Splunk for SOC. Each sink is independent — failures don't cascade.

defenseclaw setup local-observability up
defenseclaw setup splunk --enterprise --hec-endpoint ... --hec-token ...

The gateway will fan out every event to both rails. Verify with defenseclaw tui (audit panel) or tail -f ~/.defenseclaw/gateway.jsonl | jq.

Common questions