Observability
Every prompt, tool call, scan finding, and HITL decision lands in your SIEM and your dashboards. DefenseClaw ships local Grafana/Loki/Tempo and local Splunk stacks you can stand up with one command.
The DefenseClaw gateway is observable by design. Every decision — admission, guardrail verdict, judge call, scanner finding, HITL approval — is structured, correlated by trace_id/span_id, and emitted on three independent rails:
Guided example · No delivery to unconfigured services is implied
Follow one decision across every telemetry rail
The homepage enforcement event keeps shared correlation dimensions as it fans out to configured sinks.
audit: sqlite: enabled jsonl: enabledexporters: otlp: configured_only splunk: configured_only webhooks: configured_onlyaudit: sqlite: enabled jsonl: enabledexporters: otlp: configured_only splunk: configured_only webhooks: configured_only
Shared dimensions preserve decision context
Fan out to configured sinks
What DefenseClaw did — and did not do
What it did
- Reuse the homepage event for narrative continuity
- Show the supported local and external telemetry rails
What it did not do
- Invent dashboard metrics
- Claim delivery to services that have not been configured
- Display production data
What you just saw
The same synthetic Cursor enforcement decision used on the homepage kept its trace, span, session, connector, decision, rule, and severity dimensions across local evidence and configured export rails. The example does not imply delivery to an OTLP, Splunk, or webhook destination that an operator has not configured.
You can use any combination of those — they're independent sinks, not exclusive backends. Most teams start with one of the bundled local stacks while they're building policy, then add their production SIEM as a second sink without rewriting anything.
The two bundled stacks
Local OTel + Grafana stack
One-command Compose stack: OTel Collector, Loki, Tempo, Prometheus, Grafana — all wired to the gateway by setup local-observability up.
Splunk (local + Enterprise)
defenseclaw setup splunk runs three independent pipelines: --logs (local Splunk in Docker), --enterprise (remote HEC), --o11y (Splunk Observability Cloud).
Galileo (Cloud + self-hosted)
First-class traces-only OTLP destination with a guided wizard, canary test, and native coexistence with the local stack.
Agent360 drill-down
Visual per-agent guide to lifecycle, models, tools, tokens, cost, security decisions, topology, and Tempo trace drill-down.
Grafana dashboard catalog
Purpose, ownership, data sources, drill-down paths, and empty-state semantics for every bundled dashboard.
What lands in observability
Every event has the same envelope so you can correlate across rails:
{
"ts": "2026-05-08T12:00:01.234Z",
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
"span_id": "00f067aa0ba902b7",
"connector": "codex",
"agent": "claudecode",
"session_id": "...",
"kind": "guardrail.verdict",
"decision": "deny",
"rule": "secrets/aws_access_key",
"severity": "high",
"actor": { "operator": "vineeth", "host": "..." },
"payload": { "matched": "AKIA****", "tool": "fs.write" }
}The connector field is the multi-connector dimension: it names which hook connector (codex, claudecode, antigravity, …) the event came from. One gateway can serve several connectors at once, so every rail carries it — see Filter by connector below.
The kind field is your top-level taxonomy:
kind | Emitted when |
|---|---|
admission.scan | Skill or MCP scanner finishes a scan |
admission.decision | Watcher applies an OPA verdict |
guardrail.verdict | A regex / LLM judge / inspector resolves |
judge.call | The LLM judge is invoked (with model + token usage) |
hitl.prompt | Operator is asked for approval |
hitl.decision | Operator answered allow/deny |
tool.call / tool.result | The agent invoked an MCP/native tool |
proxy.request / proxy.response | Gateway-proxied LLM call |
gateway.lifecycle | Start / reload / shutdown |
You don't pick the schema in setup — you pick the destinations. Process-wide
traces, metrics, and logs use named otel.destinations[]; audit-only forwarding
uses audit_sinks[]. Both live in ~/.defenseclaw/config.yaml, and the bundled
commands maintain them without storing credential values there.
Manage destinations safely
Destination names are identities: a new name adds a route, while an existing name updates only that route. Inspect and preview before writing:
defenseclaw setup observability list
defenseclaw setup observability list --json
defenseclaw setup observability add otlp --name production-tempo \
--endpoint collector.example.com:4317 --signals traces --dry-runThe list distinguishes otel from audit_sinks and shows kind, enabled state,
signals, preset, and endpoint. The TUI Overview's full-width
Observability Destinations · Runtime panel shows the runtime-loaded version
of the same inventory, including process/global/connector scope, explicit
connector suppression, and Galileo schema eligibility plus acknowledged,
rejected, and failed delivery. It never shows
headers or credential values. Open 0 Setup → Observability / Galileo to
add, update, enable, disable, test, or remove destinations.
Audit contracts
gateway.jsonl is still the live structured runtime log. audit_events is the SQLite audit store used by the TUI, exports, and audit sinks. The details column is intentionally backward-compatible text; new machine-readable audit payloads live in structured_json and export as structured.
Connector hook audit rows use schemas/hook-audit-envelope.json (schema: "defenseclaw.hook.v1"). During migration, the same hook envelope is also embedded in details_json= inside details so existing Splunk SPL and grep workflows keep working. New consumers should read structured.
Filter by connector
When one gateway enforces guardrail policy for several hook connectors at once (see Multi-connector), every rail carries a connector dimension so you can slice telemetry per connector:
| Rail | Where the connector lives | Example filter |
|---|---|---|
| OTel metrics | connector metric label | defenseclaw_connector_hook_invocations_total{connector="codex"} |
| OTel spans / logs | defenseclaw.connector.source attribute | filter spans on defenseclaw.connector.source = "codex" |
| Audit rows / OTLP-ingest audit rows | top-level connector field, plus structured.connector on hook rows | connector="codex" |
| Splunk HEC | top-level connector, and structured.connector on hook events | structured.connector="codex" |
The OTel resource attribute defenseclaw.claw.mode reports multi when more than one connector is active, so you can also tell a fan-out gateway apart from a single-connector one. Grafana's Connectors (Overview) and Connector Detail boards are built on the connector metric label, and the Guardrail Evaluations board is connector-filterable. See Local observability stack and Splunk.
Reading multi-connector charts
On a multi-connector gateway, totals show the whole active roster. Use the connector label or field to split Codex, Claude Code, Hermes, OpenCode, OmniGent, and other hook peers; the same dimension appears in Grafana, Splunk, JSONL, and the TUI.
Pick a starting point
The fastest way to see what DefenseClaw is doing. One command brings up Grafana on :3000 with pre-built dashboards.
defenseclaw setup local-observability upYou already have Splunk in the org. Point HEC at the gateway and you're done — no extra infra to operate.
defenseclaw setup splunk --enterprise \
--hec-endpoint https://splunk.example.com:8088 \
--hec-token "$DEFENSECLAW_SPLUNK_HEC_TOKEN"Run the local Grafana stack for engineering teams and forward the same events to Splunk for SOC. Each sink is independent — failures don't cascade.
defenseclaw setup local-observability up
defenseclaw setup splunk --enterprise --hec-endpoint ... --hec-token ...The gateway will fan out every event to both rails. Verify with defenseclaw tui (audit panel) or tail -f ~/.defenseclaw/gateway.jsonl | jq.
Common questions
OpenClaw integration
How DefenseClaw integrates with OpenClaw end-to-end — fetch interceptor, before_tool_call hook, correlation headers, plugin-mediated HITL approvals, and the audit loop.
Local observability stack
One-command OpenTelemetry + Loki + Tempo + Prometheus + Grafana stack, pre-wired to the DefenseClaw gateway. Grafana on :3000, dashboards seeded, no manual config.