Local observability stack

defenseclaw setup local-observability brings up a full local OpenTelemetry stack and wires the gateway to it in one shot. It's what the project itself uses for development, demos, and rule-pack tuning.

What it brings up

The stack runs from bundles/local_observability_stack/docker-compose.yml:

All five run under one Compose project (defenseclaw-local-observability). They share a Docker network and the bundled Grafana datasources point at each container by service name.

Bring it up

defenseclaw setup local-observability up

defenseclaw setup observability test local-observability

up also refreshes the host-mounted stack copy in ~/.defenseclaw/observability-stack/ from the bundled version before Compose starts. That includes Grafana dashboards, Prometheus rules, Loki, Tempo, and OTel Collector config. Deleting Docker containers is not enough to update those files because they live on the host; pass --no-refresh-config only when you intentionally want to keep local edits.

Useful flags

The up subcommand exposes the bridge configuration directly:

defenseclaw setup local-observability up \
  --endpoint 127.0.0.1:4317 \
  --signals traces,metrics,logs \
  --service-name defenseclaw-gateway \
  --with-audit-sink \
  --timeout 90

The OTLP protocol is read from the bridge readiness contract (currently grpc); there is no --protocol flag on up. To use http/protobuf, configure that protocol on the named local-observability destination.

What ends up in config.yaml

The example below is the result of the sample up command using --service-name defenseclaw-gateway; without that flag, service.name is defenseclaw.

otel:
  enabled: true
  destinations:
    - name: local-observability
      preset: local-otlp
      enabled: true
      endpoint: 127.0.0.1:4317
      protocol: grpc
      tls: { insecure: true }
      traces: { enabled: true }
      metrics: { enabled: true }
      logs: { enabled: true }
  resource:
    attributes:
      service.name: defenseclaw-gateway

audit_sinks:
  - kind: otlp_logs
    name: local-otlp-logs
    endpoint: 127.0.0.1:4317
    protocol: grpc
    insecure: true

If a sink with the same name already exists, the bridge updates it in place rather than adding a duplicate. Other sinks (Splunk HEC, JSONL, webhooks) are left untouched.

Bundled Grafana dashboards

The bundle ships a full board set in bundles/local_observability_stack/grafana/dashboards/ — every *.json file here is auto-loaded:

See the complete Grafana dashboard catalog for the owner and purpose of every board, the drill-down map, backend usage, and the exact difference between 0, No data, and Not reported.

For a multi-connector gateway, start on Connectors (Overview) to compare connectors side by side, then click a connector cell to drill into Connector Detail.

Agent Directory and Agent360

Open Agent identity → Runtime Agent Directory. Every hook lifecycle transition updates the stable defenseclaw.agent.last_seen inventory metric, so a newly observed root agent or subagent appears automatically. Click its Agent ID to open the reusable Agent360 dashboard; DefenseClaw does not create or provision a separate dashboard file for each agent.

Agent360 correlates the same stable fields across all three backends:

Use Scope = This agent for a single agent, or Agent tree to include all descendants. Prometheus supplies directory, state, throughput, latency, token, reported-cost totals, and the dynamic agent/subagent/model/tool graph. Loki supplies compact lifecycle/model/tool/network/security summaries with the full event available in log details. Tempo supplies the complete request waterfall; the trace view also exposes Grafana's built-in span graph. Select a row in Recent traces to populate the waterfall. Token and cost panels only display values explicitly reported by the connector. Input tokens include the prompt/context usage reported by the provider and may be much larger than output for long-context agents. Missing values remain Not reported and are never estimated.

See the Agent360 dashboard guide for a visual tour of every section, the agent/model/tool topology, trace drill-down, and the difference between 0, No data, and Not reported.

They're auto-provisioned via bundles/local_observability_stack/grafana/provisioning/. The folder is named "DefenseClaw" inside Grafana so they're easy to find. Edits you make in the UI are not persisted back to disk by default — copy them out with Dashboard → JSON Model if you want to keep them.

Tear it down

defenseclaw setup local-observability down

Stops all five containers and removes the Compose project. Dashboards and Grafana state live in named Docker volumes so they survive down/up cycles.

To wipe everything including the volumes:

defenseclaw setup local-observability reset --yes

Status and logs

defenseclaw setup local-observability status
defenseclaw setup local-observability logs --service grafana --follow

status shows container health and the resolved OTLP endpoint. logs --service <name> tails one of the five containers — useful when Grafana isn't picking up dashboards or the OTel collector is dropping spans. Drop --service to fan out logs from every container at once.

Use it alongside Splunk

Local observability and Splunk are independent sinks. A common pattern:

• Engineers and SREs use the local Grafana stack for live investigation.
• The same gateway also forwards every event to the org Splunk for retention and SOC.

Just run both setup commands — they edit different blocks in config.yaml:

defenseclaw setup local-observability up
defenseclaw setup splunk --enterprise --hec-endpoint ... --hec-token ...

See Splunk integration for the dashboards we ship for the local Splunk app and for tips on building your own SPL queries against the gateway sourcetypes.

Troubleshooting