Local observability stack

defenseclaw setup local-observability brings up a full local OpenTelemetry stack and wires the gateway to it in one shot. It's what the project itself uses for development, demos, and rule-pack tuning.

What it brings up

The stack runs from bundles/local_observability_stack/docker-compose.yml:

All five run under one Compose project (defenseclaw-observability). They share a Docker network and the bundled Grafana datasources point at each container by service name.

Bring it up

defenseclaw setup local-observability up

defenseclaw observability destination test local-observability

up also refreshes the host-mounted stack copy in ~/.defenseclaw/observability-stack/ from the bundled version before Compose starts. That includes Grafana dashboards, Prometheus rules, Loki, Tempo, and OTel Collector config. Deleting Docker containers is not enough to update those files because they live on the host; pass --no-refresh-config only when you intentionally want to keep local edits.

Useful flags

The up subcommand exposes the bridge configuration directly:

defenseclaw setup local-observability up \
  --endpoint 127.0.0.1:4317 \
  --signals traces,metrics,logs \
  --service-name defenseclaw-gateway \
  --timeout 90

The OTLP protocol is read from the bridge readiness contract (currently grpc); there is no --protocol flag on up. To use http/protobuf, configure that protocol on the named local-observability destination.

What ends up in config.yaml

The example below is the result of the sample up command using --service-name defenseclaw-gateway; without that flag, service.name is defenseclaw.

config_version: 8
observability:
  resource:
    attributes:
      service.name: defenseclaw-gateway
  destinations:
    - name: local-observability
      kind: otlp
      endpoint: 127.0.0.1:4317
      protocol: grpc
      network_safety:
        allow_private_networks: true

Because send and routes are omitted, this OTLP destination receives all reviewed buckets and all three signals, unredacted. That is both the v8 capability default and the complete local-observability-v1 dashboard profile. If a destination with the same name already exists, the bridge updates it in place rather than adding a duplicate. Other telemetry destinations and notifier webhooks are left untouched.

Bundled Grafana dashboards

The bundle ships a full board set in bundles/local_observability_stack/grafana/dashboards/ — every *.json file here is auto-loaded:

See the complete Grafana dashboard catalog for the owner and purpose of every board, the drill-down map, backend usage, and the exact difference between 0, No data, and Not reported.

For a multi-connector gateway, start on Connectors (Overview) to compare connectors side by side, then click a connector cell to drill into Connector Detail.

Agent Directory and Agent360

Open Agent identity → Runtime Agent Directory. Every hook lifecycle transition updates the stable defenseclaw.agent.last_seen inventory metric, so a newly observed root agent or subagent appears automatically. Click its Agent ID to open the reusable Agent360 dashboard; DefenseClaw does not create or provision a separate dashboard file for each agent.

Agent360 correlates the same stable fields across all three backends:

Use Scope = This agent for a single agent, or Agent tree to include all descendants. Prometheus supplies directory, state, throughput, latency, token, reported-cost totals, and directed phase aggregates. Loki supplies the prompt-to-outcome lifecycle DAG plus compact lifecycle/model/tool/network/security summaries with the full event available in log details. Session and durable lineage anchors may be recovered from the prior 24 hours so selected-range boundaries remain renderable; a recovered relationship edge is kept only when its child has graph-eligible activity in the selected range. Active typed parent_of and inverse delegated_by relationship records establish agent parentage; raw parent fields and trace topology do not. Repeated model work is grouped per owning agent, provider, and model. Repeated tools are grouped per owning agent as Bash, MCP, Skills, Collaboration, File edits, Web/browser, Visual, or Task control; an unrecognized tool keeps its reported name. Exact collaboration.send_message requests are excluded from generic Collaboration summaries so they appear only as message groups; other collaboration tools remain. Summary nodes show total requests and link to every exact raw canonical OTEL record, including their terminal counterparts. Stable agent nodes use agent/root/parent identity, name, type, depth, and connector; optional session fields stay in lifecycle/session/ordered/raw detail rather than splitting one agent's total. One per-root Prompt inputs node counts distinct depth-zero model.request facts, with replay-safe turn/model-request/request/operation/occurrence identity; Session start remains a separate lifecycle anchor. The ordered/raw views retain the individual initial and follow-up request records. For each sender, real collaboration updates to /root or /root/* collapse into one Messages to root node that resolves the exported root agent ID. Exact root task paths and calls remain in ordered/raw drill-downs; non-root targets remain exact groups without an invented target-agent mapping. Tempo supplies request-bounded waterfalls; it does not pretend an asynchronous lifetime is one trace. Select a row in Operation and enforcement traces to populate the waterfall. Token and cost panels only display values explicitly reported by the connector. Input tokens include the prompt/context usage reported by the provider and may be much larger than output for long-context agents. Missing values remain Not reported and are never estimated.

The local stack performs no additional redaction, masking, or field hiding. DefenseClaw applies the selected v8 profile centrally before canonical OTEL export. Grafana displays or links every field that Loki or Tempo actually receives, including content when the producer supplies it; fields removed or transformed before export are not recoverable in a dashboard.

See the Agent360 dashboard guide for a visual tour of every section, the prompt-to-agent lifecycle DAG, trace drill-down, and the difference between 0, No data, and Not reported.

They're auto-provisioned via bundles/local_observability_stack/grafana/provisioning/. The folder is named "DefenseClaw" inside Grafana so they're easy to find. Edits you make in the UI are not persisted back to disk by default — copy them out with Dashboard → JSON Model if you want to keep them.

Managed AI Defense fail-open monitoring

Managed-enterprise mode deliberately allows a request when the authoritative AI Defense inspection cannot return a verdict. DefenseClaw exposes every such decision through the generated OpenTelemetry counter defenseclaw.managed_aid.fail_open.decisions (int64, unit {decision}, delta temporality). The counter has one bounded defenseclaw.metric.reason attribute; the local-observability projection exports it as the Prometheus reason label on defenseclaw_managed_aid_fail_open_decisions_total.

The bundled DefenseClawManagedAIDFailOpenSustained rule uses a positive availability-reason allowlist, so no_content and unknown cannot accidentally become paging conditions:

sum by (reason) (
  rate(defenseclaw_managed_aid_fail_open_decisions_total{reason=~"inspector_unwired|aid_unavailable"}[5m])
) > 0

The expression must remain true for 10 minutes before the critical alert fires. It is traffic-gated by construction: an idle gateway produces no fail-open decisions and does not page. If it fires, verify that the gateway is running in the intended deployment mode, confirm the managed inspector was wired at startup, test AI Defense credentials and endpoint reachability, and check the provider status. The allow behavior is intentional during recovery; restore authoritative inspection before considering the incident resolved.

Tear it down

defenseclaw setup local-observability down

Stops all five containers and removes the Compose project. Dashboards and Grafana state live in named Docker volumes so they survive down/up cycles.

To wipe everything including the volumes:

defenseclaw setup local-observability reset --yes

Status and logs

defenseclaw setup local-observability status
defenseclaw setup local-observability logs --service grafana --follow

status shows container health and the resolved OTLP endpoint. logs --service <name> tails one of the five containers — useful when Grafana isn't picking up dashboards or the OTel collector is dropping spans. Drop --service to fan out logs from every container at once.

Use it alongside Splunk

Local observability and Splunk are independent destinations. A common pattern:

• Engineers and SREs use the local Grafana stack for live investigation.
• The same gateway also forwards every event to the org Splunk for retention and SOC.

Just run both setup commands. They add separate entries to the same observability.destinations list:

defenseclaw setup local-observability up
defenseclaw setup splunk --enterprise --hec-endpoint ... --hec-token ...

See Splunk integration for the dashboards we ship for the local Splunk app and for tips on building your own SPL queries against the gateway sourcetypes.

Troubleshooting