Local observability stack

One-command OpenTelemetry + Loki + Tempo + Prometheus + Grafana stack, pre-wired to the DefenseClaw gateway. Grafana on :3000, dashboards seeded, no manual config.

The local stack gives you 18 DefenseClaw dashboards (19 with the Fleet assets) on localhost:3000 in one command, or 19 from a source checkout (editable install), which also seeds the optional Fleet dashboard. It is meant for development and evaluation. Every port is bound to 127.0.0.1.

sends OTLP
writes
writes
writes
queried by
queried by
queried by
Gatewaydefenseclaw-gateway
OTel Collector:4317 gRPC · :4318 HTTP
Prometheusmetrics · :9090
Lokilogs · :3100
Tempotraces · :3200
Grafanadashboards · :3000

List view for small screens. Use the expand button to open the drawing.

  1. Gatewaydefenseclaw-gateway
    • sends OTLPOTel Collector
  2. OTel Collector:4317 gRPC · :4318 HTTP
    • writesPrometheus
    • writesLoki
    • writesTempo
  3. Tempotraces · :3200
    • queried byGrafana
  4. Lokilogs · :3100
    • queried byGrafana
  5. Prometheusmetrics · :9090
    • queried byGrafana
  6. Grafanadashboards · :3000
The gateway sends logs, metrics and traces over OTLP to the collector, which writes each signal to its own store. Grafana queries the three stores. Ports are bound to 127.0.0.1.

Before you start

  • Docker with Compose v2, running.
  • Ports 3000, 3100, 3200, 4317, 4318 and 9090 free. There is no flag to move them.
  • On Windows, Docker Desktop with Linux containers on Windows Pro, Enterprise or Education. Windows Server and WSL2-only Docker are not supported; send to an existing collector instead (Send to an OTLP collector).

What it brings up

defenseclaw setup local-observability runs the bundled Compose project defenseclaw-observability.

ServiceImageRolePort on 127.0.0.1
otel-collectorotel/opentelemetry-collector-contrib:0.153.0Receives OTLP, writes to Prometheus, Loki and Tempo4317 (gRPC), 4318 (HTTP)
prometheusprom/prometheus:v2.54.1Metrics9090
lokigrafana/loki:3.2.0Logs3100
tempografana/tempo:2.6.0Traces3200
grafanagrafana/grafana:11.3.0Dashboards3000

Bring it up

  1. Start the stack.

    defenseclaw setup local-observability up

    The command starts the five containers, waits for their health checks, and adds a destination named local-observability to your observability config. A running gateway picks it up without a restart.

  2. Print the addresses.

    defenseclaw setup local-observability url
    Grafana:    http://localhost:3000
    Prometheus: http://localhost:9090
    Tempo API:  http://localhost:3200
    Loki API:   http://localhost:3100
    OTLP gRPC:  127.0.0.1:4317
    OTLP HTTP:  127.0.0.1:4318
  3. Open Grafana and sign in as admin. The generated password is in ~/.defenseclaw/observability-stack/.grafana-admin-password. A stack that was already running with anonymous access keeps it, read-only (Viewer): to edit dashboards, rerun setup with --password. The setup summary says which mode applies.

  4. Run one agent turn, for example ask Claude Code or Codex to list the files in a folder. Dashboards fill only from real traffic; a destination test adds nothing.

  5. Open Dashboards → DefenseClaw → Overview. Hook Connectors shows the connector's hook calls. Agent identity → Runtime Agent Directory lists the agent; click its Agent ID to open Agent360.

Check it worked

defenseclaw observability destination test local-observability

The test checks the connection without adding events to history or dashboard counts.

Useful flags

Subcommands: up, down, reset, status, logs, url, env. up takes:

FlagWhat it does
--endpointOTLP endpoint written to config.yaml. Default 127.0.0.1:4317.
--signalsSubset of traces,metrics,logs. All three are needed for full dashboard coverage. up checks the list first: an unknown name stops it with a usage error before it touches the running stack.
--service-nameSets observability.resource.attributes.service.name. Default defenseclaw.
--no-configStart containers without editing config.yaml.
--password / --no-passwordRequire the generated Grafana admin login, or allow anonymous read-only (Viewer) access on the loopback bind: dashboards show, nothing can be edited, and anonymous access is never Admin. New stacks default to a password; an anonymous stack keeps anonymous access, read-only, until you pass --password.
--refresh-bundle / --no-refresh-bundleRefresh ~/.defenseclaw/observability-stack/ from the bundle before starting. On by default.
--refresh-config / --no-refresh-configAlso overwrite dashboards, rules and service config. Pass --no-refresh-config to keep local edits.
--no-wait, --timeoutSkip the readiness wait, or set it in seconds (default 180).

--no-password lets every process on the workstation read the dashboards without signing in. The controller still refuses a Grafana bind other than 127.0.0.1. A raw docker compose up bypasses that check.

What ends up in config.yaml

up writes this destination (shown for --service-name defenseclaw-gateway):

observability:
  destinations:
    - name: local-observability
      kind: otlp
      enabled: true
      endpoint: 127.0.0.1:4317
      protocol: grpc
      tls:
        insecure: true
      network_safety:
        allow_private_networks: true
      send:
        signals: [traces, metrics, logs]
        buckets: ['*']
  resource:
    attributes:
      service.name: defenseclaw-gateway

The name local-observability selects the local-observability-v1 metric projection, which carries the short Prometheus labels the dashboards filter on (connector, scanner, target_type, rule_id, severity). An OTLP destination with any other name exports canonical labels (defenseclaw_connector_source, defenseclaw_scan_scanner, ...), and connector- and scanner-filtered panels stay empty. To send a gateway on another computer to a shared stack, use the local-otlp preset, which writes the same name with tls.insecure: true. In a managed administrator config, use the name local-observability too.

Every signal carries host.name from the operating system. To tell computers with the same name apart, add --environment <name> to setup observability add; it sets observability.resource.attributes.deployment.environment.name.

If up finds a container name that belongs to another copy of the stack, it stops and names the other stack. A port held by a foreign process also stops it: up names the port and says nothing was started; status reads a listener of another process as not running (port held by another process); edits to the stack's docker-compose.yml are overwritten on the next refresh.

Keep the destination name

Do not rename local-observability. The bundled dashboards read metric labels that DefenseClaw only exports to a destination with that name. To feed the stack from a gateway on another computer, add the destination with the preset:

defenseclaw setup observability add local-otlp --endpoint <stack-host>:4317 --non-interactive

Bundled Grafana dashboards

Dashboards: what each board shows and how to read empty panels.

Agent Directory and Agent360

Agent360 shows one agent, or an agent and all its descendants, from one stable identity. New root agents and subagents appear as hook events arrive; DefenseClaw creates no dashboard file per agent. Open it from Agent identity → Runtime Agent Directory by clicking an Agent ID.

Managed AI Defense fail-open monitoring

In managed enterprise mode, a request is allowed when the AI Defense inspection returns no verdict. Each such decision increments defenseclaw.managed_aid.fail_open.decisions (delta, unit {decision}). In the local stack it is the Prometheus counter defenseclaw_managed_aid_fail_open_decisions_total with a reason label.

reasonMeaningPaging
inspector_unwiredThe gateway entered managed inspection without an AI Defense inspector.Page when sustained.
aid_unavailableAI Defense returned no verdict (transport, timeout or token failure).Page when sustained.
no_contentThe request had no inspectable messages.Diagnostic only.
unknownAn invalid producer reason, normalized at the metric boundary.Diagnostic only.

The bundled DefenseClawManagedAIDFailOpenSustained rule fires after 10 minutes of rate(...{reason=~"inspector_unwired|aid_unavailable"}[5m]) > 0. An idle gateway produces no decisions. If it fires, check the deployment mode, the inspector wiring at startup, AI Defense credentials and reachability, and provider status.

Tear it down

defenseclaw setup local-observability down

down stops the containers and keeps their data. The destination stays in your config, so the gateway keeps trying to send; add --disable-config to turn it off too. To delete the stored metrics, logs and traces, run defenseclaw setup local-observability reset --yes.

Use it alongside Splunk

Send to Splunk: run Splunk next to this stack.

Troubleshooting