Observability

DefenseClaw v8 uses one pipeline for audit evidence, security findings, guardrail decisions, agent lifecycle, model/tool activity, platform health, and diagnostics:

agent · guardrail · scan · policy · platform
                     |
                     v
       bucket collection (logs/traces/metrics)
                     |
                     v
              canonical record
                     |
       +-------------+--------------+----------------+
       |             |              |                |
       v             v              v                v
 local SQLite   select/redact   select/redact    select/redact
  all logs          OTLP         Splunk HEC        Galileo
                  L/T/M             logs            traces

Collection happens once; every destination independently selects, redacts, and delivers its own projection.

The source of truth is observability: under config_version: 8. It owns collection, local retention, resource attributes, sampling, metric policy, redaction profiles, destination routing, and transport settings. Notification webhooks remain a separate product surface.

Connector-native OTLP ingress is separate from outbound destination routing. For Codex, setup writes a connector-scoped bearer and X-DefenseClaw-Source: codex header for the loopback /v1/{logs,metrics,traces} endpoints; see the Codex connector guide for credential, permission, and teardown behavior.

Full-fidelity defaults

The minimal configuration is:

config_version: 8
observability: {}

It resolves to four important defaults:

• all registered logs, traces, and metrics are collected;
• mandatory local SQLite stores every collected log unredacted;
• there is no remote export until a destination is configured;
• an enabled destination with no send or routes exports every bucket and every signal its kind supports, unredacted.

Use generated views instead of expanding every default into YAML:

defenseclaw config validate
defenseclaw config show --effective --section observability
defenseclaw config reference observability
defenseclaw observability plan

Buckets

Every canonical record belongs to one bucket:

Evaluations and findings are not process/summary copies of the same object. A guardrail evaluation records an inspection and its decision; zero or more durable security findings may result. An asset scan records the scan operation; findings remain in security.finding. An asset lifecycle transition also produces an enforcement.action only when a policy action is attempted.

Destination capabilities and fan-out

This configuration sends everything supported to both destinations:

config_version: 8
observability:
  destinations:
    - name: engineering
      kind: otlp
      protocol: grpc
      endpoint: otel.example.com:4317
      headers:
        Authorization: {env: OTEL_AUTHORIZATION}
    - name: soc
      kind: splunk_hec
      endpoint: https://splunk.example.com:8088/services/collector/event
      token_env: SPLUNK_HEC_TOKEN

engineering receives all logs, traces, and metrics. soc receives all logs because Splunk HEC is logs-only. Both receive unredacted projections because no policy overrides the default. Local SQLite also receives every collected log.

Adding destinations is fan-out, not failover or a selection of one backend. Each leg has independent filtering, redaction, queueing, delivery, and health.

Narrow a destination

Use concise send for the normal case:

observability:
  destinations:
    - name: soc
      kind: splunk_hec
      endpoint: https://splunk.example.com:8088/services/collector/event
      token_env: SPLUNK_HEC_TOKEN
      send:
        signals: [logs]
        buckets: [compliance.activity, security.finding, enforcement.action]
        redaction_profile: strict

Use ordered routes for source/action/severity filters or exclusions:

observability:
  destinations:
    - name: archive
      kind: otlp
      protocol: http/protobuf
      endpoint: https://otel.example.com
      routes:
        - name: drop-diagnostics
          signals: [logs, traces, metrics]
          selector: {buckets: [diagnostic]}
          action: drop
        - name: high-security
          signals: [logs, traces]
          selector:
            buckets: [security.finding, enforcement.action]
            min_severity: HIGH
          action: send
          redaction_profile: sensitive

Routes are first-match-wins per destination and signal. Different selector fields are ANDed; values in one field are ORed. An unmatched record is not sent to that destination. Supported selectors are buckets, sources, connectors, actions, event_names, and min_severity.

Collection is earlier and controls runtime cost. If a bucket has collect.traces: false, no destination route can recreate those traces. A small SQLite-only mandatory compliance floor survives normal log collection disablement.

Tune bounded delivery

Every optional destination has independent routing, projection, queueing, retry, and health. JSONL and console accept queue count/byte controls. Splunk HEC, HTTP JSONL, and OTLP also accept push-batch count/byte/delay controls. Prometheus is pull-based and rejects batch.

If a queue count or byte limit would be exceeded, DefenseClaw drops the newest attempted enqueue, preserves older FIFO work and mandatory SQLite history, and records bounded health telemetry. Sibling destinations continue independently. Transient or ambiguous acknowledgement failures retry the exact immutable projection; a lost acknowledgement can create a duplicate, so consumers use the record ID for deduplication. See the generated configuration reference for kind-specific fields.

Destination failure circuits

Each optional destination signal route (logs, traces, or metrics) has an independent in-memory delivery circuit. By default, three consecutive terminal transient or permanent-payload batches open that route for 30 seconds. A single payload rejection therefore does not suppress later valid telemetry. Authentication and unsafe-endpoint failures open the affected route immediately for 24 hours because retrying unchanged configuration would only repeat expensive or unsafe work.

While a circuit is open, new and already queued work for that route is rejected before adapter delivery and size-estimation work. Sibling signals, destinations, and mandatory local SQLite persistence continue. When cooldown expires, exactly one record is admitted as a half-open recovery probe. Success closes the circuit; another terminal failure reopens it.

Gateway health and defenseclaw doctor expose the circuit state, consecutive failures, bounded failure class, and open deadline. Transient-open and half-open states are warnings; an open authentication, permanent-payload, or unsafe-endpoint circuit is a failed Doctor check.

The circuit is runtime suppression, not a silent policy edit. It does not rewrite config.yaml or permanently disable a route. Repair the destination credential or endpoint and reload the gateway. If export is no longer wanted, disable that named optional route explicitly:

defenseclaw setup observability disable NAME

Edit bucket and redaction policy safely

Bucket names form a closed fourteen-name catalog; operators override collection or delivery but cannot add names in config.yaml. Back up the source with private permissions, edit only deliberate overrides, and validate before activation:

defenseclaw config validate
defenseclaw config show --effective --section observability
defenseclaw observability plan
defenseclaw-gateway restart
defenseclaw doctor

Stop if validation fails. A global or bucket redaction profile also changes local SQLite. To preserve full-fidelity local history while redacting a remote trust boundary, apply the profile only to that destination's send or matching route.

Centralized redaction

Use defenseclaw setup redaction for the guided policy editor. The first screen covers broad policy choices; Show advanced settings? opens bucket collection, custom profiles, destination policy, and ordered routes. For automation, start with defenseclaw setup redaction status --json and preview changes with --dry-run.

defenseclaw setup redaction remove-all --yes

This selects none for every configurable log and trace projection while leaving the managed enterprise destination locked.

Profile resolution is route, then bucket, then global default, then catalog:

observability:
  defaults:
    redaction_profile: sensitive
  buckets:
    model.io:
      redaction_profile: content
  redaction_profiles:
    soc:
      extends: sensitive
      detectors: [pii, credentials, secrets]
      field_classes:
        content: detect
        evidence: detect
        path: hash
        credential: remove

detect replaces only sensitive substrings; whole redacts the full field; hash creates a nonreversible same-install correlation token; remove omits; and preserve retains. See Redaction for built-ins, field classes, examples, and migration behavior.

A field-processing failure replaces the complete affected field with a safe fail-closed token and never falls back to raw content. A classification, projection-context, traversal, or complete-record serialization/size failure rejects that destination's projection; independently successful destinations continue. Profile none intentionally preserves content but still enforces schema, type, size, and serialization limits.

Local history

Exactly one generated local-sqlite destination stores every collected log and mandatory floor event. It cannot be disabled or filtered:

observability:
  local:
    path: ~/.defenseclaw/audit.db
    judge_bodies_path: ~/.defenseclaw/judge_bodies.db
    retention_days: 90

retention_days: 0 means retain indefinitely and produces a capacity warning. Raw judge-body capture is controlled separately by guardrail.retain_judge_bodies.

Choose an operator view

The defenseclaw-genai-rich-v1 trace profile preserves root agents, subagents, turns, workflows, lifecycle/execution/phase IDs, model and tool operations, retrieval, approvals, guardrail/judge spans, links, events, and stable correlation. The local-observability-v1 consumer profile protects the Agent360 and bundled dashboard query contract. If an operator narrows the local destination, plan and setup output report partial dashboard coverage.

Local dashboards consume the canonical OTEL projection as delivered and do not redact, mask, or hide fields again. DefenseClaw centrally applies the selected profile before export; Grafana displays or links every field present after that projection, including content when the producer supplies it.

Upgrade from v7

Run the normal command:

defenseclaw upgrade --yes

The upgrader backs up the source, converts and validates the complete v8 candidate, preserves narrower v7 collection/routing/redaction behavior, refreshes owned local dashboard assets without resetting data volumes, restarts, and checks health. No separate migration approval or apply command is required. The v8 gateway does not rewrite v7 configuration at startup or run both formats in parallel.

Live v8 observability policy lives in YAML. Ambient DEFENSECLAW_OTEL_*, standard OTEL_EXPORTER_OTLP_*, and DEFENSECLAW_DISABLE_REDACTION values are upgrade-only inputs, not runtime routing/redaction controls. YAML names credential environment references with token_env, bearer_env, or {env: NAME}; secret values stay out of the source and rendered plan.

See Upgrade DefenseClaw for rollback and verification.

Alert runbooks

These procedures are the canonical targets used by the bundled alert rules. Keep connector IDs, trace IDs, and destination names in incident evidence, but do not paste prompt content, credentials, or raw audit databases into tickets.

Runbook: schema violations

1. Open Runtime & Reliability → Schema violations by event type/code in the bundled Grafana dashboard and identify the producing event and violation code.
2. Run defenseclaw doctor, defenseclaw config validate, and defenseclaw observability plan. Resolve configuration or registry errors before restarting anything.
3. Inspect correlated gateway or Collector logs without copying event content into the incident. A sustained nonzero count normally indicates that a producer and the generated telemetry registry disagree.
4. For a source checkout, run make telemetry-check, make check-schemas, and the owning producer tests. Roll back an incompatible producer or regenerate and validate its registry change before redeploying.

Runbook: block SLO

1. Confirm that the alert window contains real guardrail traffic; an idle histogram is not a latency breach.
2. Open Admission block SLO compliance, then pivot through Guardrail Evaluations and Connector Detail for the affected connector and policy.
3. Compare regex, AI Defense, judge, policy, and finalization latency to isolate the slow stage or upstream dependency.
4. Preserve the configured enforcement action during diagnosis. Change to a fail-open posture only under the incident commander's approved procedure, and retain the connector and trace IDs used to justify that decision.

Runbook: exporter stalled

1. Run defenseclaw observability plan and confirm that the named destination is enabled and that its selectors include the expected signals and buckets.
2. Check defenseclaw status, the gateway log, and local Collector logs. For a remote destination, also verify DNS, TLS, credentials, and network reachability.
3. Run defenseclaw observability destination test <name>. Add --write-probe only when a bounded delivery write is appropriate; the probe is isolated and does not create normal dashboard traffic.
4. Inspect that destination's queue, retry, drop, and circuit telemetry. An open circuit suppresses only that route's adapter work while mandatory local SQLite and sibling destinations continue. Repair and reload the destination, or run defenseclaw setup observability disable <name> when disabling it is the intended policy change.

Runbook: audit sink

For compatibility, this alert can still use the historical audit.sink metric name. In live config v8 the affected target is a named observability.destinations entry; there is no live audit_sinks section.

1. Identify the sink_kind and destination name, then inspect its delivered, dropped, retry, and circuit-state telemetry.
2. Verify its referenced credential, endpoint, network_safety, send, and routes without revealing secret values.
3. Run defenseclaw observability destination test <name>; use --write-probe only for an approved isolated write.
4. Confirm mandatory local SQLite remains healthy. Repair or disable only the affected optional destination; do not remove local retention or broaden routing/redaction merely to clear the alert.

Common questions