Observability

Configure logs, traces, metrics, redaction, local history and independent export destinations from one observability graph.

DefenseClaw collects each event once and builds one record from it. Log records are stored in local history. Every record, logs, traces and metrics, goes to the destinations you add: an OpenTelemetry (OTLP) collector, Splunk, Galileo, a file, or a webhook. Each destination picks the buckets it wants and applies its own redaction.

collected buckets
logs only
redacted copy
Event sourcesagent · guardrail · scanpolicy · platform
Bucket collectionlogs · traces · metrics
Record
Local SQLiteevery log
Your destinationsOTLP · Splunk HEC · Galileofile · webhook

List view for small screens. Use the expand button to open the drawing.

  1. Event sourcesagent · guardrail · scanpolicy · platform
    • Bucket collection
  2. Bucket collectionlogs · traces · metrics
    • collected bucketsRecord
  3. Record
    • logs onlyLocal SQLite
    • redacted copyYour destinations
  4. Your destinationsOTLP · Splunk HEC · Galileofile · webhook
  5. Local SQLiteevery log
Each event becomes one record. Local SQLite keeps every collected log; traces and metrics go only to destinations. Each destination selects and redacts its own copy, limited to the signals its kind supports.

How it works

  • Buckets. Every record belongs to one of 14 buckets, such as security.finding, enforcement.action or model.io. You can turn off collection per bucket. The full list is in the reference.
  • Local history. A SQLite database in the DefenseClaw data directory keeps every collected log, 7 days by default. It cannot be turned off or filtered. A small set of content-free compliance events is always written, even when you turn off collection for their bucket.
  • Destinations. With no destination configured, nothing leaves the computer. An enabled destination with no filter receives every bucket and every signal its kind supports. Adding a second destination adds a second copy. It is not a failover.
  • Independent delivery. Each destination has its own bounded queue, retry and health. A slow or failing destination does not block agents or other destinations.
  • Redaction. DefenseClaw redacts a record once, per destination, before it is sent. Four built-in profiles run from none (the default) to strict. See Redaction.

Full-fidelity defaults

The smallest valid config is:

config_version: 9
observability: {}

It collects every registered log, trace and metric, stores every log unredacted in local SQLite, and exports nothing. These commands show the generated policy instead of making you write it out:

defenseclaw config validate
defenseclaw config show --effective --section observability
defenseclaw config reference observability
defenseclaw observability plan

The config reference observability output starts with DEFENSECLAW CONFIGURATION v9, the current config_version. Its schema is in the schema family v8, which covers config_version 8 and 9.

The default sends full content

With no redaction profile, an enabled destination receives prompts, responses, tool arguments, file paths and identifiers as recorded. Choose a profile before you send to anything outside your trust boundary.

What it does not do

  • It does not keep traces or metrics locally. Local history holds logs; traces and metrics exist only at the destinations you send them to.
  • It does not replace the audit log. For logs, local history is the durable record; destinations are copies.
  • It does not backfill. Records that a destination missed while it was down or while the gateway restarted are not resent. Export the local history if you need them.
  • Dashboards add no redaction of their own. They show what the destination received.
  • Webhook alerts to Slack, PagerDuty or Webex are a separate feature. They notify; they do not export telemetry.

Check your setup

defenseclaw setup observability list
  Observability destinations
  --------------------------
  NAME                     KIND         STATE     SIGNALS                BUCKETS  POLICY               REDACTION
  local-sqlite             sqlite       enabled   logs                   14       implicit_local       unredacted (none)
  engineering              otlp         enabled   logs,traces,metrics    14       capability_default   unredacted (none)
  Retention: 7 days
  Plan digest: 9cf948dc…

defenseclaw observability plan shows the same policy per bucket and signal.

Buckets

Every record belongs to one bucket. The catalog is closed at 14 names; you can change collection and delivery for a bucket but cannot add one.

BucketWhat it records
compliance.activityA control-plane change that was attempted, applied, rejected or failed.
security.findingA durable risk, with status, evidence and remediation.
guardrail.evaluationA runtime inspection and its decision, including clean ones.
enforcement.actionA block, quarantine, disable, approval or other policy action.
model.ioA model operation, with permitted content, usage, latency and outcome.
tool.activityA tool run, with permitted arguments and results, status and latency.
asset.scanA skill, MCP server, plugin or source scan.
asset.lifecycleDiscovery, install, enable, quarantine, restore or removal of an asset.
network.egressAn outbound operation or egress decision.
agent.lifecycleAgents, subagents, turns, workflows and executions.
ai.discoveryAI components and runtimes found. Includes one ai.runtime.plane_health log per running plane per poll (about once a minute).
telemetry.ingestAn inbound OTLP record that was accepted, normalized, rejected or re-exported. Also carries correlation.relationship.changed.
platform.healthGateway, storage, exporter, queue, guardrail and sidecar health.
diagnosticA debug fact that fits no other bucket.

A guardrail evaluation records the inspection; zero or more security.finding records may follow. A scan records the operation; its findings go to security.finding. A lifecycle change also produces an enforcement.action only when a policy action is attempted. OpenShell sandboxes write into the same buckets.

To stop collecting a bucket's signal, set observability.buckets.<bucket>.collect.<signal>: false. Collection runs before routing, so no destination route can bring those records back. The mandatory floor (below) is still written.

Destination capabilities and fan-out

Destination kindSignals it receives
jsonl, console, splunk_hec, http_jsonllogs
prometheusmetrics
otlplogs, traces, metrics
otlp with preset: galileotraces

An enabled destination with no send or routes receives every bucket and every signal its kind supports. Each destination has its own filtering, redaction, queue, delivery and health.

observability:
  destinations:
    - name: engineering
      kind: otlp
      protocol: grpc
      endpoint: otel.example.com:4317
      headers:
        Authorization: {env: OTEL_AUTHORIZATION}
    - name: soc
      kind: splunk_hec
      endpoint: https://splunk.example.com:8088/services/collector/event
      token_env: SPLUNK_HEC_TOKEN

engineering receives all logs, traces and metrics. soc receives all logs. Both are unredacted.

Send to an OTLP collector

An OTLP destination receives logs, traces and metrics from the gateway. Use it for Datadog, Honeycomb, New Relic, Grafana Cloud or your own collector.

Before you start

  • The collector address, and a header or token if it needs one.
  • Store the token in an environment variable, not in a command line. Command-line values are visible to other users in ps.

Steps

  1. Add the destination.

    defenseclaw setup observability add otlp \
      --name engineering \
      --endpoint otel.example.com:4317 \
      --protocol grpc \
      --non-interactive
      Generic OTLP: added as destination 'engineering'
      Test it with: defenseclaw setup observability test engineering
      Saved. The gateway isn't running; it loads this destination when it starts (defenseclaw-gateway start).

    For a collector on this computer or a private network, add --allow-private-networks. If it has no TLS, add --plaintext too. Use --environment <name> to tell computers with the same host name apart.

  2. Narrow what it receives. With no filter, it gets every bucket and every signal. This sends only security logs and traces:

    observability:
      destinations:
        - name: engineering
          kind: otlp
          protocol: grpc
          endpoint: otel.example.com:4317
          send:
            signals: [logs, traces]
            buckets: [security.finding, enforcement.action]

    Edit ~/.defenseclaw/config.yaml, then run defenseclaw config validate. A running gateway applies a valid edit by itself. For filters by connector, action or severity, see routes and selectors.

  3. Review the result.

    defenseclaw observability plan --bucket security.finding --signal logs
    BUCKET            SIGNAL  COLLECT  DESTINATION   DECISION  ROUTE                                   REDACTION  …
    security.finding  logs    yes      local-sqlite  send      all-collected-logs-and-mandatory-floor  none       …
    security.finding  logs    yes      engineering   send      send                                    none       …

Check it worked

defenseclaw observability destination test engineering

--write-probe is not supported for OTLP: it would create ordinary logs, traces or metrics. The handshake checks connectivity and does not prove delivery from the gateway.

To check delivery, use an event in a bucket the destination receives. A clean agent call may record nothing in security.finding or enforcement.action, so with the step 2 filter it may send nothing. Check before you add the filter, or pick an event in one of the filter's buckets: run step 3 with its bucket and confirm engineering shows send. Then find its event ID in the local audit log, and that ID at the collector or its destination.

The test connects from your shell, while the gateway delivers through the proxy it was started with. If the test passes but defenseclaw doctor reports delivery_failed, look for export failed lines in the gateway log.

Undo

defenseclaw setup observability disable engineering

disable keeps the entry. defenseclaw setup observability remove engineering deletes it.

Next

Narrow a destination

Use send for the common case:

observability:
  destinations:
    - name: soc
      kind: splunk_hec
      endpoint: https://splunk.example.com:8088/services/collector/event
      token_env: SPLUNK_HEC_TOKEN
      send:
        signals: [logs]
        buckets: [compliance.activity, security.finding, enforcement.action]
        redaction_profile: strict

Use ordered routes for filters by source, action or severity, and for exclusions:

observability:
  destinations:
    - name: archive
      kind: otlp
      protocol: http/protobuf
      endpoint: https://otel.example.com
      routes:
        - name: drop-diagnostics
          signals: [logs, traces, metrics]
          selector: {buckets: [diagnostic]}
          action: drop
        - name: high-security
          signals: [logs, traces]
          selector:
            buckets: [security.finding, enforcement.action]
            min_severity: HIGH
          action: send
          redaction_profile: sensitive
RuleBehavior
OrderThe first matching route wins, per destination and signal.
MatchingDifferent selector fields are ANDed. Values in one field are ORed.
No matchThe record is not sent to that destination.
Selectorsbuckets, sources, connectors, actions, event_names, min_severity.

To keep relationship records out of a destination, drop them ahead of the routes that send:

observability:
  destinations:
    - name: soc
      kind: splunk_hec
      endpoint: https://splunk.example.com:8088/services/collector/event
      token_env: SPLUNK_HEC_TOKEN
      routes:
        - name: drop-relationship-changes
          signals: [logs]
          selector: {event_names: [correlation.relationship.changed]}
          action: drop
        - name: everything-else
          signals: [logs]
          selector: {buckets: ["*"]}
          action: send

The mandatory floor is a small set of content-free compliance events. They are always written to local SQLite and never exported.

Tune bounded delivery

JSONL and console destinations take queue count and byte limits. Splunk HEC, HTTP JSONL and OTLP also take batch limits. Prometheus is pull-based and rejects batch.

FieldDefaultBounds
batch.max_queue_size20481..65536 records
batch.max_queue_bytes671088644198400..268435456 bytes
batch.max_export_batch_size5121..8192, no greater than queue count
batch.max_export_batch_bytes83886084263936..67108864 encoded bytes
batch.scheduled_delay_ms50001..600000. The Galileo preset defaults to 1000.
timeout_ms10000Positive, bounded milliseconds

When a queue is full, DefenseClaw drops the newest record, keeps older queued work and local history, and records the drop. Other destinations continue.

A transient failure keeps the exact projection queued for at most 24 attempts, with exponential backoff from 2 seconds to a cap of 30 seconds. A batch still unacknowledged after that is counted as dropped. Queued records are lost on gateway restart, and optional destinations are not backfilled from local history. A lost acknowledgement can cause a duplicate, so consumers deduplicate on the record ID.

Destination failure circuits

Each destination and signal (logs, traces or metrics) has its own in-memory circuit.

FailureCircuit opens
Three consecutive terminal transient or permanent-payload batches30 seconds
Authentication failureAuthentication failures open the affected route immediately for five minutes; a half-open probe then retries it
Unsafe endpoint24 hours

While a circuit is open, work for that route stays in its bounded queue. When the queue fills, the newest record is dropped and counted. Other signals, other destinations and local SQLite continue. The oldest queued batch is the recovery probe: success closes the circuit and another failure reopens it.

defenseclaw doctor and gateway health show the state, failure count, failure class and open deadline. Transient-open and half-open states are warnings. An open authentication, permanent-payload or unsafe-endpoint circuit fails the check.

The defenseclaw.queue.drops metric counts lost records per destination.<name>.<signal> queue, with reason dropped (overflow, retry exhaustion, shutdown) or rejected. When the gateway stops before a destination takes its queue, gateway.log names each destination and count, and the next start adds them to the counters.

A circuit does not edit config.yaml. Fix the credential or endpoint and reload. To stop export, run defenseclaw setup observability disable NAME.

Edit bucket and redaction policy safely

Back up the file with private permissions, edit only deliberate overrides, and validate before relying on it:

defenseclaw config validate
defenseclaw config show --effective --section observability
defenseclaw observability plan
defenseclaw doctor

A running gateway applies a valid edit to buckets, redaction profiles and destinations without a restart, and keeps the previous policy if it rejects the edit. The Policy row of doctor shows which. The RELOAD column of observability plan says live_reloadable or restart_required per route. Only a Prometheus destination and observability.local.path need defenseclaw-gateway restart.

A global or bucket redaction profile also changes local SQLite. To redact only a remote destination, put the profile in that destination's send or route. See Redaction.

Centralized redaction

DefenseClaw redacts each record once, at the boundary between the record and a destination, using the profile chosen for that destination. One event can stay whole in local history, go whole to a forensic backend, and go with only sensitive substrings removed to a SOC backend. For the steps, see Redact before export.

A new install uses profile none everywhere, local SQLite included.

Local history

One generated local-sqlite destination stores every collected log and the mandatory floor. It cannot be disabled or filtered.

observability:
  local:
    path: ~/.defenseclaw/audit.db
    judge_bodies_path: ~/.defenseclaw/judge_bodies.db
    retention_days: 7
SettingBehavior
local.pathMust resolve inside the DefenseClaw data directory. doctor reports an outside path as a custody failure. A ~ or a relative path is resolved against the data directory by the gateway, status and doctor. Changing it needs defenseclaw-gateway restart.
retention_daysDefault 7. A reaper runs a minute after startup and every six hours and deletes in small batches. 0 keeps history forever and doctor warns about capacity.
max_store_mbOff by default (0): history is kept by days only. Set it (64 or more, in MiB) to also cap the size of audit.db: when the used pages pass the cap, the reaper deletes the oldest eligible history the same way retention_days does, aiming for nine tenths of the cap. The newest hour is always kept, so the file can pass the cap during a flood. A running gateway applies a new value, including setting it back to 0, without a restart, and status stops showing the cap. A retention run that fails shows retention_state: degraded and retention_failure: run_failed; the gateway log (the service journal on a managed computer; Managed command availability) names the cause. The run is tried again after a minute, then after waits that double up to 30 minutes, and at once after the next change to the retention settings. Releases before 1.0.6 refuse this key: remove it before going back with defenseclaw upgrade --version (defenseclaw rollback restores the previous configuration). status shows it with the retention (7 days, size cap 64 MiB); on a managed computer defenseclaw-gateway status shows it on the Telemetry retention: line and enterprise linux status or enterprise macos status on the audit store retention: line (audit_store in --json, with the size on disk).
judge_bodies_pathRaw LLM-judge request and response text, kept while guardrail.retain_judge_bodies is true (the default). Redaction profiles do not apply. Turn it off with defenseclaw config set guardrail.retain_judge_bodies false, then restart the gateway.
InventoryThe same window prunes AI discovery scan history and unseen agent identities (agt-) in inventory.db. The newest scan is always kept. doctor warns at 1 GiB.

On per-user and standalone installs, if a completed retention pass still leaves the used pages of audit.db above max_store_mb, gateway health reports retention_state: warning, retention_cap_used_bytes, retention_cap_bytes, and a bounded retention_cap_reason. defenseclaw status, defenseclaw doctor, and defenseclaw-gateway status show the size, cap, reason, and one suggested step. Active correlation cursors and pending operations can keep old graph rows for a long session, and the newest hour is always kept. End unused sessions or raise observability.local.max_store_mb if the warning persists. This is a warning only; it does not block tool calls. A later pass that meets the cap clears the warning.

1.0.7 retention and memory limits

GAP-0408: active correlation state or recent history can keep used pages above max_store_mb after retention. The warning names the used size, configured cap, and reason; it does not stop tool calls. GAP-0327: on a per-user macOS install, idle gateway memory can grow gradually with the audit store even while performance remains within its release budget. Retention bounds the stored history, but does not immediately return all memory.

From 1.0.6, a standalone install stores the field classes of a record (the redaction class of each field) once per set in audit.db, and each row names its set by digest. The row hash and signature cover that digest, so a row whose set is missing or changed fails the integrity check. Destinations still get the full record. Releases 1.0.4 and 1.0.5 read these rows after a rollback. A Secure Client install keeps the classes in each row.

Local history is authoritative against other OS accounts: they cannot change it. It is not tamper-proof against its own account, which owns the files. Use a remote SIEM or collector as the system of record, and deduplicate on the record ID.

If a write to local history fails, for example on a full disk, the record still goes to destinations and health counts it as dropped on local-sqlite. When writes resume, the gateway writes one sqlite.write_failed record with the number of missing records. The count survives a crash in local-write-losses.journal in the data directory (64 KiB, reserved in advance, counts and failure classes only). If the journal cannot be written, status and doctor report the in-memory count.

The gateway checks the write-ahead log (audit.db-wal) every two seconds and copies it into audit.db; it trims the file when idle. From 88 MiB of active log it copies and syncs the log without holding the audit writer until little is left, then holds the writer for up to 180 ms to restart the log. If the active log still passes 176 MiB, ordinary catch-up holds the audit writer for up to 100 ms. At 208 MiB, a flooded log can use one fixed hold of up to 1 second to restart before the 256 MiB ceiling. Only the wait to acquire the writer grows after a missed pass, up to 10 seconds; the hold cap never grows. A SQLite statement can run past that hold, delaying hook answers. If the file is still over 256 MiB after a check, the gateway warns, repeats the warning every ten minutes while it stays over, and reports when it falls back under. A slow disk or long-lived reader can still push it past 256 MiB; this is an alarm threshold, not a guaranteed disk bound. Secure Client does not run this check. The defenseclaw.sqlite.wal.bytes metric reports the file size.

Choose an operator view

Reference

Upgrade from v7

Run defenseclaw upgrade --yes. It converts and validates the v7 settings. It reads ambient DEFENSECLAW_OTEL_* and standard OTEL_EXPORTER_OTLP_* values only during that upgrade. The Galileo preset keeps your project and Log stream. See Upgrade DefenseClaw.

Alert runbooks

These are the targets of the bundled alert rules. Keep connector IDs, trace IDs and destination names in incident evidence; do not paste prompts, credentials or raw audit databases into tickets.

Runbook: schema violations

  1. In Grafana, open Runtime & Reliability → Schema violations by event_type × code and find the producing event and code.
  2. Run defenseclaw doctor, defenseclaw config validate and defenseclaw observability plan. Fix configuration or registry errors before restarting anything.
  3. Read the gateway or collector logs without copying event content. A sustained count means a producer and the telemetry registry disagree.
  4. In a source checkout, run make telemetry-check and make check-schemas, then the producer's tests. Roll back the producer or regenerate its registry change.

Runbook: block SLO

  1. Confirm the alert window has real guardrail traffic. An idle histogram is not a breach.
  2. Open Admission block SLO compliance, then Guardrail Evaluations and Connector Detail for the connector and policy.
  3. Compare regex, AI Defense, judge, policy and finalization latency to find the slow stage.
  4. Keep the configured enforcement action during diagnosis. Fail open only under your incident commander's procedure.

Runbook: exporter stalled

  1. Run defenseclaw observability plan. Confirm the destination is enabled and selects the signals and buckets you expect.
  2. Check defenseclaw status, the gateway log and collector logs. For a remote destination, check DNS, TLS, credentials and the network.
  3. Run defenseclaw observability destination test <name>. Add --write-probe only where supported; OTLP supports the handshake only. To prove gateway delivery, match a local audit event ID at the destination. The test connects directly from your shell. The gateway delivers through the proxy it started with (HTTPS_PROXY and NO_PROXY, or the enterprise proxy). If the test passes but doctor shows delivery_failed, find [observability] <name> ... export failed lines in the gateway log; each says whether the connection was direct or through a proxy. If it used a proxy, restart the gateway from a shell with the right proxy settings.
  4. Inspect the destination's queue, retry, drop and circuit telemetry. Repair and reload it, or run defenseclaw setup observability disable <name>.

Runbook: audit sink

The alert keeps the historical audit.sink metric name. The target is a named destination.

  1. Find the sink_kind and destination name, then its delivered, dropped, retry and circuit telemetry.
  2. Check its credential, endpoint, network_safety, send and routes without printing secrets.
  3. Run defenseclaw observability destination test <name>; use --write-probe only for a supported, approved write. OTLP supports the handshake only.
  4. Confirm local SQLite is healthy. Repair or disable only the failing destination. Do not remove local retention or widen routing to clear the alert.