Observability

Galileo

Send DefenseClaw GenAI traces to Galileo Cloud or self-hosted Galileo while keeping local observability enabled.

DefenseClaw sends its generated galileo-rich-v2 trace projection to Galileo over OTLP HTTP/protobuf. Galileo is a traces-only destination; metrics and logs continue flowing to the bundled local collector or another destination. The projection retains root-agent/subagent topology, workflows, turns, model and tool operations, retrieval, guardrail/judge work, structural events/links, and stable lifecycle/execution correlation when those fields are present.

Cloud setup

export GALILEO_API_KEY='...'

defenseclaw setup galileo \
  --non-interactive \
  --project defenseclaw \
  --logstream production \
  --persist-api-key

defenseclaw setup galileo test

--persist-api-key copies the environment value into the owner-only ~/.defenseclaw/.env. The key is never written to config.yaml; the OTLP header contains ${GALILEO_API_KEY}.

Run defenseclaw setup galileo without flags for the guided wizard, or open Setup → Observability → Galileo Cloud / Self-hosted in defenseclaw tui.

Self-hosted setup

DefenseClaw derives the API hostname by replacing a leading console. with api. or console- with api-, then appends /otel/traces:

export GALILEO_API_KEY='...'

defenseclaw setup galileo \
  --deployment self-hosted \
  --console-url https://console.galileo.example.com \
  --project defenseclaw \
  --logstream production

For deployments with a custom hostname or path convention, pass the complete endpoint explicitly:

defenseclaw setup galileo \
  --deployment self-hosted \
  --trace-endpoint https://api.example.com/galileo/otel/traces \
  --project defenseclaw \
  --logstream production

TLS verification is always enabled for Galileo setup and canary requests.

Keep local observability and Galileo together

defenseclaw setup local-observability up
defenseclaw setup galileo
defenseclaw setup observability list

The resulting configuration contains two independent destinations:

config_version: 8
observability:
  destinations:
    - name: local-observability
      kind: otlp
      protocol: grpc
      endpoint: 127.0.0.1:4317
      network_safety:
        allow_private_networks: true

    - name: galileo
      kind: otlp
      preset: galileo
      protocol: http/protobuf
      endpoint: https://api.galileo.ai/otel/traces
      batch:
        scheduled_delay_ms: 1000
      headers:
        Galileo-API-Key: {env: GALILEO_API_KEY}
        project: defenseclaw
        logstream: production

Omitted policy on local-observability selects all logs, traces, metrics, and buckets, unredacted. Omitted send and routes on preset: galileo have a more specific meaning: the compiler creates a traces-only capability-default route whose event_names are exactly the currently available families in the generated galileo-rich-v2 compatibility profile. The setup command deliberately writes this omitted form. Do not add send: {signals: [traces]} merely to restate the default; that changes the policy from compiler-owned membership to operator intent.

The capability-default Galileo projection is unredacted. To narrow one Galileo destination by bucket and redact it, author an explicit policy:

observability:
  destinations:
    - name: galileo
      kind: otlp
      preset: galileo
      protocol: http/protobuf
      endpoint: https://api.galileo.ai/otel/traces
      headers:
        Galileo-API-Key: {env: GALILEO_API_KEY}
        project: defenseclaw
        logstream: production
      routes:
        - name: redacted-model-and-tool-spans
          signals: [traces]
          selector:
            buckets: [model.io, tool.activity]
            event_names: [span.model.chat, span.tool.execute]
          action: send
          redaction_profile: sensitive

Explicit send or advanced routes replace the generated capability-default route. DefenseClaw treats them as operator intent and does not silently intersect them with compatibility membership. If an explicit selector admits a nonmember, the Galileo projector rejects it as unsupported_shape; destination failure accounting and runtime health report it. Use the effective configuration view to read the generated profile's available families, then run defenseclaw observability plan --signal traces --event-name NAME to review each explicit family before activation. Recheck the effective membership after an upgrade if an advanced route enumerates event names.

If the same redaction posture is acceptable for local history and every matching destination, a bucket-level redaction_profile can preserve Galileo's omitted capability-default route. Use destination policy, as above, when only Galileo must be narrowed or redacted.

Each destination has its own bounded export queue. A Galileo outage does not stop local Tempo/Grafana export, and a local collector outage does not stop Galileo export. Credential-bearing routes are rejected at gateway startup if a hand-edited endpoint uses remote plaintext transport or URL userinfo; loopback collectors may still use plaintext.

Manage and verify

defenseclaw setup observability list
defenseclaw setup observability list --json
defenseclaw setup galileo status
defenseclaw setup galileo status --json
defenseclaw setup galileo test
defenseclaw setup galileo disable
defenseclaw setup galileo enable
defenseclaw setup galileo remove --yes

Destination names are identities. A new name appends a route; the same name updates only that route. The dedicated Galileo command always manages the destination named galileo, so rerunning it reports UPDATE and never replaces local-observability. Preview the exact result without writing:

defenseclaw setup galileo --project defenseclaw --logstream production --dry-run

To configure a second Galileo project, use the unified command with a unique name:

defenseclaw setup observability add galileo \
  --name galileo-security \
  --endpoint https://api.galileo.ai/otel/traces \
  --project defenseclaw \
  --logstream security

The inventory prints every v8 destination's kind, preset, enabled signal set, bucket coverage, and effective redaction profile.

test uses an internal Go helper to resolve the gateway bearer and authenticate only to the loopback gateway API. The bearer is never placed in subprocess argv, helper stdout, or rendered output. The helper asks the running gateway to emit canonical agent/chat spans, flushes the real destination processor, and prints the trace ID only after the Galileo exporter records an OTLP acknowledgement. This is an external write: each invocation creates one content-free, two-span diagnostic trace in the selected Galileo project and Log stream.

The TUI Overview includes a full-width Observability Destinations · Runtime panel. It lists every runtime-loaded named v8 destination with target, process/global/connector scope, kind/preset, state, signals, schema eligibility, delivery/rejection/failure counts, and endpoint. Explicit per-connector suppression is visible too. Headers and credential values are never displayed. The compact Services row remains as a roll-up. defenseclaw status shows each destination, while defenseclaw doctor reads bounded runtime health for every destination and emits the same external two-span canary through the first four enabled Galileo routes in compiled-plan order. If more than four are enabled, doctor leaves their runtime-health checks in place and emits a bounded coverage warning instead of adding unbounded sequential network latency. Every doctor invocation can therefore create up to four diagnostic traces in Galileo.

What you see in Galileo

DefenseClaw exports completed GenAI operations as Galileo-compatible chat, agent, and tool spans. Stable conversation, root-agent, lifecycle, and execution attributes let Galileo group the real-time short traces into one coherent session.

Agent graph: understand delegation and dependencies

Galileo Agent graph showing root agents, delegated agents, models, MCP calls, and native tools

The Agent graph turns span relationships into an execution map. Agent nodes show roots and delegated subagents, chat nodes show model calls, and tool nodes show native tools or MCP operations. Select an edge to inspect traffic volume and relative frequency.

This view is useful when an apparently simple task fans out: it makes delegated researchers, repeated model calls, patch operations, file reads, and MCP dependencies visible without reading every message. Unexpected edges often identify a tool or subagent that policy should constrain more tightly.

Session messages: reconstruct the lifecycle

Galileo session Messages view with invoke-agent lifecycle spans and tool calls

The session view reconstructs the conversation from correlated spans. The outer Session groups all activity with the same conversation identity; invoke_agent rows mark lifecycle boundaries, while nested chat and tool rows carry the model input/output or tool arguments/results.

Lifecycle envelopes can legitimately show sub-millisecond duration: they record a hook transition, not the time the long-running agent remained alive. Completed chat and tool operations carry their measured duration and are sent as soon as the connector exposes completion, so operators do not need to wait for a final Stop event to see a multi-hour agent's progress.

Model detail: inspect input, output, latency, and reported usage

Galileo model span detail showing rendered input and assistant output inside a long-running agent session

Open a chat row to compare the exact rendered input and assistant output, provider/model identity, latency, token usage, and cost when the connector reports them. Tool rows expose their structured call arguments and results. DefenseClaw applies the Galileo route's v8 redaction profile before this data leaves the gateway. Fresh v8 destinations default to none (unredacted), so shared deployments should explicitly select sensitive, content, strict, or a reviewed custom profile.

Galileo and Agent360 are complementary rather than competing views:

UseGalileoLocal Agent360
Session reconstructionRich message-centric GenAI session view.Lifecycle/model/tool summaries correlated from Loki.
Agent relationshipsGalileo Agent graph and edge analytics.Cross-trace root/parent tree plus aggregate model/tool topology.
Individual requestTrace graph, latency, inputs, and outputs.Tempo waterfall, ordered sequence, phase graph, and raw event stream.
Fleet operationsGalileo projects, log streams, metrics, controls, and alerts.Local Prometheus/Loki/Tempo health, policy, reliability, and connector dashboards.
Non-GenAI gateway spansExcluded by the Galileo destination profile.Retained in the local OTLP destination.

The last row is why multi-destination routing matters: Galileo receives the GenAI operations it can index cleanly, while local observability keeps the full DefenseClaw operational and security trace set.

DefenseClaw and GenAI semantic conventions

DefenseClaw keeps its defenseclaw.* security attributes as the authoritative policy and forensic schema, then adds a standards-compatible projection on AI inference spans:

DefenseClaw dataGenAI projection
provider and request modelgen_ai.provider.name, gen_ai.request.model
response model and token countsgen_ai.response.model, gen_ai.usage.*
session IDgen_ai.conversation.id
prompt and responsegen_ai.input.messages, gen_ai.output.messages
agent identitygen_ai.agent.*
tool name and call IDgen_ai.tool.*

Prompt, response, document, tool-argument, and tool-result values follow the Galileo destination's v8 profile. Fresh v8 configuration exports registered content unchanged under none; redacting profiles either replace detected sensitive substrings or transform/remove whole field classes.

The Galileo preset binds the generated galileo-rich-v2 compatibility profile. Eligibility, projected attributes, events, links, instrumentation scope, and content states come from the canonical telemetry registry rather than a hand-authored span_filter. Its omitted capability-default route selects only available profile members, so non-eligible operational spans still flow to local observability and other matching OTLP destinations without being offered to Galileo. Explicit send and advanced routes are not compatibility filters: when they select a nonmember, the projector rejects that span as unsupported_shape and destination health exposes the error.

Hook connectors expose prompts, model completions, and tool calls as separate events. DefenseClaw gives each hook delivery a short canonical invoke_agent anchor and parents that delivery's completed chat, execute_tool, or lifecycle span to it. The pair is exported as an independently indexable trace; DefenseClaw does not append children for hours to a trace a backend may already have finalized. gen_ai.conversation.id and the stable agent, root-session, lifecycle, and execution attributes group these short traces into the same Galileo session and Agent360 identity. This keeps long-running agents real-time without disconnected operations or duplicated Bash → Bash rows. Connectors with post-model hooks export chat output at that hook; Codex and Claude Code use their turn-completion/Stop signal as the chat fallback. Completed tools export immediately during long-running turns. Correlation caches and content are bounded, duplicate completions are suppressed, and restart gaps use explicit unavailable-input placeholders.

The one authoring source is schemas/telemetry/v8/registry.yaml. It generates the v8 telemetry schema/catalog, builders, fixtures, and the logical galileo-rich-v2.json compatibility profile. Git stores the reproducible runtime JSON as deterministic gzip under schemas/telemetry/runtime/; release staging restores the exact raw JSON resource names. Runtime contract tests compare real exports to those generated artifacts, including name, kind, attributes, events, links, scope, topology, and dropped-structure accounting.

Read the routing and delivery funnel

defenseclaw.telemetry.destination.spans counts every destination-profile decision with destination, outcome, and reason attributes. For Galileo:

observed → eligible → attempted → delivered | rejected | failed

eligible means the operation-specific schema matched; it is not remote success. collector_accepted (and the compatibility delivered counter) is a batch-level span count inferred from OTLP responses and partial-success rejection counts; it is not per-span attribution. The runtime canary is stricter: its trace is isolated into its own export request and must appear in the destination's acknowledged trace-ID set. Indexing remains unverified until the trace appears in Galileo Logs. Protocol partial success increments rejected, while authentication, transport, TLS, timeout, and other exporter errors increment failed. /health, setup galileo status, and the TUI expose the same process-lifetime counters without headers or credentials. Local compatibility rejection such as unsupported_shape is also a reported destination failure; it is never counted as a successful delivery or hidden as a policy drop.

Upgrade from v7

Run the ordinary upgrade command:

defenseclaw upgrade --yes

The required migration converts legacy otel and audit_sinks settings into the v8 observability graph, preserves Galileo project/logstream headers, endpoint, explicit batching, routing, signal selection, and effective redaction posture, and keeps every other destination. The v8 Galileo preset uses a deliberate 1-second batch delay; an explicit v7 value is preserved, and the upgrade summary discloses when the old inherited 5-second default becomes the preset value.

No separate apply command is required. The v8 gateway is strict and does not rewrite v7 configuration on startup or operate both formats in parallel. A failed required conversion restores the backed-up source and does not start the v8 gateway against a v7 file.