Agent Substrate

defenseclaw substrate reference

Every defenseclaw substrate command, its exit codes and environment variables, the substrate configuration section, the chart values that matter, and the files the launcher keeps.

defenseclaw substrate runs on your laptop or in CI. It needs no config.yaml on the laptop: every command except the admin ones reads a small context file. defenseclaw substrate hands each command to defenseclaw-gateway substrate, so the same commands also run under that name.

Commands

CommandWhat it does
login, logout, whoamiSign in with your organization's OIDC provider (device flow; --no-browser prints the code), in CI with --workload and --workload-token-file, or with --kube on an open-source cluster that has no OIDC provider (your current kubeconfig; the gateway verifies it with a TokenReview). login saves a context: --admin-url, --issuer, --client-id, --scope, the CA files (--admin-ca-file, --issuer-ca-file), --default-profile, --default-template, --name. --kube-context and --kubeconfig choose the Kubernetes identity for --kube; --kube-namespace NAME is the namespace DefenseClaw runs in, required on the first --kube sign-in (without it, login prints the namespace whose endpoint record names --admin-url, with the command to run, and exits with code 2); --kube-service-account NAMESPACE/NAME signs you in as that ServiceAccount with a short-lived token the launcher gets for it (a client-certificate kubeconfig uses defenseclaw-launcher when none is named, and you need create on serviceaccounts/token); --workload-audience names the audience of a CI token. Tokens go in the operating system keychain
run <agent> [-- args]Start an agent in an actor and attach. --task "..." runs headless, --detach doesn't attach (with --task or --batch), --timeout stops waiting for a task, --batch FILE --parallel N runs one actor per task, --clone URL --ref REF starts from a repository instead of a copy of the folder, --unmask and --exclude change what is uploaded, --no-upload, --max-upload-mb (default 500), --secret NAME, --profile, --template, --name (a readable start; a random suffix is always added), --keep, --no-asks, --transport auto|websocket|longpoll, --workload-token-file, --yes
connect <name>Attach to an actor, resuming it first if it is suspended. It reconnects by itself if the connection drops. Keys typed while the connection is down are not typed into the agent; it says how many so you can type them again, and how many keys typed just before the actor suspended may not have reached it. --no-asks leaves asks for approvals; --transport picks the relay transport. Only the agent's own person can connect: an administrator gets not_your_agent for another person's agent
listYour actors with NAME, AGENT, PROFILE, STATE, AGE and BATCH. --all (admins) adds OWNER: the person, or not launcher (atespace only) for an agent made outside DefenseClaw; --json has attribution (person, workload or atespace) and foreign. --batch
status [name]Without a name, the cluster and account in use and how many agents you have. With a name, that agent. --batch shows a batch. status --cluster is the cluster and fleet document for admins (--json; it reads the cluster with kubectl, so give --install-file)
verifyAdmins: checks the cluster side. Reads the cluster with kubectl (--install-file) and the snapshot bucket with your aws or gcloud CLI (what verify needs); --json follows the management contract. --managed-atespaces A,B names the chart's managedAtespaces, so verify also checks DefenseClaw's editor grant there (doctor and status --cluster take it too). The issuer_unique check compares the actor-token issuer with the other DefenseClaw clusters of your kubeconfig (--fleet-contexts A,B to name them, none for this cluster only). The policy_conflict check fails while an agent is in policy_conflict, naming it and the policy reset, and --json (also status --cluster --json) carries policy_conflicts: the count and, per launcher and managed atespace (and any other with a conflict), conflicts. The count is read with your DefenseClaw sign-in (--context) and needs an administrator's: without one it is unknown (read false, agents null, a reason), never 0, and the check warns. The actor_keys check: the keys URL answers, Substrate signs actor tokens with substrate.identity.issuer, and the install file's actorJWT.issuer is the same value. If it fails, every actor token is refused. The sign_in_server check is the gateway's own check of the OIDC sign-in server (substrate.identity.oidc.issuer), from the gateway's network: the discovery address answers, TLS verifies against the gateway's trust bundle and the certificate names the issuer host, the discovery issuer is the configured one exactly, and the key set loads. It fails only when a failure repeats within one check (a retry 2 s later; the gateway's doctor runs the same retry, so it shows an outage within about 12 s and hides a one-off failure), with the cause (issuer_unreachable, issuer_tls_name, issuer_tls_untrusted, issuer_unavailable, issuer_http, issuer_mismatch, issuer_keys); the detail is in the gateway log and the gateway's substrate.oidc_issuer doctor row
logs <name> [-f]What the headless task printed, and its exit code; --task picks a task
pull <name>See the changed files, flagged for risk, then --diff prints the diff, --apply writes the changes into the folder, --branch NAME puts them on a new branch, --patch-out FILE writes a patch. Flagged files reach the folder only on a yes to the question or with --allow-flagged; --yes takes the safe answer (no). --timeout, --yes, --allow-flagged
checkpoint <name> [--every D]Save a restore point of the agent's folder, inside the agent. With --every it saves on that schedule for as long as the command runs
undo <name> [--to ID]Put the agent's folder back to its last restore point, or to a given one, and discard what the agent changed since. It asks first
delete <name>... | --batch IDDelete actors, their allow lists and their snapshots. Pull first
migrate <name>Move an actor with an old kit to the current kit
approvals, approve <id> [--always] [--reason], reject <id> [--reason]Answer asks, including for detached and unattended runs. approvals lists ID, AGENT, WHAT, SEVERITY, AGE and RUN; an ask an unattended run queued shows unattended and the time left in RUN
usage [--batch ID] [--all]Your own usage and cost. Not available yet: the cluster answers "does not offer that yet" (see Feature coverage)
secrets list [--mine], set NAME (--stdin | --from-env VAR), rm NAME, test NAMENames of your secrets and their bindings. Values are never shown. See Secrets
policy explain NAME, show, test NAME --host H [--method M] | --tool TYour own effective policy and where each value comes from; test asks whether a host, method or tool would be allowed for an agent
policy reset ATESPACE/NAME [--reason TEXT] [--yes]Administrators: clear a policy_conflict once the other writer is stopped. The controller writes the full policy again and the count of three reverts starts over. Audited under your own name
doctorChecks this computer and your sign-in. Administrators add --install-file FILE (the SubstrateInstall file given to ate-setup) for the cluster's own checks, the same ones verify runs, read with kubectl (--kube-context, --kubeconfig, --namespace, --substrate-namespace, --managed-atespaces); --cluster without it says what is missing. Ends with ready for agents, or not ready and exit 1
setupAdministrators: adds the callout and the router client to the SubstrateInstall file (--install-file, --namespace). The result is written as FILE.new; --yes replaces the file and keeps FILE.bak; --dry-run shows the changes; --remove takes them out; --print-access-policies --plan FILE prints the access policies; --capability body-mode|route-timeouts says what your Substrate build has; --issuer sets this cluster's actor-token issuer; --callout-port is the port of the callout Service (default 9443). With --issuer it also checks, through kubectl, that the keys URL answers (--keys-url, --kube-context, --kubeconfig) and warns when it doesn't
template build|verify|scan|publish|keygenAdministrators: build an agent image with the kit and a signed attestation, check or scan it, publish it to the cluster's registry, make the signing key. They run on this computer and need no sign-in. On a managed laptop's launcher they are not offered

Every read command has --output text|json and --json, and every command takes --context to pick a saved cluster (the default is the one you signed in to last). Errors say what failed and how to fix it. --yes answers questions.

What verify needs

verify (and status --cluster and doctor --install-file, which run the same checks) changes nothing, but a check that cannot read what it needs warns and says why. It uses your own tools and identity:

  • kubectl: create a token for the defenseclaw-gateway ServiceAccount (serviceaccounts/token in the DefenseClaw namespace), pods/portforward in the DefenseClaw and Substrate namespaces, and read clustertrustbundles. verify reads the access policies as the gateway's own ServiceAccount with a 10-minute token, and probes the actor Service through a port-forward without a token. It reads the gateway's sign-in server check from the admin Service's /api/v1/substrate/info through the API server's Service proxy (services/proxy in the DefenseClaw namespace), so it works while nobody can sign in. It also reads the defenseclaw-config ConfigMap of the other clusters in your kubeconfig for issuer_unique. For policy_conflict it lists every person's agents through the admin address with your DefenseClaw sign-in (defenseclaw substrate login as an administrator).
  • The snapshot bucket on S3: the aws CLI with s3:GetBucketLocation, s3:GetEncryptionConfiguration, s3:GetBucketPublicAccessBlock and s3:GetLifecycleConfiguration, and for an account-level public access block s3:GetAccountPublicAccessBlock with sts:GetCallerIdentity. On Cloud Storage: the gcloud CLI with storage.buckets.get.

There is no --api: the cluster checks come from kubectl with --install-file, and the bucket check is skipped when the file names no storage.snapshotLocation. Troubleshooting lists each warning.

Two warnings are in the text output of verify and status --cluster, and in doctor and doctor --json (rows cluster.atespace_globs and group_forms), but not in verify --json: an atespace that holds actors, that DefenseClaw did not create and that a match.substrate glob meets (the profile does not apply there; list the atespace by name), and a profile assignment whose groups are in another form than the sign-in server sends. The verify --json management contract is version 1, with exactly ten checks, and stays as it is in 1.0.10; the two warnings are planned for its version 2 (schema_version 2). doctor --json is not a management contract.

Template flags

CommandFlags that matter
template build--connector (claudecode, codex, opencode, copilot, hermes, kiro, antigravity, cursor, amp or devin; default claudecode), --from, --to, --hook-binary, --relay-binary, --key, --session-key (repeatable; the gateway's relay session public key), --session-issuer, --session-audience, --block-at critical|high|medium|low|none (default high), --accept-finding RULE-ID with --reason, --trusted-base sha256:..., --allow-unverified-base, --mcp-host, --task-stall (how long a --task run may make no progress before the relay ends it with exit code 124; 1m to 24h, default 10m), --provider (the model route: Claude Code bedrock or anthropic; Codex bedrock, which is GPT-5.6 Luna, us.openai.gpt-5.6-luna, on Bedrock's OpenAI-compatible API, or openai; the other agents have bedrock only. Without it: bedrock, unless substrate.llm.providers lists the vendor's API and not Bedrock), --model, --kit-version (default dev), --policy-generation N (recorded in the attestation, default 1), --builder-id ID, --mcp-analyzers LIST (mcp-scanner analyzers, default yara), --no-mcp-static (do not run mcp-scanner over tool descriptions), --gateway-url URL (the actor listener; default the substrate.services.actor name of --config), --verify-url URL (ask the gateway to check each session token instead of --session-key; an image that starts the relay needs one of the two), --keep-entrypoint (for tests: a real actor image starts the relay), --dry-run, --json
template verifyIMAGE --anchors FILE, --min-policy-generation, --max-age, --revoke KEYID, --allow-tag. Exit 0 admitted, 1 not admitted
template scanIMAGE, --connector, --block-at, --mcp-analyzers, --no-mcp-static. Exit 0 clean, 1 blocked
template publishSOURCE@DIGEST TARGET:TAG --anchors FILE, --template ATESPACE/NAME creates the ActorTemplate, with --class gvisor|microvm, --cpu, --memory, --env K=V, --worker-label K=V, --dry-run, and --api-* flags for the Substrate API. Exit 75 means the Substrate API can't be reached; run it again
template keygen--out DIR makes attestation.key (0600) and attestation.pub

All four registry commands take --registry-user, --registry-password-file, --registry-auth-host, --insecure-registry, --config and --platform. The login goes only to the host named by --registry-auth-host; for a --to on Docker Hub that host is auth.docker.io.

Exit codes

CodeMeaning
0Done
1The command failed
2Usage error
3Unsupported or refused (a profile or setting forbids it)
4Not signed in; run defenseclaw substrate login
75The cluster is unreachable (verify, status --cluster, template publish)

Environment variables

VariableMeaning
DEFENSECLAW_SUBSTRATE_CONTEXTWhich saved cluster to use
DEFENSECLAW_SUBSTRATE_ADMIN_URL, DEFENSECLAW_SUBSTRATE_ISSUER, DEFENSECLAW_SUBSTRATE_CLIENT_ID, DEFENSECLAW_SUBSTRATE_PROFILE, DEFENSECLAW_SUBSTRATE_TEMPLATEOverride the saved context, for CI
DEFENSECLAW_SUBSTRATE_KUBE, DEFENSECLAW_SUBSTRATE_WORKLOAD_TOKEN_FILE, DEFENSECLAW_SUBSTRATE_WORKLOAD_AUDIENCESign in with the Kubernetes identity, or as a CI workload, without flags
DEFENSECLAW_SUBSTRATE_TOKEN_STORE=fileKeep tokens in a 0600 file instead of the keychain (headless Linux and CI). The launcher never falls back to this on its own; it says so and names the variable

Files

run copies your folder and pull brings the agent's work back with git on your computer (Git for Windows on Windows): git's ignore rules decide what is copied, secret files stay behind, and the bytes are copied as they are, line endings included. --clone URL instead clones the repository inside the agent, through the cluster's network rules, which add the credential for the hosts your profile allows; nothing of your computer's credentials is sent. Put no user name or token in the address. To bring a clone's changes home, run pull in a checkout of the same repository: it reviews what the agent changed since the clone, and --apply writes it into that folder.

substrate/contexts.json under the DefenseClaw state directory holds your contexts: addresses, issuer, client ID and the CA files. It has no secrets and mode 0600. Tokens live in the keychain (macOS Keychain, Windows Credential Manager, Linux Secret Service). On macOS the key that seals them is an item only the DefenseClaw program may read without asking; macOS asks once after you install a new DefenseClaw build. Over SSH the keychain can't show that question, so the launcher stops with: "the macOS keychain is locked and this session cannot ask you to unlock it (for example over SSH); unlock it with: security unlock-keychain". Unlock it, or set DEFENSECLAW_SUBSTRATE_TOKEN_STORE=file.

Configuration

The substrate section of the DefenseClaw configuration. On a cluster it comes from the chart's config: values.

KeyDefaultMeaning
substrate.enabledfalse (the chart sets true)Turns the Substrate components on
substrate.api.address, .server_name, .ca_file, .authapi.ate-system.svc:443, none, in-cluster CA, serviceaccountHow DefenseClaw reaches the Substrate control API (.auth is serviceaccount or mtls)
substrate.supported_versions>=1.0.0 <1.1.0The Substrate versions verify accepts
substrate.identity.issuerhttps://idp.ate-system.svcIssuer of the actor tokens. Make it unique per cluster. The gateway compares iss with it as a string; it need not resolve
substrate.identity.keys_urlhttps://idp.<substrate namespace>.svc/openid/v1/jwksWhere the gateway reads the keys that sign actor tokens. The namespace comes from substrate.api.address. The token's iss is still compared with substrate.identity.issuer exactly
substrate.identity.keys_ca_file/run/servicedns.podcert.ate.dev/trust-bundle.pemCA bundle (PEM) for the keys URL's certificate. The default is the servicedns trust bundle the chart mounts, which signs the idp Service certificate. The gateway flag --issuer-ca is deprecated; use this key
substrate.identity.audiencedefenseclawAudience of actor tokens
substrate.identity.token_lifetime_seconds300Lifetime of the token the egress gateway adds. 300 s is the upstream minimum, and egress caches the token by actor name, so a recreated actor with the same name is refused until the old token expires. run never reuses a name for that reason
substrate.identity.oidc.issuer, .client_id, .groups_claimnone, none, groupsThe OIDC provider people sign in with. Unset on an open-source cluster to accept Kubernetes identities (login --kube) instead. Managed installs require it
substrate.identity.admin_groups[]Groups (the OIDC groups claim, or Kubernetes groups with the Kubernetes sign-in) whose members are Substrate administrators. The gateway's --admin-groups flag (chart gateway.adminGroups) adds to it
substrate.identity.workload_oidc[]Issuers accepted for CI workloads: issuer, audience, subject, claims
substrate.services.actor, .admindefenseclaw-actor.defenseclaw-system.svc.cluster.local, defenseclaw-admin.defenseclaw-system.svc.cluster.localService names. Use the full name: an actor resolves only the names in its allow list
substrate.launcher_atespaces{}Profile name to the atespaces the launcher may create actors in. Only DefenseClaw edits these
substrate.require_attested_templatestrueActors from unattested templates get no network
substrate.templates.trust_anchorsthe chart's anchors Secret mountPublic keys that sign attestations. Re-read when the Secret changes, so a revoked key applies at the next poll
substrate.templates.registry_auth.docker_confignoneA Docker config.json (the chart's controller.templateRegistrySecret mount). Each auths entry is sent only to its own registry host, after that host asks. Re-read when it changes; a missing file is anonymous, a broken one refuses
substrate.templates.registry_auth.ecr_hosts[]Amazon ECR registry hosts (<account>.dkr.ecr.<region>.amazonaws.com) logged in to with ecr:GetAuthorizationToken through the AWS SDK default chain (IRSA or EKS Pod Identity). Other hosts are refused when the configuration loads
substrate.templates.allowed_images, .profile_images, .volume_images[]Repository globs (registry/team/*, no tag) an attested image must match, narrowed per profile; and images a template may mount read-only without an attestation
substrate.templates.min_policy_generation, .max_age, .revoked_digests, .allow_unverified_base, .plain_http_registries0, none, [], false, []Refuse older scans, old builds, listed digests, untrusted bases; registries reached over plain HTTP (a dev registry only)
substrate.default_sandbox_classgvisorgvisor or microvm
substrate.chain_egress_actionblockOn Agent Substrate a completed chain refuses the egress by default, in every mode (open source, strict and managed), because actors are unattended: the actor's tool call that uploads is refused by its hook, and its request that leaves the cluster is refused by the egress callout (category chain_egress). That includes a chain whose every completing read names no one file (a read of several files, a read in a call that also makes a write with its own commands or redirects, a relative path in a call that is not proven to keep its directory): the agent chooses how it reads, so that form refuses the same way. The callout sees no file: after a credentials read it refuses every request of that actor that leaves the cluster, benign or not, until the chain's window ends (8 steps and 30 minutes of the actor's running time: wall time while the actor runs, whether or not it makes requests; a suspend, however long, doesn't count, nor does a time the leader can't see it running, such as a controller outage) or the opt-in is set; an upload of a different, known file is allowed only at the hook. The one setting is the config key substrate.chain_egress_action: block (the default) or alert (an opt-in under which every chain only alerts), set with the chart value config.substrate.chain_egress_action, which the gateway and the leader both read. A request counts as a Substrate actor's only with a verified actor token (the actor listener or the callout route). Hooks on a person's own computer or in an OpenShell sandbox keep the rule pack's disposition: an alert in the default pack, also for a chain whose read names no one file. The leader logs the action it uses at startup. Which reads start the chain: the secret-read chain
substrate.idle_suspend_after30mIdle time before DefenseClaw suspends an actor; 0s turns it off. A running --task run is exempt unless its profile sets idle_suspend_tasks: true; a finished one is suspended about a minute after its task ends, which frees its worker
substrate.idle_suspend_without_cpufalseWithout a CPU feed (the controller's --usage-listen), an actor with a terminal attached is never idle. true makes 30 minutes with no hook, model or terminal traffic idle for it too
substrate.workspace_max_bytes1 GiBLargest folder upload the gateway takes
substrate.unattended.asksdenyWhat an unattended run does with an ask: deny or queue
substrate.unattended.ask_timeout30mHow long a queued ask waits for a person
substrate.unattended.telemetry_sampling1.0Fraction of routine events kept for --task runs. Blocks and lifecycle records are never sampled
substrate.rate_limits.per_actor_rps, .per_atespace_rps20, 500Fair-share limits of each actor and each atespace on the actor Service, whatever the number of gateway replicas: every second the replicas count their requests through the leader, and each holds an actor to its share of the budget, in proportion to the requests it gets. Over budget answers 429 with Retry-After. In the first second of a flood, and on a replica that has not reached the leader for 3 seconds, each replica applies the whole budget. One actor also has at most 16 requests open at a time on each replica. Requests to hosts in an actor's allow list (the egress callout) are not counted
substrate.extproc.enabledtrueThe egress callout
substrate.extproc.failure_modeclosedWhat the egress gateway does when the callout fails. Managed installs are always closed
substrate.extproc.body_inspectionautoRead bodies of POSTs to MCP and listed hosts when Substrate supports it. on says it does; off checks headers only. In managed enterprise, auto refuses requests to listed MCP servers until the callout has received a body
substrate.extproc.body_max_bytes1048576Largest body the callout reads
substrate.extproc.mcp_hosts, .body_hosts[]Hosts whose bodies are read as MCP calls (host or host=name), and other hosts whose bodies get the secret scans
substrate.extproc.request_accessnoneThe contact a refusal names
substrate.extproc.chain.check_budget, .send_budget, .failure_mode20ms (at most 500ms), 250ms (at most 500ms, never shorter than check_budget), autoHow long the tool-chain check may take per request, and per request whose miss would refuse it (one that can carry data out, such as a git push or fetch; any request of an unattended actor or on a managed or closed install; one that reuses a cached registration), and what a request does when it can't be checked: auto refuses for unattended actors and on a managed install and alerts for the rest, closed always refuses, open always alerts. The alert covers GET and HEAD only: a request with a body or any other method is refused in every mode while its check is missing
substrate.extproc.ask.hold1.5sHow long a request waits for an ask. Keep it under the egress filter's 2 s message timeout. The ask comes after the tool-chain check, within the callout's 1.8 s: with a slow leader, ask.hold plus chain.send_budget over 1.8 s shortens the hold
substrate.extproc.dlp.credential_headersalertalert or deny for a secret-shaped credential header on a host no secret is bound to
substrate.extproc.actor_secrets_dirset by the chart from callout.actorSecrets.secretNameWhere the callout reads the values of profiles' actor_secrets
substrate.llm.providersall the gateway proxy servesProviders actors may reach, only through the proxy: anthropic, openai, bedrock, gemini
substrate.llm.bedrock_models[]Bedrock model IDs actors may call. A call that names another model goes to the first entry instead, through the same guardrails. That holds for /model/<id> and for the model named in the body of the OpenAI-compatible calls (/llm/bedrock/openai/v1/responses and /chat/completions). A body that can't be read as a JSON object with a model, or that is compressed or too large, is refused
substrate.llm.credential.provider_urinoneWhere the model key comes from (ate-secret://...)
substrate.egress.allow, .ecosystem_packs, .registries[], every pack, []The base egress every actor gets before its profile tightens it
substrate.secrets[]See Secrets
substrate.allow_passthrough_for[]Hosts that are not inspected. Never given a secret
substrate.adminnoneThe cluster floor, the same keys as openshell.admin plus the Substrate keys below. Profiles tighten it

Profile keys for Substrate: profile_assignments[].match.substrate (atespaces, templates, workloads). To assign a profile to a Kubernetes ServiceAccount (open source, login --kube), use match.users: [system:serviceaccount:NAMESPACE:NAME] for one ServiceAccount, or match.groups: [system:serviceaccounts:NAMESPACE] for all of a namespace's. The sandbox section of a profile is the same as for OpenShell (required_for, allowed_harnesses, egress_allow_only, allow_unblock, user_approvals and the rest). See Managed enterprise.

Keys of the sandbox section (and of substrate.admin) for the runtime:

KeyValuesMeaningHow a profile tightens substrate.admin
runtimes[openshell, substrate]Where the profile's agents may run. Left out: both. Without substrate, its actors get no policyIntersection
sandbox_classgvisor, microvmThe weakest isolation an actor may run in. Unset: substrate.default_sandbox_classmicrovm wins
idle_suspend_afterGo duration, 1m or moreReplaces substrate.idle_suspend_afterThe minimum
idle_suspend_taskstrue or falseA running --task or --batch run of the profile may be idle-suspended (by default it is exempt). Without the controller's CPU feed (--usage-listen), idle follows substrate.idle_suspend_without_cputrue wins
unattended_asksdeny, queueWhat an unattended run's ask does. Unset: substrate.unattended.asksdeny wins
ask_timeoutGo duration, 1m to 24hHow long a queued ask waits before it is denied. Unset: substrate.unattended.ask_timeoutThe minimum
ask_hostshost names or *.<host>Allowed hosts an agent reaches only after a person approves (the egress callout asks; substrate approvals answers). They never open a host the agent's allow list doesn't haveAdded together
url_rules{name, hosts, methods, path_prefix, action}Requests to a host the callout refuses (deny) or asks about (ask), by method and path prefix. action is deny or ask; a rule with action: allow is refused when the configuration loads. Every matching rule counts, whatever its order or which side (substrate.admin or a profile) wrote it, and the strictest wins: deny, then ask. An approval never lifts a deny. An approval covers the rule and the methods it was asked for (the ask shows METHOD host /prefix*); an ask_hosts approval covers the host. Rule names must be unique (an approval is keyed by the name; a duplicate is refused when the configuration loads). See the next row for methods and pathsAdded together
url_rules methods and pathshow a request is matchedA request counts as every method a server framework could run it as: its own, each value of X-HTTP-Method-Override, X-HTTP-Method and X-Method-Override, _method or _HttpMethod in the query, _method in the body of a POST read as a form (no Content-Type, urlencoded or multipart, gzip and declared charsets decoded), and the top-level _method or _HttpMethod of a JSON POST body (Laravel, Express method-override) where a deny or ask rule on that host and path names a method other than POST. Each rule is checked for every one of those methods, and an ask names them all; its approval covers only those. The path is matched after decoding, with . segments and repeated slashes removed, without case for deny and ask. A .. segment or an encoded /, \, ; or % is refused where a path rule applies. When such a POST's body can't be read (multipart, over body_max_bytes, body inspection off, or auto before Substrate has sent a body), failure_mode decides: closed refuses, open allows with an alert. On a headers-only cluster in closed mode, JSON POSTs to such paths are refused; use body_inspection: on. Other JSON and typed POSTs are not held-
actor_secretskey namesValues a program inside the actor may read, keys of the Secret in callout.actorSecrets.secretName. Matched by exact value on prompts and egressIntersection

openshell.admin refuses sandbox_class, idle_suspend_after, idle_suspend_tasks, unattended_asks, ask_timeout, actor_secrets, ask_hosts and url_rules. substrate.admin refuses allow_mount, allow_host_ports, process_tree, driver_fallback, require_copy_for and locked workdir.* keys.

When a config change takes effect

Change the config with helm upgrade or GitOps. helm upgrade also restarts every DefenseClaw pod (each pod carries a checksum of the config), so every setting below applies within one rollout.

A direct edit of the defenseclaw-config ConfigMap, for example with kubectl edit, needs no restart for most settings. The controller, the gateway and the leader read the ConfigMap through the Kubernetes API every 2 seconds. An edit that does not load is refused with an alert (sandbox.config.refused), and the config in force stays. That includes a key the config schema does not declare, such as a misspelled egres:, which a pod start would refuse too. So does a failed read of the ConfigMap: once the API has answered, a component never falls back to its mounted file, which on a managed install keeps the config it started with.

SettingWhen running and new agents get it
substrate.egress (allow list, registries, ecosystem packs), substrate.allow_passthrough_forWithin about 5 seconds. The controller renders each agent's egress policy again on its next pass
substrate.admin and guardrail.profiles.<name>.sandbox (egress block and allow-only lists, allowed, runtimes, sandbox class, idle suspend)Within about 5 seconds
guardrail.profile_assignments, substrate.launcher_atespacesWithin about 5 seconds. An agent that is no longer managed loses DefenseClaw's policy. New agents get the profile from the new list
substrate.secretsThe egress bindings within about 5 seconds. The gateway's injection list at its next restart
substrate.templates, substrate.require_attested_templatesTemplates are attested again on the next pass
substrate.chain_egress_actionWithin about 2 seconds, on the gateway's hook answers and the leader's callout verdicts
substrate.unattended.asks (and the profile unattended_asks)On the gateway's next ask. The callout's copy only through helm upgrade
guardrail mode and rules (hooks and model calls)When the gateway reloads its mounted config file. The kubelet updates the file about a minute after the edit. Managed installs mount the file in a way the kubelet never updates, so they apply it at the next gateway restart
substrate.identity, substrate.services, substrate.api, substrate.bundles, substrate.rate_limits, substrate.llm, llm_providers, substrate.workspace_max_bytes, substrate.unattended.ask_timeoutAt the next restart of the pods. The controller keeps the identity, services, API and llm_providers hosts it started with too, so it stays in step with the gateway. Each component logs the keys that wait for a restart
deployment_mode and the enterprise profileNever live. An edit that changes them, for example one that drops managed_enterprise, is refused whole with a sandbox.config.refused alert that names deployment_mode_change (or managed_downgrade); none of the edit applies, the guardrail part included. It applies only when the pods restart with it, through helm upgrade
substrate.extproc, and the callout's copy of url_rules, ask_hosts, actor_secrets and unattended asksOnly through helm upgrade, which renders the callout's own ConfigMap. The controller keeps the mcp_hosts it started with, so an edit opens no egress the callout does not inspect yet

After a direct edit of a setting that waits for a restart, run kubectl -n defenseclaw-system rollout restart deploy/defenseclaw-gateway (or deploy/defenseclaw-gateway-leader). With substrate.bundles on, the stage bundles are layered over the profiles that config.yaml had when the controller started, so a profile change in config.yaml applies when the controller restarts.

Chart values

The signed chart is oci://ghcr.io/cisco-ai-defense/charts/defenseclaw-substrate. Every value is checked by its schema. The ones you will set:

ValueMeaning
configThe DefenseClaw configuration, rendered into one ConfigMap
deploymentModemanaged_enterprise sets config.deployment_mode, mounts config.yaml as one read-only file and gives each pod a private data volume (managed.serviceAccountName, managed.dataSizeLimit)
bootstrap.ownerTokenSecretA Substrate global owner's token for the one-time access-policy bootstrap
managedAtespacesOther atespaces DefenseClaw writes policy in
substrate.namespace, substrate.apiAddressWhere Substrate runs
gateway.hpa.*, gateway.pdb.*Scaling and disruption budget of the gateway. The gateway and callout autoscalers scale on CPU, so they need the Kubernetes metrics API (for example metrics-server): without it kubectl get hpa shows cpu: <unknown> and they stay at their minimum replicas, and a busy atespace slows every actor's hooks instead of adding replicas
gateway.relaySessionSecretThe Secret holding the relay session key pair (private.pem signs attach tokens, public.pem goes into agent images). The relay-session hook creates it when absent, never replaces it, and uninstall keeps it. Default defenseclaw-relay-session
gateway.llmCredentialsSecretThe Secret with one model key per provider (anthropic, openai, bedrock, gemini). Default defenseclaw-llm
gateway.authDelegatorBindingName of the ClusterRoleBinding of the gateway to system:auth-delegator, rendered only for the Kubernetes sign-in (no OIDC issuer). Default defenseclaw-auth-delegator-<namespace>-<release>
gateway.adminGroupsGroups whose members are DefenseClaw administrators (--admin-groups). Adds to config.substrate.identity.admin_groups
gateway.oidcCA.configMap or .secret, gateway.oidcCA.keyThe trust bundle of a private OIDC issuer, from a ConfigMap or a Secret (the Secret wins; key default ca.crt). Mounted read-only at /etc/defenseclaw/oidc-ca/, and --oidc-ca-file is set
gateway.extraArgsFlags added to the gateway's serve flags, for example --chain-retention=48h. The chain egress action is not a flag: set config.substrate.chain_egress_action
gateway.extraEnv, gateway.extraVolumes, gateway.extraVolumeMountsAdded to the gateway container and pod
gateway.argsReplaces the whole argument list (serve --substrate ... included). Experts only; use gateway.extraArgs to add a flag
controller.templateRegistrySecretA kubernetes.io/dockerconfigjson Secret with logins for private template registries, mounted in the controller and the gateway. Empty reads registries anonymously
serviceAccounts.annotations, serviceAccounts.componentAnnotations.<gateway|controller|callout>Annotations on every ServiceAccount, and on one component's (merged over the first), for example an IRSA role on gateway and controller only
leader.store.size, leader.store.keepOnUninstallThe leader's volume
controller.templateAnchorsSecretThe admin-only Secret with the template trust anchors. Default defenseclaw-template-anchors
callout.judge.tokenSecretThe callout's token for the MCP tool-rules route. Default defenseclaw-callout-judge, which the chart creates. "" turns the route off: MCP tool calls then raise an alert, and a managed install refuses them
callout.actorSecrets.secretNameA Secret whose keys are the values profiles' actor_secrets expose. Mounted in the callout and the gateway only
admin.expose.type, admin.expose.ingress.*, admin.url, admin.tlsSecretNameHow laptops reach the admin Service, and the https address published for login --kube
credentialProvider.secretsNamespaceThe namespace that holds the per-actor secret copies. Default defenseclaw-actor-secrets. Two releases in one cluster, for example a managed one next to an open-source one, each need their own. Only one of them can register the egress callout, because Substrate has one callout slot
substrate.actorKeys.urlSets substrate.identity.keys_url. Empty is the idp Service
substrate.actorKeys.caConfigMapA ConfigMap (key ca.crt) with the CA of the keys URL, mounted on the gateway as keys_ca_file. Empty is the servicedns trust bundle
networkPolicy.egress.extraExtra NetworkPolicy egress rules for every pod with an egress limit (gateway, leader, callout, controller). The default allows DNS, the Kubernetes and Substrate APIs, the release's own Services, OTLP and model providers on 443 only. Use it for a private OIDC issuer, a private registry or EKS Pod Identity. See the EKS page
networkPolicy.adminFromWho can reach the admin Service
clusterNameThe cluster name on the gateway's telemetry (the k8s.cluster.name resource attribute; k8s.namespace.name comes from the pod). Empty leaves it out
metrics.prometheusRule.enabledInstall the alert rules
metrics.prometheusRule.namespaceLabelThe Prometheus label that carries the k8s.namespace.name resource attribute. Empty (the default): the alerts on the defenseclaw.substrate.* metrics read every release's series, which works with any OTLP receiver. Set it to k8s_namespace_name so they select this release's namespace and releases that share a Prometheus don't alert on each other, after you promote the attribute where your metrics enter Prometheus (otlp.promote_resource_attributes, or the collector's resource_to_telemetry_conversion). Without the promotion the label is absent and those alerts never fire
homeSizeLimitThe disk volume each pod keeps its local audit database in (4Gi; managed.dataSizeLimit in managed mode; homeMedium: Memory makes it a memory volume). It must hold config.observability.local.max_store_mb twice over plus 256 MiB. When the store outgrows a disk volume, the kubelet evicts the pod and the store is lost; only a memory volume or a full node disk gives ALERT audit_store_full in the gateway log, with hooks that need an audit record failing closed. It is lost when the pod is replaced. See Turn on OTLP export
observability.local.enabledA local collector for trying the chart, and a destination chart-collector that exports to it. See Turn on OTLP export

Turn on OTLP export

On Agent Substrate, export is the audit record. Each gateway pod keeps a local audit database in a pod volume (an emptyDir on disk, or in memory with homeMedium: Memory). It is lost when the pod is replaced, for example by a rollout, an eviction or a node move, and each replica holds only the requests it handled. The chart has no persistent volume for it: a volume per replica would still split the history across replicas. Send the records to a collector or a SIEM that keeps them. A gateway started without an export destination logs no export destination once.

Export destinations go under config.observability.destinations in the chart values. The chart renders config into the defenseclaw-config ConfigMap, and the gateway reads it from there. There is no config.otel key.

config:
  observability:
    destinations:
      - name: my-collector
        kind: otlp
        protocol: http/protobuf
        endpoint: http://opentelemetry-collector.otel-system.svc:4318
        tls:
          insecure: true
        network_safety:
          allow_private_networks: true
  • TLS. An http:// endpoint needs tls.insecure: true. An https:// endpoint must not set it. config validate refuses a mismatch.
  • Private addresses. A Kubernetes Service name resolves to a cluster address, and on most clusters that address is private. The gateway sends to a loopback or private address only when the destination sets network_safety.allow_private_networks: true. For an address in 100.64.0.0/10, set network_safety.allow_cgnat: true. Without the option, the gateway still starts. It runs without that destination and logs one line that names the destination and the option.
  • Protocol. protocol defaults to grpc, the collector's port 4317. For port 4318, set protocol: http/protobuf.
  • Network policy. The chart's egress policy allows OTLP on ports 4317 and 4318 (networkPolicy.egress.otlpPorts). Add other ports there.

defenseclaw-gateway config validate warns about the last two mistakes. private_service_endpoint is a Service name without allow_private_networks. otlp_protocol_port is grpc on port 4318.

To try it out, set observability.local.enabled: true. The chart then installs a collector that prints what it receives, and adds a destination chart-collector that points at it with the settings above. If you name your own destination chart-collector, the chart keeps yours.

Check that it works:

  1. defenseclaw substrate status --cluster --install-file install.yaml shows every gateway replica ready.
  2. If a gateway pod restarts instead, kubectl -n <namespace> logs deploy/defenseclaw-gateway --previous names the destination and the cause, for example observability v8 bootstrap failed: runtime_unavailable: destination "my-collector": ....
  3. A line observability destination "my-collector": endpoint host ... in the gateway log means the destination is not sending. The line says why: the option to set, or a host that does not resolve.
  4. With the local collector, kubectl -n <namespace> logs deploy/defenseclaw-otel-collector shows the records arriving.

Telemetry

The new attributes, families and metrics are in Sandbox telemetry. Metrics carry no actor, atespace or person labels.