defenseclaw substrate reference
Every defenseclaw substrate command, its exit codes and environment variables, the substrate configuration section, the chart values that matter, and the files the launcher keeps.
defenseclaw substrate runs on your laptop or in CI. It needs no config.yaml on the laptop: every command except the admin ones reads a small context file. defenseclaw substrate hands each command to defenseclaw-gateway substrate, so the same commands also run under that name.
Commands
| Command | What it does |
|---|---|
login, logout, whoami | Sign in with your organization's OIDC provider (device flow; --no-browser prints the code), in CI with --workload and --workload-token-file, or with --kube on an open-source cluster that has no OIDC provider (your current kubeconfig; the gateway verifies it with a TokenReview). login saves a context: --admin-url, --issuer, --client-id, --scope, the CA files (--admin-ca-file, --issuer-ca-file), --default-profile, --default-template, --name. --kube-context and --kubeconfig choose the Kubernetes identity for --kube; --kube-namespace NAME is the namespace DefenseClaw runs in, required on the first --kube sign-in (without it, login prints the namespace whose endpoint record names --admin-url, with the command to run, and exits with code 2); --kube-service-account NAMESPACE/NAME signs you in as that ServiceAccount with a short-lived token the launcher gets for it (a client-certificate kubeconfig uses defenseclaw-launcher when none is named, and you need create on serviceaccounts/token); --workload-audience names the audience of a CI token. Tokens go in the operating system keychain |
run <agent> [-- args] | Start an agent in an actor and attach. --task "..." runs headless, --detach doesn't attach (with --task or --batch), --timeout stops waiting for a task, --batch FILE --parallel N runs one actor per task, --clone URL --ref REF starts from a repository instead of a copy of the folder, --unmask and --exclude change what is uploaded, --no-upload, --max-upload-mb (default 500), --secret NAME, --profile, --template, --name (a readable start; a random suffix is always added), --keep, --no-asks, --transport auto|websocket|longpoll, --workload-token-file, --yes |
connect <name> | Attach to an actor, resuming it first if it is suspended. It reconnects by itself if the connection drops. Keys typed while the connection is down are not typed into the agent; it says how many so you can type them again, and how many keys typed just before the actor suspended may not have reached it. --no-asks leaves asks for approvals; --transport picks the relay transport. Only the agent's own person can connect: an administrator gets not_your_agent for another person's agent |
list | Your actors with NAME, AGENT, PROFILE, STATE, AGE and BATCH. --all (admins) adds OWNER: the person, or not launcher (atespace only) for an agent made outside DefenseClaw; --json has attribution (person, workload or atespace) and foreign. --batch |
status [name] | Without a name, the cluster and account in use and how many agents you have. With a name, that agent. --batch shows a batch. status --cluster is the cluster and fleet document for admins (--json; it reads the cluster with kubectl, so give --install-file) |
verify | Admins: checks the cluster side. Reads the cluster with kubectl (--install-file) and the snapshot bucket with your aws or gcloud CLI (what verify needs); --json follows the management contract. --managed-atespaces A,B names the chart's managedAtespaces, so verify also checks DefenseClaw's editor grant there (doctor and status --cluster take it too). The issuer_unique check compares the actor-token issuer with the other DefenseClaw clusters of your kubeconfig (--fleet-contexts A,B to name them, none for this cluster only). The policy_conflict check fails while an agent is in policy_conflict, naming it and the policy reset, and --json (also status --cluster --json) carries policy_conflicts: the count and, per launcher and managed atespace (and any other with a conflict), conflicts. The count is read with your DefenseClaw sign-in (--context) and needs an administrator's: without one it is unknown (read false, agents null, a reason), never 0, and the check warns. The actor_keys check: the keys URL answers, Substrate signs actor tokens with substrate.identity.issuer, and the install file's actorJWT.issuer is the same value. If it fails, every actor token is refused. The sign_in_server check is the gateway's own check of the OIDC sign-in server (substrate.identity.oidc.issuer), from the gateway's network: the discovery address answers, TLS verifies against the gateway's trust bundle and the certificate names the issuer host, the discovery issuer is the configured one exactly, and the key set loads. It fails only when a failure repeats within one check (a retry 2 s later; the gateway's doctor runs the same retry, so it shows an outage within about 12 s and hides a one-off failure), with the cause (issuer_unreachable, issuer_tls_name, issuer_tls_untrusted, issuer_unavailable, issuer_http, issuer_mismatch, issuer_keys); the detail is in the gateway log and the gateway's substrate.oidc_issuer doctor row |
logs <name> [-f] | What the headless task printed, and its exit code; --task picks a task |
pull <name> | See the changed files, flagged for risk, then --diff prints the diff, --apply writes the changes into the folder, --branch NAME puts them on a new branch, --patch-out FILE writes a patch. Flagged files reach the folder only on a yes to the question or with --allow-flagged; --yes takes the safe answer (no). --timeout, --yes, --allow-flagged |
checkpoint <name> [--every D] | Save a restore point of the agent's folder, inside the agent. With --every it saves on that schedule for as long as the command runs |
undo <name> [--to ID] | Put the agent's folder back to its last restore point, or to a given one, and discard what the agent changed since. It asks first |
delete <name>... | --batch ID | Delete actors, their allow lists and their snapshots. Pull first |
migrate <name> | Move an actor with an old kit to the current kit |
approvals, approve <id> [--always] [--reason], reject <id> [--reason] | Answer asks, including for detached and unattended runs. approvals lists ID, AGENT, WHAT, SEVERITY, AGE and RUN; an ask an unattended run queued shows unattended and the time left in RUN |
usage [--batch ID] [--all] | Your own usage and cost. Not available yet: the cluster answers "does not offer that yet" (see Feature coverage) |
secrets list [--mine], set NAME (--stdin | --from-env VAR), rm NAME, test NAME | Names of your secrets and their bindings. Values are never shown. See Secrets |
policy explain NAME, show, test NAME --host H [--method M] | --tool T | Your own effective policy and where each value comes from; test asks whether a host, method or tool would be allowed for an agent |
policy reset ATESPACE/NAME [--reason TEXT] [--yes] | Administrators: clear a policy_conflict once the other writer is stopped. The controller writes the full policy again and the count of three reverts starts over. Audited under your own name |
doctor | Checks this computer and your sign-in. Administrators add --install-file FILE (the SubstrateInstall file given to ate-setup) for the cluster's own checks, the same ones verify runs, read with kubectl (--kube-context, --kubeconfig, --namespace, --substrate-namespace, --managed-atespaces); --cluster without it says what is missing. Ends with ready for agents, or not ready and exit 1 |
setup | Administrators: adds the callout and the router client to the SubstrateInstall file (--install-file, --namespace). The result is written as FILE.new; --yes replaces the file and keeps FILE.bak; --dry-run shows the changes; --remove takes them out; --print-access-policies --plan FILE prints the access policies; --capability body-mode|route-timeouts says what your Substrate build has; --issuer sets this cluster's actor-token issuer; --callout-port is the port of the callout Service (default 9443). With --issuer it also checks, through kubectl, that the keys URL answers (--keys-url, --kube-context, --kubeconfig) and warns when it doesn't |
template build|verify|scan|publish|keygen | Administrators: build an agent image with the kit and a signed attestation, check or scan it, publish it to the cluster's registry, make the signing key. They run on this computer and need no sign-in. On a managed laptop's launcher they are not offered |
Every read command has --output text|json and --json, and every command takes --context to pick a saved cluster (the default is the one you signed in to last). Errors say what failed and how to fix it. --yes answers questions.
What verify needs
verify (and status --cluster and doctor --install-file, which run the same checks) changes nothing, but a check that cannot read what it needs warns and says why. It uses your own tools and identity:
kubectl: create a token for thedefenseclaw-gatewayServiceAccount (serviceaccounts/tokenin the DefenseClaw namespace),pods/portforwardin the DefenseClaw and Substrate namespaces, and readclustertrustbundles.verifyreads the access policies as the gateway's own ServiceAccount with a 10-minute token, and probes the actor Service through a port-forward without a token. It reads the gateway's sign-in server check from the admin Service's/api/v1/substrate/infothrough the API server's Service proxy (services/proxyin the DefenseClaw namespace), so it works while nobody can sign in. It also reads thedefenseclaw-configConfigMap of the other clusters in your kubeconfig forissuer_unique. Forpolicy_conflictit lists every person's agents through the admin address with your DefenseClaw sign-in (defenseclaw substrate loginas an administrator).- The snapshot bucket on S3: the
awsCLI withs3:GetBucketLocation,s3:GetEncryptionConfiguration,s3:GetBucketPublicAccessBlockands3:GetLifecycleConfiguration, and for an account-level public access blocks3:GetAccountPublicAccessBlockwithsts:GetCallerIdentity. On Cloud Storage: thegcloudCLI withstorage.buckets.get.
There is no --api: the cluster checks come from kubectl with --install-file, and the bucket check is skipped when the file names no storage.snapshotLocation. Troubleshooting lists each warning.
Two warnings are in the text output of verify and status --cluster, and in doctor and doctor --json (rows cluster.atespace_globs and group_forms), but not in verify --json: an atespace that holds actors, that DefenseClaw did not create and that a match.substrate glob meets (the profile does not apply there; list the atespace by name), and a profile assignment whose groups are in another form than the sign-in server sends. The verify --json management contract is version 1, with exactly ten checks, and stays as it is in 1.0.10; the two warnings are planned for its version 2 (schema_version 2). doctor --json is not a management contract.
Template flags
| Command | Flags that matter |
|---|---|
template build | --connector (claudecode, codex, opencode, copilot, hermes, kiro, antigravity, cursor, amp or devin; default claudecode), --from, --to, --hook-binary, --relay-binary, --key, --session-key (repeatable; the gateway's relay session public key), --session-issuer, --session-audience, --block-at critical|high|medium|low|none (default high), --accept-finding RULE-ID with --reason, --trusted-base sha256:..., --allow-unverified-base, --mcp-host, --task-stall (how long a --task run may make no progress before the relay ends it with exit code 124; 1m to 24h, default 10m), --provider (the model route: Claude Code bedrock or anthropic; Codex bedrock, which is GPT-5.6 Luna, us.openai.gpt-5.6-luna, on Bedrock's OpenAI-compatible API, or openai; the other agents have bedrock only. Without it: bedrock, unless substrate.llm.providers lists the vendor's API and not Bedrock), --model, --kit-version (default dev), --policy-generation N (recorded in the attestation, default 1), --builder-id ID, --mcp-analyzers LIST (mcp-scanner analyzers, default yara), --no-mcp-static (do not run mcp-scanner over tool descriptions), --gateway-url URL (the actor listener; default the substrate.services.actor name of --config), --verify-url URL (ask the gateway to check each session token instead of --session-key; an image that starts the relay needs one of the two), --keep-entrypoint (for tests: a real actor image starts the relay), --dry-run, --json |
template verify | IMAGE --anchors FILE, --min-policy-generation, --max-age, --revoke KEYID, --allow-tag. Exit 0 admitted, 1 not admitted |
template scan | IMAGE, --connector, --block-at, --mcp-analyzers, --no-mcp-static. Exit 0 clean, 1 blocked |
template publish | SOURCE@DIGEST TARGET:TAG --anchors FILE, --template ATESPACE/NAME creates the ActorTemplate, with --class gvisor|microvm, --cpu, --memory, --env K=V, --worker-label K=V, --dry-run, and --api-* flags for the Substrate API. Exit 75 means the Substrate API can't be reached; run it again |
template keygen | --out DIR makes attestation.key (0600) and attestation.pub |
All four registry commands take --registry-user, --registry-password-file, --registry-auth-host, --insecure-registry, --config and --platform. The login goes only to the host named by --registry-auth-host; for a --to on Docker Hub that host is auth.docker.io.
Exit codes
| Code | Meaning |
|---|---|
| 0 | Done |
| 1 | The command failed |
| 2 | Usage error |
| 3 | Unsupported or refused (a profile or setting forbids it) |
| 4 | Not signed in; run defenseclaw substrate login |
| 75 | The cluster is unreachable (verify, status --cluster, template publish) |
Environment variables
| Variable | Meaning |
|---|---|
DEFENSECLAW_SUBSTRATE_CONTEXT | Which saved cluster to use |
DEFENSECLAW_SUBSTRATE_ADMIN_URL, DEFENSECLAW_SUBSTRATE_ISSUER, DEFENSECLAW_SUBSTRATE_CLIENT_ID, DEFENSECLAW_SUBSTRATE_PROFILE, DEFENSECLAW_SUBSTRATE_TEMPLATE | Override the saved context, for CI |
DEFENSECLAW_SUBSTRATE_KUBE, DEFENSECLAW_SUBSTRATE_WORKLOAD_TOKEN_FILE, DEFENSECLAW_SUBSTRATE_WORKLOAD_AUDIENCE | Sign in with the Kubernetes identity, or as a CI workload, without flags |
DEFENSECLAW_SUBSTRATE_TOKEN_STORE=file | Keep tokens in a 0600 file instead of the keychain (headless Linux and CI). The launcher never falls back to this on its own; it says so and names the variable |
Files
run copies your folder and pull brings the agent's work back with git on your computer (Git for Windows on Windows): git's ignore rules decide what is copied, secret files stay behind, and the bytes are copied as they are, line endings included. --clone URL instead clones the repository inside the agent, through the cluster's network rules, which add the credential for the hosts your profile allows; nothing of your computer's credentials is sent. Put no user name or token in the address. To bring a clone's changes home, run pull in a checkout of the same repository: it reviews what the agent changed since the clone, and --apply writes it into that folder.
substrate/contexts.json under the DefenseClaw state directory holds your contexts: addresses, issuer, client ID and the CA files. It has no secrets and mode 0600. Tokens live in the keychain (macOS Keychain, Windows Credential Manager, Linux Secret Service). On macOS the key that seals them is an item only the DefenseClaw program may read without asking; macOS asks once after you install a new DefenseClaw build. Over SSH the keychain can't show that question, so the launcher stops with: "the macOS keychain is locked and this session cannot ask you to unlock it (for example over SSH); unlock it with: security unlock-keychain". Unlock it, or set DEFENSECLAW_SUBSTRATE_TOKEN_STORE=file.
Configuration
The substrate section of the DefenseClaw configuration. On a cluster it comes from the chart's config: values.
| Key | Default | Meaning |
|---|---|---|
substrate.enabled | false (the chart sets true) | Turns the Substrate components on |
substrate.api.address, .server_name, .ca_file, .auth | api.ate-system.svc:443, none, in-cluster CA, serviceaccount | How DefenseClaw reaches the Substrate control API (.auth is serviceaccount or mtls) |
substrate.supported_versions | >=1.0.0 <1.1.0 | The Substrate versions verify accepts |
substrate.identity.issuer | https://idp.ate-system.svc | Issuer of the actor tokens. Make it unique per cluster. The gateway compares iss with it as a string; it need not resolve |
substrate.identity.keys_url | https://idp.<substrate namespace>.svc/openid/v1/jwks | Where the gateway reads the keys that sign actor tokens. The namespace comes from substrate.api.address. The token's iss is still compared with substrate.identity.issuer exactly |
substrate.identity.keys_ca_file | /run/servicedns.podcert.ate.dev/trust-bundle.pem | CA bundle (PEM) for the keys URL's certificate. The default is the servicedns trust bundle the chart mounts, which signs the idp Service certificate. The gateway flag --issuer-ca is deprecated; use this key |
substrate.identity.audience | defenseclaw | Audience of actor tokens |
substrate.identity.token_lifetime_seconds | 300 | Lifetime of the token the egress gateway adds. 300 s is the upstream minimum, and egress caches the token by actor name, so a recreated actor with the same name is refused until the old token expires. run never reuses a name for that reason |
substrate.identity.oidc.issuer, .client_id, .groups_claim | none, none, groups | The OIDC provider people sign in with. Unset on an open-source cluster to accept Kubernetes identities (login --kube) instead. Managed installs require it |
substrate.identity.admin_groups | [] | Groups (the OIDC groups claim, or Kubernetes groups with the Kubernetes sign-in) whose members are Substrate administrators. The gateway's --admin-groups flag (chart gateway.adminGroups) adds to it |
substrate.identity.workload_oidc | [] | Issuers accepted for CI workloads: issuer, audience, subject, claims |
substrate.services.actor, .admin | defenseclaw-actor.defenseclaw-system.svc.cluster.local, defenseclaw-admin.defenseclaw-system.svc.cluster.local | Service names. Use the full name: an actor resolves only the names in its allow list |
substrate.launcher_atespaces | {} | Profile name to the atespaces the launcher may create actors in. Only DefenseClaw edits these |
substrate.require_attested_templates | true | Actors from unattested templates get no network |
substrate.templates.trust_anchors | the chart's anchors Secret mount | Public keys that sign attestations. Re-read when the Secret changes, so a revoked key applies at the next poll |
substrate.templates.registry_auth.docker_config | none | A Docker config.json (the chart's controller.templateRegistrySecret mount). Each auths entry is sent only to its own registry host, after that host asks. Re-read when it changes; a missing file is anonymous, a broken one refuses |
substrate.templates.registry_auth.ecr_hosts | [] | Amazon ECR registry hosts (<account>.dkr.ecr.<region>.amazonaws.com) logged in to with ecr:GetAuthorizationToken through the AWS SDK default chain (IRSA or EKS Pod Identity). Other hosts are refused when the configuration loads |
substrate.templates.allowed_images, .profile_images, .volume_images | [] | Repository globs (registry/team/*, no tag) an attested image must match, narrowed per profile; and images a template may mount read-only without an attestation |
substrate.templates.min_policy_generation, .max_age, .revoked_digests, .allow_unverified_base, .plain_http_registries | 0, none, [], false, [] | Refuse older scans, old builds, listed digests, untrusted bases; registries reached over plain HTTP (a dev registry only) |
substrate.default_sandbox_class | gvisor | gvisor or microvm |
substrate.chain_egress_action | block | On Agent Substrate a completed chain refuses the egress by default, in every mode (open source, strict and managed), because actors are unattended: the actor's tool call that uploads is refused by its hook, and its request that leaves the cluster is refused by the egress callout (category chain_egress). That includes a chain whose every completing read names no one file (a read of several files, a read in a call that also makes a write with its own commands or redirects, a relative path in a call that is not proven to keep its directory): the agent chooses how it reads, so that form refuses the same way. The callout sees no file: after a credentials read it refuses every request of that actor that leaves the cluster, benign or not, until the chain's window ends (8 steps and 30 minutes of the actor's running time: wall time while the actor runs, whether or not it makes requests; a suspend, however long, doesn't count, nor does a time the leader can't see it running, such as a controller outage) or the opt-in is set; an upload of a different, known file is allowed only at the hook. The one setting is the config key substrate.chain_egress_action: block (the default) or alert (an opt-in under which every chain only alerts), set with the chart value config.substrate.chain_egress_action, which the gateway and the leader both read. A request counts as a Substrate actor's only with a verified actor token (the actor listener or the callout route). Hooks on a person's own computer or in an OpenShell sandbox keep the rule pack's disposition: an alert in the default pack, also for a chain whose read names no one file. The leader logs the action it uses at startup. Which reads start the chain: the secret-read chain |
substrate.idle_suspend_after | 30m | Idle time before DefenseClaw suspends an actor; 0s turns it off. A running --task run is exempt unless its profile sets idle_suspend_tasks: true; a finished one is suspended about a minute after its task ends, which frees its worker |
substrate.idle_suspend_without_cpu | false | Without a CPU feed (the controller's --usage-listen), an actor with a terminal attached is never idle. true makes 30 minutes with no hook, model or terminal traffic idle for it too |
substrate.workspace_max_bytes | 1 GiB | Largest folder upload the gateway takes |
substrate.unattended.asks | deny | What an unattended run does with an ask: deny or queue |
substrate.unattended.ask_timeout | 30m | How long a queued ask waits for a person |
substrate.unattended.telemetry_sampling | 1.0 | Fraction of routine events kept for --task runs. Blocks and lifecycle records are never sampled |
substrate.rate_limits.per_actor_rps, .per_atespace_rps | 20, 500 | Fair-share limits of each actor and each atespace on the actor Service, whatever the number of gateway replicas: every second the replicas count their requests through the leader, and each holds an actor to its share of the budget, in proportion to the requests it gets. Over budget answers 429 with Retry-After. In the first second of a flood, and on a replica that has not reached the leader for 3 seconds, each replica applies the whole budget. One actor also has at most 16 requests open at a time on each replica. Requests to hosts in an actor's allow list (the egress callout) are not counted |
substrate.extproc.enabled | true | The egress callout |
substrate.extproc.failure_mode | closed | What the egress gateway does when the callout fails. Managed installs are always closed |
substrate.extproc.body_inspection | auto | Read bodies of POSTs to MCP and listed hosts when Substrate supports it. on says it does; off checks headers only. In managed enterprise, auto refuses requests to listed MCP servers until the callout has received a body |
substrate.extproc.body_max_bytes | 1048576 | Largest body the callout reads |
substrate.extproc.mcp_hosts, .body_hosts | [] | Hosts whose bodies are read as MCP calls (host or host=name), and other hosts whose bodies get the secret scans |
substrate.extproc.request_access | none | The contact a refusal names |
substrate.extproc.chain.check_budget, .send_budget, .failure_mode | 20ms (at most 500ms), 250ms (at most 500ms, never shorter than check_budget), auto | How long the tool-chain check may take per request, and per request whose miss would refuse it (one that can carry data out, such as a git push or fetch; any request of an unattended actor or on a managed or closed install; one that reuses a cached registration), and what a request does when it can't be checked: auto refuses for unattended actors and on a managed install and alerts for the rest, closed always refuses, open always alerts. The alert covers GET and HEAD only: a request with a body or any other method is refused in every mode while its check is missing |
substrate.extproc.ask.hold | 1.5s | How long a request waits for an ask. Keep it under the egress filter's 2 s message timeout. The ask comes after the tool-chain check, within the callout's 1.8 s: with a slow leader, ask.hold plus chain.send_budget over 1.8 s shortens the hold |
substrate.extproc.dlp.credential_headers | alert | alert or deny for a secret-shaped credential header on a host no secret is bound to |
substrate.extproc.actor_secrets_dir | set by the chart from callout.actorSecrets.secretName | Where the callout reads the values of profiles' actor_secrets |
substrate.llm.providers | all the gateway proxy serves | Providers actors may reach, only through the proxy: anthropic, openai, bedrock, gemini |
substrate.llm.bedrock_models | [] | Bedrock model IDs actors may call. A call that names another model goes to the first entry instead, through the same guardrails. That holds for /model/<id> and for the model named in the body of the OpenAI-compatible calls (/llm/bedrock/openai/v1/responses and /chat/completions). A body that can't be read as a JSON object with a model, or that is compressed or too large, is refused |
substrate.llm.credential.provider_uri | none | Where the model key comes from (ate-secret://...) |
substrate.egress.allow, .ecosystem_packs, .registries | [], every pack, [] | The base egress every actor gets before its profile tightens it |
substrate.secrets | [] | See Secrets |
substrate.allow_passthrough_for | [] | Hosts that are not inspected. Never given a secret |
substrate.admin | none | The cluster floor, the same keys as openshell.admin plus the Substrate keys below. Profiles tighten it |
Profile keys for Substrate: profile_assignments[].match.substrate (atespaces, templates, workloads). To assign a profile to a Kubernetes ServiceAccount (open source, login --kube), use match.users: [system:serviceaccount:NAMESPACE:NAME] for one ServiceAccount, or match.groups: [system:serviceaccounts:NAMESPACE] for all of a namespace's. The sandbox section of a profile is the same as for OpenShell (required_for, allowed_harnesses, egress_allow_only, allow_unblock, user_approvals and the rest). See Managed enterprise.
Keys of the sandbox section (and of substrate.admin) for the runtime:
| Key | Values | Meaning | How a profile tightens substrate.admin |
|---|---|---|---|
runtimes | [openshell, substrate] | Where the profile's agents may run. Left out: both. Without substrate, its actors get no policy | Intersection |
sandbox_class | gvisor, microvm | The weakest isolation an actor may run in. Unset: substrate.default_sandbox_class | microvm wins |
idle_suspend_after | Go duration, 1m or more | Replaces substrate.idle_suspend_after | The minimum |
idle_suspend_tasks | true or false | A running --task or --batch run of the profile may be idle-suspended (by default it is exempt). Without the controller's CPU feed (--usage-listen), idle follows substrate.idle_suspend_without_cpu | true wins |
unattended_asks | deny, queue | What an unattended run's ask does. Unset: substrate.unattended.asks | deny wins |
ask_timeout | Go duration, 1m to 24h | How long a queued ask waits before it is denied. Unset: substrate.unattended.ask_timeout | The minimum |
ask_hosts | host names or *.<host> | Allowed hosts an agent reaches only after a person approves (the egress callout asks; substrate approvals answers). They never open a host the agent's allow list doesn't have | Added together |
url_rules | {name, hosts, methods, path_prefix, action} | Requests to a host the callout refuses (deny) or asks about (ask), by method and path prefix. action is deny or ask; a rule with action: allow is refused when the configuration loads. Every matching rule counts, whatever its order or which side (substrate.admin or a profile) wrote it, and the strictest wins: deny, then ask. An approval never lifts a deny. An approval covers the rule and the methods it was asked for (the ask shows METHOD host /prefix*); an ask_hosts approval covers the host. Rule names must be unique (an approval is keyed by the name; a duplicate is refused when the configuration loads). See the next row for methods and paths | Added together |
url_rules methods and paths | how a request is matched | A request counts as every method a server framework could run it as: its own, each value of X-HTTP-Method-Override, X-HTTP-Method and X-Method-Override, _method or _HttpMethod in the query, _method in the body of a POST read as a form (no Content-Type, urlencoded or multipart, gzip and declared charsets decoded), and the top-level _method or _HttpMethod of a JSON POST body (Laravel, Express method-override) where a deny or ask rule on that host and path names a method other than POST. Each rule is checked for every one of those methods, and an ask names them all; its approval covers only those. The path is matched after decoding, with . segments and repeated slashes removed, without case for deny and ask. A .. segment or an encoded /, \, ; or % is refused where a path rule applies. When such a POST's body can't be read (multipart, over body_max_bytes, body inspection off, or auto before Substrate has sent a body), failure_mode decides: closed refuses, open allows with an alert. On a headers-only cluster in closed mode, JSON POSTs to such paths are refused; use body_inspection: on. Other JSON and typed POSTs are not held | - |
actor_secrets | key names | Values a program inside the actor may read, keys of the Secret in callout.actorSecrets.secretName. Matched by exact value on prompts and egress | Intersection |
openshell.admin refuses sandbox_class, idle_suspend_after, idle_suspend_tasks, unattended_asks, ask_timeout, actor_secrets, ask_hosts and url_rules. substrate.admin refuses allow_mount, allow_host_ports, process_tree, driver_fallback, require_copy_for and locked workdir.* keys.
When a config change takes effect
Change the config with helm upgrade or GitOps. helm upgrade also restarts every DefenseClaw pod (each pod carries a checksum of the config), so every setting below applies within one rollout.
A direct edit of the defenseclaw-config ConfigMap, for example with kubectl edit, needs no restart for most settings. The controller, the gateway and the leader read the ConfigMap through the Kubernetes API every 2 seconds. An edit that does not load is refused with an alert (sandbox.config.refused), and the config in force stays. That includes a key the config schema does not declare, such as a misspelled egres:, which a pod start would refuse too. So does a failed read of the ConfigMap: once the API has answered, a component never falls back to its mounted file, which on a managed install keeps the config it started with.
| Setting | When running and new agents get it |
|---|---|
substrate.egress (allow list, registries, ecosystem packs), substrate.allow_passthrough_for | Within about 5 seconds. The controller renders each agent's egress policy again on its next pass |
substrate.admin and guardrail.profiles.<name>.sandbox (egress block and allow-only lists, allowed, runtimes, sandbox class, idle suspend) | Within about 5 seconds |
guardrail.profile_assignments, substrate.launcher_atespaces | Within about 5 seconds. An agent that is no longer managed loses DefenseClaw's policy. New agents get the profile from the new list |
substrate.secrets | The egress bindings within about 5 seconds. The gateway's injection list at its next restart |
substrate.templates, substrate.require_attested_templates | Templates are attested again on the next pass |
substrate.chain_egress_action | Within about 2 seconds, on the gateway's hook answers and the leader's callout verdicts |
substrate.unattended.asks (and the profile unattended_asks) | On the gateway's next ask. The callout's copy only through helm upgrade |
guardrail mode and rules (hooks and model calls) | When the gateway reloads its mounted config file. The kubelet updates the file about a minute after the edit. Managed installs mount the file in a way the kubelet never updates, so they apply it at the next gateway restart |
substrate.identity, substrate.services, substrate.api, substrate.bundles, substrate.rate_limits, substrate.llm, llm_providers, substrate.workspace_max_bytes, substrate.unattended.ask_timeout | At the next restart of the pods. The controller keeps the identity, services, API and llm_providers hosts it started with too, so it stays in step with the gateway. Each component logs the keys that wait for a restart |
deployment_mode and the enterprise profile | Never live. An edit that changes them, for example one that drops managed_enterprise, is refused whole with a sandbox.config.refused alert that names deployment_mode_change (or managed_downgrade); none of the edit applies, the guardrail part included. It applies only when the pods restart with it, through helm upgrade |
substrate.extproc, and the callout's copy of url_rules, ask_hosts, actor_secrets and unattended asks | Only through helm upgrade, which renders the callout's own ConfigMap. The controller keeps the mcp_hosts it started with, so an edit opens no egress the callout does not inspect yet |
After a direct edit of a setting that waits for a restart, run kubectl -n defenseclaw-system rollout restart deploy/defenseclaw-gateway (or deploy/defenseclaw-gateway-leader). With substrate.bundles on, the stage bundles are layered over the profiles that config.yaml had when the controller started, so a profile change in config.yaml applies when the controller restarts.
Chart values
The signed chart is oci://ghcr.io/cisco-ai-defense/charts/defenseclaw-substrate. Every value is checked by its schema. The ones you will set:
| Value | Meaning |
|---|---|
config | The DefenseClaw configuration, rendered into one ConfigMap |
deploymentMode | managed_enterprise sets config.deployment_mode, mounts config.yaml as one read-only file and gives each pod a private data volume (managed.serviceAccountName, managed.dataSizeLimit) |
bootstrap.ownerTokenSecret | A Substrate global owner's token for the one-time access-policy bootstrap |
managedAtespaces | Other atespaces DefenseClaw writes policy in |
substrate.namespace, substrate.apiAddress | Where Substrate runs |
gateway.hpa.*, gateway.pdb.* | Scaling and disruption budget of the gateway. The gateway and callout autoscalers scale on CPU, so they need the Kubernetes metrics API (for example metrics-server): without it kubectl get hpa shows cpu: <unknown> and they stay at their minimum replicas, and a busy atespace slows every actor's hooks instead of adding replicas |
gateway.relaySessionSecret | The Secret holding the relay session key pair (private.pem signs attach tokens, public.pem goes into agent images). The relay-session hook creates it when absent, never replaces it, and uninstall keeps it. Default defenseclaw-relay-session |
gateway.llmCredentialsSecret | The Secret with one model key per provider (anthropic, openai, bedrock, gemini). Default defenseclaw-llm |
gateway.authDelegatorBinding | Name of the ClusterRoleBinding of the gateway to system:auth-delegator, rendered only for the Kubernetes sign-in (no OIDC issuer). Default defenseclaw-auth-delegator-<namespace>-<release> |
gateway.adminGroups | Groups whose members are DefenseClaw administrators (--admin-groups). Adds to config.substrate.identity.admin_groups |
gateway.oidcCA.configMap or .secret, gateway.oidcCA.key | The trust bundle of a private OIDC issuer, from a ConfigMap or a Secret (the Secret wins; key default ca.crt). Mounted read-only at /etc/defenseclaw/oidc-ca/, and --oidc-ca-file is set |
gateway.extraArgs | Flags added to the gateway's serve flags, for example --chain-retention=48h. The chain egress action is not a flag: set config.substrate.chain_egress_action |
gateway.extraEnv, gateway.extraVolumes, gateway.extraVolumeMounts | Added to the gateway container and pod |
gateway.args | Replaces the whole argument list (serve --substrate ... included). Experts only; use gateway.extraArgs to add a flag |
controller.templateRegistrySecret | A kubernetes.io/dockerconfigjson Secret with logins for private template registries, mounted in the controller and the gateway. Empty reads registries anonymously |
serviceAccounts.annotations, serviceAccounts.componentAnnotations.<gateway|controller|callout> | Annotations on every ServiceAccount, and on one component's (merged over the first), for example an IRSA role on gateway and controller only |
leader.store.size, leader.store.keepOnUninstall | The leader's volume |
controller.templateAnchorsSecret | The admin-only Secret with the template trust anchors. Default defenseclaw-template-anchors |
callout.judge.tokenSecret | The callout's token for the MCP tool-rules route. Default defenseclaw-callout-judge, which the chart creates. "" turns the route off: MCP tool calls then raise an alert, and a managed install refuses them |
callout.actorSecrets.secretName | A Secret whose keys are the values profiles' actor_secrets expose. Mounted in the callout and the gateway only |
admin.expose.type, admin.expose.ingress.*, admin.url, admin.tlsSecretName | How laptops reach the admin Service, and the https address published for login --kube |
credentialProvider.secretsNamespace | The namespace that holds the per-actor secret copies. Default defenseclaw-actor-secrets. Two releases in one cluster, for example a managed one next to an open-source one, each need their own. Only one of them can register the egress callout, because Substrate has one callout slot |
substrate.actorKeys.url | Sets substrate.identity.keys_url. Empty is the idp Service |
substrate.actorKeys.caConfigMap | A ConfigMap (key ca.crt) with the CA of the keys URL, mounted on the gateway as keys_ca_file. Empty is the servicedns trust bundle |
networkPolicy.egress.extra | Extra NetworkPolicy egress rules for every pod with an egress limit (gateway, leader, callout, controller). The default allows DNS, the Kubernetes and Substrate APIs, the release's own Services, OTLP and model providers on 443 only. Use it for a private OIDC issuer, a private registry or EKS Pod Identity. See the EKS page |
networkPolicy.adminFrom | Who can reach the admin Service |
clusterName | The cluster name on the gateway's telemetry (the k8s.cluster.name resource attribute; k8s.namespace.name comes from the pod). Empty leaves it out |
metrics.prometheusRule.enabled | Install the alert rules |
metrics.prometheusRule.namespaceLabel | The Prometheus label that carries the k8s.namespace.name resource attribute. Empty (the default): the alerts on the defenseclaw.substrate.* metrics read every release's series, which works with any OTLP receiver. Set it to k8s_namespace_name so they select this release's namespace and releases that share a Prometheus don't alert on each other, after you promote the attribute where your metrics enter Prometheus (otlp.promote_resource_attributes, or the collector's resource_to_telemetry_conversion). Without the promotion the label is absent and those alerts never fire |
homeSizeLimit | The disk volume each pod keeps its local audit database in (4Gi; managed.dataSizeLimit in managed mode; homeMedium: Memory makes it a memory volume). It must hold config.observability.local.max_store_mb twice over plus 256 MiB. When the store outgrows a disk volume, the kubelet evicts the pod and the store is lost; only a memory volume or a full node disk gives ALERT audit_store_full in the gateway log, with hooks that need an audit record failing closed. It is lost when the pod is replaced. See Turn on OTLP export |
observability.local.enabled | A local collector for trying the chart, and a destination chart-collector that exports to it. See Turn on OTLP export |
Turn on OTLP export
On Agent Substrate, export is the audit record. Each gateway pod keeps a local audit database in a pod volume (an emptyDir on disk, or in memory with homeMedium: Memory). It is lost when the pod is replaced, for example by a rollout, an eviction or a node move, and each replica holds only the requests it handled. The chart has no persistent volume for it: a volume per replica would still split the history across replicas. Send the records to a collector or a SIEM that keeps them. A gateway started without an export destination logs no export destination once.
Export destinations go under config.observability.destinations in the chart values. The chart renders config into the defenseclaw-config ConfigMap, and the gateway reads it from there. There is no config.otel key.
config:
observability:
destinations:
- name: my-collector
kind: otlp
protocol: http/protobuf
endpoint: http://opentelemetry-collector.otel-system.svc:4318
tls:
insecure: true
network_safety:
allow_private_networks: true- TLS. An
http://endpoint needstls.insecure: true. Anhttps://endpoint must not set it.config validaterefuses a mismatch. - Private addresses. A Kubernetes Service name resolves to a cluster address, and on most clusters that address is private. The gateway sends to a loopback or private address only when the destination sets
network_safety.allow_private_networks: true. For an address in 100.64.0.0/10, setnetwork_safety.allow_cgnat: true. Without the option, the gateway still starts. It runs without that destination and logs one line that names the destination and the option. - Protocol.
protocoldefaults togrpc, the collector's port 4317. For port 4318, setprotocol: http/protobuf. - Network policy. The chart's egress policy allows OTLP on ports 4317 and 4318 (
networkPolicy.egress.otlpPorts). Add other ports there.
defenseclaw-gateway config validate warns about the last two mistakes. private_service_endpoint is a Service name without allow_private_networks. otlp_protocol_port is grpc on port 4318.
To try it out, set observability.local.enabled: true. The chart then installs a collector that prints what it receives, and adds a destination chart-collector that points at it with the settings above. If you name your own destination chart-collector, the chart keeps yours.
Check that it works:
defenseclaw substrate status --cluster --install-file install.yamlshows every gateway replica ready.- If a gateway pod restarts instead,
kubectl -n <namespace> logs deploy/defenseclaw-gateway --previousnames the destination and the cause, for exampleobservability v8 bootstrap failed: runtime_unavailable: destination "my-collector": .... - A line
observability destination "my-collector": endpoint host ...in the gateway log means the destination is not sending. The line says why: the option to set, or a host that does not resolve. - With the local collector,
kubectl -n <namespace> logs deploy/defenseclaw-otel-collectorshows the records arriving.
Telemetry
The new attributes, families and metrics are in Sandbox telemetry. Metrics carry no actor, atespace or person labels.
How it works
Where DefenseClaw runs in an Agent Substrate cluster, how a tool call, a model call and a web request are checked, and how agent identity, network policy, state and suspend and resume fit together.
Troubleshooting Agent Substrate
Symptoms, causes and fixes for DefenseClaw on Agent Substrate - no free worker, the ask delay, unattended templates, drift and policy conflicts, another callout, version, issuer, bucket and kit skew - and a section written for the agents themselves.