Agent Substrate

Unattended and long-running runs

Run coding agents without a terminal attached - one task, a batch from a file, or a job in CI - and what DefenseClaw does differently when nobody is there to answer.

An agent doesn't need your terminal. Give it a task, leave, and read the result later. Every connector that has a headless mode can run this way, in its own actor. The headless command of each is in Agents and features.

One task

defenseclaw substrate run codex --task "make the failing tests in pkg/parser pass" --detach
defenseclaw substrate logs codex-2c9e41a7 -f
defenseclaw substrate status codex-2c9e41a7

--task runs the connector's headless command (for Codex, codex exec) instead of the interactive program. --detach returns as soon as the agent is running; without it run waits and prints what the agent printed (--timeout stops waiting after a while, and the task keeps running). The task text is registered with the cluster when the actor is created, so the guardrails know the user's intent from the start, from a source the agent can't change.

When the task ends, run says so, and logs shows what the agent printed and its exit code:

The task finished. Bring changes home with: defenseclaw substrate pull codex-2c9e41a7

defenseclaw substrate run claude --task finishing in the terminal: Started dc-ops/nightly-528031bf (Claude Code, profile ops), Copying this folder into the agent, the agent's Bash step and its report of three tries (403, 403, then 200), and the closing line The task finished with the defenseclaw substrate pull command to bring changes home

The task ended with exit code 1. Review it with: defenseclaw substrate logs codex-2c9e41a7; bring changes home with: defenseclaw substrate pull codex-2c9e41a7

If the cluster refused the agent's calls, run says The task did not run instead and exits with code 1 (see Troubleshooting).

A run that makes no progress for 10 minutes ends with exit code 124, and logs shows why (see When a run stops making progress):

defenseclaw: the task made no progress for 10m0s (no output, CPU or I/O); the cluster gateway answers, so the agent is stuck on a connection of its own. The task is ended with exit code 124.

A batch

# tasks.yaml
tasks:
  - name: fix-parser
    task: make the failing tests in pkg/parser pass
  - name: bump-deps
    task: update the Go dependencies and run the tests
defenseclaw substrate run claude --batch tasks.yaml --parallel 8 --clone https://git.example.test/team/app.git --ref 4f1c2ab
defenseclaw substrate list --batch BATCH
defenseclaw substrate status --batch BATCH
defenseclaw substrate delete --batch BATCH

The file is a list of tasks, under tasks: or bare. A task is a line of text, or name, task and agent (a task may name its own agent). One batch takes at most 500 tasks. Each task gets its own actor, with a shared batch ID, and --parallel (default 4) limits how many run at once. A batch can be larger than the worker pool: a task that finds every worker taken waits up to 15 minutes for one, and a finished task gives its worker back about a minute after it ends. With --parallel at most the number of free workers, no task waits. --clone and --ref start every task from the same commit; without them every task starts from a copy of the same folder. status --batch shows each agent's state. usage --batch is not available yet: see Feature coverage.

From CI

CI has no person to sign in. The launcher signs in with a workload identity instead: a GitHub Actions OIDC token or a Kubernetes service-account token. The cluster accepts the issuers you list in substrate.identity.workload_oidc. The "person" on every event is then the workload principal, and a profile picks the policy by its match.substrate.workloads entry.

A workload_oidc subject and a workloads entry are exact or end in :* or /*, for example repo:acme/web:*. A raw prefix such as repo:acme/web* is refused, because it also matches repo:acme/web-evil, and so is a bare *.

Write the subject exactly as the token carries it. GitHub can put the owner and repository ids in it (repo:acme@1234/web@5678:ref:refs/heads/main, not repo:acme/web:ref:refs/heads/main), and an entry written the other way matches nothing: the workload then gets the default profile. login --workload asks the cluster which entry the token's subject matches and warns no match.substrate.workloads entry matches <sub>; it gets the default profile when none does. Read the sub of a real token from your workflow (login --workload prints it as "Signed in as"), or bind on the repository and ref claims in workload_oidc instead. login --workload checks the token with the cluster and says at once when the cluster does not accept it.

A workload's profile needs a launcher atespace: list one for it under substrate.launcher_atespaces, or run refuses with "your profile ... has no launcher atespace on this cluster".

defenseclaw substrate login --workload --workload-token-file "$TOKEN_FILE" --admin-url ... --issuer ... --client-id ...
defenseclaw substrate run claude --task "triage the open dependabot PRs" --detach --workload-token-file "$TOKEN_FILE"

run also accepts --workload-token-file, so a job can skip login and pass its token directly. Tokens are never printed or logged. In CI without a keychain, set DEFENSECLAW_SUBSTRATE_TOKEN_STORE=file.

What is different when nobody is there

SituationWhat happens
The gateway can't be reachedHooks fail closed, always, whatever the interactive setting. A hook that waits retries first, then blocks: each hook call gives up within about 21 seconds (2 s to connect, 9 s, then one 12 s retry). Model calls are the agent's own: it retries them, and the run carries on once the gateway is back
The run stops making progressAfter 10 minutes with no output (retry notices don't count) and almost no CPU or I/O, the relay ends the agent: exit code 124, a line in logs that says why and whether the gateway answers, and an audit row. This is what ends a run whose model call never answers after a gateway outage
A hook call got no answer during the runThe hook refused the step closed, so it may not have run. If the agent still exits 0, the relay reports the task with exit code 75 (a temporary failure: run it again), not a success, and logs and the audit row say how many hook calls the gateway did not answer and when the first one was. An agent that exits with an error keeps its own exit code, with the same reason
The agent asks to reach a new site, or a tool needs approvalDenied by default. A profile can set unattended_asks: queue to hold the ask for a person for up to ask_timeout (30 minutes by default), and a person answers with defenseclaw substrate approvals and approve or reject. The egress gateway can hold one request for under 2 seconds only, so the request is refused with a note that the ask is open until its expiry: the agent waits, sends it again and keeps going once it is approved; a reject or a timeout is a deny. approve --always also answers the same host (or url_rules rule) for that person's later runs on the same profile
The agent has been quiet for a whileNo idle suspend for a running --task run unless its profile sets idle_suspend_tasks: true, and never while an ask is pending. An attached terminal keeps the actor running. CPU use counts as activity when the controller has a usage feed (--usage-listen). Without a CPU feed, a detached actor running a silent job can be suspended after idle_suspend_after; attach, or run it with --task, to keep it running
The task endedThe end (exit code and cause) goes to the audit log. About a minute later the actor is suspended, which frees its worker; pull, logs and connect wake it. If something reached the gateway after the end (a pull right after the run), it is suspended after idle_suspend_after without activity instead. This happens with idle suspend off (idle_suspend_after: 0s) too
A step crashes the actorThe actor shows as crashed, and pending asks close. Substrate brings a crashed actor back only with a revert (kubectl ate revert); DefenseClaw doesn't do it for you
You want to know it is alivestatus, list (it has a STATE column) and logs -f show it. checkpoint NAME --every 1h saves a restore point on that schedule for as long as the command runs
The resultHave the agent push a branch (the git credential is injected on the git host only), or defenseclaw substrate pull the changes

A queued ask shows unattended and the time left in the RUN column of approvals, and every configured webhook gets an approval-request post for it, whatever its min_severity.

A rejected or expired ask is not remembered. If the agent asks for the same site again, a new ask opens at once; there is no cooldown. To stop the asking, refuse the site with a url_rules rule (action: deny) or leave it off the allow list.

defenseclaw substrate approvals listing one ask waiting: ID ask-4c49616b8f92eb8c85c8fc2b, agent nightly-528031bf, what GET example.com, age 7s, run unattended with 4m left, and the line with the approve and reject commands; then defenseclaw substrate approve for that ID prints Approved

When a run stops making progress

Nobody watches an unattended run, so DefenseClaw makes sure it never hangs without a word. The relay in the actor watches the agent. If for 10 minutes the agent prints nothing but retry notices and its processes use almost no CPU or I/O, the relay checks whether the cluster gateway answers, prints the reason, stops the agent and everything it started (TERM, then KILL after 10 seconds) and the task exits with code 124. The relay counts every process the agent started, its whole process tree, including tools the agent runs in sessions of their own (Claude Code's Bash tool, Codex's exec server), so a long silent build or test run counts as progress through its CPU and I/O. A running tool of Claude Code also prints a heartbeat every 30 seconds. A tool that waits silently with no CPU or I/O for longer than the bound, such as a long sleep in a tool that prints no heartbeat, is ended. The other way round, a background process that keeps running, such as a dev server or a file watcher the agent started, counts as progress, so it keeps a run whose agent hangs going for as long as it runs: this is a no-progress bound, not a time limit. To bound the whole task, give it a maximum run time from outside, for example run --task ... --timeout 2h and then defenseclaw substrate delete NAME, or the CI job's own timeout.

The typical cause is a gateway outage: hooks give up within about 21 seconds and fail closed, and the agent retries its model calls. If a model call was cut off without an answer, Claude Code waits about 6 minutes for that attempt before it retries, and a path that keeps hanging would leave the run silent for much longer. The bound is set when the template is built: defenseclaw substrate template build --task-stall 30m (1 minute to 24 hours; default 10 minutes). It is the kit's relay.task_stall_ms.

Every task end, finished, failed or ended for no progress, is reported to the gateway: a substrate audit row. Every report is kept; a second report for the same actor is marked as a repeat (repeat), because the agent shares the actor's identity and could post one itself. The row has event task_end, the actor, batch, exit code and reason, with severity INFO for a finish, MEDIUM for a failure and HIGH for a run the relay ended. The controller then frees the actor's worker (sandbox.task_end_suspend).

Telemetry volume

A batch of hundreds of actors produces a lot of events. Set substrate.unattended.telemetry_sampling (1.0 keeps everything) to sample the routine ones. Blocks, asks, refusals and lifecycle records are never sampled.

Limits

  • A finished --task agent stays until you delete it, suspended: it gives its worker back about a minute after its task ends, and waking it for pull, logs or connect takes a free worker again. Pull its changes first: what is not pulled is gone when you delete it.
  • A task runs as the connector's headless mode allows. run --task for a connector that cannot run one says so. See Agents and features.
  • The cluster's worker pools bound how many actors run at once. A full pool answers that no worker is free, and run starts nothing. See Troubleshooting.