A gateway restart used to erase your subagent — now it keeps a record and picks itself back up

You hand a subagent a refactor that spans twelve files. It has been running for 25 minutes and you can see it on file eight. Then you restart the gateway — an upgrade, a config change, or it simply fell over. When you come back, the subagent row says unknown. No summary, no diff, no hint about what comes next. Did it get through eight files? Did it leave one half-written?
The expensive part is not the 25 minutes. It is losing the answer to “how far did it get”, which is exactly the information you need to decide what to do now.
What actually happens at the restart
A background subagent runs inside the process that owns the conversation. When the gateway stops, the executor is gone, and Hermes cannot prove whether the child’s last tool call landed. So the outcome is recorded as unknown rather than “failed”. A failure can be retried. unknown used to mean a human reading git status and guessing.
The fix is not to make crashes impossible. It is to write down enough before the crash that the work can be picked up afterwards.
Every child now has a durable row in state.db
Since the v0.23.2 window, each dispatched subagent keeps a record in the profile’s state.db (the resumable_subagents table):
record_startwhen its workspace is seeded;checkpointon each heartbeat, whenever its terminal’s working directory moves;record_stopafter its cleanup ran;- and, if the owning process died outright, boot recovery marks the record abandoned.
The field that matters most is the pin. A child that did not complete and was not stopped by a person keeps its worktree — the directory its terminal was in when it stopped, plus any worktree it created — out of the startup sweep. That lasts delegation.resumable_ttl_hours (default 48 hours), for at most delegation.max_resumable records per profile (default 32); past the cap the oldest loses its pin first. When the pin ends, the normal sweep archives the tree to refs/hermes/archive/ before removing it. Completed children and children you stopped are not pinned at all.
resume_mode: why it stopped decides how it continues
Each record carries a resume_mode, which is the difference between “retry it” and “retry it and waste another 25 minutes”:
| resume_mode | Typical cause | What continuing needs |
|---|---|---|
AUTO_AT |
usage or rate limit with a reset time, transient provider errors | wait for failure_resets_at |
AUTO_ON_BOOT |
gateway shutdown, crash, hard kill | nothing; check git state and the last tool call before re-running side effects |
ON_FIX |
credits, auth, model access, account policy, request-shape errors | you fix the cause first |
REPAIR_THEN_CONTINUE |
the conversation no longer fits, or an image/reasoning part was rejected | the transcript needs repairing first |
HANDOFF |
a provider safety refusal of this conversation | another model, or the parent takes over |
A limit failure also carries failure_resets_at, and usage_limit when a plan window is involved — so “when can I try again” is a value in the record, not a guess.
Continuing by hand
{"action": "continue", "subagent_id": "sa-0-1a2b3c4d", "message": "finish the remaining 4 files; do not rewrite what is already done"}
What that does is more careful than dispatching a fresh child:
- it rebuilds the child on the same route — model, provider, profile and toolsets;
- its history is loaded from the saved transcript, and
messagebecomes the next turn; - it keeps the same
subagent_idand session, so the transcript appends the way--resumedoes; - but it is a new background delegation with its own
delegation_id, so the result arrives as a normal completion.
A few refusals are worth knowing. A child that is still running cannot be continued — steer it instead. A child whose inherited route now resolves to a different provider or endpoint, because the parent switched models since it ran, is refused too. Only the owning conversation, or an ancestor in its spawn tree, can continue a child. Its background processes are long gone, so it gets a fresh terminal starting in the directory and worktree it stopped in.
A continue is an ordinary dispatch, which means a closed chat, max_concurrent_children, max_spawn_depth and the global spawn pause all refuse it exactly like they refuse a spawn.
Automatic resume
Manual continue assumes you noticed. The opt-in switch covers the case where you did not:
delegation:
auto_resume: true # default: false
auto_resume_deadline_minutes: 360
resumable_ttl_hours: 48
max_resumable: 32
With it on, the gateway checks every profile it serves every 15 seconds — each under that profile’s own configuration, credentials and state.db — and continues the records that are due through the same claim and continue path as the manual action. The resume message names the real cause: “Your run was stopped by a provider limit; continue where you left off.”
Only two kinds resume automatically: AUTO_AT once the limit’s reset has passed, and AUTO_ON_BOOT once the gateway is back up.
What never resumes: ON_FIX, REPAIR_THEN_CONTINUE and HANDOFF children (those need you, a repair, or a different model); anything you or the parent stopped; a child whose stop happened while auto_resume was off, or under a parent no gateway serves — those get the manual hint instead; expired records; records whose worktree pin the max_resumable cap dropped; and nested children, because the orchestrator that spawned them is the thing that resumes.
Silent while it waits, one report at the end
A child promised an automatic resume sends the parent no completion when it stops — otherwise the parent would think the work finished and act on half a result. While it waits, delegate_task(action='list') shows it as waiting_to_resume with a line like:
waiting to resume at 14:05 (attempt 1 of 2); gives up at 19:40
In a batch where only some children are waiting, the completion lists them with that same status, and the failed-delegations view does not count them as failures.
The parent hears once per child: either the resumed run’s own result, or one terminal failure when the scheduler gives up. Giving up happens when the two automatic attempts are used, when delegation.auto_resume_deadline_minutes (default 360) has passed since the first stop, when the known reset lands after that deadline (it gives up immediately rather than waiting for nothing), or when a permanent reason appears — a dropped worktree pin, for example.
That failure is not a shrug. It carries the reason, the child’s last summary, the tail of its transcript, and how to continue it by hand (delegate_task(action='continue', subagent_id=..., message=...)) or re-dispatch it. Automatic resume failing still leaves you with usable context.
Three safety rules that matter more than resume itself
No duplicate work. If you dispatch a new child whose goal is exactly the goal of a child waiting to resume (whitespace aside), the spawn is refused with the waiting subagent_id, when it will resume, and the two ways forward: wait, or action='stop' it first and re-dispatch. In a batch only that task is refused; its siblings run.
Unconfirmed side effects are never replayed. The resume message tells the child: if its last tool call has no recorded result and it cannot establish whether that call ran, it must not repeat it, and must finish with a result starting BLOCKED: naming that call. That is a much better answer than a confident-looking duplicate.
Attempts are bounded, and counted before they run. At most two automatic attempts per record, written into the record before the attempt starts — so a child whose resume crashes the gateway every time stops after two boots instead of looping. Children waiting on the same reset are started one per provider endpoint per pass, 15 seconds apart, not all at once.
When to leave it off
auto_resume defaults to false for a reason: a resumed child spends tokens with nobody watching, and so does the parent turn its result starts. If your budget is metered and uncapped, or your children keep failing on the same error, keep it off until you know which failure class you are dealing with. If your pain is purely environmental — laptop lid, gateway restart, a nightly rate limit — this switch is built for exactly that.
To understand how a child’s result wakes the parent, see background subagent failure wake; for concurrency and live steering, subagent live orchestration. The full change set is on the v0.23.2 release page.
The short version
“Half my subagent disappeared” used to have one fix: run it again, pay the tokens, hope the side effects were idempotent. The path now looks like: a durable record and a pinned worktree at stop time → resume_mode explaining the cause → manual continue or automatic auto_resume → silence while waiting, one report at the end → uncertain tool calls marked BLOCKED instead of replayed.
Crashes still happen. What changes is the cost: instead of 25 wasted minutes, it is four remaining files.