After kill -9, some processes never leave: Hermes now makes every child die with its parent


It is 2am, the laptop fans are screaming, and you kill -9 the Hermes gateway that went off the rails. Next morning ss -ltnp still shows port 8080 taken, and ps lists a llama-server, two browser daemons and a language server whose parent died hours ago. None of them left. You kill them one by one and wonder whether tomorrow repeats the same ritual.

That is not user error — it is how Unix works. When a parent process dies, its children are re-parented to init (PID 1) and keep going. For a one-shot command that is irrelevant. For something like Hermes, which starts a whole cast of long-lived children, it is a real problem. A recent change (PR #137057) replaced a pile of per-feature workarounds with one mechanism: when the owner dies, every long-lived child it started goes with it.

Why children “forget” to die

Almost everything an agent does spawns a process: npm run dev, a browser automation daemon, a local model router, a language server, a code-execution cell, an MCP server over stdio. Those processes should live exactly as long as the Hermes process that started them. By default they do not — they just get a new parent.

The bill arrives in concrete ways: memory creeps up, ports stay held so the next start fails, a local model keeps burning GPU after the session ended, and an old daemon fights the new one over the same pid file. Debugging is worse: you see an unfamiliar python process and have no idea who started it.

Hermes used to patch this family by family — an orphan sweep at the next start, a per-owner poller, a parent-pid watchdog, a death pipe, an atexit hook. None of them acted at the moment the owner actually died; each waited for the next boot or the next poll, and each copy had its own bugs.

One pipe tells you the owner is gone

The new design keeps exactly one mechanism. Every Hermes process holds the write end of a pipe; a supervisor (a POSIX sh script, roughly 0.5 MB resident when idle, down from 8.1 MB for the earlier Python version) holds the read end. When a child starts, Hermes registers its process group with the supervisor; when the child exits normally, it unregisters.

The trick is EOF. The moment the Hermes process disappears — clean exit, crash, kill -9, the OOM killer, a systemd stop — the write end is closed and the supervisor reads EOF immediately. That is the kernel telling you the holder of the write end no longer exists, and it cannot be ignored or missed. On EOF the supervisor sends TERM to every registered group, waits up to 3 seconds, then sends KILL.

It also refuses unsafe targets: its own group, Hermes’s group, and anything with a pgid of 1 or below. It starts lazily on the first registration and exits after 30 idle seconds, so a foreground command that registers and releases its group does not leave a /bin/sh behind on every run.

Two engineering details came straight from production failures. The spawn ledger’s concurrent writes: three processes spawning 15 children each used to keep 18 of 45 rows because read-modify-write was unserialized; it now takes a cross-process file lock that waits at most 2 seconds, so one stalled writer cannot freeze the others. The supervisor being killed itself: pkill -f hermes takes it along, so it is restarted with every group replayed while registrations exist — but a supervisor that exits by itself within 5 seconds of starting (a broken install) is not restarted in a loop.

The API is small: spawn_owned() is the one way to start a long-lived child (it leads its own process group, registers it, and records it in the ledger); own_process_groups() / release_process_groups() adopt groups something else spawned, such as the MCP SDK or a browser harness; release_process_group_once_empty() hands a group back only once nothing is left in it.

14 child families moved over

These are the numbers from the PR. The test in each row kills the owner with kill -9 and counts what is still alive afterwards; every row has a live control owner so a fix cannot pass by killing everything.

Child family Before (main) After
stdio MCP servers covered by the old MCP-only supervisor covered by the shared one
llama-server router and models 2 0
Browser harness daemons 2 (and 4 left after a clean exit) 0 (0)
Language servers (descendants that ignore stdin EOF) 1 0
Code-cell subprocesses inside the kernel 3/3 0/3
Slash-command worker 1 0
Terminal foreground command 2 0
Terminal background jobs (in a non-gateway process) 3 0
WhatsApp / Raft bridges 1 / 1 0 / 0
verify app server / command TTS-STT 3 / 3 0 / 0
Embedded cua-driver (Linux) 1 0
Dashboard WhatsApp pairing bridge 1 0
Bounded git maintenance (repack, commit-graph) 2 0

There is also a CI ratchet: scripts/check_detached_spawns.py walks the production trees and fails the build on any new detached-spawn call (start_new_session, process_group, os.setsid and friends) unless the line or the line above carries a detached-spawn: ok — <why> marker. The baseline is empty. So the fix cannot quietly erode later.

Three families are supposed to outlive the owner

Uniform cleanup is not “kill everything”. Some processes are deliberately kept:

  • Background jobs of any gateway. CLI, TUI, Desktop, ACP, cron and messaging all run tools inside a gateway, and every gateway boot calls recover_from_checkpoint, which re-adopts its jobs along with their output logs. Killing your npm run dev & because the gateway happened to idle out would be a disaster, so only a non-gateway process owns its background jobs.
  • persist_on_release jobs (#41225) — the name is the contract; they are meant to survive.
  • The Bot Desktop screen. One virtual screen per profile, shared across turns, open to human takeover, re-attached by a restarted gateway after an identity check.
  • Plus the updater and /update hand-offs, the gateway/dashboard/launchd services themselves, cron and kanban workers, the egress daemon and a /browser connect Chrome — things that exist precisely because they must outlive their launcher. Each carries a written reason in the code.

Short version: what you start for one task dies with Hermes; what you explicitly asked to persist does not.

Checking your own machine

You can verify this today, without a new release. Take a snapshot before you restart the gateway, then compare after a kill -9:

# long-lived children Hermes tends to leave behind
ps -eo pid,ppid,pgid,etime,cmd | grep -Ei 'llama-server|chromium|puppeteer|language-server' | grep -v grep

# ports those daemons hold
ss -ltnp | grep -E ':8080|:9222|:8000'

A ppid of 1 means init adopted it — on an older build, that is the signal that cleanup never happened. If you want to sweep what is already there, the agent-janitor walkthrough covers a third-party plugin that grades orphans by evidence before killing anything. The two are complementary: the supervisor prevents new leftovers, the janitor cleans up history.

What changes day to day

Three practical differences:

  1. Restarts after a crash are clean. No more pkill -f llama-server before starting, no more “port already in use” caused by a daemon from the previous build.
  2. Local models stop burning GPU. llama-server and its model children end with the session.
  3. The memory curve flattens. Every crash used to leave residue; a full day of restarts could cost gigabytes.

Boundaries worth knowing: Windows uses a different layer (job objects) and is unchanged; only long-lived children are owned, so one-shot commands behave exactly as before; and gateway background jobs can still survive a killed gateway by design, not by oversight.

Release status

PR #137057 merged into main on 2026-10-29 and shipped in v0.22.6 (2026-11-02) — a normal upgrade gets you the supervisor, no main install needed. On an older build, do the manual check above and pair it with the gateway loop watchdog so a wedged process is at least restarted instead of silently rotting.

If you find a new family of leftovers on your machine — a plugin’s own daemon, say — an issue is worth filing. Core code now has a CI gate in front of it; plugins still rely on their authors being careful.