One gateway per host: the multiplex-only migration, three real incidents, and a post-upgrade checklist


You run three Hermes agents on one small box: a personal one, a coding one, and one watching production alerts. After every upgrade you used to type hermes -p coder gateway restart without thinking, because you knew it touched exactly one process. Since v0.21.4 that command means something else: it now targets the one gateway on this machine, and that gateway is serving the other two profiles at the same time. If you would rather know what you are standing on before you upgrade, here is the new topology, the migration command, what it refuses, and the three real incidents the convergence cleaned up along the way.

The new rule: one hermes gateway run per host, per OS user

The architecture directive is blunt: multiplex-only. One gateway process per machine receives platform messages, and it serves every profile under profiles/. This is not a recommendation — it is the shipped topology as of v0.21.4. hermes gateway migrate --multiplex converges an older box onto it, and a newly created named profile can no longer install a gateway of its own (see the forced case below).

What changed with it is the meaning of every lifecycle verb. The decision function is attach → rescan → refuse, and the rule is that a second process is never started:

  • hermes -p X gateway run while X is served: prints the host gateway’s PID and served set, exits 0, spawns nothing.
  • hermes -p X gateway run while X is not served: asks the host gateway over its control socket to re-scan profiles/, then attaches once the answer includes X. Exit 0.
  • The owner cannot serve X (say, a duplicate credential): refuse with the reason, exit 78. A transient refusal — owner alive but silent — exits 75 so a supervisor retries.
  • hermes gateway start --all / restart --all mean the one host multiplexer. They no longer sweep away running gateway processes; a profile still holding its own gateway is reported, never killed, with the hermes gateway migrate --multiplex one-liner attached.
  • hermes gateway stop on a served secondary is no longer “no gateway running for this profile” with exit 0 — it names the host process and tells you to stop that instead, exit 78.
  • hermes gateway run --replace decides by the live served set: whichever profile launched the process takes the host role over. --force is the escape hatch that starts a separate gateway without asking the owner at all.

One detail that bites people: under a supervisor the attach path exits 75, not 0. systemd, launchd and s6 all restart a 75 after a short delay, so the unit keeps retrying and takes over by itself the moment the host process disappears.

Migrating: read the plan, then let it resume itself

On a box upgraded from a per-profile topology, the standard move is two commands:

hermes gateway migrate --multiplex --dry-run   # print the plan, blockers and the restarts it implies
hermes gateway migrate --multiplex             # converge (add --yes in non-interactive shells)

The dry run lists each profile’s gateway PID and service type as a table, then spells out what it will do — stop pid 4101 and uninstall the systemd (user) unit for coder, set gateway.multiplex_profiles: true in the default profile’s config.yaml, start the gateway and verify it serves three profiles — and records the previous state in ~/.hermes/gateway_migration.json. It never SIGTERMs a gateway silently: the plan says which processes will be terminated, and they drain in-flight turns before exiting.

Two design decisions are worth remembering, because they decide what you do when something goes wrong:

  1. Convergence is defined by topology, not by the config flag. A converged host has no secondary profile owning a gateway process or a supervisor unit. The older check only asked “is the flag on?”, so a half-migrated host — flag flipped, unit left behind, a crash in between — reported as already multiplexed, and the command did nothing on exactly the machine that needed it. Now the flag alone never satisfies the check, which means re-running the command is the recovery.
  2. The manifest is the resume record: present means unfinished. When the command finds one on disk it resumes from it instead of overwriting it (it is the only record of which units existed), and only a confirmed convergence clears it.

Migration covers systemd (user and system, reading User= before removing the unit), launchd, and Windows Scheduled Tasks — Windows used to be a flat refusal with instructions to do it by hand, and now also cleans up the Startup-folder entry that gateway install writes when it cannot register a task (removing only the task would leave that fallback launching a second gateway at the next logon). s6 containers are still refused, correctly: there the per-profile gateways are slots the container’s own boot registers, so the fold belongs to that boot.

Three real incidents: why one backend must serve one profile

This convergence is P1 work because “one backend, many profiles” amplifies a very quiet failure class. Three instances landed in this window with names attached.

First, REST routes mutated the wrong profile’s data. With one backend serving everyone, a handful of handlers read the process’s get_hermes_home() and ignored the profile the request named. The result was not a wrong read but a wrong delete: POST /api/memory/reset?profile=worker_beta removed the launch profile’s memories/MEMORY.md and USER.md while worker_beta was untouched. The same shape covered deleting a webhook subscription, deleting an ops hook (which also revoked the launch profile’s consent), pruning checkpoints, importing a backup over the launch profile, deleting a credential-pool entry, and running the curator against the launch profile’s skills. Every route now honours the profile it is given, and the destructive ones return 400 when a request names no profile on a multi-profile host instead of picking one.

Second, named profiles became credential islands. An earlier change made every named profile read only its own auth.json, so after an update Desktop bots started failing with No Codex credentials stored. Run hermes -p <bot> auth add openai-codex --type oauth while the root profile was logged in. Named profiles now resolve the root profile’s OAuth login again, while keeping their own credential scope.

Third, Desktop started a backend per profile. The app used to spawn a hermes serve for each profile — a pile of backend processes and duplicated MCP boot on a multi-profile machine. Desktop now attaches to the running host backend. That change explicitly depends on the profile-aware REST routes landing first: with the order reversed, the collapse sends unscoped destructive calls at the primary’s home directory, and the PR says so.

The update path converged too: hermes -p coder update and hermes -p writer update now restart the host gateway once between them, and the restart notice reaches every served profile.

The storage half: hermes sessions set-journal-mode

The same campaign (tracking issue #109417, 70-plus PRs across it) added a self-service command for converting state.db between WAL and DELETE journal mode offline:

hermes sessions set-journal-mode delete                          # WAL -> rollback journal
hermes sessions set-journal-mode wal                             # back to WAL
hermes sessions set-journal-mode delete --db ~/.hermes/kanban.db # another Hermes store

The premise matters: Hermes never live-downgrades a WAL database at open — the gateway, cron or a worker may hold uncheckpointed commits, and a downgrade underneath them destroys those commits. So a configured database.journal_mode: delete against an existing WAL store has always meant “did not apply, one ERROR per process”. The command’s admission test is simply “no process holds the file”: it lists every holder as pid — command: path and refuses otherwise, switches the mode without waiting out openers (one that appears mid-way makes SQLite refuse rather than race), and reads the header bytes back to confirm the mode. It reminds you to align database.journal_mode when the config disagrees, because the next open re-applies the configured value.

Post-upgrade checklist

hermes version                                  # expect v0.21.4 (tag v2026.9.21)
hermes gateway status                           # who serves which profiles
hermes gateway migrate --multiplex --dry-run    # upgrades from a per-profile topology: plan + blockers
hermes -p coder gateway run                     # should attach and print the host PID, not spawn a second process
hermes doctor                                   # storage and gateway-topology warnings surface here

Two more behaviours worth knowing: profiles created after the migration are served by the running multiplexer automatically (it re-scans profiles/ on create and delete, and every 30 seconds otherwise — around half a second in practice), and a host with an explicit gateway.multiplex_profiles: true is never second-guessed by the boot preflight — your decision, or the migration’s, stands. The GATEWAY_MULTIPLEX_PROFILES environment variable overrides the unset-key decision the same way.

The forced case: when --force is still correct

Multiplexing is on by default, but an unset flag is a request the host gateway settles at boot, not a verdict. Each start runs the same preflight as the migration: two or more profiles, no secondary still running its own gateway (a live process or an installed unit both count), no duplicate bot credential, no port-binding platform without a /p/<profile>/ ingress, and a host type the migration understands. If any check fails it comes up exactly as before, serving the default profile only, logging the blocker and the migration one-liner — and changing nothing on disk, converging by itself once the blocker is gone.

That leaves exactly two situations where --force is the right answer: a fleet split across UNIX users, and a HERMES_HOME outside profiles/. For a named profile, hermes -p <name> gateway install, start and run all exit 78 without --force and point you at the default profile’s hermes gateway install or the migration command; only a machine that already installed a unit with --force keeps reaching it from a plain gateway start.

The short version

The value of this convergence is not “one fewer process” — it is the boundary. One backend serving one profile means credentials, memory, sessions, hooks and cron all have to be profile-scoped, and every missed one turns into deleted data or a leaked key. So if your box runs more than one profile: run the dry run after upgrading, then walk the checklist above. For the wider picture of what else changed in this tag, read what v0.21.4 actually shipped; for the gateway surface itself, see the hermes gateway command reference and the earlier multiplex-by-default guide.