Monday morning, you open your laptop and find that last night’s cron jobs never ran — not with errors, just no execution records at all. You SSH in and the gateway process is clearly alive, the port is open, yet every message you send vanishes into silence and the log sits frozen on its last line. That is far more frustrating than a crash: when a process dies, systemd restarts it in seconds, but a process that is alive yet wedged looks perfectly healthy to your monitoring.

Hermes Agent ships with a built-in answer to exactly this scenario: the loop watchdog. A dedicated OS thread watches the gateway’s event loop and, when it detects the loop is frozen, deliberately kills the process with a service-restart exit code so your supervisor revives it. And in PR #92317, merged on August 22, 2026, the team finally wired this mechanism’s tuning parameters through the real config loader — previously you could write them all day and nothing would change.

Crashes are easy to fix; wedges are not

Two words first. A crash means the process exits: the port closes, monitoring notices instantly, and systemd/launchd KeepAlive pulls it right back up. A wedge is different — the process stays alive, but the asyncio event loop (the “heart” of an async program, where every task queues up to run) is blocked by some call that never returns.

Here is the trap: every recovery path built on the event loop — timeout retries, state rewrites, error logging — needs the event loop to be running in order to fire. The more you need recovery, the less you can have it. The process doesn’t die, so no alarm rings, and a half-dead gateway just sits there until a human shows up.

How the watchdog watches the loop

Hermes’ approach (source: gateway/shutdown_watchdog.py) sidesteps the loop entirely: a plain OS-level daemon thread keeps watch from outside, in three steps:

1. Probe — every loop_watchdog_probe_interval_s seconds (default 30), the watchdog injects a probe into the loop via call_soon_threadsafe. That call is thread-safe, so it lands even when the loop is busy.
2. Strike — the watchdog then waits up to loop_watchdog_probe_timeout_s seconds (default 10) for the probe to be processed. Healthy loop → probe handled instantly, strike counter resets. Frozen loop → probe never handled, one strike.
3. Hard exit — after loop_watchdog_max_strikes consecutive misses (default 3), the watchdog dumps all-thread stack traces via faulthandler (invaluable evidence for later forensics), records reason=loop_liveness_watchdog in the lifecycle ledger, and force-exits with exit code 75 — the dedicated service-restart code, which tells systemd/launchd to bring the gateway back up.

At the defaults, roughly 90–120 seconds of sustained loop blockage triggers automatic recovery. Compared with discovering the failure the next morning, that reaction time is genuinely useful.

The three new knobs — actually wired this time

Before #92317, this mechanism had an awkward flaw: the gateway.loop_watchdog switch and its parameters had existed in the config defaults for a while, but the config loader never read any of them — no matter what you wrote, the watchdog ran on hardcoded defaults, and you couldn’t even turn it off. The PR bridges the keys into the real load path (top-level-wins with nested fallback) and adds bounded validation: NaN, Infinity, and oversized values now degrade safely to defaults instead of crashing the config load.

Four settings are now tunable under the gateway section of config.yaml:

Key	Default	Meaning
gateway.loop_watchdog	true	Master switch; set false to disable entirely
gateway.loop_watchdog_probe_interval_s	30.0	Seconds between liveness probes
gateway.loop_watchdog_probe_timeout_s	10.0	How long a probe may go unprocessed before counting as a miss
gateway.loop_watchdog_max_strikes	3	Consecutive misses before the hard exit

Tuning from the CLI

No need to hand-edit YAML — hermes config handles it:

# Check the current value
hermes config get gateway.loop_watchdog_max_strikes

# Your machine is heavily loaded and occasionally stalls:
# give a probe more time before counting a miss
hermes config set gateway.loop_watchdog_probe_timeout_s 20

# Allow 5 consecutive misses before acting
# (~2-3 minutes of sustained block before recovery)
hermes config set gateway.loop_watchdog_max_strikes 5

# Prefer no watchdog at all?
hermes config set gateway.loop_watchdog false

# Changed your mind — back to defaults
hermes config unset gateway.loop_watchdog

Restart the gateway after changing the config. Note that hermes config unset removes the key entirely so it falls back to the built-in default — cleaner than manually setting true again.

When to loosen, when to hold the line

Loosen when: you talk to remote or slow model providers whose single requests occasionally hang for a while, or your machine is so loaded that the event loop stalls for seconds at a time. Occasional missed probes are not deadlocks, and a larger probe_timeout_s or max_strikes cuts down on false kills.

Do not loosen to hide a real wedge. The whole point of the watchdog is fast recovery; raising max_strikes from 3 to 8 stretches recovery 2–3×, and every extra minute a wedged gateway sits there means more piled-up cron jobs and undelivered messages. The team is also fixing the false-positive class at the root (moving the watchdog’s own heartbeat write off-loop, with a two-witness probe — PR #90502 is still under review), which is why the default stays tight.

And if “tasks stall mid-run” is your recurring pain, that’s a different layer: the session-level heartbeat and goal gates cover it.

A heartbeat file for your own monitoring

If you run external monitoring (Uptime Kuma, Prometheus, or a one-line cron check), the gateway also atomically rewrites a heartbeat file at <HERMES_HOME>/state/gateway.heartbeat on a cadence. It serves two purposes: it lets external supervision distinguish “process alive” from “loop working”, and it doubles as a rolling pre-death telemetry snapshot — after an unclean death, the last heartbeat is the closest thing to a crime-scene photo.

Check freshness with one command:

# Linux
stat -c %Y ~/.hermes/state/gateway.heartbeat
# macOS
stat -f %m ~/.hermes/state/gateway.heartbeat

Compare that timestamp with the current time. If it hasn’t moved in a minute or two, the loop has likely stopped doing real work — the watchdog is probably about to act, and you just got an early warning.

Release status

These tuning knobs live on main right now (PR #92317, merged 2026-08-22) and are not yet in any release tag — the latest v0.20.5 (tag 2026.8.19) still ships the old fixed behavior. If you want to try them, wait for the next release and run hermes update, or follow the GitHub releases page — our v0.20.5 release notes summarize that release.

The gateway is the heart of every Hermes cron job, messaging bot, and remote session (new to it? see our cron automation guide and the hermes-gateway command reference). Give it a watchdog configuration that fits your environment, and at least you know the system will pick itself up at 3 AM — instead of waiting for you to find out at 9.