The network blipped overnight and your Hermes gateway lost its Telegram connection. In the morning the gateway is up, every other platform is fine, and Telegram has been sitting at “reconnecting” ever since. You restart the gateway and it comes back. This class of bug is the worst to diagnose — nothing crashed, there is just a retry queue that will never make progress.

Three fixes merged on October 5, 2026 (#133289, #133228, #133230) cover exactly three such cases. They share a shape: the reconnect machinery was fine, but three exits were not covered, so “it heals itself” turned into “it stays down” under specific conditions.

One: a platform whose plugin failed to load never retried the import

The gateway already handles the first half: an enabled platform whose plugin fails to load at startup gets flagged and queued on the reconnect watcher as retrying / adapter_unavailable. That queue keeps retrying.

The problem was what it retried. Platform plugin loading is deferred: the registry holds a load lease, and the actual import happens at the first lookup. If that deferred load raises, or overruns its deadline, the lease is disposed and the registry simply forgets the platform. The watcher kept checking for an adapter, kept finding none, and had nothing that could re-run the import — only a forced re-discovery (reload-plugins, the dashboard, or the plugin tool) re-leases the loader. The platform stayed queued indefinitely while the log printed Reconnect <platform>: no adapter yet, next retry in Ns.

The fix adds rearm_failed_platform(): it finds the platform manifest recorded as a failed load and re-registers the deferred platform, so the next registry lookup retries the import. Three safety valves matter:

• It re-gates before re-arming. A placeholder (disabled, not enabled, catalog-removed) and a plugin that fails requires_hermes have the same error-set shape as a failed load, and they are never re-armed.
• It never stacks threads. If a hung plugin-load: worker for that plugin is still alive, the re-arm is skipped rather than leaking a second abandoned thread.
• It is capped. The watcher re-arms at most 3 times per queued platform (_MAX_PLUGIN_LOAD_REARMS = 3): a transient load failure heals, a genuinely broken plugin does not re-import forever.

Both the re-arm and the registry lookup that materialises the plugin run in an executor, off the event loop, with profile context variables kept. The line to look for in your log is Re-arming failed platform plugin load: <name>.

Two: Telegram messages held during a disconnect were stranded on the retired adapter

Telegram has an awkward property: an update that python-telegram-bot has already acknowledged cannot be fetched again. So Hermes holds acknowledged-but-unprocessed events in a queue on the adapter instance, and that instance drains the queue itself when it reconnects.

The queue belongs to the adapter instance — and a retryable fatal error (a network error, a polling stall, and since a later change an ingress stall) makes the runner build a new adapter. The old reconnect path carried the dedup caches over to the replacement but not the hold queue, so events held before the rebuild — plus anything teardown salvaged onto the retired adapter afterwards — were stranded there forever.

The gap dates back to August 11, when the per-instance hold queue was introduced; a September 23 change carried dedup state over but not the queue. The fix keeps the replaced adapter on the reconnect queue entry as predecessor and, at publish time — when the replacement is actually registered — calls hand_over_held_inbound() to move the held events across. Worth knowing:

• a candidate adapter that fails to connect never takes ownership of the queue, and delivery waits until the replacement is registered;
• the handover rebinds only the transport-layer reference; profile and role authorization stay as they were resolved;
• if the retired adapter still receives salvaged events after the handover, it forwards them to the live successor, first accepting its own in-flight update claim so the receipt is still recorded;
• the helper is a no-op for adapters that have no hold queue — only Telegram has one.

Three: in-flight /v1 runs were killed after 0.00s on gateway stop

The third is not “stuck”, it is “killed too early”. Stopping or restarting the gateway runs a drain phase so in-flight work can finish. That drain put api_server (/v1) runs on the same budget as chat turns, and that budget — restart_drain_timeout — defaults to 0, because an interrupted chat turn is acceptable: it gets announced and can resume.

A /v1 run cannot resume. It has an HTTP caller blocked on the result. With the 0 default the drain loop never actually ran, so the shutdown path interrupted every in-flight /v1 run and failed the waiting caller.

The fix puts api_server runs on the same deadline as cron jobs (the agent.cron_drain_timeout floor, watchdog-clamped). No new counter and no new config key. The stop-path “waiting up to Ns” log line now also fires when only api_server runs are in flight, and includes their count. Note the opt-out: cron_drain_timeout: 0 opts /v1 runs out of the drain as well, and an interrupted /v1 run fails its waiting caller.

Symptom to behaviour

Symptom you may have seen	What happens now
A platform plugin fails to load at startup, the platform sits at adapter_unavailable, and only a restart or reload-plugins brings it back	The reconnect watcher re-arms the load each tick, up to 3 times, and the platform heals once the cause is gone
A genuinely broken plugin (version mismatch, removed from the catalog) spamming the log	It is not re-armed, and retries do not pile up
Telegram reconnects and messages sent during the disconnect seem to vanish	Held events are handed to the new adapter and delivered
Restarting the gateway fails every in-flight /v1 call	They finish within the drain deadline; only a timeout or an explicit opt-out interrupts them

The knobs for gateway behaviour in general (reconnect intervals, caps, the watchdog) are in the gateway command reference. If you have chased a “process alive, platform silently gone” failure, the silent disconnect heartbeat post covers the adjacent half, and the loop watchdog guide explains how the drain and watchdog work together.

Release status

All three fixes are on main and in no published tag (the newest release is v0.21.5, tag v2026.9.24). Git installs get them by pulling the latest main; Docker images and hosted deployments get them with the next tag. If you run several gateways, the one gateway per host rule still applies.

The design lesson

All three come down to the same judgement: a retry has to retry the right layer. A live queue with nothing that re-runs the import is not a retry; carrying dedup state but not the payload is not a handover; a drain loop with a zero budget is a kill switch. When you audit your own self-healing paths, do not stop at “is there retry logic” — check that each attempt actually moves the state forward, and that it has a ceiling, so a recoverable fault and a permanent one do not get treated the same.