You ask your bot something in a group chat, get an answer, and then — ten minutes later, right after you restarted the gateway — the same answer arrives again. Your first thought is about money: did the model run twice, and did I pay twice? Your second thought is about trust: if it answers twice now, what else is it repeating?

Both questions have concrete answers, and the mechanism behind them is one of the more carefully designed parts of the gateway. Two bugs in it were fixed this week (#120444 and #120450), so this is a good moment to understand what you are looking at.

The delivery ledger, in plain words

Around every platform send, Hermes records the final response in a durable delivery ledger living in state.db. If the gateway crashes or restarts between producing a reply and the platform confirming receipt, the next boot has three options: lose the reply, re-run the whole turn (costly and possibly wrong), or redeliver what it stored. It redelivers.

The semantics are deliberately honest rather than clever — at-least-once, with ambiguity labelled instead of guessed at:

• A response whose send never started is redelivered as-is, with nothing added. This one is safe to send exactly once.
• A response that was mid-send when the gateway died — the platform may or may not have received it — is redelivered with a visible ♻️ Recovered reply — … may be a duplicate prefix.
• A send refused by flood control (Telegram rate limits, for example) is retried automatically after the recorded penalty expires, no reconnect or restart needed, and the recovery prefix warns that earlier chunks may already have arrived. The ledger cannot infer partial delivery from message length, so it says so.
• Any other rejected final send — a platform 5xx, an unclassified error — is retried after a growing backoff of 30 seconds, then two minutes. The last budgeted attempt is left for the next gateway start, so an outage that outlasts the timer never strands the reply. A permanently unreachable chat (blocked bot, deleted group) is not retried at all.

Redelivery is bounded: three attempts and 24 hours of freshness, after which the row is abandoned. Delivered rows are pruned after seven days. Retries keep the original bot profile, chat and thread, and a multiplexed gateway keeps every served profile’s replies in the home it was started from.

What the marker tells you

This is the practical takeaway: a labelled duplicate means the model did not run twice. The gateway recognised that a send was already in flight and chose to resend the stored reply rather than re-run the turn. The reply you are seeing twice is one model run, delivered twice, with a prefix that admits it.

So when you get a double answer, look for the ♻️ Recovered reply prefix before you blame your provider bill. If it is there, the honest accounting is a duplicate message, not duplicate spend. If it is not there and the two replies are separated by a restart, you were hitting the bug below.

Bug one: a crash mid-redelivery sent a second, unmarked copy

The ledger marks rows as it claims them. A gateway killed while its boot sweep was redelivering a stored reply used to leave a row that the next boot could claim again and send without the recovery prefix — the worst outcome, because the duplicate was unlabelled and therefore indistinguishable from a real second run.

The fix moves every claimed row to attempting inside the same owner-stamp compare-and-swap that claims it. Previously only flood rows got that treatment. The needs_marker decision still reads the row’s pre-claim state, so a row that was never claimed still goes out plain and exactly once — the behavior the first bullet above promises is preserved, not traded away. A leftover pending row with attempts greater than zero, which an older build had already claimed and possibly sent, is now marked too.

There was a sibling race in the same mechanism: a failed row claimed at boot stayed failed and owned by the live process while its boot send was in flight. The runtime reconnect sweep — which takes failed rows — could then claim it and send it a second time. The row now stays attempting for the whole send, so the boot claim is exclusive.

Bug two: a reconnected adapter forgot what it had already seen

The second duplicate had nothing to do with outbound delivery. When a messaging platform connection drops, the gateway’s reconnect watcher replaces the failed adapter with a new instance. Inbound deduplication — the MessageDeduplicator that drops an inbound message ID it has already seen — lived on the adapter instance. A brand-new adapter therefore started with an empty cache.

That matters because platforms replay. A websocket resume, a webhook retry or an unacknowledged poll batch can deliver the same inbound message again right after the reconnect. With an empty cache, the gateway admitted it as new, ran a turn, and answered a second time.

The fix seeds the rebuilt adapter’s dedup caches from the instance it replaces, adopting the still-live IDs at their original seen times. The subtle part is “live”: the cache entry keeps its original timestamp rather than being refreshed, so the dedup window does not silently extend just because the adapter was rebuilt. It is a small, complete fix for a whole family of “why did it answer twice” reports.

What to do about it

• Read the marker first. A ♻️ Recovered reply prefix means one model run and two deliveries. Budget-wise you are fine; the reply is duplicated, not the spend.
• Unmarked duplicates after a restart or a reconnect are what these two fixes address. Pulling main — or waiting for the next tagged release, since both landed after v0.21.4 — is the remedy.
• If you would rather lose a reply than ever duplicate one, gateway.delivery_ledger: false turns the ledger off and restores the older behavior: in-flight responses are lost on a crash. It is a real trade, not a free switch — for a bot where a duplicate is unacceptable and a lost reply is acceptable, it is the right one; for a support bot, it is probably not.
• Nothing else to configure. There is no dedup window to tune and no retry count to set; the bounds above are the design.

The wider picture on how replies reach a platform is in our messaging delivery cadence guide (batching and backlog behaviour), with media attachment delivery for files and images and background notification batching for how scheduled work reports back. If you run a gateway that restarts unexpectedly, the loop watchdog guide explains the recovery path a restarting gateway goes through.