One 16 KB Script Froze a Gateway for 23 Seconds: The Command-Scan Postmortem

You hand Hermes a few-hundred-line script — a Python heredoc that inlines document text into JSON payloads — and ask it to run. The terminal goes quiet. A while later you ask for progress in Telegram and the bot does not answer. In the log you find a set of lines that explain nothing: deadline (15s) expired but the event loop has not processed the expiry, the loop thread appears BLOCKED in a synchronous call, then the shutdown watchdog, an exit, a restart — and then the agent retries the same command and the whole thing starts over. It looks like a network problem or a messaging-platform failure. It is actually about something that sounds purely like safety: dangerous-command detection.
The symptom: not “slow”, but an event loop held down
Before it runs a command, Hermes has to decide whether that command is dangerous — which determines whether it sails through, prompts you, or is blocked outright. That check runs in Python’s regex engine and holds the GIL. The longer the command, the longer it occupies the process; and what it occupies is not one command’s slot but the entire event loop.
A user reported it with full reproduction material (#113535, 16 September 2026, official image 0.20.6), and the failure chain is worth reading once:
- Detection takes minutes on a long command, starving the event loop.
- Telegram polling times out; the
telegram-init15-second deadline expires but the loop cannot process the expiry. - The shutdown watchdog concludes the process is wedged, exits 75, the gateway restarts.
- On restart the agent faces the same unfinished command and runs the same detection again — and the cycle repeats.
That last step deserves emphasis: “stall, restart, retry, stall again” looks exactly like a broken messaging platform, which sends people looking in entirely the wrong place.
Root cause: one regex starting at O(n²), multiplied by a variant explosion
The reporter’s analysis is blunt. Two factors compound:
First, an unanchored whole-string lookahead. The launchctl gateway-lifecycle rule was written as two independent whole-string lookaheads:
(r'(?=[\s\S]*\blaunchctl\s+(?:stop|kickstart|bootout|unload|kill|disable|remove)\b)'
r'(?=[\s\S]*\b(?:hermes|ai\.hermes)\b)', "stop/restart hermes launchd service ...")
The intent is “both a launchctl stop-ish verb and the word hermes/ai.hermes appear, in any order”. But neither lookahead is anchored, so re.search retries them at every start offset, and each attempt rescans to the end of the string — one search is O(n²). Measured on a 60,018-character string: 16.5 seconds for that single pattern; with an anchor, 0.001 seconds.
Second, “command word variants” explode with the number of quotes. _command_detection_variants yields a full-length variant for every quoted or escaped command word so it can match the deobfuscated form. A quote-heavy heredoc ("key": "value" style JSON) has hundreds of quoted words: 230 lines produce 233 variants. Detection scans every variant with every pattern — and with both passes (hardline and dangerous), the total is O(n³).
The reported measurements:
| Command length | Variants | Total regex time |
|---|---|---|
| 3,000 chars | 41 | 2.7 s |
| 6,000 chars | 82 | 27.8 s |
| 15,369 chars | 201 | about 7 minutes (tool timed out at 420 s) |
So this was never “a bit slow” — it is quadrupling or worse for every doubling of length.
The fix: one anchor and one combined variant, with identical verdicts
PR #114208, merged on 17 September 2026, dismantles the explosion with two changes:
- Anchor the launchctl lookaheads with
\A. This does not change semantics: a lookahead evaluated at offset 0 already sees the whole string, so the match set is identical — reversed token order still matches, multiline input still matches. One search simply goes from O(n²) to O(n). - Build one combined variant instead of one per word. Every command word is deobfuscated and applied as one candidate; spans arrive out of order (loop bodies after their keywords) and can nest (a backtick word and the command inside it), so replacements are applied sorted, and a span overlapping an already-applied one keeps its own per-word variant; overlapping spans wait for the next round (one combined variant per nesting level), so nested-substitution scripts cannot re-explode either. On the reporter’s input the variant count went from 923 to 4.
The measured result: a 16 KB / 460-line heredoc drops from roughly 23 seconds (7.8 s hardline + 15 s dangerous) to about 0.4 seconds, with identical verdicts. The hardline pass keeps its unbounded YOLO semantics — a benign heredoc is neither blocked nor prompted, before or after.
That “identical verdicts” part is the most important claim in the whole fix. An approval gate is a security boundary, and you do not make it fast by loosening it — letting one should-have-been-blocked command through is far worse than stalling for 23 seconds. The right repair removes wasted work, never the check itself.
Why the rule is so broad in the first place
Read the comment in the source and the conservatism makes sense: an earlier version required “hermes” to appear after the verb, and a for-loop that assembled the label from a list defined earlier (for item in 'ai.hermes...'; do launchctl bootout "$label") slipped past and restarted four gateways with zero approvals. For an approval gate, erring broad is correct — an extra prompt is cheap; a miss can kill every running agent.
That is exactly why this fix only touches performance and not breadth. It is the reason it can be trusted.
When you will meet it
- Long scripts. Few-hundred-line heredocs and commands inlining large JSON or document text hit this path. Everyday short commands do not.
- Agents on a gateway. Because detection holds the GIL, the victim is not just the current command but every messaging channel, heartbeat and watchdog in the same process. In a CLI you wait; on a gateway the bot goes dark and restarts in a loop.
- Log keywords to grep for.
loop thread appears BLOCKED in a synchronous call, expireddeadline, watchdog exit code 75. When those appear together, suspect synchronous blocking, not the network.
Release status
The fix merged into main on 17 September 2026. The reporter confirmed the offending pattern exists unchanged in v2026.9.14 — that is the current release, v0.21.3 — so the build you are running today still stalls; the fix is not in any tag yet.
hermes --version # v0.21.3 or earlier: long heredocs still hit this
hermes update # fixed once the next tag lands
Until then, if you need the agent to run a very long script, write it to a file first and have the agent execute that file rather than pasting the whole body onto the command line — that path skips the whole-string scan entirely.
Further reading: for the gate as a whole, approvals when nobody is watching covers modes and policy; why SSH config edits need approval is one real rule’s journey from incident to code; dry-run testing approvals lets you check whether your rules misfire; and if you are curious how a wedged gateway gets brought back at all, the gateway loop watchdog is the operations side of the same story.
Summary
- Symptom: a long heredoc pins the gateway’s event loop inside synchronous regex, showing up as a silent Telegram bot, expired
telegram-initdeadlines, watchdog exit 75, and a retry loop after restart. - Root cause: two unanchored whole-string lookaheads made one search O(n²) (16.5 s on a 60 K-character string), while
_command_detection_variantsproduced one full-length variant per quoted word, taking the total to O(n³) (82 variants and 27.8 s at 6,000 characters). - Fix (PR #114208, 17 September 2026): a
\Aanchor plus a single combined variant, 923 variants down to 4, a 16 KB heredoc from 23 s to 0.4 s — with identical verdicts. - Status: the released build (v0.21.3) is still affected; the fix is on
mainand not in a tag.