Three quiet gateway bugs: a typo that killed replies, a mask that ate job keys, and a sys.path shuffle


Your alert router pages you about a job landing in the wrong queue. You open the log to find the key, and it reads ITEM-000042:***. So redaction did that. You audit security.redact_secrets, your vendor prefixes, the bearer-token rules, and everything looks right. It takes a second pass to see it: nothing secret was matched. The pattern started in the middle of an ordinary identifier, treated the digits at its tail as a credential, and masked the whole key with them.

Three bugs, one layer

The gateway is the process behind Desktop and behind every chat-platform integration. The platform adapters — Telegram, WhatsApp, Discord, Slack, Raft — are thin translators between that process and a specific network API. They read config out of the environment, they write log lines that pass through the redaction filter, and they import sibling modules from the repo. Three ordinary jobs, three bugs.

All three were fixed in the same gateway-layer pull request (#151738, merged 2026-11-09). They look unrelated, and in a sense they are. But they share a layer and a personality: each one fails somewhere other than where the fault lives, so none of them ever surfaces as an error you can search for.

That is the first reason this class of defect is hard to find. An adapter regression does not look like an adapter problem: it looks like a bad job key, a bot that stopped answering, or a library reporting the wrong version. You spend the first hour in the wrong file.

Case one: one wrong character in .env killed the reply

Telegram’s “follow-up while busy” path needs to know how long to wait before deciding a message is a genuine follow-up rather than part of the run already in progress. That value comes from HERMES_TELEGRAM_FOLLOWUP_GRACE_SECONDS, and the code converted it with a bare float call.

So a value like abc — a typo, a leftover placeholder, or a 3s someone wrote because it reads nicer — raised ValueError: could not convert string to float: 'abc' at run_inbound.py:697. That exception was not caught at the boundary. It unwound the whole inbound handler, which means the user’s message got no reply at all. Not a late follow-up. A dead turn.

The fix routes that read through gateway.run._float_env, the same helper the other timeout knobs in that module already use. Bad input no longer raises: it falls back to 3 seconds, the follow-up is queued, and the agent keeps working.

# Wrong — all three are strings, not numbers
HERMES_TELEGRAM_FOLLOWUP_GRACE_SECONDS=abc
HERMES_TELEGRAM_FOLLOWUP_GRACE_SECONDS=3s
HERMES_TELEGRAM_FOLLOWUP_GRACE_SECONDS=
# Right — a bare number of seconds
HERMES_TELEGRAM_FOLLOWUP_GRACE_SECONDS=3

Why it hid: the path only runs when a second message arrives while a run is in flight, so most testing never touches it. When it fires, the error surfaces as “the bot ignored me”, which sends people to the token, then the webhook, then the network — not to a config value they typed weeks ago. Env vars have no type system, and nothing validates them until the line that parses them runs.

Case two: redaction started matching mid-word

Redaction is pattern matching with a promise attached: find the secrets, leave everything else alone. The second half is the part that breaks quietly.

A regex does not know what a word is. Unless you tell it to start only at a boundary, it will happily begin a match at any character that satisfies the pattern. The old Telegram pattern _TELEGRAM_RE was willing to start in the middle of an ASCII identifier, at the digit run attached to its tail. So an ordinary job key — something shaped like ITEM-000042:job-a12345678:q- followed by a long run of a characters — matched from the middle and was replaced, entire, with ***.

Nothing raised. Nothing warned. The log line lost its key and the on-call engineer lost twenty minutes.

The fix constrains where a match may begin: at an identifier boundary, or after one of the fixed-width escape separators the adapter actually emits (\n, \uXXXX, %XX, and friends). The digit-run lookbehind that stops the scanner from re-walking the same digits is still there — dropping it is how a redaction pass becomes quadratic on a long numeric string.

One test assertion had to be loosened, and the reason is worth knowing: JSON key-name handling on main now masks credential values before key names are processed, so an assertion expecting a key name to appear intact no longer describes what the code does. That is a test catching up to a deliberate change, not a hole being covered.

The lesson generalizes: if your redaction rules are a list of regexes and none of them anchor at a boundary, you have a bug waiting for the first identifier that ends in eight digits.

Case three: importing an adapter reshuffled the import path

sys.path in one sentence: the ordered list of directories Python walks, front to back, when you write import something, where the first directory containing a match wins. Position zero beats everything, including packages properly installed into the environment.

Installing any platform adapter — telegram, whatsapp, discord, slack, raft, plus gateway.platforms.base, cron.scheduler and tools.cronjob_tools — used to insert the repository directory at sys.path[0]. From that moment, any module name existing both in the repo and in the installed environment resolved to the repo copy.

The reproduction is unusually clean. Import the real evals package, then an adapter, then import evals again: it now binds to the repository’s own evals directory instead of the installed distribution. Ten of ten runs red before the fix, ten of ten green after, and the adapters no longer touch sys.path ordering at all.

This is the quietest of the three. No exception, no warning, no deprecation notice. The import succeeds and returns an object — just a different one than every other code path in the process expects. Anything depending on the installed distribution’s identity (version metadata, entry points, plugin discovery) starts returning answers belonging to another package, and the symptom shows up in whichever subsystem consulted that module, nowhere near the adapter that moved the path.

Before and after, in plain terms

Stripped of the test harness, here is what changed:

Probe Before After
HERMES_TELEGRAM_FOLLOWUP_GRACE_SECONDS=abc, message arriving mid-run ValueError kills the reply Bad value falls back to 3 seconds; the follow-up still queues
An ordinary job key like ITEM-000042:job-a12345678:q- plus a long run of a The whole key becomes *** The key survives verbatim
A real credential in the same log stream Masked Still masked
import evals after an adapter import, evals installed from PyPI Binds to the repo copy, 10 of 10 Binds to the installed package, 10 of 10

Read the third row before trusting any redaction change: tightening a pattern is easy, tightening it without dropping what it protected is the work.

Checking your own install

Two quick checks, because both bugs stay invisible until they bite.

# Which `evals` does this interpreter resolve?
python3 -c "import evals; print(evals.__file__)"

# Confirm the follow-up grace value is a bare number
grep -n FOLLOWUP_GRACE .env

If the first prints a path inside your checkout rather than inside site-packages, you are looking at exactly the failure above. For the second, a non-numeric value now degrades to the 3 second default instead of raising — but a silent fallback is still a wrong configuration, so fix the value rather than leaning on the fallback.

For the redaction half, grep your own logs for *** and read the surrounding lines. A masked Authorization header is the system working. A masked job key, request id or trace id is the system working too hard, and no amount of configuration review will reveal it.

Who fixed what

Three fixes, three origins, merged together:

  • The redaction boundary fix (#145664) was cherry-picked from @liuhao1024, with a patch from @yuzilongleif-collab.
  • The follow-up grace fix (#50120) was cherry-picked from @Bartok9’s #51324.
  • The sys.path fix (#117698) builds on @beardthelion’s #117699.

The short version

A bug is hard to find in proportion to how far its symptom sits from its cause. A typo’d float in an env var looks like a broken bot. A regex that starts mid-word looks like a working mask. A reordered import path looks like a working import.

Take the checking habits rather than the three patches. Print __file__ for the modules you depend on. Never let a config value reach a parser without a fallback. Anchor every redaction regex at a boundary and re-read what it matched the last time it fired. All three would have been caught in minutes by any of those habits.

Related reading: why a cookie in a log line is a login, what a Telegram 409 used to cost, how blocking calls froze the gateway event loop, and the configuration reference.