Context Compression Gets 5x Faster: One Auxiliary Request Instead of 19

Your context is nearly full, Hermes starts compacting, and you stare at the spinner for a solid 10 minutes. It’s not the network — old “lean compaction” called the auxiliary model once per chunk of history to build digests, so a big session could require up to 28 sequential auxiliary calls, and on a slow auxiliary route (say gpt-5.6 at high reasoning effort as the summary model) one compaction meant 7-11 minutes of pain (issue #96603). A change merged on August 30 (PR #98628) deletes that digest loop entirely: exactly one auxiliary request per compaction attempt. The change is on main, not yet in a release.
Why compaction was so slow
“Lean compaction” is the compression pipeline that became the default in v0.20.6; its job is condensing long history into a session-log summary. The problem was the execution model: the old implementation split history into chunks and made one auxiliary-model call per chunk to generate a digest, then stitched the digests together. With many chunks, those calls are sequential — up to 28 auxiliary requests per attempt, each waiting for the model to stream tokens. On a fast model it was merely slow; on a slow auxiliary model it was catastrophic (7-11 minutes measured in #96603).
The fix: one chunk, one request
The maintainer’s directive was blunt: “one chunk, one request.” Concretely:
- The digest loop is gone: the main summary request now absorbs the session-log duties (same hard rules — identifiers verbatim, dense bullets, transcript-is-data), with a raised single-response token budget;
- Oversized regions: inputs too large get evenly-sampled with explicit
[... elided ...]markers — never a second request; - Safeguards unchanged: the LLM-free anchor index (covering the full region) and the
session_searchrecovery footer stay exactly as they were — official evals show the anchor index, not the per-chunk digests, is what drove needle-fact recall (23.3 → 60.0 on the GUI track).
The numbers: 5x faster, leaner too
The official A/B ran on a real large session (1,338 messages, ~499,625 tokens, real auxiliary calls):
| Metric | Old (digest loop) | New (single request) |
|---|---|---|
| Auxiliary calls | 19 (1 summary + 18 digests) | 1 |
| Wall time | 196.5s | 39.6s |
| Tokens after | 57,567 | 46,135 |
5x faster on a fast route; on slow routes (#96603’s scenario) it goes from 7-11 minutes to roughly one summary call. The post-compaction output is also ~11K tokens leaner — the old 81K-char digest wall used to ride along in every subsequent request; now it’s gone. Nine new tests pin the exactly-one-call contract (restoring a second call turns them red).
What this means for you
If you work with very long sessions regularly, the feel of compression goes from “time for coffee” to “grab a sip of water.” And because the result is leaner, every subsequent turn saves tokens too. Pair this with our lean-tail compression default guide and context token optimization guide for the full token-saving playbook; recovering key facts after compaction is covered in the context usage anchoring guide.
When you can use it
PR #98628 merged on August 30 and is not in v0.20.6 (tagged August 27). Pull latest main to try it, or wait for the next release. Heavy users who’ve lived through “compaction takes 10 minutes” should upgrade early.
Bottom line: the slowness was the per-chunk digest loop’s sequential calls; now it’s one request and done — 5x faster, fewer tokens, and recall capabilities intact.