You know the session that stayed open for three days? The API naming you argued about, the exact error string you pasted, the reason you rejected one approach — all of that was there on day one. On day three you ask “how did we fix that timeout again?” and get a confident, brand-new answer. You did not change models. You did not change the prompt. What changed is the context: it was compacted many times while you were not looking.

The real question is what compaction grinds away, and at what point that starts to hurt. Feelings are a bad instrument for this. On September 19 the repository gained a full controlled eval (evals/compaction/) that answers it with numbers from real sessions.

How the eval works

The material is three real 500K-token session lineages, reconstructed from a local state.db: a PR review campaign, a system-prompt token analysis, and a SIGSEGV investigation. Compaction rolls history into a summary, so the eval first writes 15 factual recall questions about the region that is about to be summarised, then hands only the post-compaction context to one model that answers them, and a second model that grades against gold. Two points for correct, one for partial, zero for wrong.

That produces two axes together: recall accuracy and how many tokens were retained to get it. Keeping more context and scoring higher proves nothing on its own; only the pair tells you whether the compression was worth it.

Results: the shipping path scores 78.9% and costs less per turn

Policy	Mean recall	Retained tokens	Per-compaction time / cost
Summary only (no recovery pointer)	43.3%	55K	37 s / $0.061
Summary + session-search recovery (shipping)	78.9%	55K	37 s / $0.061
Lean tail, recency only	41.1%	55K	33 s / $0.062
Third-party Jev arm (default threshold)	75.5%	115K	1.4 s / $0.007
Jev with a 60K tool budget	77.8%	176K	1.5 s / $0.007
Plain recency with the same 60K budget	77.8%	177K	0 s / $0

First, “session-search recovery.” When Hermes compacts, you do not only get a summary: the summary carries a pointer, because the archived region is still in the store and the same full-text index can pull it back. The eval gives the answering model exactly one search round-trip — that is the second row. The identical summary with no retrieval scores 43.3%. Those 35 points are the difference between “the archive still exists” and “the archive does not.”

Now the third-party policy, which comes from a community idea: write no summary at all, use a small decision model to score every tool call and result, drop the stale ones, and keep user and assistant text verbatim. The pitch is strong — closed-book it beats the summary by 32 points (75.5% vs 43.3%) at one ninth the compaction cost and one twenty-fifth the wall time.

Put the retained tokens next to it and the accounting changes. It carries 2.1x the context, and that context gets re-billed on every subsequent turn. More interesting: the eval found the model’s judgement did not actually do the work. At the default threshold it dropped 100% of 851 candidates, which makes the default arm behaviourally identical to one crude rule: “delete every old tool result, keep all dialogue text verbatim.” What rescued the score was verbatim text, not scoring. Widen the tool budget to match the recency arm and the two tie exactly (77.8 vs 77.8) — the ranked judgement lands inside the noise floor.

The real failure mode is not one compaction, it is the hundredth

A one-shot score cannot see the long-term failure mode, so the eval added a repeated-compaction simulator: feed a lineage chronologically, compact every time the estimate crosses the threshold, and record how much each round freed and how much the floor grew.

Run	Cycles	Freed per cycle	Text floor	End state
PR review @ 500K window	32	89% → 55%	43K → 139K	still working
System-prompt analysis @ 500K	40	76% → 20%	91K → 240K	degrading
SIGSEGV investigation @ 500K	40	63% → 8%	175K → 359K	40K freed per cycle
PR review @ 160K window	60	87% → 22%	12K → 87K	compacting every ~14 rows
SIGSEGV @ 160K window	11	52% → 0%	85K → 151K	STUCK (floor reached the threshold)

Look at the third column. Freed space decays because dialogue text is never removed, and it only grows. So each cycle can free at most “threshold minus text floor,” while the floor climbs. On a 200K-window host the SIGSEGV lineage hit the hard stop after 11 cycles — having consumed only 0.42M tokens of work. The 500K-window run reached cycle 40 by compacting every ~40K tokens of new work, with every turn billed at roughly 460K input tokens.

And one transcript with 541 tool calls never fit the plugin’s 25K-token state ceiling even once, falling back to a summary on the very first attempt.

The verdict is “change nothing” — but three things follow for you

The eval’s recommendation is to keep the current design: the shipping path (summary plus recovery pointer plus a lean tail) wins on both recall and retained cost, and carrying more context per turn just to delete fewer tool results is a bad trade. None of that is a knob you configure, but three practical consequences are.

One: keep identifiers in dialogue text. The facts the summary lost were strikingly regular — delegation session ids, root causes, config key names, verbatim error strings. They sat in assistant text the whole time; it is tool output that gets rolled away. Next time you ask the agent to report, ask it to paste the id and the raw error, and stop hunting through tool output later.

Two: session search is a free safety net, so use it. That is where the 35 points came from. After a compaction, “how did we fix that timeout” is still answerable by the full-text index, as long as it really happened — a point the session-boundaries post also makes. The commands live in the session management reference.

Three: split long jobs into phases. Both the hard stop and “compacting every 14 rows” come from piling an entire project into one session. Finish a phase, start a new session, carry the conclusions across — that is far more stable than making one session carry everything.

If what you care about is when compaction fires, the two knobs are compression.threshold (default 0.50; models below a 512K window are floored at 0.75) and compression.tail_mode (default lean). We unpacked their behaviour in the lean-tail post and the one-call compaction post. This eval did not change any default — it is the first time anyone has shown, with numbers, why the defaults look the way they do.