You set up a cron job on Sunday night. Monday morning you open a terminal to read the results, and the first command just sits there. Twenty seconds later it gives up with database is locked. You are not running two Hermes instances. You never touched the database by hand. So who took the lock?

A batch of fixes merged on September 11 answers that, and the answer is counterintuitive: the lock was Hermes taking write authority on itself every time it opened the session store — and a handful of processes were never supposed to touch that file in the first place. Below: who was taking the lock, who was quietly writing, and what got misjudged.

1. Opening the store took a write lock even when nothing needed writing

Background: Hermes keeps every session in a SQLite file called state.db — session_search, hermes sessions, exports, and insights all read from it. SQLite’s concurrency rule is blunt: a statement that could write takes the write lock, even when it matches zero rows.

That was the trap. Every read-write open of state.db used to also run a set of statements that changed nothing on a settled database:

• inserting the generation stamp (INSERT OR IGNORE);
• UPDATE messages SET active=1 WHERE active IS NULL;
• upserting the FTS storage-version marker;
• DROP TRIGGER IF EXISTS plus CREATE TRIGGER for four display triggers.

So a perfectly healthy database with nothing pending still made sibling processes — a cron tick, a hermes sessions call, a doctor check — wait out its whole transaction. Past the patience window, the open failed outright with database is locked.

The fix is to read first and write only when something is genuinely stale: insert the stamp only when it is absent, heal active IS NULL only when a NULL row exists, skip the marker when the stored value already matches, and recreate triggers only when the definition stored in sqlite_master.sql differs from the desired one.

The numbers from the pull request make the point:

Scenario	Before	After
Settled database, sibling holds the write lock for 4 s	open took 4.14 s	0.01 s
Holder outlives the patience window	database is locked after 20.5 s	opens normally

The stall was never “someone else is using the database”. It was “we asked for write permission we did not need”.

2. Who was quietly acting as a second writer

The same batch found four roles treating the main session store as scratch paper:

• The cross-profile group-chat coordination store lived on the root gateway’s state.db, while every profile gateway starts that worker unconditionally — which made each profile process a long-lived writer on the master session store. It now uses shared-state.db under the install root.
• The web dashboard now reconciles the schema on startup through a read-only open. A healthy store sees zero writable opens; only a genuinely broken schema takes the single writable pass.
• The cron lifecycle guard scanned command text for path-like tokens and read them with a raw os.open. A command that merely mentioned the gateway’s state.db could cancel the gateway’s POSIX locks. Those reads now go through the tracked offline-file-access registry.
• hermes doctor --fix now checks for a live writer before its WAL checkpoint. Under a running gateway it skips the checkpoint with an actionable finding instead of joining the WAL as a second connection.

Together these explain why database is locked kept coming back on real machines: not one bug, but one database with several writers.

3. A healthy WAL was declared dead

The third class is subtler. Hermes has a guard: if the WAL sidecar files of state.db get unlinked, it falls back to JSONL storage — slower, but safe. The old evidence was the " (deleted)" suffix in /proc/<pid>/fd entries.

On OpenZFS, a live and still-linked -wal/-shm file is reported exactly that way (dentry unhashed, nlink=1). So the guard fired permanently and the gateway ran on JSONL forever — degraded, and completely silent about it.

The check now compares the file descriptor’s (st_dev, st_ino) against the path currently being watched. A real orphan kept alive by a hard link is still caught, so the guard stays fail-closed.

Two races in the same family were fixed as well:

1. The write path ran its “sidecars disappeared” probe before taking its own lock, while close() unlinks the sidecars under that lock. A writer racing a clean shutdown tagged itself with the sticky “WAL generation lost” flag. The probe now runs inside the lock — eleven lines.
2. When the WAL probe was indeterminate, the helper returned "wal" as the assumed mode, and SessionDB enabled its lock-free read pool on that assumption. If the file was actually in DELETE mode, the pool was wrong. The header must now be confirmed; when it cannot be, reads queue on the writer lock instead.

4. One bad row should not destroy the whole table

Locking aside, there was an avalanche point: a single bad row could kill an entire command. The shapes seen in the wild include a TEXT garbage timestamp, malformed model_config JSON, and a bulk delete that exceeded SQLite’s bound-variable ceiling.

The casualties were hermes sessions list, hermes sessions browse, session export (JSONL / Markdown / HTML), hermes insights, and sessions prune.

Now there are guards on both sides:

• Reads go through hermes_cli.timefmt.coerce_epoch(), which coerces a stored cell into epoch seconds inside a sane window (1970 up to roughly 2103, inside 32-bit time_t) or returns nothing — after a WARNING naming the session id. The bad cell renders as ? or N/A and the command finishes.
• Writes apply the same window: an out-of-window timestamp falls back to now, so Hermes cannot persist a new bad row.
• JSON markers are guarded with json_valid wherever model_config is read.
• Bulk work chunks its IN lists to avoid “too many SQL variables”, and the exporter’s N+1 query now hydrates messages in 900-id batches.

5. Sessions no longer visit other profiles

If you run more than one profile, two of these fixes are directly yours:

• The desktop launch backend called acquire() without a path, so its first touch resolved HERMES_HOME under whatever context override was active at that instant. If another profile’s override window was open, the process-lifetime handle pointed at that profile’s store forever. It is now pinned to the launch home.
• The bare-id fallback scan in session_search is gone. Previously, a miss in your own store made it open every profile’s database and return the first owner’s full transcript — without naming the profile. A miss is now a miss, with a hint to pass profile= or @session:<profile>/<id>.

Deleting a profile also closes this process’s shared handles before removing the directory, which fixes the WinError 32 (file in use) failure on Windows.

6. A damaged search index is not a damaged transcript

One more practical one: a corrupted FTS5 index inside state.db used to be reported as whole-file corruption, pushing people toward the heavy recovery path.

Errors are now classified by provenance: a failure raised from the messages_fts* shadow tables becomes the persistence cause fts_index, and the surface says “search degraded, canonical transcript intact”. On top of that:

• hermes doctor resolves damaged tree ids through sqlite_master.rootpage, so it can tell you whether the index or the structural tables are damaged before recommending recovery;
• hermes sessions repair --check-only exits non-zero on an unhealthy store, which makes it scriptable;
• the recovery lane handles the shapes real incident databases actually arrive in: a torn page-1 header, phantom NOT NULL rows, a damaged rowid edge, a CJK index family, and orphan shadow tables left behind by .recover.

What you should do

Update to the latest build (hermes update). To check your own machine, three commands cover most of it:

hermes sessions list --limit 5   # does the list still read cleanly?
hermes sessions repair --check-only   # non-zero exit on an unhealthy store
hermes doctor --fix              # names index vs structural damage, then the next step

If you run multiple profiles, name the profile explicitly:

hermes -p worker doctor --fix

The takeaway

What ties these fixes together is that “the lock” was never one thing. It was unneeded write authority, several processes competing to be the single writer, and one false alarm that silently downgraded storage forever. For users the most visible change is simple: a settled database should open instantly, not make you watch a prompt for twenty seconds.

For the earlier round of self-healing — detection, quarantine, repair — see state.db heals itself. For why long-running sessions are the wrong default and how memory is designed around session boundaries, read session boundaries. The day-to-day commands live in the hermes sessions reference, and this batch is on main for now — it will show up in the release notes once the next version ships.