It is 3am and your weekly digest job starts running. It pulls a dozen data sources, writes files, sends messages — two and a half hours of work on a normal week. At 5:02am the log prints this:

RuntimeError: Cron job 'weekly-digest' lost its durable fire claim ownership

The run stops. That week’s output is gone and jobs.json records last_status: error. You check the process: the external worker PID is alive, and a moment ago it was mid tool-call stream.

PR #133791, merged in early October, fixes exactly this. The root cause is worth reading on its own: the mechanism deciding whether a job is stuck was asking how long it had been running, not how long it had been quiet.

Where the two hours came from

Every cron execution first writes a row to the profile-local ~/.hermes/cron/executions.db. Attempts move from claimed to running and end in exactly one terminal state: completed, failed, or unknown. The ledger exists to survive a process dying mid-run: after a restart, Hermes marks abandoned attempts unknown so they are not mistaken for live work.

The problem was in the “prove it” part. Alongside the hard evidence — the original PID and process-start fingerprint showing the owner is gone — there is a time-based fallback: if the owner still looks alive but has been quiet for too long, treat the claim as lost. “Too long” was derived like this:

max(3 × HERMES_CRON_TIMEOUT, script timeout, 2 hours)

HERMES_CRON_TIMEOUT is an inactivity limit, not a wall-clock limit — 600 seconds by default, and 0 means unlimited. So 3 × 600 = 1800 seconds, but a 7200-second floor (LIVE_OWNER_STALE_CLAIM_FLOOR_SECONDS) sits under it, and the result is two hours.

Every healthy long job therefore tripped the same wire. Any run past two hours, no matter how actively it was calling tools, got swept up as a dead owner: the ledger row was flipped to unknown with _OWNER_WEDGED_REASON, the gateway’s wait loop saw a terminal row, _FireOwnership.lost() failed the heartbeat, and the run was interrupted. The host in the report had 61 cron jobs, and two weekly ones died the same day in exactly the same way — 2h02m after start, seconds apart in error terms.

The irony is precise: the inactivity watchdog already knew the agent was active — it interrupts runs based on idle time. The ledger simply was never told. Two components read the same signal, and only one of them used it.

The fix: measure silence instead of run length

Three changes:

1. cron/executions.py gains a nullable progress_at column on executions (added additively, so existing databases upgrade in place) plus touch_execution_progress(execution_id), stamped by the process that owns the row.
2. The sweep now branches on _stale_age_seconds(claimed_at, progress_at) — the last stamp wins, falling back to claim age only for legacy rows that have no stamp.
3. A new cron/scheduler_liveness.py holds the stamper: it rides the existing 5-second monitor loop inside _run_agent_with_watchdog and stamps every 60 seconds, but only while agent.get_activity_summary() reports activity recent enough.

Point 3 is the design that makes it safe. A worker genuinely stuck on a lock reports growing idle time, so it stops stamping, and it is still released once the bound passes:

Situation	progress_at	Outcome
Six hours of real work, tools always running	Refreshed every minute	Claim is kept; never reclaimed
Blocked on a lock, two hours of silence	Stops updating	Released once the bound passes
Legacy row with no progress_at	Absent	Falls back to claim age (previous behaviour)

The cost is one extra UPDATE per minute per running job. As a side effect of the refactor, cron/scheduler.py shrank from 4457 to 4436 lines with the watchdog loop moved to the new topical sibling.

What you can tune

No new config key and no new switch — this changed the evidence, not the policy. The knob is still the environment variable:

# How much silence counts as "wedged", in seconds. Default 600; 0 = unlimited
export HERMES_CRON_TIMEOUT=1800

Note the semantics: it is idle time, not total duration. A job that keeps calling tools is not idle, so you no longer need to raise it just to let long jobs finish — which is the point of the fix. Invalid values are ignored and fall back to 600 seconds.

Script-based jobs also carry their own script timeout, which participates in the derived bound.

Checking whether you were hit

hermes cron runs weekly-digest --limit 20   # alias: hermes cron history

Look for two things: a terminal state of unknown, and how long the attempt sat in running. An unknown that ended at almost exactly 2h02m is this bug.

One thing to know: unknown is an audit record — Hermes never reruns it automatically. The lost cycle is yours to recover:

hermes cron run weekly-digest     # fire it again now
hermes cron list                  # check last_status and the next run time
hermes cron status                # is the scheduler alive, is anything OVERDUE

If this has bitten you more than once, the cron troubleshooting guide covers scheduler and delivery failures, and memory continuity for scheduled jobs explains why long-running jobs need context to survive between runs. Command reference: hermes cron.

The short version

• Symptom: a healthy long job is interrupted roughly 2h02m after it starts with lost its durable fire claim ownership, losing that run’s output and recording last_status: error;
• Root cause: the live-owner stale-claim bound measured claim age, with an upper bound of max(3 × HERMES_CRON_TIMEOUT, script timeout, 2 hours) — two hours by default — while the active-work signal never reached the ledger;
• Fix: a progress_at column stamped every 60 seconds but only while the agent is genuinely active, and a sweep that measures silence; a deadlocked worker stops stamping and is still released;
• What to do: check hermes cron runs for unknown rows near the two-hour mark, and use hermes cron run to recover the cycles you lost.

Put simply: to decide whether a job is dead, ask how long it has been quiet, not how long it has been alive.