One Junk Line in jobs.json Stopped Every Cron Job You Own

It is Monday morning and the daily briefing never arrived. You check Wednesday — empty. Tuesday — empty too. hermes cron list shows every job still there, next_run_at looks sane, and no line in the status output is screaming at you. Nothing runs anyway. When every job goes silent at once, the cause is usually not your job definitions: it is one null that got into ~/.hermes/cron/jobs.json. One bad entry is enough to stop the entire scheduler. A fix merged on 2026-09-26 (PR #124034) handles that class of input; here is how it broke, what happens now, and how to check your own machine.
How a single null takes down the scheduler
jobs.json is the cron job store. Every tick, the scheduler calls load_jobs() to read it. If that read raises, the tick fails — and the next tick reads the same file and fails again. The jobs themselves are perfectly healthy; they simply never get their turn.
Three distinct inputs could each produce that outcome:
| Input | Old behaviour | Consequence |
|---|---|---|
A non-object entry in the jobs list (null, a string, a number) |
load_jobs() raised |
Every tick failed, so no job ran at all (issue #123281) |
A scalar jobs field, e.g. {"jobs": null} |
Returned to the caller as-is | Callers blew up somewhere else |
A job whose repeat.completed was null |
mark_job_run / update_job raised TypeError: NoneType + int |
Recording a run crashed (issue #122114) |
The third one carried a display bug that made it easy to ignore: with a broken counter, the cronjob tool showed a run-once job as 1/1, and hermes cron showed the count as None/3. Odd-looking, not error-looking — so it stayed broken.
What happens now
- Bad entries are skipped and the file is repaired. The loader no longer raises over one piece of junk; it drops non-object entries and writes the cleaned store back to disk — including when no valid job is left, in which case the file becomes
[]. - A scalar
jobsfield is normalised.{"jobs": null}becomes[]and is saved, so every caller gets a list. - A non-integer counter is normalised.
repeat.completedis reset to a valid count when it is not a non-negative integer (0 when it cannot be read), once, at load — so every reader sees an integer and the1/1/None/3display errors disappear with it. - The repair runs under the lock. The save re-reads the file after acquiring the job-store lock, so a concurrent
update_jobis not reverted. That was the blocking review point on the original patch, and it is the difference between “repair” and “silently undo someone’s change”. - Warnings log types, not contents. Each repair warning is emitted once, and it names the type of the skipped value rather than the value itself.
The documentation now states it plainly: if a hand edit leaves jobs.json malformed, the scheduler repairs it on the next load instead of stopping.
Checking your own machine
First confirm the scheduler and the jobs themselves are healthy:
hermes cron list # jobs and their status
hermes cron doctor # failed runs, failed deliveries, overdue next_run_at, missing scripts
To look for values that should not be there, this read-only pass reports the shapes:
python3 - <<'PY'
import json, os
p = os.path.expanduser("~/.hermes/cron/jobs.json")
d = json.load(open(p))
jobs = d.get("jobs")
print("jobs type:", type(jobs).__name__)
if isinstance(jobs, list):
bad = [(i, type(j).__name__) for i, j in enumerate(jobs) if not isinstance(j, dict)]
print("entries:", len(jobs), "| non-object entries:", bad or "none")
for j in jobs:
c = (j.get("repeat") or {}).get("completed")
if not isinstance(c, int) or c < 0:
print(" bad counter:", j.get("name") or j.get("id"), "->", repr(c))
PY
It never writes. The repair itself is Hermes’ job: upgrade, let the scheduler tick, and the bad entries are gone.
Do not hand-edit jobs.json
The docs are explicit: manage jobs through the cronjob_manage tool, hermes cron edit, or /cron — not by patching jobs.json directly. Direct edits can fail silently when the path is blocked by file write safety (for example with HERMES_WRITE_SAFE_ROOT set), and the file-mutation verifier footer is the only authoritative signal that nothing was saved.
That is exactly why this fix matters: the accidents that happen outside the tool path — an external script, a sync tool, a half-finished manual edit — should not cost you every scheduled job on the box.
What is still open
The PR is explicit about one thing it deliberately did not do: issue #123120 proposes failing closed on a corrupt store instead of repairing it, which is a different behavioural stance and a maintainer decision. It remains open. Today’s policy is self-heal first.
The takeaway
jobs.json went from “one junk entry kills the fleet” to “skip it, repair the file, keep firing”, and repeat.completed went from a crash to a guaranteed integer. If you run unattended scheduled work, this is worth taking early — it fixes the failure mode that is hardest to diagnose, the one where everything looks configured and nothing happens. For the surrounding ground: one command that gives the whole fleet a checkup, timezone and DST traps in day-crossing schedules, cron jobs that keep notes between runs, and the hermes cron command reference.