The thing that clicked today: the service manager's green status tells you what the most recent invocation did, not whether the thing you wanted exists, and those are different questions when the most recent invocation was a manual test.
Loupe's databases are snapshotted nightly by a timer into an archive on network storage. This was a side-quest that became the main event: while doing cleanup on the app's state, the newest snapshot in the destination was from three days earlier, and the timer said it had succeeded 22 hours ago.
Built / shipped
The diagnosis from the journal. Two consecutive scheduled runs had failed with the same error: the snapshot tool could not load the vector-search extension that one of the databases requires. The only successful run in the window was a manual verification run that wrote to a temporary directory, not the real destination. The service manager's status faithfully reflected that manual run, so the unit looked healthy while producing nothing where it mattered. The extension fix had been committed the day before; a later commit touched the same script and nothing had re-run it end to end.
The rail, proven rather than trusted. Ran the service against the real destination: a 639,548,259-byte archive written, the oldest pruned, exit 0, and the verify pass green across all eleven databases including the vector one.
A restore drill, for real. Extracted the fresh snapshot to scratch in 5.5 seconds and checked four properties, not one, because extraction succeeding proves almost nothing: SQLite's own quick check on every database, the journal mode, the hash of each file against the snapshot's manifest, and per-table row counts against both the manifest and the live databases. All eleven passed. Scratch removed, runbook written.
A cap on the preview cache, while in there. The cache was 536 MB across 1,001 files with no eviction. Eviction now runs from the one function that writes into the cache, with a default cap well above the current size so it does not start evicting immediately, verified by a temporary 400 MB cap: 536 MB and 1,001 files down to 361 MB and 682 files on the next preview build, against a 360 MB target, with the eviction line landing in the server log. Temporary cap removed, default left.
Problems & fixes
The drill failed first, on my own assumption. It reported one database as "live absent." My script had joined the state root to the database's relative path, but the snapshot tool deliberately stores that one database beside the application code rather than in the state directory, because that is where the server reads it from. Corrected, and that asymmetry is now the most load-bearing paragraph in the runbook, since a naive restore onto a fresh machine would put that database somewhere the search feature cannot find it.
The worklist was stale. Over half of the planned cleanup had already been resolved as a side effect of the move to a new machine the week before: an orphan database and an old archive directory no longer existed. Auditing the list against live state before executing it saved doing work that was already done.
Decisions
The first thing the runbook says: check the newest file in the destination, not the unit status.
Four properties for a restore, every time: integrity, journal mode, manifest hash, row counts against manifest and live.
Eviction from the single writer into the cache, with a default that does not fire on day one.
Audit an old worklist against live state before executing it.
Learned
A green unit status is not a green backup. The most recent invocation was a manual run to a temporary directory, and the status told me about that.
A fix committed and a fix exercised are different states. Three days passed between the two.
A restore drill that fails on the drill's own assumption is still useful; it found the one place a fresh-machine restore would have gone wrong.
Still open / next
The snapshot rail has no off-site copy; the archive lives on one storage box. And the drill is a command I run, not a schedule with a loud failure, which is the same gap the earlier restore-drill post ended on.