The thing that clicked today: I had backups I had never restored, which means I had a belief about my backups rather than a backup.
Built / shipped
An end-to-end restore drill, run for real. Not a checksum, not a file listing. Decrypt the archive with the escrowed key, open the database, verify it is internally consistent, confirm every record is still individually encrypted, and then actually start the application against the restored copy and log in with the master password in a browser. Four things have to be true together for a restore to mean anything: the key, the archive, the password, and a clean restore. Testing them one at a time proves none of them.
One command for the archive half. Verify the archive against its sidecar checksum, decrypt it with the escrowed identity, run the database's own integrity and foreign-key checks, confirm exactly one vault header, confirm every record still carries its encryption envelope, and re-hash every encrypted attachment against the checksum recorded when it was uploaded. That last one is the check that would catch silent corruption, which is the failure mode that scares me most because nothing announces it.
A separate mode that serves the restored copy in isolation. On its own address, against the restored data only, so I can enter the real master password in a real browser without going anywhere near the live store. A restore you have to trust is not a restore you have tested.
Print the shape, never the secret. The drill reports the key-derivation settings so I can confirm the vault is protected the way I expect, and deliberately never prints the salt or the wrapped key. A diagnostic that leaks the thing it is checking is worse than no diagnostic.
Problems & fixes
The older archive was 1,471 bytes. It had been taken while the vault was still empty, so it was a perfectly valid, perfectly verifiable, completely worthless backup. The first archive that ever contained my actual records was taken the morning of this drill, at seven megabytes. Between populating the vault and that morning there was no recoverable copy of anything, and the backup system reported healthy the entire time. Nothing was wrong with the backups. What was wrong was that "a backup exists and its checksum matches" was the only question anyone was asking.
Size is a signal and I was not looking at it. A backup that is three orders of magnitude smaller than the data it supposedly contains is not a subtle failure. I did not catch it because I was checking whether the process succeeded, not whether the result was plausible.
Decisions
The rule was written before the vault was allowed to hold anything real. The note governing this vault said plainly not to store irreplaceable records until a restore drill had proved all four pieces together. That rule is the only reason this got done rather than deferred forever, and it is the pattern I want to copy: write the gate down while it is still cheap to honour.
Restore into isolation, never over the live copy. The drill can never damage what it is meant to protect. That constraint is what makes it safe to run often.
Re-hash the contents, do not just count them. Counting files catches truncation. Re-hashing catches corruption, which is the quiet one.
Learned
An untested backup is a hypothesis. Mine happened to be false in the most boring possible way, and no amount of checksum verification would ever have revealed it, because the empty archive was internally perfect.
Verifying the process is not verifying the outcome. Every check I had was about whether the backup ran. None was about whether the result contained anything.
The drill found the real problem, not the thing it was written to find. It was built to prove recovery works. What it actually surfaced was that there had been nothing to recover. That is the argument for running drills rather than reasoning about them.
Still open / next
The drill is a command I have to remember to run. It should be on a schedule with a loud failure, and until it is, this remains a good process that depends on me.
The plausibility check is still my eyes. A rule that fails when an archive is drastically smaller than the last one would have caught this months earlier, and it does not exist yet.