The thing that clicked today: a running service is not a working service, and a test that cannot fail guards nothing, and I found both out on the same afternoon.
Loupe is my self-hosted photo app. It had grown for two months with hand-checks and no test suite, and the roadmap said a quality harness had to exist before the next round of changes, especially one rename that touched the search path and had been put off precisely because nothing could prove the rename changed nothing.
Built / shipped
A test suite of 40 tests in five files, using only Python's built-in test runner. No test framework got installed, on purpose: the app runs from a pinned, locked set of packages that was committed hours earlier so that drift could be detected, and installing a framework into that same environment would have changed the exact thing the lock exists to protect.
The static tests read the server's source code as a tree of syntax rather than importing it. Importing the server opens databases and loads the machine-learning stack; the properties worth guarding, like "is there any route that skips the access check," are questions about the code's shape, so reading the shape answers them in milliseconds on a laptop with nothing mounted. The other files cover data invariants on synthetic fixtures (for example, that the exclusion query drops no asset, so reinstating an old exclusion has to be deliberate rather than drift), the live API contract including the whole security matrix that had only ever been checked by hand, the backup rail (a recent file has to exist in the destination, which encodes that day's other outage as a test), and dependencies. Fast path 0.17 seconds, slow path 11.4 seconds.
Testing the tests. A small script breaks one invariant at a time in a copy of the server and asserts that the matching test fails. Five mutations, five caught: the access check removed from the write handler, a route added without the network check, the token compared with plain equality, two of three required headers dropped, the cache-eviction call removed.
The rename, done against known-good answers. Before renaming the three search modules, I captured four reference vectors (the numbers the search produces for a fixed input) and their fingerprints. After the rename, the same inputs produced byte-identical vectors on all four. A rename "should not" change numbers, but search quality degrades silently if it does, and now there was proof instead of a shrug.
Problems & fixes
The test that could not fail. The mutation script immediately caught that the test meant to guard "the token is compared in constant time" was passing by finding the name of the constant-time function in the function's own documentation string. Replace the safe comparison with plain equality and the test still passed, because the docstring still mentioned the safe one. Rewritten to look at the actual call in the code and to reject a plain equality on the token. I would not have spotted that by reading the test; it looked fine.
The search that had been dead for two days. Capturing the reference vectors failed on a missing text-cleanup library. That led to the live service: the search feature was switched on, and a real query returned status 000. Not a 500. The handler raised, and the built-in HTTP server dropped the connection with no response at all, which reads like a network problem and is not one. The library had not survived the move to a new machine two days earlier, and because the search module only imports it on the first search, the service started perfectly clean, the front page returned 200, and the status tool said active the whole time. Installed the library, restarted, verified two real queries return sensible results, regenerated the lock (58 to 60 packages).
A command that half-ran. One compound shell command with a nested block failed to parse and silently took two file copies with it, so the new reference-vector tests were never installed. The only visible sign was the suite reporting 38 tests instead of 40. Re-sent the files one at a time. A partially executed command is more dangerous than a failed one, because the run looks successful.
Decisions
Built-in test runner, not a framework, because the production environment is locked and a harness that changes the artifact it protects is a bad trade.
Read the source as syntax rather than importing it, so the structural tests run anywhere in milliseconds.
Fix the missing dependency rather than route around it. The library is a real runtime dependency, so install it and relock.
Import eagerly in tests what the app imports lazily. Both of the day's outages were lazy imports that a healthy-looking service concealed. That is a category, not two coincidences, so it got its own test file.
Learned
A running service is not a working service. Startup proved nothing, the service manager's "active" proved nothing, the front page returned 200 throughout. Only exercising the specific feature found the break.
Status 000 from a Python HTTP server means an unhandled exception dropped the connection. I had seen a 000 earlier that day from my own shell quoting, which is exactly the kind of coincidence that trains you to dismiss the real one.
A test that cannot fail guards nothing, and you cannot tell which kind you wrote by reading it. Breaking the code on purpose and watching for the failure is the only check that is not itself a guess.
Stale names mislead; stale reasons mislead worse. The renamed modules carried documentation justifying a design choice with a fact about the old machine that is false of the new one.
Still open / next
The reference vectors were captured after the library was reinstalled, so they pin behaviour going forward but cannot prove parity with whatever version built the image-side vectors originally. That deserves a bounded check before I trust any claim about search precision. And the backup ledger still has no off-site copy, which remains the largest open resilience gap in the project.