← All posts

July 27 - the migration command that defaulted to production

The thing that clicked today: a default is a decision you make once and then forget you made, and the safest default is usually no default at all.

Prospect is a local-first job-application tracker. A browser extension captures a posting into structured fields while you are looking at it, and the whole thing writes into a SQLite database I own. Today I was building a scratch clone to test a schema change safely. I did the opposite.

Built / shipped

The advisory feature I was actually working on: a bounded per-listing judgment pass, gated behind an environment flag and off by default, writing into its own append-only table so a re-survey adds a generation instead of overwriting one. That part went fine.

The schema mechanism it sits on is worth describing, because it is the reason today was recoverable. Migrations are numbered files, migrations/NNN_something.sql, applied by a small explicit runner and keyed on SQLite's PRAGMA user_version. They are additive: add a column, create a table, never rewrite or drop. And they are run by hand. Nothing applies a migration on service boot, because a service that migrates itself on restart is a service that can rewrite your data because you power-cycled a machine.

Problems & fixes

While assembling the scratch-clone command sequence, one command ran the migration runner bare, with no database path set for that specific call. The runner's default is the live database. Migration 013 applied to production.

I caught it immediately, because the next thing I did was check PRAGMA user_version and saw a number I did not expect. What followed was the part I actually want to remember:

I established what had happened before deciding what to do about it. The migration was purely additive: one ALTER TABLE adding a nullable column, one CREATE TABLE. I checked the new column was NULL on every existing row, the new table was empty, and every other column was byte-identical to what it had been. PRAGMA integrity_check came back clean. The running service had never restarted, so the live process did not even know the file had changed.

Then I took a WAL-safe backup using SQLite's own .backup() rather than copying the file out from under a live writer, and recorded its hash.

Then I deliberately did not roll it back. A rollback is itself a live mutation with its own failure modes, and the state I was in was inert and identical to what a legitimate enable pass would have produced anyway. Undoing it would have been motion for the sake of feeling better.

The genuine cost was subtler: the live schema was now ahead of git, since the migration file was written but not committed. That is the kind of drift that bites you weeks later when a deploy assumes the database is behind the code and it is not.

Decisions

The runner keeps no default database path. If you do not say which database, it refuses rather than guessing. A default that is right 95% of the time and catastrophic the other 5% is worse than no default, because it trains you to stop supplying the argument.

Verification comes before remediation. Every instinct said undo it. Ten minutes of checking established there was nothing to undo, which is a much better place to make that decision from.

Learned

"I caught it immediately" is doing a lot of work in that sentence, and it was luck as much as discipline: I happened to check user_version next because that is how the scratch-clone sequence continues. If the next command had been anything else, I would have found this days later, by which time "was the schema always like this?" is a genuinely hard question.

Additive-only migrations turned an incident into a footnote. The same mistake against a migration that dropped a column would have been a restore-from-backup evening.

Still open / next

The environment-variable literals for the model endpoint are now duplicated across five files, which is exactly the sort of thing that drifts. Folded into a follow-up rather than smuggled into this build.