← All posts

August 6 - the migration that stopped being a migration and became a retirement

The thing that clicked today: map what actually depends on what before you move anything, and be willing to find out that the thing you parked as too hard was already half-moved.

Three applications and a pile of history lived on the backup machine. The goal on the first day was modest: move two of them to the control-plane machine. By the end of the second day the backup machine hosted nothing.

Built / shipped

Scoping did the useful work first. Two apps had no blockers: one is pure standard-library Python with no native code, no machine learning and no GPU; the other's only GPU use is an outbound call to the GPU box, which works from anywhere. Neither touches network storage. A third, the photo app, was explicitly parked: it depends structurally on the storage network the control-plane machine lacks by design, and on a machine-learning stack whose support on that machine's processor architecture was unverified. Copying native addons across architectures would silently break at runtime. The two easy ones moved the same day with shadow-then-cutover, and the parked one was recorded as parked with the reasons.

The next day, reconnaissance reframed the parked one. The photo app was already a two-host application. Its 3.9 GB machine-learning environment and three of its data directories already lived on the GPU box, and the backup machine's own Python could not even import the face library. A whole apparatus existed only because the app and its models were on different machines: CPU-side mirrors of the GPU recipe, plus a "cross-machine proof" comparing the two. Moving the app to the GPU box collapsed that split rather than relocating a dependency. Features that had silently degraded on the old machine (faces, content screening) now had their imports satisfied locally.

A mount alias, not a path rewrite. The metadata database holds 102,678 asset rows whose file paths all begin with one mount prefix, with sibling references in three more databases. Both machines mount the same share, at different paths. One symlink on the new machine made every stored path resolve unchanged. Rewriting a hundred thousand paths across four databases would have been slower, riskier, and would have destroyed the ability to roll back. The photo corpus itself never moved; it lives on the storage box. The app's mount guard was checked rather than assumed: it tests for a sentinel file's existence, which a symlink satisfies, verified live before cutover.

Run under the right interpreter. The new machine's system Python was two minor versions ahead of the old one's, on an app with lazy machine-learning imports. The existing environment matched the old minor line and supplied the stack in-process, so that is what the service runs under.

The long tail. With the bulk gone, the old machine's remaining clutter became visible: three input files the photo app's enrichment builder reads by documented name, still loose in the home directory while the code that reads them had moved, consolidated into an inputs folder beside the code so they travel with it. The last three repositories relocated. The sync topology repaired so the laptop pairs with the control plane directly. About 51 GB reclaimed on the old machine, 85% to 30% used. Six categories of migration straggler identified and listed, and one container-written directory that a relay worker cannot remove because it needs a human with elevated rights, which I cleared by hand.

Problems & fixes

Two live credentials, caught before they replicated. Preserved config from two retired projects was scanned as a matter of course before it could sync into the notes. One environment file carried four populated username and password variables: deleted from the preserved copy rather than redacted, with the example file kept. One application config carried a plaintext third-party API token: redacted. Both would have landed in a folder that replicates to four machines, is committed to version control nightly, and pushes to a code host on an allowlist. The generic key-prefix scan returned zero matches on both; service passwords and vendor tokens of other shapes do not match vendor prefixes. What caught them was the broader sweep by filename and content for the words password, token, key and secret. The old copies are deleted and the token flagged for rotation.

Deploy keys left in place, deliberately. The old machine's keys are inert, removed from its push manifest, and the rollback window is still open; retiring them is a follow-up for when it closes. A deliberate divergence from the previous day's precedent, recorded as one.

Decisions

Map dependencies, move the unblocked ones, park the blocked one with reasons, and re-examine the parked one with fresh reconnaissance rather than treating "parked" as final.

Alias the mount rather than rewrite the data.

Only read-only requests against a shadow instance of an app that trusts local peers for writes and can mutate storage.

Scan preserved material for secrets by shape and by words, before it can sync.

Learned

"Too hard to move" was wrong because the hard part had already moved. Reconnaissance found it; the plan did not.

A symlink that makes stored paths resolve unchanged is a rollback-preserving migration for free.

Vendor-prefix secret scanning misses most secrets. Passwords and other vendors' tokens have no prefix; scan by the words that label them.

Preserving a retired project's configuration is exactly where secrets hide.

Still open / next

One third-party token should be rotated if the account still matters. The old machine's deploy keys retire when the rollback window closes. And the decision about whether the old machine gets a new job or stays a pure backup node is its own item.