Self-hosting

Six of my repositories had gone public straight from their working trees, carrying internal names, addresses, paths and real personal data in test fixtures. Rewriting files does not purge history, so every repo was deleted and recreated from a sanitized single-commit snapshot, and the publishing pipeline was folded into the nightly pusher with a fail-closed gate that is deliberately allowed to tolerate two things.

I have been building an agent platform on my own hardware one post at a time. This is the whole thing in one place: how a job gets dispatched, which model runs it, and what it has to prove before I believe it.

A phone screenshot of an error turned into a day of storage work: an audit of 4,921 pulled photo edits found 3,777 byte-identical to their originals, 108 GB reclaimed, a 3.4 TB library moved with an instant rename, and five separate ways a copy or a delete can lie to you.

A single readiness percentage is a number you can fool yourself with. Waypoint replaced it with ten independent gates that each pass or fail on their own evidence.

While reading my dashboard's code as a precedent for something else, a note that said 'stale comments' turned into a live path where anyone on my home network could launch an AI worker with tool auto-approval. Fixed at two layers, then the fix locked me out, which was its own lesson.

A live practice-exam submission returned a 502 and the retry said 'already submitted', leaving a permanent error in front of a button that could never succeed. The proxy had a five-second default nobody had questioned, the submit path took 29 seconds, and the first optimization made it slower. The fixes preserved output exactly or were reverted.

Listing sites edit and pull their own pages. Homestead keeps every capture as immutable evidence and rebuilds the searchable database from it, so a delisted house keeps its record.

The fleet dashboard kept raising conformance warnings. Genuine state edges were being amplified by four independent defects in how it observed and reported. Then two screenshots from my phone showed the first fix was incomplete.

Loupe got its first real test suite. Writing it found two things: one of the new tests could not fail, and the search feature had been silently broken since a move two days earlier while the service reported healthy the whole time.

The timer that snapshots Loupe's databases reported success while the newest snapshot in the destination was three days old. Two scheduled runs had failed and a manual test run had overwritten the status. Then a real restore drill, which checked four properties and first failed on my own script's assumption.

Breadcrumbs grew a price-history service over 17 months of my own grocery orders. Two facts about the data shaped the whole design, a passing test suite hid a false alert that only a replay against real orders exposed, and raising the threshold provably could not fix the promo problem.

Two passes over the memory layer my AI tools search. First, an audit of how eight knowledge bases preserve working transcripts, which ended with part of the index quarantined and a refusal to backfill history. Then importing 708 conversations from a cloud account into that same layer without publishing things to myself that should stay private.

A multi-day video conversion job hit two unrelated failures in one day, and both had the same shape: one broken precondition outside any file, turned by the script into thousands of fast, confident per-file failures. A circuit breaker, a pinned driver, and a watcher that looks at rates instead of names.

Porting a working file-sync pattern from one project to another would have replicated a 188 MB authenticated browser session and a secrets file across the whole machine mesh, because the sync tool does not read the ignore file that version control does. Three options, one structural fix, verified by hashes and an absence check.

Every mode of my study assistant returned a server error before the model was ever called. The context packet's fixed part had grown past the safety limit on its own, and one unbounded field guaranteed it would keep growing. The fix went into the shared layer three different callers depend on, with caps chosen by measurement and a degrade path before any error.

What started as moving two small apps to the control-plane machine became the complete retirement of a third machine as an application host. Dependency mapping first, an explicit 'park that one' that evidence later overturned, a symlink instead of rewriting a hundred thousand paths, 51 GB reclaimed, and two live credentials caught before they replicated.

A sweep after the watchdog repair found the fleet's automation registry listed 13 timers on a machine running 16, two of the missing ones backups. It also nearly 'discovered' a hole that did not exist, because a grep over unit files cannot see configuration that arrives through drop-in directories.

The service whose only job is to tell me when a device stops checking in was dead. The previous handoff said it had died the day before. It had never worked once: its sandbox was built before its script ran, and it was the one unit in its batch that nobody wired to the failure notifier.

A small domain knowledge base built from an official document and three books, with a coverage gate that reports its own holes instead of hiding them. The useful part was the second pass: of eight apparent gaps, three turned out to be my own bad phrasing and five survived as real, and telling those apart is the whole job.

A session meant to make a backup claim checkable first read the config wrong and nearly 'repaired' a healthy setup, then proved one restore, then opened an alarm about unbounded growth. The follow-up the same day measured the alarm and found two of its load-bearing facts wrong. Ruling: accept.

I had encrypted backups of my private records and had never once put one back. Proving it worked also revealed that the older archive contained nothing at all.

The redaction fix from the day before had quietly made my search index 21% bigger. The cause was one line in the wrong order, the fix was two changes that were useless alone, and the proof was a measurement that matched the prediction to the character.

The instruction text my AI tools read lived in a dozen places across the fleet: files on each machine, symlinks, importers, and text boxes in someone else's web UI, with no record of which was current. Now there is one registry that owns every surface by path and hash, a checker that verifies 103 of them, and a ruling about what an embedded version token actually means.

My notes sync between three machines went through one of them, which made that machine a single point of failure for the other two. I paired the outer two directly, then proved the claim both ways by pausing the middle machine and timing a canary file across. The procedure I had approved turned out not to exist yet, and the documented emergency command had the wrong verb.

Three live API keys had made it into the searchable index of my own working history, straight through a redaction step that ran on every byte. The fix was small; deciding how to measure it before trusting it was the real work.

A routine tidy-up of my project notes was handed to an automated worker. It went past its brief, so it was stopped, its own success report was thrown out, and the recovery was done by hand against an approved list with every deletion checked twice. The good suggestions it made were kept.

Two passes in a row had shipped the wrong artifact one layer apart: a service running a process older than the source, then a deployed bundle older than the feature commit. The fix was a guard validated by running it against the actual stale artifact, a release workflow with automatic rollback, and, the same day, an API that only three bounded classes of caller can reach.

Jobs were queuing behind a busy machine for no better reason than habit about which machine owned that kind of work, so I separated who runs a job from where the job lands.

The free model on my own hardware kept failing halfway through jobs. The cause was not on my side, and the fix was to stop asking the model to write its own proof of work.

The standby for my agent server was designed as a warm standby, formally ratified, amended five times, and thrown out the same day for an always-on mesh with edge steering. That died too, on cost and on a Pi with 905 MB of RAM. What shipped is a dormant cold standby with its own free tunnel, a structural split-brain fence, and a health endpoint that tells the truth about what it cannot do.

A standing read-only collector that checks contract parity, machine-to-machine reachability, required services and timers, and required paths across the fleet. Three design choices carry it: it keeps last-known-good evidence, it separates drift from probe failure, and it never repairs anything.

Prospect's two AI background workers were quietly overwriting each other's status, so every service restart redid work that was already done. The handoff called it a one-row cosmetic cleanup. Reading the two workers against each other showed something worse, and the fix cost no GPU time.

Prospect's migration runner picks the live database when you forget to tell it otherwise. I found that out by running it without the environment variable, against production.

My notes could answer what I decided but not what I actually did, so I built a second searchable layer over the working history itself, and a rule for when it is allowed to be used.

I stopped picking which AI model runs each job by hand and built a router that scores the options on what each one has left, reserves the winner, and writes down why it chose.

Three database backups written to network storage two weeks earlier were all zero bytes. The cause was an idle timeout on the mount tearing it down mid-write. The obvious retry then failed instantly for a second, unrelated reason, and the worker stopped and asked instead of improvising.

Hashing every size-collision candidate on my NAS found fifty-two thousand duplicate groups and about 254 GB reclaimable. The interesting part was everything that nearly made the number wrong.

I assumed every machine could reach every other. Checking every direction properly found a quarter of the paths broken, and one repair job reported success while having quietly undone itself.

The harness that runs my local model was reporting success unconditionally, no matter what happened. Hardening it went in four phases, each one proving the defect on the unmodified code before patching, and ended with a single terminal contract that maps every way a run can end to an honest status and exit code.

Two things about working with AI agents that only show up after you have run them for a while: the cost of what you read at the start of a conversation is paid again on every later turn, and a detailed handoff from another agent is a claim until something the author could not have shaped confirms it.

The free model on my GPU box was unreliable in a way I could not name. Before swapping it, I found the reason by reading its transcripts, then ran a controlled A/B of six scored tests in an isolated copy of the real harness. The candidate won 5 to 2, a longer soak passed 21 of 22, and the swap was one line.

The control-plane machine had a config file explicitly setting persistent log storage, and its logs were still vanishing. A vendor file with a higher number won by sort order alone, roughly nine days of history had already rotated away, and half the reported symptom turned out to be my own wrong command.

Prospect got meaning-based search fused with its keyword search, and a semantic check for reposted listings. Both had to be designed so they could fail without taking anything down, and the duplicate threshold was calibrated on a copy before it touched the real database.

Working with AI assistants for months across several projects, the failure I kept hitting was not bad answers, it was lost context: every conversation restarted from zero or dragged a bloated history. The fix was a method: each conversation does exactly one bounded section, and continuity lives in three small documents, not in the chat.

I set out to train a small local model to reproduce Apple's photo-quality score so Loupe could rank photos for anyone. It matched Apple almost perfectly. Then a cheap probe showed what the score actually measures, and it was not beauty.

Loupe can now search photos by describing them, with every model running on my own hardware. The interesting work was not the model. It was proving that the index machine and the query machine compute exactly the same numbers, and finding out the plan's premise was false.

My entire knowledge base had no undo, and adding one turned out to be a real distributed-systems puzzle because the same files live on four machines at once.

For an app that looks at your private photos, the machine learning cannot phone home, so every model in Loupe runs on your own machine, on a single consumer graphics card.

One machine showed permanently critical on the fleet dashboard while every other check against it was green. The probe was judging health on a ping command's exit code, and a ping that succeeds through a relay returns a real answer, a real latency, and a non-zero exit code.

Culling a hundred-thousand-photo library should never come down to one mysterious score, so I split the decision into three plain roles that each answer a different question.

Turned the fleet dashboard into an installable phone app with alerts tiered by urgency, a critical channel that survives the app itself failing, and a dead-man's switch that can announce its own death.

My busiest compute machine was cooking itself under real load, so instead of just turning the work off, I built layers of protection that let it keep running without frying.

I taught my notes to answer questions by meaning, so I can ask what I decided about X and why and land on the right note even without remembering a single word from it.

I write a lot of scattered notes across a lot of projects, and instead of ever organizing them by hand, I built a pipeline that rebuilds and reorganizes the whole thing every night while I sleep.

A lot of the small jobs I was paying a hosted model to do did not need a frontier model at all, so I started running open-weight models on my own GPU box and let the routine work run for free.

I stopped doing my machines' work at the keyboard and built a way to hand a task to whichever machine should run it, let it run on its own, and pick up the result later.

I caused an outage by running a heavy job on the one machine that holds everything together, then spent the night proving my failover plan actually works.

Pulled fleet health and my entire notes vault onto one gated dashboard, then added semantic search over my own work by borrowing an embedding model already running elsewhere on the network.

Every cheap health check told me my storage link was perfectly healthy while it was actually running at a tenth of its speed, and the real work was proving which piece was lying.

Turned the fleet into a full mesh where every machine can reach every other one directly, passwordless, without hand-managed keys, laying the groundwork for running work anywhere in the fleet.

Adding one wireless card to the GPU box renumbered the PCI bus and renamed two interfaces, not one, which took down the isolated storage network's gateway through a chain nothing had advertised. The rebuild made the two radios a primary and hot spare, with a measured failover gap and an idempotent service.

Moved the public tunnel entrance off the busy compute box onto its own dedicated Raspberry Pi with its own uplink, so a reboot of any other machine can no longer take the whole public site down.