The thing that clicked today: I have been describing this system one piece at a time for months and never once written down what the whole thing is - one fleet, one MCP server, and a rule that nothing counts as done until it can prove it.
Built / shipped
One server, 21 tools. The control plane is an MCP server built on FastMCP. It exposes 21 typed tools over the Model Context Protocol: semantic search across my notes, reading and writing individual notes, appending to a journal, dispatching a job to a machine, inspecting a run that is still going, and auditing what has run recently. Any MCP client can connect to it and use those tools. That is the whole interface - there is no separate API to keep in sync.
Jobs go to whichever model should run them. When I dispatch work, I do not usually name a model. The router looks at live quota across three cloud providers and picks one, or it sends the job to an open-weight model running locally on my GPU box when the work is deterministic enough that paying for it would be silly. Cheap local work costs nothing and does not touch a quota at all. The expensive models get saved for the jobs that actually need judgment.
Retrieval is tiered on purpose. Searching my notes hits compact summary cards first. Only if that is not enough does it go to a second index built over normalised conversation history, and only then does anything read a raw source. Most questions get answered at the first tier for almost nothing. The deep path exists, but it has to be earned.
Everything is behind a real access gate. The server sits behind Cloudflare Access. It verifies the access token on every single call rather than trusting the tunnel, so a request that somehow reaches the origin without a valid token gets rejected at the server.
Problems & fixes
The router sent free work to a paid model. One table mapped worker names to machines, and the same table was being reused to pick which launcher to call. The local zero-cost model lives on the GPU box, so looking it up in that table returned the GPU box's cloud launcher, and deterministic work that should have cost nothing quietly went to a paid model instead. Two lookups that happened to return the same shape are not the same lookup. It is now two explicit tables.
A find-and-replace ate a word inside other words. The rename of a component replaced a bare string across the tree. The old name was a substring of translate, so every CSS transform in the project turned into nonsense, along with a vendored library I was not even thinking about. Twelve files had to be repaired, and the replace now matches on word boundaries. I have now let this exact mistake through more than once, which is the only reason it is written down here.
I broke the SSH mesh by being helpful. Trying to make the fleet independent of one naming service, I pointed hosts at raw addresses. Known-host entries are hashed against the name, so every machine lost its trust anchor at once. The fix was to keep the addresses and pin the original name as an explicit alias so the existing keys still match.
A guardrail refused a deploy and was right. Twice now the agent has declined to deploy when it was acting on an old instruction rather than something I said in the moment. Both times I was briefly annoyed and both times it was correct. That behaviour stays.
Decisions
Serialise by what a job touches, not by which machine runs it. My first version locked per machine: one write job per box at a time. That was wrong in both directions. Two unrelated jobs on the same machine blocked each other for no reason, and a job that reached across to a second machine was completely unprotected. Now every mutating job declares exactly what it will touch - this repo, this service, this path - and the lock is on that. Unrelated work on one machine runs in parallel. A job touching two machines declares both.
Single-use tokens, permanently. Every job carries a token that is valid exactly once, including if the launch fails. I cannot re-fire a token to retry. That sounds annoying and occasionally it is, but it killed an entire category of bug where I acted on a result from a previous run without noticing.
Read-only and mutating jobs are different things. A recon job can look at anything and report, and cannot change a thing. A build job is allowed to make changes and is subject to all the locking above. Keeping them separate means I can fire off an investigation without any thought about what it might break.
A job has to return evidence. Exit code zero means the process ended, not that the work is right. Each run writes an immutable request and a durable response, and the response has to carry the exact token plus the actual command output that backs its claims. Then I check one material fact myself against the source. A confident summary from a model is not proof of anything.
Learned
Routing on cost changes what you build. Once cheap work is genuinely free, I stopped rationing it. Broad sweeps, inventories and cross-checks that I would never have paid a frontier model to do now just run, and they surface things I would not have gone looking for.
The interesting engineering is not the prompt. It is the boundary around the model: what a job is allowed to touch, what it must return, what happens when two run at once, and how a wrong answer gets caught. The model is one component. The rest of it is the system.
Serialising by resource rather than by host is the change I would make first if I started over. It is the difference between a lock that describes the hardware and a lock that describes the work.
Still open / next
I do not have evaluations, and I should be precise about that: I have release gates and verification discipline, not a dataset with metrics I can point at and say the system got better. Those are related and they are not the same thing, and it is the most obvious hole in the platform.
The other one is that I am the only user. Everything here was designed for an operator who already knows what all of it means. The next real test is putting it in front of somebody who does not.