The thing that clicked today: a safety control that was added as a convenience, a client-sent "are you sure" flag, had become the only thing between the home network and a worker that runs commands, because the real control, a loopback-only bind, had quietly stopped being true three weeks earlier.
The fleet dashboard is a small web service on the control-plane machine. It shows fleet health, ingests events from the other machines, and can launch bounded AI workers on a named host. I was reading it as a precedent while building a private-network path for the agent server, and its documented access model no longer matched its code.
Built / shipped
Severity re-called upward after checking. The first report described the mismatch as stale comments. Reading the routes showed the worker-launch endpoint was guarded by a known host name and a confirmed: true flag supplied by the client, a UI "are you sure," not a credential. A read-only probe from another machine on the home network confirmed the control router answered with 404, not 403: reachable and ungated. The fleet workers run with tool auto-approval, so that was arbitrary code execution across the fleet from any peer on the home network. The flag reading like a safety control is exactly what made it easy to skim past.
Why it existed. The service's documented justification for listening on every interface was that remote machines post events to it "over the trusted private network," and a comment stated the wider bind was "safe because an access check gates every read." That check had been deleted on July 20. The bind stayed. One implicit layer had been assumed to do a job it had stopped doing.
Fixed at two independent layers, either sufficient. First, bind each address explicitly, so the home network cannot open a socket at all. Second, a peer-trust middleware that rejects untrusted peers regardless of bind, so a future bind change cannot silently reopen the hole. Because the hole was caused by trusting one layer, one replacement layer was not enough.
What actually depended on the wildcard, checked before removing it. Both remote hooks already posted to the private-network address, not the home-network one. The intent was always the private network; the wildcard merely also handed the routes to the home network. Nothing legitimate broke.
Narrow trust, and WebSockets too. Trust is the private overlay network's range only, not every private address, because the home network is where guest wifi and IoT devices live. And the WebSocket channel that drives the fixed-command terminals was gated too; gating only HTTP would have left the more dangerous surface open.
Verification: 17 new tests asserting the rejections (home network, public, boundary addresses, an unrelated private range, missing or malformed peer, WebSocket), full suite 642 green, control-router probe from the home network now connection-refused (was 404), the public hostname still redirects to the access login, and 41 real events ingested in the three minutes after restart with zero peer-trust rejections, meaning nothing legitimate was caught in the net.
Problems & fixes
The gate locked me out within a minute of deploying, and I found it before the agent did. My phone got the gate's own 403 JSON. The log named the rejected peer immediately: the phone's public address, not the tunnel's loopback. The web server's default behaviour is to trust forwarded-for headers from a loopback proxy and rewrite the client address accordingly, so every request arriving through the tunnel reached the gate wearing the end user's address. Those requests had already been authenticated at the edge; the gate should never have been judging them. Latent, not new: the previous launcher enabled the same behaviour, and nothing noticed because nothing used to read the peer address. Adding a gate that reads it turned a long-standing default into a fault.
The fix: disable proxy-header rewriting in the launcher, plus two regression tests, one asserting the launcher disables it and one asserting headers cannot grant trust even if it were re-enabled. Verified: a loopback request carrying forwarded headers now 200 (was 403); a home-network peer sending a forged forwarded-for loopback still refused at the socket; private-network 200 from both remote machines; event ingest resumed with zero rejections. Suite 644.
A design question answered by reading, not guessing. Would new middleware break the 642-test suite? It could not, because nearly every test builds its own app from the routers rather than importing the main app. Checking that first made a risky-looking change safe, and also noted that the suite therefore could not have caught the original hole.
Decisions
Defence in depth because one layer had already failed silently. Explicit binds and a peer gate, each sufficient alone.
Trust the overlay network's range, not "private" in general.
Read what depends on a setting before removing it, and verify with real traffic afterwards.
Test the rejections, not just the acceptances.
Learned
A control that looks like a safety check can be the only check. "Are you sure" is a UX affordance, not authentication.
Justifications in comments go stale. The bind was defended by a check that had been deleted three weeks earlier.
Adding a gate that reads the peer address exposes every upstream default about what the peer address means. The proxy-header rewrite had been there all along; nothing consumed it until the gate did.
Verify with real traffic in both directions: that the untrusted is refused, and that the trusted still gets through.
Still open / next
The same review found one more service on another machine listening on every interface by inheritance rather than by choice, with no documented access application. It is on the list, with the same two-layer treatment.