The thing that clicked today: I was queuing work behind a busy machine out of habit, when three other machines were sitting idle and perfectly able to do it.
Built / shipped
Who runs a job and where it lands are now separate questions. Each machine used to own a category of work, so a job for that category waited for that machine even when the work itself did not care. Now a request names the machine it needs to affect, and the system picks any free runner to carry it out, reaching across to the target when the runner is not already there.
Prefer local, spill over when busy. If the machine that owns the target is free, it does the work, because that is the shortest path. If it is busy, the longest-idle free machine takes the job instead and reaches the target remotely rather than everything waiting.
Locks describe the work, not the hardware. Any job that changes something has to declare what it will change: a specific repository, a specific service, a specific path, or the whole machine when that is honestly the scope. Two jobs touching the same thing serialise. Two jobs touching genuinely different things on one machine run at the same time. Read-only jobs declare nothing and never block anyone.
Selection and launch happen under one lock. Choosing a free runner and starting it are a single step, because checking availability and then acting on it is a race that resolves in exactly the wrong way when two requests arrive together.
A written route for every job. Which runner was chosen, which machine was targeted, how it got there, and why, recorded per run alongside the original request. The original request is preserved unchanged; the routing details are added beside it rather than edited into it.
Problems & fixes
Reaching further does not mean being allowed more. The obvious risk in letting any machine act on any other is that remote access quietly becomes broader permission. So the rule is explicit: remote access is transport only. It does not widen what the job may touch, which paths it may use, what it may restart, or what it is allowed to do without asking. The approval boundary belongs to the original request and travels with it unchanged.
Declaring only the most important machine leaves the others unprotected. A job that changes things on two machines used to name the one that mattered most. That silently left the second one unguarded against a job running elsewhere. Now every affected machine must be declared, and claims are compared as whole sets in both directions, so an undeclared second machine is genuinely unprotected rather than merely undocumented.
Machine-wide locks are honest sometimes. Some changes really do affect an entire machine. There is an explicit scope for that, and it deliberately conflicts with everything else on that machine. The point of finer-grained locks is accuracy, not always claiming less.
Decisions
Never migrate a job that is already running. Once something has started, later information will eventually suggest a different runner would have been better. Moving it buys very little and risks the work.
Pin a specific runner only for a real capability. Not for preference, and not because that machine historically owned that kind of task. Habit was the thing this change existed to remove.
Idleness beats ownership. If the machine that "owns" a task is busy and another is free and capable, the free one takes it. The old arrangement optimised for a tidy diagram rather than for work getting done.
Learned
Locking by machine was both too strict and too loose at once. Too strict, because unrelated jobs on one machine blocked each other for no reason. Too loose, because a job spanning two machines was only protected on one of them. Locking by what the work touches fixed both directions at the same time.
Convenience and permission need separating on purpose. Making every machine able to reach every other is useful and says nothing about what any of them should be allowed to do. Writing that distinction down as a rule was more valuable than the routing itself.
Still open / next
Longest-idle is a crude way to choose a runner. It works because my machines are similar enough, and it will stop working the moment one of them is meaningfully better at something.
I have no fairness guarantee. Nothing prevents a long job from repeatedly losing to short ones, and I have not seen it happen yet, which is not the same as it being impossible.