The thing that clicked today: the machine showing as offline was fine, and the machine showing as fine was the one that had failed.
Built / shipped
A diagnosis that started in the wrong place and got corrected by evidence. The visible symptom was my storage box reading offline, last seen two hours ago. Checking it directly showed it had been up for four and a half days on the same boot, its own network interface had not flapped once, and it was serving files to another machine the entire time at well under a millisecond. It was not offline. It was unreachable by one particular path.
The real fault was a network port on the compute box. Its onboard interface had gone into a continuous hardware transmit hang, starting about two and a half hours after that machine booted, and it stayed there unrecovered for nearly three hours across more than five thousand kernel messages. The interface never came back on its own.
The reason it looked like a storage problem is that one port had two jobs. That same interface is both the compute box's connection to the storage network and the route the storage box uses to reach the rest of the world. When it stalled, the compute box lost its storage peers and the storage box lost its only path out, at exactly the same moment, from one fault.
A third machine was completely unaffected and that was the giveaway. It sits on the storage network directly instead of routing through the compute box, so it kept talking to the storage box the whole time. Two machines seeing completely different realities about a third is the shape of a routing fault, not an endpoint fault.
Problems & fixes
The recovery was resetting the stalled interface, and I could not do it from where I was standing. The fix itself is trivial, and the automation I was driving from was not permitted to run it, so I ran it by hand from my laptop. Worth writing down because the diagnosis being remote and the repair needing to be local is a normal situation, not a failure.
The alert named the wrong machine, and it was not wrong to. The monitoring reported what it observed: it could not reach the storage box. That was true. It has no way to know the unreachability was caused two hops away. An alert reports a symptom, and I had been reading symptoms as causes.
Decisions
One interface carrying both the storage network and another machine's only route out is a single point of failure I had not written down as one. It is now recorded as such. That does not fix it, but an untracked dependency is worse than a known one.
Interface hangs on this hardware get treated as expected rather than surprising. This is a recurrence, not a first occurrence, so the useful question is detection and recovery rather than root cause I cannot fix in the driver.
Learned
Ask which machines disagree, not which machine is broken. Two observers with different views of the same third machine locates the fault at the difference between them, which is where the routing is.
Uptime and interface counters are stronger evidence than a dashboard. The dashboard was reporting a genuine failure to reach something. The box itself knew it had been up for days and never lost carrier, and those two facts together are what redirected the search.
A shared interface makes one fault look like two. Because the stall hit the storage path and the route out simultaneously, the symptom pattern suggested something wrong in the middle. There was, but it was a port, not a network.
Still open / next
The port still hangs. It has done it before and it will do it again, and I have no fix beyond resetting the interface when it happens.
Nothing detects the hang automatically yet. The kernel says so clearly and repeatedly, and nothing is watching for it. A watcher that catches the message and resets the interface is the obvious next step, and until it exists this recurs as a manual outage.