← All posts

August 6 - checking whether the job ran, not whether the timer is armed

The thing that clicked today: a timer can be perfectly armed and completely useless, a check that always returns a success-shaped value is worse than no check, and the service manager's effective configuration lives in places a file search does not reach.

The day after fixing the watchdog that had never run, the question was what else on that machine was unregistered, unprotected, or unmonitored.

Built / shipped

The inventory gap. The control-plane machine runs 16 user timers; the fleet's automation registry listed 13. The three missing were exactly the ones suspected, and two of them were backups, which is the worst possible place for an inventory gap, because a backup that stops is invisible until the day the copies are needed. The other two machines matched the registry exactly once a package-owned unit is excluded by existing policy, so the gap was bounded to one machine rather than fleet-wide rot. Both directions now proven clean: nothing live is unregistered, nothing registered is a phantom.

The near-miss, which is the useful part. A search across the unit files for the failure-notification directive found one hit, the watchdog. That would have meant the previous day's account was backwards and the three siblings were unprotected. It was the search being too narrow, not a finding: the rebuild and both backups carry the declaration through drop-in directories, which a pattern over the unit files cannot see. Querying the loaded property on each unit showed all four wired correctly. Had the first result been written up, a non-existent hole in the backup notification path would have been "discovered." The lesson went into the index: query the effective configuration, never the file, because the service manager assembles it from places a search does not reach.

A deliberate non-registration. The failure handler itself is a static unit with no enablement or activity state. It is never enabled, never appears in a timer listing, and exists only to be reached by a failing sibling. Every probe the registry checker owns would return a success-shaped value for it whether or not the notification path actually works, so registering it would manufacture false assurance. It stays unregistered, with the reason written down.

Two signals, not one. The checker now also reads each triggered service's result, so "the timer is armed" and "the last run succeeded" are both required for green.

Problems & fixes

A static web server unit turned out to be enabled and actively running on the machine with no registry entry and no obvious owner. Recorded for a ruling, not stopped in passing.

The failure notifier deliberately omits journal lines from its payload, because they carry paths and command lines and the payload crosses a third-party push service. That is itself a discretion decision worth keeping: what you let leave your network.

Decisions

Registry parity in both directions, per machine.

Query loaded properties, never unit files, for anything drop-ins can supply.

Do not register a unit the checker can only ever report as fine.

Green requires the timer armed and the last result successful.

Learned

Armed is not working. A timer is a promise; the service result is the evidence.

A grep over config files is a guess about configuration. The effective state is what the manager says it is.

A check that always passes is a liability, not a metric.

Still open / next

The unowned static server needs a ruling. And the same drop-in blindness applies to every other property the fleet checks by reading files; those checks want the same conversion to loaded-property queries.