The thing that clicked today: a script defending itself is no defence against a sandbox built before it starts, and the unit whose job is to report silent failures is the one most likely to fail silently.
A small watchdog service on the control-plane machine checks that a device is still syncing its data and notifies me if the check-ins stop. The handoff described it as dead since the day before.
Built / shipped
The correction first. It had not died; it had never worked. The unit file was written at 06:19:58 on the fourth, fired for the first time at 09:19:13 the same morning, and failed with a namespace error. That single failure was its complete journal history.
The cause. The unit declared a writable state path for its sandbox. The service manager builds the mount namespace before the process starts, and a writable-path directive can only bind a path that already exists. The directory did not exist yet, even though the Python script creates it itself, correctly, with the right permissions, on its first run. The script's own care could not help, because the sandbox was assembled first.
The fix. Use the directive that asks the service manager to create the state directory itself before building the namespace, and recreate it if it is ever removed, verified by deleting it and restarting. With an explicit private mode, because the directive's default is wider than what the script had chosen, and silently loosening a permission the script had deliberately set would have been a small real regression introduced by the fix.
Why nobody heard it fail. Its three sibling units, a rebuild and two backups, each carry a drop-in that routes a failure to the notifier. All three drop-ins were created at 06:19 on the same morning, the same minute this unit was written. It was the one in that batch that did not get one. So the unit whose whole job is to report a silent failure failed silently. Now declared inline in the unit rather than as a fourth untracked drop-in, since the unit is tracked.
Proven, not assumed. Fifty-four tests, including a new one that compares the unit's declared state directory against the script's default rather than a hardcoded string. The notification path was exercised by making the real unit fail once behind a self-removing drop-in, because "is notified" is not known to work until it has notified. And the timer fired unattended the next morning and the service succeeded, the first scheduled success in its history.
Problems & fixes
The notifier template and all three sibling drop-ins were untracked in version control, two of them on the backup path. Noted as a separate repair rather than folded into a watchdog fix.
Decisions
Let the service manager own the state directory, with an explicit mode.
Declare the failure hook inline in the tracked unit.
Prove the notification path by failing once on purpose.
Learned
Order matters in a sandbox. Anything the process creates for itself is too late for directives that bind paths before it starts.
A fix can widen a permission the original chose. Read the directive's default.
The monitor of silent failures needs its own monitor, and it is the one most easily forgotten when the batch is created.
Still open / next
Track the notifier template and the sibling drop-ins, and audit every unit on the machine for the same missing failure hook, which became the next day's work.