The thing that clicked today: two configuration files can both be correct and one of them still wins purely because of how its name sorts, and the effect can be nine days of logs that were never on disk.
The control-plane machine is a small single-board computer whose logs I rely on to reconstruct incidents. A drop-in config explicitly set the system log to persistent storage. The logs were still disappearing.
Built / shipped
The cause. The local drop-in was named with a low number. The board vendor ships its own drop-in, setting volatile storage to spare the SD card, with a higher number. Drop-ins apply in filename order, so the vendor file applied after mine and won. Every log file lived in a memory-backed filesystem. No reboot since the sixth, and roughly nine days of history had already rotated away.
The fix, smallest safe. Rename, not a new file: the local drop-in moved to a number that sorts last, and the vendor file was left untouched, because the override is the mechanism and the right place to express intent. Size caps set deliberately, 64 MB total and 16 MB per file, because the persistent log directory sits inside a 128 MB RAM-backed filesystem on this board and an uncapped journal would have eaten it. Confirmed exactly one persistence drop-in remained afterwards, and the original was archived with a hash and a manifest before anything moved.
Activation without a reboot. Restarting the log daemon with the new config still showed the logs in the volatile path, because the daemon does not retroactively migrate files that are already open. An explicit flush moved them. Cross-reboot durability depends on the board's RAM-to-card sync layer, which is a separate piece of the hardware work.
Problems & fixes
Half the reported symptom was a false alarm worth recording honestly. The ticket said logs could not be read. Read access was already granted through the active login session; the historical complaint was a wrong-invocation artifact, not a permission wall. One real bug, one wrong command, same symptom.
Decisions
Rename to sort last; never edit the vendor's file.
Cap the journal to fit the RAM-backed directory it lives in.
Flush explicitly after the config change, because the daemon will not move open files on its own.
Learned
Configuration that applies in sort order is configuration you can lose by choosing the wrong number. Check what applies after yours.
A constraint-aware cap matters: the right storage mode with the wrong size limit is a different outage.
Report the half of a symptom that was your own mistake alongside the half that was real.
Still open / next
Making the persistent journal survive a power cut depends on the card-sync layer, which belongs to the hardware batch; until then, persistence means "survives a daemon restart," and the notes say exactly that.