The thing that clicked today: I had been carrying an estimate of how much duplicate data was on my NAS, and every step of actually measuring it moved the number.
Built / shipped
A hash pass over the candidates, not the whole array. Two files can only be identical if they are the same size, so the only files worth hashing are the ones whose size collides with another file. That turned a multi-terabyte problem into about 184,000 files and 696 GB, which is a job that finishes.
The result: 52,777 groups of true duplicates covering 110,803 files, about 254 GB reclaimable. Nothing was deleted. The output is a report and a spreadsheet, because deciding what to remove is a separate decision from finding out what is redundant, and collapsing those two steps is how people lose files.
Resumable and checkpointed. It ran detached with its progress written down as it went, so a disconnect or a reboot would have cost minutes rather than the whole pass.
A single lock for heavy storage work. There was no mechanism preventing two heavy jobs from hitting the array at once. Now there is one, and the pass ran under it.
Problems & fixes
My first query was wrong by a factor of eight. The initial number came back at about 5,460 GB, which was a naive total rather than the size-collision total I actually wanted. It was caught and discarded before it went anywhere, but it would have been an easy number to repeat confidently. The corrected figure reconciles exactly against two independently known baselines, which is the only reason I trust it.
Two thirds of a job I thought was covered was unreachable. The candidate set spanned nine shares. Only two were mounted, because an earlier pilot had only needed those two. About 382 GB of candidates were sitting on shares nothing was connected to, so a pass that looked complete would have quietly skipped them. Mounted everything and re-ran for real coverage.
The rising error count was the storage vendor's own bookkeeping. Partway through, errors were climbing steadily and the rate was dropping, which looks exactly like a job going wrong. Almost all the errors turned out to be the NAS's internal thumbnail and metadata pseudo-paths, which do not exist as real files from the client's perspective. The slowdown was file-size mix: half an hour grinding through enormous video files, then thousands of tiny ones a minute. A live read-rate check confirmed the link was saturated and the machine was otherwise idle. Both alarming signals were benign, and I only know that because I checked instead of aborting.
A detached job outlived the session that started it. The job kept running correctly. What did not happen was the finishing work, because the session that launched it ended rather than staying alive to pick it up, so nothing wrote the durable result and nobody ran the follow-up steps. Caught by checking the actual evidence rather than the status line, which said done and exit zero.
Decisions
Exclude the backup blob at the query level, not by hoping. Machine-backup bundles are enormous, internally chunked, and full of legitimately identical pieces. They are excluded explicitly and the numbers are reported both ways.
Never delete in the same pass that discovers. Finding duplicates and choosing which copy is canonical are different jobs with different risk. One produces a report; the other requires a human looking at it.
Photos are excluded from general passes. One folder is roughly half the bytes on the array, which distorts every aggregate it appears in. It gets handled deliberately or not at all.
Learned
Every check I ran moved the number, and one moved it by a factor of eight. A figure that has not been reconciled against something independent is a guess with decimal places.
"The job succeeded" and "the job did what I meant" are separate claims. The coverage gap is the clearest example: the pass completed cleanly over two thirds of what it should have looked at, and reported success.
Alarming metrics deserve five minutes before they deserve an abort. Both scary signals here were normal, and killing the pass on either would have cost hours and taught me nothing.
Still open / next
Nothing has been reclaimed. There is a report and a decision to make, and the decision is the slow part because it needs judgement about which copy is the real one.
The heavy-job lock is a convention, not an enforced gate. It works because I respect it, which is the weakest kind of works.