The thing that clicked today: I had reproduced a score to within a hair and still did not know what it measured, and a twenty-minute probe told me more than the week of training did.
Loupe's ranking inside a group of photos came from Apple's own aesthetic score, which only exists for my library, on my Mac. For the app to work for anyone, that score had to be replaced with one that runs on their own hardware. The plan was distillation: use the image vectors Loupe already computes for every photo (a list of numbers describing what the picture contains) as input, and train a small model to predict Apple's score from them.
Built / shipped
The small model, and a parity check. 45,737 pairs of image vector and Apple score, split so every score range was represented in training and testing, and a small network trained on a CPU in minutes, no GPU needed for something this size. On the held-out test set it ranked photos in the same order as Apple with a rank correlation of 0.903. Ordering across the bulk of the library, where culling actually happens, was strong and well calibrated. The soft spot was the very top: Apple's own labels are scarce there, so the best-of-the-best recall was lower.
A reframe. With parity in hand, the goal changed from "match Apple" to "beat Apple, using open models anyone can run." Apple became the baseline and the teacher, not the ceiling.
A blind preference tool. A small standalone web page, only reachable on my home network, that shows me two photos and asks which is better, with no scores visible. It seeds pairs specifically where two candidate scores disagree, because those are the pairs that settle which score is closer to a human eye. 342 pairs seeded, 40,400 assets scored by both candidates, agreement measured with proper confidence intervals rather than a raw percentage.
An offline test of a stronger open teacher. A 7-billion-parameter vision-language model under an open license, with a published add-on trained for photo quality, run locally in reduced precision, scored a 4,499-photo sample drawn evenly across the score range on two axes: overall quality and composition.
Problems & fixes
The probe that changed the question. Before committing to any of that, I ran a free test on the vectors I already had: compare every photo to a handful of text descriptions and see which description's similarity tracks Apple's score. "Beautiful, well-composed photograph" versus "poor photograph" correlated with Apple at 0.098. "Worth keeping" versus "delete" correlated at 0.011. The two aesthetic phrasings only agreed with each other at 0.428. But "sharp, in-focus, well-exposed" versus "blurry, poorly exposed" correlated at 0.464, four to five times stronger than either aesthetic framing. The top-1000 photos by the aesthetic probe overlapped Apple's top-1000 by 10.1%.
The reading: Apple's "overall" score, and therefore my model distilled from it, is substantially a technical-quality score. Focus and exposure. Not composition or beauty. I had spent a week faithfully reproducing a sharpness detector.
The stronger teacher measured something else entirely. On the partial sample (about 1,800 photos, 40% of the run), its overall score correlated with Apple at 0.145 and its composition score at 0.052, while its two axes agreed with each other at 0.800. Roughly 10% of its overall answers failed to parse into a number; composition failed at 0.1%. Nearly orthogonal to Apple. Which is a stronger independent signal than the text probe, and also not proof of anything on its own: different from Apple is not the same as better than Apple. Only the blind comparison can settle that.
A status that lied. The scoring job's launcher reported done and exit 0 while the actual scoring process was still running at 40%. Verified at the source on the machine before believing it. And a shared single-slot response file was overwritten by an unrelated job, which is a latent data-loss bug in the relay that got flagged for the fleet work.
Decisions
Baseline first, personalization later. A score that works for a stranger out of the box comes before one tuned to my taste, and my taste is calibration on top, not the foundation.
Evaluate with human preference, not keep/reject history. My cull decisions are keyed to a hundred things besides quality; a blind A/B on disagreement pairs isolates the one question.
Licensing is part of the model choice. The open vision-language model's base license was verified at the actual license file, not the tag; a popular alternative turned out to inherit a research-only dataset. The training data behind the add-on still needs its terms confirmed before anything ships.
Keep the image vectors. Every experiment runs on the same vectors; nothing gets re-embedded.
Learned
Matching a number is not understanding it. A correlation of 0.903 to a teacher says nothing about what the teacher measures.
The cheapest probe was the most informative. A text-similarity test on vectors I already had reframed a multi-week plan in twenty minutes.
"Different" needs a judge. Two scores that disagree tell you there is a question, not which one is right. The blind tool exists to answer that, and I am the judge.
Verify a long job at the source. The launcher's exit code was about the launcher.
Still open / next
Let the stronger-teacher scoring finish, compute the full-sample correlations, and seed Apple-versus-teacher disagreement pairs into the blind tool. Then sit down and judge them. That verdict decides whether the second training run targets composition, technical quality, or both as separate axes.