The verifier that said yes to a PDF

SkinAtlas’s catalog pipeline sources product images automatically, and because automated image lookup is wrong more often than it’s right, every candidate passes through a verification step before it’s accepted. A vision model compares the candidate against a reference and votes.

Two bugs got through anyway. One candidate was a PDF from Wikimedia Commons. Another was a 404 page wearing an image URL. Both ended up accepted into the catalog.

The verifier hadn’t approved them. It had errored on them, because you can’t run image verification on a PDF or on an error page. And the code treated “verification didn’t complete” as “no objection raised.” Fail open.

A check that shrugs is not a check

The failure mode is worth naming precisely, because it hides in a lot of pipelines. The verification step existed, ran, and logged. Anyone auditing the pipeline would see a verification stage and assume everything in the catalog had passed it. But the gate only stopped things it could positively reject. Anything that confused it walked through.

Security people have a name for this: fail open versus fail closed. A door that unlocks when the power cuts. My verifier was that door. The fix inverted the default: a candidate is rejected unless verification positively succeeds. Errors, timeouts, weird MIME types, all of it now lands on the reject pile, where a human can look at it later.

Cheap filters in front of expensive ones

The second half of the fix was structural. The vision model is the expensive, smart check, and it was being asked to handle garbage that a dumb check could have caught. Now candidates from Commons are pre-filtered by MIME type and by keyword overlap in the file title before the vision pass ever runs. The PDF never reaches the model; it dies at the MIME check, which costs nothing.

That ordering, cheap-and-dumb before expensive-and-smart, is the same shape as the QA hierarchy I wrote about in the pipeline note. The pattern keeps reappearing: every expensive judgment deserves a cheap bouncer in front of it, and the bouncer’s job isn’t to be clever. It’s to make sure the expensive judgment only ever sees questions it can actually answer.