A pipeline returns a perfectly-shaped JSON object. Every field present, every type valid, Pydantic green across the board, not a parse error in sight — and the number inside it is wrong. Not malformed. Wrong. The model misread a figure, or invented one, or pinned it to the wrong entity, and nothing in the system can tell, because nothing in the system was ever built to ask. The output was indistinguishable from a correct output, so it shipped as one. Somewhere downstream, a person made a decision on it. If you have shipped an extraction pipeline, this one probably looks familiar — it is the most common way a demo that dazzled becomes a number nobody can defend.
An extraction pipeline has to hold up on the whole corpus, not the slice in the demo — and the demo is the one place that gap stays invisible. The vendor demo ran clean: a dozen hand-picked PDFs went in, structured fields came out, the fields were correct, and everyone in the room exhaled. Somebody said the word “magic.” A six-figure number got attached to a roadmap. Then the pipeline met the real corpus — the ten thousand documents the business actually owns, the ones nobody curated — and the output quietly fell apart. Not with an error. With confident, plausible, wrong results that nobody caught until they were already downstream.
Transfer Learning is a powerful tool allowing us to use huge pre-trained models and tune them for our task. In this post we review a case of transferring weights of a CNN architecture trained ImageNet to detect chest X-Ray infected with pneumonia.