Every medical imaging model I have worked on has a number attached to it. Usually an AUC, sometimes a sensitivity at a fixed operating point, occasionally a headline comparison against a panel of readers. That number is the thing everyone remembers, and it is almost always the least interesting fact about the model.
What is interesting is the shape of the failures. Not how often the model is wrong, but when — because clinical risk is not uniformly distributed across a test set. A model that is 95% accurate overall can be nearly useless if its 5% of errors are concentrated in exactly the cases a clinician most needs help with.
This is the part of the work that does not make press releases, so I want to spend an article on it.
Failure mode one: the model learned the hospital, not the disease
The cleanest demonstration of this remains a 2018 cross-site study. Researchers trained pneumonia-detection CNNs on chest radiographs from one health system and then tested them on radiographs from another. Internal performance looked strong. External performance did not: in three of five natural comparisons, the model performed significantly worse at a hospital it had not trained on. The detail that should bother anyone building these systems is the control experiment — a CNN could identify which hospital system a radiograph came from with near-perfect accuracy, from the image alone.
That means the image carries a strong, learnable signal about its own provenance. Scanner make and model, exposure settings, processing pipeline, the physical way a portable unit gets positioned in a crowded ward, even the burned-in laterality markers. If disease prevalence differs between sites — and it always does — the model has a shortcut available. It can partly infer "sick" from "this looks like the kind of image taken at the hospital where sick people go."
The shortcut is not a bug in the architecture. It is a rational solution to the optimisation problem you handed it. You asked for the label; you did not ask that the label be predicted from pathology.
I have seen internally validated models lose a meaningful chunk of their operating-point performance on a new site's data, and the drop is rarely symmetric. Sensitivity often survives reasonably well. Specificity is what collapses, because the decision threshold was calibrated against a prevalence and an image distribution that no longer hold. A model that starts producing three false alarms for every true one gets switched off within a month, regardless of what its AUC says.
Failure mode two: shortcut learning under time pressure
The COVID-19 period produced a natural experiment in what happens when models are built fast on whatever data is available. A 2021 analysis took several published COVID chest-radiograph classifiers and asked what they were actually keying on. The answer: the models relied substantially on confounding factors — laterality markers, text annotations, patient positioning, image edges — rather than on lung pathology. Some of them were, in effect, detecting which dataset an image came from, because the COVID-positive and COVID-negative images had been pooled from different sources.
What makes this case instructive is that the reported metrics were good. If you had only the performance table, you would have concluded these were useful systems. The failure was invisible until someone ran saliency analysis and generative counterfactuals against them.
This is why I am sceptical of any validation story that consists only of numbers on a held-out split. A held-out split drawn from the same pool inherits the same confounders. Confirming that a model attends to plausible anatomy is not a nice-to-have interpretability exercise; it is the only cheap way to catch this class of error before deployment does it for you.
Failure mode three: the long tail
Training sets are shaped like clinical reality, which means they are enormously imbalanced. You will have tens of thousands of normals, thousands of the common findings the label taxonomy was designed around, and a handful of everything else. A rare entity with a few dozen examples is not something a network learns; it is noise that the loss function is happy to absorb.
The clinical consequence is uncomfortable and specific. The cases where a reader most wants a second opinion are the unusual ones — the finding that does not look like anything in the textbook. Those are precisely the cases where the model has the least basis for an opinion, and, worse, where it will not say so. A softmax will produce a confident-looking number for an input drawn from a distribution it has never seen. The model does not know it is out of its depth, and the interface does not tell the reader that either.
A radiologist encountering something genuinely strange has a feeling about it. That feeling — "I have not seen this before, I should look harder or ask someone" — is doing real diagnostic work. It is the piece of human performance that current systems most conspicuously lack, and calibration and out-of-distribution detection are still open problems rather than solved features.
Failure mode four: the image is not the case
Most deployed imaging models see a study and nothing else. A clinician sees a patient.
Consider a small pulmonary nodule flagged on a CT. Whether that finding matters depends on things that are nowhere in the pixel data: whether the patient had a pneumonia six weeks ago in that lobe, whether there is a prior study showing it stable, smoking history, age, immune status, whether the patient is currently being treated for something that produces exactly this appearance. The same pixels support very different actions depending on that context.
Systems that pull in priors and structured clinical data exist, and multimodal models are improving. But the gap between "the model processes the current image" and "the model reasons about the patient" is wider than most product marketing suggests, and closing it is mostly a data-integration problem rather than a modelling one — which is to say it is slow, unglamorous, and hospital-specific.
What follows from this
None of the above is an argument that these systems do not work. They do, in narrow, well-specified tasks, and I would not spend my working life on them otherwise. But it changes what a responsible deployment looks like.
Validate where you will deploy, not where you trained. External validation at the actual target site, on the actual scanners, is not a formality. Treat a model that has only internal validation as unvalidated.
Recalibrate thresholds locally. Prevalence differs between a screening programme and an emergency department. An operating point tuned on one is wrong for the other, and the resulting false-positive burden lands on people who will quickly stop trusting the tool.
Monitor after go-live, continuously. Scanners get replaced, protocols change, referral patterns shift. Performance degrades quietly. If nobody is measuring, nobody finds out until a miss becomes an incident report.
Design the interface around uncertainty, not around a score. A number between 0 and 1 invites the reader to treat it as a probability of disease. It usually is not one. Showing what region drove the output, and flagging when an input looks unlike the training distribution, is more honest and more useful than another decimal place.
Match the task to the failure profile. Triage and worklist prioritisation tolerate false positives reasonably well. Autonomous rule-out does not. Choosing the deployment pattern that fails safely matters more than squeezing another point of AUC.
The framing I keep coming back to is that these models are not junior radiologists. They are instruments, and like any instrument they have a range over which they are accurate and conditions under which they give a reading that is confidently wrong. Nobody expects a pulse oximeter to work through nail polish. The equivalent knowledge for imaging AI — what makes it fail, and how you would notice — is still being assembled, and the field has a bad habit of publishing the range and not the failure conditions.
That is the honest state of things. It is genuinely useful technology with a documentation problem.
Tags
Taresh Sharan