Every clinical machine learning project I have worked on reaches the same meeting. The model performs well, someone from the clinical side asks how it decides, and an engineer opens a laptop and shows a heatmap. The room relaxes. The heatmap is glowing over roughly the right anatomy, so the model must be looking at the right thing.
I have been the engineer in that meeting, and I want to explain why that reassurance is largely unearned — and what I now think we should be showing instead.
What a Saliency Map Is Not
Grad-CAM and its relatives produce a map of where the model's output was sensitive to changes in the input. That is a real and precise quantity. It is not an account of the model's reasoning, and the two get conflated constantly.
The distinction matters because sensitivity and reasoning can come apart completely. A model can be sensitive to the region containing the lesion while actually deciding on the basis of something correlated with it — the imaging protocol used for sicker patients, a marker burned into the corner of the film, the fact that portable chest radiographs come from inpatients who are more likely to be unwell. The map will still light up over the lung. It looks like a correct explanation for a model that has learned something you would not endorse.
This is not hypothetical. Work published in 2018 showed a pneumonia detection model that had partly learned to identify which hospital system an image came from, using image features that encoded the acquisition site, and exploited the differing disease prevalence between sites. Performance on internal data was excellent. Generalisation was not.
There is a second, more uncomfortable finding. Sanity-check experiments have shown that several widely used saliency methods produce visually similar maps even when the model's weights are randomised — meaning the map is substantially a function of the input image's edges rather than of anything the trained network learned. Not every method fails this test, but enough do that "we generated a saliency map and it looked sensible" is close to no evidence at all.
A plausible-looking explanation for a model that is wrong for the wrong reasons is worse than no explanation, because it manufactures confidence.
LIME and SHAP: What They Actually Give You
The two most-used post-hoc methods deserve a more careful reading than they usually get.
LIME fits a simple, interpretable surrogate model in the neighbourhood of one prediction by perturbing the input and observing how the output moves. For images this means occluding superpixels; for tabular clinical data, perturbing feature values.
from lime import lime_image
explainer = lime_image.LimeImageExplainer()
explanation = explainer.explain_instance(
image=scan,
classifier_fn=model.predict,
top_labels=2,
num_samples=1000,
)Two properties to keep in mind. First, the explanation describes the surrogate, not the model — the surrogate is faithful only locally, and how local is "locally" is a hyperparameter you chose. Second, LIME is stochastic. Run it twice on the same case with a different random seed and you can get materially different explanations. If you are going to show a LIME output to a clinician, run it several times first and see whether it is stable. Frequently it is not.
SHAP has a firmer theoretical footing: it allocates the prediction across features using Shapley values from cooperative game theory, with desirable properties like additivity and consistency. For tree ensembles there is an exact, fast algorithm, which is why SHAP on tabular clinical data is genuinely useful.
import shap
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(patient_features)
shap.summary_plot(shap_values, patient_features)The caveat that gets dropped: the common approximations assume feature independence, and clinical features are anything but independent. Age, comorbidity count, creatinine and medication list move together. When features are correlated, attributions get distributed among them in ways that can be misleading, and the method may evaluate feature combinations that could not occur in a real patient.
The deeper point applies to both methods. A SHAP value tells you how much a feature moved this model's output. It says nothing about whether that feature causes the outcome. If your model learned that patients seen in a particular clinic have worse outcomes, SHAP will faithfully report that the clinic feature mattered. That is a correct explanation of the model and a terrible basis for a clinical decision.
The Case for Not Needing a Post-Hoc Explanation
There is a serious argument, made most forcefully by Cynthia Rudin, that for high-stakes decisions we should stop explaining black boxes and build models that are interpretable by construction.
The usual objection is that you sacrifice accuracy. In my experience this is true sometimes and assumed always. On structured clinical data — labs, vitals, demographics, history — a well-regularised additive model or a carefully built scoring system is frequently within noise of a gradient-boosted ensemble, and you can read it. Risk scores that clinicians already use and trust work this way.
Where the argument genuinely does not apply is raw imaging. Nobody has an interpretable model that reads a chest CT as well as a deep network, and pretending otherwise would be dishonest. For imaging, post-hoc methods are what exist. That is a reason to use them carefully, not to treat them as solved.
What Clinicians Actually Want
Something I got wrong for a long time: I assumed the demand for explainability was a demand for mechanism. It usually is not.
When a radiologist asks why the model said that, the underlying question is nearly always about trust calibration — should I believe it on this case? That is a question about reliability, not about internals. And it has better answers than a heatmap.
Calibrated probabilities. If the model says 0.8, it should be right about eighty percent of the time when it says 0.8. Modern networks are typically overconfident out of the box, and calibration is cheap to fix and cheap to verify. A well-calibrated number is more actionable than any visualisation.
Honest uncertainty, including abstention. A model that can say "this case is unlike my training data, I decline to predict" is enormously more useful clinically than one that always answers. Conformal prediction is the most practical approach I know for this: it gives prediction sets with a distribution-free coverage guarantee under exchangeability, and the set size itself communicates difficulty.
Similar cases. Retrieving the nearest examples from the training set — "the model is treating this like these fourteen previously confirmed cases" — is an explanation a clinician can actually evaluate, using exactly the skills they already have. It is underused and I think it is the most promising direction in this space.
Known failure modes, documented. Where does this model degrade? Which scanner, which population, which presentation? Clinicians reason fluently about the limitations of their existing tests. Give them the same information about a model and they will integrate it sensibly.
Practical Guidance
If you are building one of these systems: treat explanations as artefacts to be validated, not produced. Test your saliency method against a randomised model and see whether the output changes. Run LIME multiple times and check stability. Sanity-check attributions against known-spurious features — deliberately train on data with a planted shortcut and confirm your explanation method reveals it. If it does not, you have learned something important about your tooling.
If you are evaluating one: ask what the explanation is an explanation of. "The model was sensitive to this region" and "this finding caused the prediction" are different claims and only the first is supported. Then ask about calibration and external validation, which are the questions the explanation was standing in for.
Regulators are moving in this direction. Transparency guidance for machine-learning-enabled devices, from the FDA and its international counterparts, focuses on documenting intended use, training population, performance across subgroups and known limitations — not on requiring a heatmap. That framing is, I think, correct. The useful form of transparency is about the model's provenance and behaviour, not a picture of its internals.
The Part That Stays Hard
None of the above closes the gap. We can characterise these models much better than we can explain them, and characterisation is genuinely enough for many clinical purposes — we deploy plenty of medicine whose mechanism is incompletely understood but whose behaviour is well documented.
What we should stop doing is showing a heatmap and calling the transparency problem solved. That practice has the specific effect of making unjustified confidence look justified, in a setting where the cost of unjustified confidence is measured in patients. The honest position is that these models are hard to explain, that we have better tools for knowing when to trust them than for knowing why they work, and that saying so out loud is more useful than a reassuring picture.
Tags
Taresh Sharan