Clinical decision support is older than machine learning by decades, and understanding why matters, because most of what goes wrong with AI-based decision support goes wrong for reasons that were already well documented in the era of if-then rules.
A decision support system is any software that puts information in front of a clinician at the moment they are making a decision. A drug interaction warning is decision support. So is a dosing calculator, a reminder that a screening test is overdue, a risk score computed from vitals, and a model that flags a suspicious region on a radiograph. A 2020 overview in npj Digital Medicine surveys the category and, usefully, gives as much space to the risks as to the benefits — the failures cluster around workflow fit, alert fatigue and over-reliance far more than around the underlying logic being wrong.
That paper's framing has held up, and the arrival of machine learning has not changed it. It has just raised the stakes.
What machine learning actually changed
The honest answer is: the input space, not the paradigm.
Rule-based support requires a human to have written down the rule. That works well for things medicine already knows explicitly — this drug interacts with that one, this patient is due for this screening. It works badly for anything where the relevant signal is a pattern nobody has articulated, or where the inputs are unstructured.
Machine learning extends decision support into three places rules could not reach. It can read free text, so a system can act on what the clinician dictated rather than only on what got coded. It can read images and waveforms directly. And it can fit relationships across many weak predictors that no one would think to encode by hand — which is how risk models for deterioration, readmission and sepsis got built.
What did not change: the system still has to interrupt a human being who is busy, at a moment they did not choose, with information they did not ask for. That interaction is where these projects live or die, and no amount of model quality rescues a bad one.
Alert fatigue is the default outcome
If you build a decision support system and do not actively fight this, you will produce alert fatigue. It is not an edge case; it is what happens by default.
The mechanism is simple arithmetic that people consistently get wrong. Suppose a model has 95% specificity — respectable — and the condition it detects has a prevalence of 1%. Out of a thousand patients, ten have the condition and the model catches most of them. It also flags around fifty of the 990 who do not. The clinician sees roughly five false alarms for every true one, and that is with a model most teams would be pleased with.
Clinicians are rational. Faced with an alert that is wrong most of the time, they learn to dismiss it, and then they dismiss the one that was right. Override rates north of ninety percent are well documented for some categories of legacy alerting, and nothing about a neural network underneath changes that behaviour.
The fixes are not sophisticated, they are just unglamorous. Tune the operating point to the local prevalence rather than shipping the vendor default. Suppress alerts the clinician has already acted on. Route non-urgent findings to a worklist instead of a modal dialog. Measure override rates as a primary system metric and treat a rising one as an incident, not as user error. And — the one people resist — be willing to fire fewer alerts and accept some misses, because a system that is ignored has a sensitivity of zero in practice regardless of what it has on paper.
Automation bias runs in the other direction
The opposite failure is quieter and worse. When a system is usually right, people stop checking. This is well established across aviation and industrial control long before it arrived in medicine, and there is no reason clinicians should be immune.
In imaging specifically, the concerning pattern is that a reader who sees no flag from a well-regarded model looks less carefully than a reader working unassisted. The model's misses then become the system's misses, and because the reader has been nudged rather than told, nobody can reconstruct afterwards why the finding was not pursued.
This is why I am wary of interfaces that present a single score. A number implies a precision the model does not have and gives the reader nothing to disagree with. Showing the evidence — the region, the contributing variables, the comparison to prior — lets a clinician exercise judgement about whether the system is looking at the right thing. It also makes the failures visible, which is the only way they get fixed.
What deployment reveals that validation does not
The gap between a published result and a working installation is the single most underestimated thing in this field.
A model validated retrospectively on curated data operates under conditions that do not hold in a live system: complete inputs, clean labels, a fixed population, no time pressure, and the outcome already known. Live, it gets partial data, arrives before the labs come back, and runs on a population that may differ substantially from the development cohort.
There is a well-publicised case of a proprietary sepsis prediction model that was deployed across hundreds of US hospitals and, when independently evaluated at an academic centre, showed discrimination substantially worse than the figures the developer had reported — while also generating a very large number of alerts per case identified. The model was not fraudulent. It was validated under conditions that did not survive contact with a different patient population and a different set of documentation habits.
The lesson I take from that is procedural rather than technical. A decision support model needs local validation before go-live and continuous monitoring after it, and the health system deploying it needs the capacity to do that itself rather than taking the vendor's numbers on trust. That capacity is expensive and almost never budgeted.
The burnout argument, handled carefully
Decision support is frequently sold on clinician burnout, and the burnout problem is real and expensive: one widely cited estimate puts the cost attributable to physician burnout in the US at roughly $4.6 billion a year, driven by turnover and reduced clinical hours.
But it is worth being clear about the direction of the arrow. A substantial contributor to that burnout is the electronic health record itself — documentation burden, inbox volume, and, yes, alerts. Adding another interruptive system to a clinician's day is at least as likely to make this worse as better.
The decision support that genuinely reduces load is the kind that removes work rather than adding information. Ambient documentation that drafts a note the clinician edits. Automatic pre-population of a structured report from measurements the system already computed. Worklist reordering so the urgent study surfaces first without anyone triaging manually. These are less exciting than diagnostic suggestion, and they are where the reliable wins have been.
What good looks like
The deployments I would consider well-designed share a few properties.
They target a decision that is genuinely hard and genuinely frequent, rather than one that is merely easy to model.
They are scoped narrowly enough that failure is bounded. Prioritising a worklist fails gracefully; autonomous rule-out does not.
They were built with the clinicians who will use them, early, including the ones who think it is a bad idea. The sceptics identify the workflow problems that kill projects, and they do it for free before you have spent the money.
They surface reasoning, not just conclusions, and they make it easy to disagree and record the disagreement — which then becomes the feedback data the next version needs.
They are monitored the way any piece of clinical equipment is monitored, with someone accountable for noticing when performance drifts.
And they have an off switch that a clinical lead, not a vendor, controls.
None of this is about how good the model is. The modelling is the part we collectively know how to do. The part the field is still bad at is everything that happens between a validated model and a clinician who is better off for having it.
Tags
Taresh Sharan