AI-Powered Medical Diagnosis: Accuracy and Limitations

On 11 April 2018, a quiet regulatory event in the United States marked a change in what machines are permitted to do in medicine. The Food and Drug Administration allowed a company called IDx to market software that could examine photographs of a diabetic patient's retina and decide, without a doctor looking over its shoulder, whether the disease had begun to eat at the back of the eye. It was the first time the FDA had authorised an artificial intelligence system to make a diagnostic call autonomously (FDA announcement, 11 April 2018). The decision did not arrive from nowhere. It rested on a decade of research, and it opened a set of questions that medicine is still arguing about eight years later: how accurate are these systems, for whom do they fail, and who answers when they are wrong.

What the machines can actually see

The strongest evidence for AI in diagnosis comes from images, because images are where the new machine-learning techniques excel. Two studies published within a year of each other became the field's touchstones. In November 2016, researchers at Google reported in JAMA that a deep-learning algorithm, trained on a very large set of retinal photographs, detected referable diabetic retinopathy with high sensitivity and specificity — that is, it caught most of the true cases while keeping false alarms relatively few (Gulshan et al., JAMA, 2016). Two months later, in Nature, a Stanford team showed that a neural network trained on roughly 129,000 images of skin lesions classified skin cancer at a level comparable to 21 board-certified dermatologists working the same cases (Esteva et al., Nature, 2017).

These results were real, and they were impressive. But it is worth stating plainly what they were not. They were not trials of doctors against machines in a working clinic. They were comparisons of performance on curated image sets, taken under conditions chosen for the algorithm's benefit: good lighting, clear photographs, a fixed set of possible answers. A dermatologist sees a patient who scratches, wears make-up over the lesion, or has a mole that has bled; the algorithm sees pixels. The moment a task moves from the laboratory to the examination room, the gap between the two begins to matter.

The clinic is not the laboratory

The best documented case of that gap is a study that Google itself published. In 2020, a team described what happened when a deep-learning system for diabetic eye screening was deployed in eleven clinics in Thailand, where nurses took the retinal photographs rather than specialist photographers (Beede et al., CHI 2020). The system's accuracy on paper was not the problem. The problems were the ones the paper had never contained: photographs rejected for poor image quality, lighting conditions the training data had never seen, slow internet connections interrupting uploads, and workflows that asked nurses to repeat a screen the machine had refused to grade. What had scored well in validation degraded in the field, not because the mathematics failed but because the world is messier than the dataset.

A second and harsher lesson came from a system that had no glamorous provenance at all. The Epic Sepsis Model was a proprietary prediction tool built into the electronic health records of hundreds of American hospitals; it warned clinicians that a patient might be sliding toward sepsis, a catastrophic reaction to infection. Its performance had never been independently tested at scale, despite that reach. In July 2021, physicians at the University of Michigan published an external validation in JAMA Internal Medicine covering nearly 30,000 hospitalisations: the model caught only about a third of the patients who actually developed sepsis, while generating alerts on roughly a fifth of all hospitalised patients — a torrent of false alarms that trained nurses to ignore it (Wong et al., JAMA Internal Medicine, 2021). One should be careful with the numbers: this was a single-centre validation, using one definition of the sepsis outcome, and the vendor disputed aspects of the analysis. But the structural finding survives every caveat. A prediction tool deployed at hundreds of hospitals had gone into production without anyone publishing proof that it worked.

The bias is in the label

The most consequential failure of medical AI found to date had nothing to do with images or sepsis scores. In October 2019, researchers reported in Science that a risk algorithm used by health systems across the United States — affecting millions of patients — systematically assigned Black patients the same risk scores as White patients who were, in fact, sicker (Obermeyer et al., Science, 2019). The cause was not malicious code. The algorithm had been trained to predict future healthcare costs, and used cost as a proxy for illness. Because less money is spent on Black patients at the same level of need, the proxy quietly imported an old inequity into a new technology. The authors calculated that correcting the proxy would have raised the share of Black patients enrolled in extra care programmes from 17.7 to 46.5 percent.

The lesson generalises. An algorithm trained on biased or unrepresentative data will reproduce the bias, often with a precision no human bigot could manage. This is the first limitation of diagnostic AI stated in the plainest terms: the system is only ever as honest as the data and the design choices behind it, and the people most likely to be missing from the training data are usually the people least likely to be asked about it.

The arithmetic of a positive result

There is also a mundane arithmetic that accuracy headlines obscure, and it deserves a worked example. Consider, purely as an illustration, a screening tool with 90 percent sensitivity and 95 percent specificity, used on a population where the disease has a prevalence of 1 percent — assumptions of the sort typical of rare-condition screening, not measurements of any real product. Out of 100,000 people screened: 900 true cases would be found (90 percent of the 1,000 who have the disease), while 4,950 healthy people would receive a false positive (5 percent of the 99,000 who do not). Fewer than one in six positive results would be a true case. The tool would be, by the standards of screening, a good one; the positive result, for most of the people who receive it, would still be wrong. When prevalence falls — when the disease is rarer, or the tool is applied to people it was never meant for — the ratio collapses further. Any honest account of "accuracy" must say which population, which prevalence, which cut-off.

Black boxes, trust and the drifting clinician

Even a good model presents a second difficulty: most deep-learning systems cannot explain themselves. A network that flags a lesion as malignant does not, in any usable sense, say why. That opacity complicates both trust and liability. If a physician defers to the machine and the machine is wrong, responsibility does not vanish; it diffuses — into the vendor's validation documents, the hospital's procurement, the clinician's judgement, the regulator's authorisation. There is also the opposite failure, documented in studies of clinical decision support going back decades: automation bias, the human tendency to go along with a confident number on a screen. The 2018 FDA authorisation acknowledged the problem from one direction by requiring the IDx-DR system to refuse to grade photographs of inadequate quality rather than guess — the machine, in effect, was designed to know when it did not know. Whether the humans supervising such systems will admit the same is a question of training and temperament, not technology.

The regulators' position

The institutions that watch this field have been explicit about the risks. The World Health Organization's first guidance on the ethics and governance of AI for health, published in 2021, set out six principles, including that AI should not narrow the health gaps between rich and poor populations, and that its developers must be transparent and accountable (WHO, 2021). Its January 2024 guidance on large multi-modal models — the generative systems now being asked to draft clinical notes and answer patient questions — warned plainly of hallucinated content, bias, and unsafe automated decisions in health contexts (WHO, 2024). In Kenya, the Data Protection Act, No. 24 of 2019, which gives effect to Article 31(c) and (d) of the Constitution, treats health data as sensitive personal data subject to stricter processing conditions (Kenya Law). The Office of the Data Protection Commissioner issued a draft guidance note on artificial intelligence in July 2026, open for public comment until 17 August 2026, setting out the obligations of controllers and developers when AI systems process Kenyans' personal data (ODPC, 2026; KICTANet, 2026). It remains draft guidance, and what it does not yet settle — consent for secondary use of medical records as training data, and the rules for automated decisions affecting care — is precisely where diagnostic AI presses hardest on the law.

What to conclude

The evidence supports a narrow and defensible claim: AI can reach expert-level performance on specific, well-defined, image-based diagnostic tasks under controlled conditions, and one such system has been used lawfully without physician review for a single indication since 2018. The evidence does not support the broader claim that AI diagnosis is, in general, accurate — the external validations of widely deployed tools have often disappointed — and it says nothing comforting about performance for populations absent from the training data. The honest position is uncomfortable on both sides: these systems are neither the fraud their critics describe nor the revolution their vendors sell. They are instruments, good at narrow things, silent about their reasoning, and shaped entirely by the choices of the people who built them. The task of medicine is to keep the clinician accountable, the data representative, the validation independent, and the claims proportionate to the evidence. That is not a limitation of the technology. It is the price of using it.

Fundus photograph used in a 2022 study of AI-assisted diabetic-retinopathy screening. Clinical image, not an AI-generated image or diagnostic output. Credit: Shaofeng Hao, Changyan Liu, Na Li, Yanrong Wu, Dongdong Li, Qingyue Gao, Ziyou Yuan, Guanyan Li, Huilin Li, Jianzhou Yang and Shengfu Fan, PLOS ONE 2022, Figure 1A, via Wikimedia Commons. Image source · CC BY 4.0 International