AI receptionists seem like a dangerous idea after a patient revealed her doctor’s system could not understand her speech

A GP surgery in South Yorkshire has withdrawn an AI receptionist after a woman recovering from a stroke could not make it understand her and moved to another practice. The NHS Accessible Information Standard has required practices to identify and meet patients’ communication needs since 2016.

A woman recovering from a stroke could not get her doctor’s AI receptionist to understand her. After several attempts to book an appointment she registered with a different practice, and the surgery has since withdrawn the system.

The details are the point. Her stroke had left her speech fragmented and affected her right side, and the automated service asked her not to use speakerphone, which she needed because she could hold the handset with only one hand.

This was not one bad call. Healthwatch Rotherham found that the system, an AI receptionist called Emma, “does not respond well to variations in speech such as regional accents or those with speech impediments.”

Other patients described giving up. “I could never get it to understand me, I ended up just hanging up and not bothering to try and book an appointment,” one told the watchdog, while others travelled to the surgery in person instead.

Its developer sees it differently. QuantumLoopAI says Emma is trained on a wide range of accents and dialects, supports 17 languages, lets callers ask for a member of staff at any time, and that more than 90% of patients report an improvement.

The national direction of travel is not away from this. The NHS App will use AI to triage patients as part of a £10B technology overhaul.

The governance answer so far is local. South Yorkshire’s integrated care board says it is working with Healthwatch on guidance and describes adopting these systems as an individual GP practice decision.

What is coming is more capable, not less. A British supplier is building a triage model for the NHS with Nvidia, designed to work out what a patient needs before a clinician sees them.

A standard for this already exists. The Accessible Information Standard obliges GP practices to identify, record, flag and meet patients’ communication needs, and has done since 2016.

Europe classifies the technology itself. The EU AI Act puts emergency healthcare patient triage in its high-risk tier, which attaches testing, logging and human oversight duties that a booking line does not carry.

Britain has the standard without the classification. What it has today is one practice that unplugged the machine, after a woman recovering from a stroke gave up and went somewhere else.