The pitch for medical AI is that it frees clinicians to care for patients. Two stories this week suggest the reality can run the other way. In New York, nurses say software replaced them. In Minnesota, a former Mayo Clinic leader says the software was not safe to trust.
Marilyn Shuler spent 39 years reading patient charts at Montefiore hospital in the Bronx. This month she was one of a dozen nurses laid off, replaced by AI software that does part of the same job, their union says.
The nurses handled utilization review. They read charts and argue with insurers over what care gets covered. The New York State Nurses Association says the layoffs broke a contract the nurses had just won through a strike, the Guardian reported.
“It should also concern every practitioner and patient who cares about… the quality of care they receive,” said Shaiju Kalathil, a Montefiore nurse and union representative. Montefiore disputes the account. It calls the union’s claims “inaccurate and misleading”, and says the change involves a nonclinical, paperwork programme.
What the whistleblower saw
If New York is about jobs, Minnesota is about trust. And it is more alarming.
The 💜 of EU tech
The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!
Traci Tamiko Eto joined the Mayo Clinic in 2023 to help build safeguards around its AI. In a new lawsuit, she says the hospital demoted her and then fired her for raising the alarm about its AI rollout, Minnesota Public Radio reported.
Her claims are specific. The team behind an AI assistant called MAYA, she alleges, deleted unflattering test results, overstated what the tool could do, and pushed it forward without proper oversight. At one point, the suit says, the tool had an error rate as high as 67%.
Mayo says it is committed to responsible AI, with privacy, security, and compliance built into its work. Eto’s account, if it holds up in court, describes something else. It describes a system rushed past the checks meant to catch its mistakes.
The gap between promise and practice
These are two cases, not a trend. But they sit either side of the same fault line.
Healthcare is where AI’s benefits are meant to be clearest, and where its failures are least forgivable. A mangled advert is embarrassing. A mangled chart is dangerous.
The wider record is already uneasy. Utah has let an AI renew prescriptions without a doctor. Research keeps finding that leaning on AI can quietly erode the skills professionals are meant to keep, and that it is cutting experienced careers short. Employers from Allianz to Thomson Reuters are already swapping staff for models.
Who carries the risk
The common thread is who ends up holding the risk. Montefiore keeps the savings; the patient inherits whatever the software misses. Mayo gets the efficiency; the whistleblower, she says, got shown the door.
Neither case has been tested in court, and both hospitals defend their record. But together they puncture the tidy story that medical AI only ever adds a helping hand. Sometimes it removes one. And sometimes the person who understood the machine best is the first to be told to leave.
Get the TNW newsletter
Get the most important tech news in your inbox each week.