Leading LLMs Demonstrate High Diagnostic Safety and Low Overuse in Choosing Wisely Scenarios

Article Impact Level: HIGH
Data Quality: STRONG
Summary of  Annals of Internal Medicine https://doi.org/10.7326/ANNALS-26-01667
Dr. Dean Ariel et al.

Points

Ben-Gurion University and Clalit Health Services researchers evaluated six leading large language models across clinical scenarios derived from fifteen Choosing Wisely guidelines.
Quantitative testing demonstrated an overall low language model overuse rate of five point four percent while completely avoiding unneeded testing in eight guideline categories.
Language model overtesting clustered primarily around thyroid ultrasounds for incidental subcentimeter nodules at forty two point eight percent and routine electrocardiograms at twenty eight point three percent.
False restraint involving the erroneous withholding of medically necessary care occurred in only zero point six percent of total responses across all evaluated artificial intelligence models.
Authors concluded that language models display strong restraint regarding redundant interventions but mirror existing human clinical disagreements during ambiguous diagnostic scenarios.

Summary

This study evaluated the ability of leading large language models (LLMs) to identify and withhold unnecessary medical tests and treatments while maintaining diagnostic safety. Published in Annals of Internal Medicine by researchers from Ben-Gurion University of the Negev and Clalit Health Services, the investigation evaluated six LLMs from Google, Anthropic, and OpenAI across clinical scenarios derived from 15 Choosing Wisely recommendations. Given that low-value care generates patient anxiety, invasive workups, and health system strain, the research sought to quantify rates of LLM-recommended overuse and false restraint when presented with redundant versus clinically indicated clinical scenarios.

Across thousands of responses, the overall rate of LLM-recommended overuse was 5.4% (95% CI, 0.6% to 12.2%), with LLMs completely avoiding overuse recommendations in 8 of the 15 Choosing Wisely categories. The majority of overuse recommendations clustered within two clinically ambiguous scenarios: thyroid ultrasound for incidental subcentimeter nodules (overuse recommended in 42.8% of LLM responses) and routine electrocardiography in low-risk adults (overuse recommended in 28.3% of LLM responses). For the remaining 5 recommendations where overuse occurred, LLMs recommended inappropriate interventions in only 0.6% to 3.3% of responses. Crucially, false restraint—withholding necessary care when an intervention was medically indicated—was rare at 0.6% (95% CI, 0.1% to 1.2%), was entirely absent in 11 of 15 recommendations, and peaked at 3.3% for prostate-specific antigen screening.

The authors conclude that leading LLMs demonstrate strong clinical restraint and high diagnostic safety, successfully recognizing low-value care in the vast majority of scenarios without withholding necessary interventions. Because LLM errors reflect existing areas of clinical debate among medical professional societies, further optimization is needed to manage expert disagreement and maintain appropriate restraint during patient interactions.

Link to the article: https://www.acpjournals.org/doi/10.7326/ANNALS-26-01667

References

Ariel, D., Endelman, S., & Laban, G. (2026). Overuse and false restraint in large language model recommendations for choosing wisely scenarios. Annals of Internal Medicine, ANNALS-26-01667. https://doi.org/10.7326/ANNALS-26-01667