Medical Interview Performance Using an AI-Based Standardized Patient vs a Human Standardized Patient: Randomized Crossover Trial.
Researchers
Yasutaka Yanagita, Daiki Yokokawa, Fumitoshi Fukuzawa, Shiichi Ihara, Ryo Yoshida, Rurika Sato, Takanori Uehara
Abstract
In Japan, standardized patients (SPs) have traditionally been recruited to teach medical interview competencies. Recent advances in large language models have enabled increasingly natural conversational interactions, raising the possibility of using AI-based SPs (AI-SPs) for medical interview training. However, medical interview performance during interactions with AI-SPs compared with human SPs remains insufficiently understood. This randomized single-center crossover trial aimed to compare medical interview performance and learner-reported experiences between encounters with a large language model-based AI-SP and a human SP. Participants were fifth-year medical students undertaking clinical clerkships at Chiba University Hospital. The AI-SP was a text-based simulated patient implemented using GPT-4o. Participants were randomized to AI-SP followed by human SP, or the reverse. A washout period of at least 3 days was set between conditions. Each participant completed 2 cases, depression and appendicitis, in both conditions. For the AI-SP interviews, chat logs were retained, and a physician provided feedback after completion of the 2 cases. Human SP interviews were video recorded, and the physician portraying the SP provided feedback immediately after each interview. Medical interview performance was assessed using selected items from the Mini-Clinical Evaluation Exercise (Mini-CEX), with ratings performed independently by 2 general medicine specialists. Participants completed a self-rated questionnaire (7-point Likert-type scale) regarding learning satisfaction, perceived rapport formation, and active engagement in history taking. Forty-eight participants were analyzed. The Mini-CEX-based mean total score was significantly higher for the human SP than for the AI-SP (23.9, SD 1.7 vs 22.7, SD 1.7, respectively; mean difference 1.24; t<sub>47</sub>=3.7; P=.001). Scores for history taking and overall clinical competence were higher in the human SP condition. Interrater reliability (intraclass correlation coefficient of 2,1) was 0.76 (95% CI 0.66-0.83). No significant differences were found between conditions for learning satisfaction (human SP: mean 5.9, SD 1.1; AI-SP: mean 6.0, SD 0.8) or perceived rapport formation (human SP: mean 5.4, SD 1.3; AI-SP: mean 5.3, SD 1.3). Active engagement in history taking was significantly higher in the AI-SP condition (human SP: mean 5.8, SD 1.1; AI-SP: mean 6.2, SD 0.7; t<sub>47</sub>=2.7; P=.009). Medical interview performance differed between the 2 SP conditions, with higher history taking and overall clinical competence scores in the human SP condition. The text-based nature of the AI-SP and the absence of nonverbal cues may have contributed to these differences. Although no significant differences between conditions were observed in most other assessment outcomes, learning satisfaction, or perceived rapport, self-rated active engagement in history taking was higher with the AI-SP condition. AI-SPs may offer a complementary modality for medical interview practice; however, this study assessed performance during simulated encounters rather than subsequent learning gains or performance with real patients. UMIN Clinical Trials Registry UMIN000056228; https://center6.umin.ac.jp/cgi-open-bin/ctr_e/ctr_view.cgi?recptno=R000064153.Source: PubMed (PMID: 43038524)View Original on PubMed