Comparison of Assessment by a Virtual Patient and by Clinician-Educators of Medical Students' History-Taking Skills: Exploratory Descriptive Study

Primary care physicians' perceptions of artificial intelligence systems in the care of adolescents' mental health

BACKGROUND

Given that mental health problems in adolescence may have lifelong impacts, the role of primary care physicians (PCPs) in identifying and managing these issues is important. Artificial Intelligence (AI) may offer solutions to the current challenges involved in mental health care. We therefore explored PCPs' challenges in addressing adolescents' mental health, along with their attitudes towards using AI to assist them in their tasks.

METHODS

We used purposeful sampling to recruit PCPs for a virtual Focus Group (FG). The virtual FG lasted 75 minutes and was moderated by two facilitators. A life transcription was produced by an online meeting software. Transcribed data was cleaned, followed by a priori and inductive coding and thematic analysis.

RESULTS

We reached out to 35 potential participants via email. Seven agreed to participate, and ultimately four took part in the FG. PCPs perceived that AI systems have the potential to be cost-effective, credible, and useful in collecting large amounts of patients' data, and relatively credible. They envisioned AI assisting with tasks such as diagnoses and establishing treatment plans. However, they feared that reliance on AI might result in a loss of clinical competency. PCPs wanted AI systems to be user-friendly, and they were willing to assist in achieving this goal if it was within their scope of practice and they were compensated for their contribution. They stressed a need for regulatory bodies to deal with medicolegal and ethical aspects of AI and clear guidelines to reduce or eliminate the potential of patient harm.

CONCLUSION

This study provides the groundwork for assessing PCPs' perceptions of AI systems' features and characteristics, potential applications, possible negative aspects, and requirements for using them. A future study of adolescents' perspectives on integrating AI into mental healthcare might contribute a fuller understanding of the potential of AI for this population.

Exploring medical students' perceptions of individual and group-based clinical reasoning with virtual patients: a qualitative study

BACKGROUND

Virtual Patients are computer-based simulations used to teach and evaluate patient interviews, medical diagnoses, and treatment of medical conditions. It helps develop clinical reasoning skills, especially in undergraduate medical education. This study aimed to and investigate the medical students' perceptions of individual and group-based clinical reasoning and decision-making processes by using Virtual Patients.

METHODS

The study group comprised 24 third-year medical students. Body Interact® software was utilized as a VP tool. The students' readiness and the courses' learning goals were considered when choosing the scenarios. Semi-structured interview forms were employed for data collection. MAXQDA 2020 qualitative analysis software was used to analyze the data. The students' written answers were analyzed using content analysis.

RESULTS

The participants perceived individual applications as beneficial when making clinical decisions with Virtual Patients, but they suggested that group-based applications used with the same cases immediately following individual applications were a more appropriate decision-making method. The results indicated that students learn to make decisions through trial and error, based on software scoring priorities, or using clinical reasoning protocols.

CONCLUSION

In group-based reasoning, the discussion-conciliation technique is utilized. The students stated that the individual decision-making was advantageous because it provided students with the freedom to make choices and the opportunity for self-evaluation. On the other hand, they stated that the group based decision-making process activated their prior knowledge, assisted in understanding misconceptions, and promoted information retention. Medical educators need to determine the most appropriate method when using Virtual Patients, which can be structured as individual and/or group applications depending on the competency sought.

Predominant aspects of knowledge and practical skills among medical students with online learning during the COVID-19 pandemic era

BACKGROUND

Social restrictions due to the COVID-19 pandemic have shifted most learning methods into online courses, especially for medical skills education. However, the effects of online courses on medical skill education amongst medical students are still arguable. The study aims to analyse medical students' knowledge, attitude, practice and satisfaction towards medical skills between online and offline courses.

METHOD

We performed a case-control study conducted among 533 medical students with online (as a case group, n = 288) and offline courses (as a control group, n = 245). We evaluated three fundamental medical skills, including history taking [HT], lung physical examination [LPE], and heart physical examination [HPE]. We tested the knowledge and skills among students through theory and practical examinations. Students' attitudes and satisfaction were assessed using a validated questionnaire.

RESULTS

The scores for knowledge and practical skills among the online group were significantly higher (p = 0.016, p = 0.004, respectively). In comparison, the scores for the students' attitudes and satisfaction were substantially lower (p = 0.000, p = 0.003, respectively) compared to the control group. Most of the students in both groups passed the exam (case vs. control = 81.94%; 83.27%, respectively). Males were the only factor associated with a higher rate of passing the examination (OR 0.42, 95% CI [0.27-0.67], p = 0.000).

CONCLUSIONS

Online learning could be an alternative approach on improving student's knowledge and practice towards medical skill especially amidst COVID-19 pandemic, however further consideration on student's attitude and satisfaction are mandatory to achieve appropriate competence as future general practitioner.

Learning Analytics Applied to Clinical Diagnostic Reasoning Using a Natural Language Processing-Based Virtual Patient Simulator: Case Study

BACKGROUND

Virtual patient simulators (VPSs) log all users' actions, thereby enabling the creation of a multidimensional representation of students' medical knowledge. This representation can be used to create metrics providing teachers with valuable learning information.

OBJECTIVE

The aim of this study is to describe the metrics we developed to analyze the clinical diagnostic reasoning of medical students, provide examples of their application, and preliminarily validate these metrics on a class of undergraduate medical students. The metrics are computed from the data obtained through a novel VPS embedding natural language processing techniques.

METHODS

A total of 2 clinical case simulations (tests) were created to test our metrics. During each simulation, the students' step-by-step actions were logged into the program database for offline analysis. The students' performance was divided into seven dimensions: the identification of relevant information in the given clinical scenario, history taking, physical examination, medical test ordering, diagnostic hypothesis setting, binary analysis fulfillment, and final diagnosis setting. Sensitivity (percentage of relevant information found) and precision (percentage of correct actions performed) metrics were computed for each issue and combined into a harmonic mean (F₁), thereby obtaining a single score evaluating the students' performance. The 7 metrics were further grouped to reflect the students' capability to collect and to analyze information to obtain an overall performance score. A methodological score was computed based on the discordance between the diagnostic pathway followed by students and the reference one previously defined by the teacher. In total, 25 students attending the fifth year of the School of Medicine at Humanitas University underwent test 1, which simulated a patient with dyspnea. Test 2 dealt with abdominal pain and was attended by 36 students on a different day. For validation, we assessed the Spearman rank correlation between the performance on these scores and the score obtained by each student in the hematology curricular examination.

RESULTS

The mean overall scores were consistent between test 1 (mean 0.59, SD 0.05) and test 2 (mean 0.54, SD 0.12). For each student, the overall performance was achieved through a different contribution in collecting and analyzing information. Methodological scores highlighted discordances between the reference diagnostic pattern previously set by the teacher and the one pursued by the student. No significant correlation was found between the VPS scores and hematology examination scores.

CONCLUSIONS

Different components of the students' diagnostic process may be disentangled and quantified by appropriate metrics applied to students' actions recorded while addressing a virtual case. Such an approach may help teachers provide students with individualized feedback aimed at filling competence drawbacks and methodological inconsistencies. There was no correlation between the hematology curricular examination score and any of the proposed scores as these scores address different aspects of students' medical knowledge.

Assessment of Diagnostic Competences With Standardized Patients Versus Virtual Patients: Experimental Study in the Context of History Taking

BACKGROUND

Standardized patients (SPs) have been one of the popular assessment methods in clinical teaching for decades, although they are resource intensive. Nowadays, simulated virtual patients (VPs) are increasingly used because they are permanently available and fully scalable to a large audience. However, empirical studies comparing the differential effects of these assessment methods are lacking. Similarly, the relationships between key variables associated with diagnostic competences (ie, diagnostic accuracy and evidence generation) in these assessment methods still require further research.

OBJECTIVE

The aim of this study is to compare perceived authenticity, cognitive load, and diagnostic competences in performance-based assessment using SPs and VPs. This study also aims to examine the relationships of perceived authenticity, cognitive load, and quality of evidence generation with diagnostic accuracy.

METHODS

We conducted an experimental study with 86 medical students (mean 26.03 years, SD 4.71) focusing on history taking in dyspnea cases. Participants solved three cases with SPs and three cases with VPs in this repeated measures study. After each case, students provided a diagnosis and rated perceived authenticity and cognitive load. The provided diagnosis was scored in terms of diagnostic accuracy; the questions asked by the medical students were rated with respect to their quality of evidence generation. In addition to regular null hypothesis testing, this study used equivalence testing to investigate the absence of meaningful effects.

RESULTS

Perceived authenticity (1-tailed t81=11.12; P<.001) was higher for SPs than for VPs. The correlation between diagnostic accuracy and perceived authenticity was very small (r=0.05) and neither equivalent (P=.09) nor statistically significant (P=.32). Cognitive load was equivalent in both assessment methods (t82=2.81; P=.003). Intrinsic cognitive load (1-tailed r=-0.30; P=.003) and extraneous load (1-tailed r=-0.29; P=.003) correlated negatively with the combined score for diagnostic accuracy. The quality of evidence generation was positively related to diagnostic accuracy for VPs (1-tailed r=0.38; P<.001); this finding did not hold for SPs (1-tailed r=0.05; P=.32). Comparing both assessment methods with each other, diagnostic accuracy was higher for SPs than for VPs (2-tailed t85=2.49; P=.01).

CONCLUSIONS

The results on perceived authenticity demonstrate that learners experience SPs as more authentic than VPs. As higher amounts of intrinsic and extraneous cognitive loads are detrimental to performance, both types of cognitive load must be monitored and manipulated systematically in the assessment. Diagnostic accuracy was higher for SPs than for VPs, which could potentially negatively affect students' grades with VPs. We identify and discuss possible reasons for this performance difference between both assessment methods.