Using virtual standardized patients to accurately assess information gathering skills in medical students

Virtual patients improve history-taking competence and confidence in medical students

PURPOSE

Klark is a novel online medical education tool (www.klark-cases.com) where students take histories from virtual patients with common presentations from multiple specialities. We investigated whether Klark could enhance student confidence and competence in history-taking, and whether students find Klark helpful.

METHODS

A single cohort of first-year clinical medical students had access to Klark for three weeks. At both ends of the trial, participants were asked to complete feedback forms and participate in two mock Objective Structured Clinical Examination (OSCE) history stations. Outcome measures included self-reported confidence and competence in history-taking, performance in OSCE stations, and qualitative user experience data.

RESULTS

Seventy participants successfully completed a case on Klark (mean 18.7), of which 63 (90% user retention) completed  ≥ 2 cases. Self-reported competence (p < 0.001) and confidence (p < 0.001) improved. Participants found Klark to be helpful, impactful, and would recommend it to other students. OSCE scores improved for medical (57% vs. 69%, p < 0.001) and surgical (58% vs. 70%, p < 0.001) histories.

CONCLUSIONS

Klark improved competence and confidence in history-taking. Students found it helpful and chose to continue using the platform. By developing confidence and competence at their own pace in the Klark simulated environment, students can then maximise benefit from in-person clinical opportunities.

A scoping review of artificial intelligence in medical education: BEME Guide No. 84

BACKGROUND

Artificial Intelligence (AI) is rapidly transforming healthcare, and there is a critical need for a nuanced understanding of how AI is reshaping teaching, learning, and educational practice in medical education. This review aimed to map the literature regarding AI applications in medical education, core areas of findings, potential candidates for formal systematic review and gaps for future research.

METHODS

This rapid scoping review, conducted over 16 weeks, employed Arksey and O'Malley's framework and adhered to STORIES and BEME guidelines. A systematic and comprehensive search across PubMed/MEDLINE, EMBASE, and MedEdPublish was conducted without date or language restrictions. Publications included in the review spanned undergraduate, graduate, and continuing medical education, encompassing both original studies and perspective pieces. Data were charted by multiple author pairs and synthesized into various thematic maps and charts, ensuring a broad and detailed representation of the current landscape.

RESULTS

The review synthesized 278 publications, with a majority (68%) from North American and European regions. The studies covered diverse AI applications in medical education, such as AI for admissions, teaching, assessment, and clinical reasoning. The review highlighted AI's varied roles, from augmenting traditional educational methods to introducing innovative practices, and underscores the urgent need for ethical guidelines in AI's application in medical education.

CONCLUSION

The current literature has been charted. The findings underscore the need for ongoing research to explore uncharted areas and address potential risks associated with AI use in medical education. This work serves as a foundational resource for educators, policymakers, and researchers in navigating AI's evolving role in medical education. A framework to support future high utility reporting is proposed, the FACETS framework.

Twelve tips for Natural Language Processing in medical education program evaluation

With the increasing application of Natural Language Processing (NLP) in Medicine at large, medical educators are urged to gain an understanding and implement NLP techniques within their own education programs to improve the workflow and make significant and rapid improvements in their programs. This paper aims to provide twelve essential tips inclusive of both conceptual and technical factors to facilitate the successful integration of NLP in medical education program evaluation. These twelve tips range from advising on various stages of planning the evaluation process, considerations for data collection, and reflections on preprocessing of data in preparation for analysis and interpretation of results. Using these twelve tips as a framework, medical researchers, educators, and administrators will have an understanding and reference to navigating applications of NLP and be able to unlock its potential for enhancing the evaluation of their own medical education programs.

Development and evaluation of a software system for medical students to teach and practice anamnestic interviews with virtual patient avatars

BACKGROUND AND OBJECTIVES

Taking a medical history is a core competence of the diagnostic process. At the beginning of their study medical students need to learn and practice the necessary techniques, initially focusing on good structuring and completeness. For this purpose, an interactive software system (ViPATalk) was developed in which the student can train to pose questions to virtual patient avatars in free conversation. At the end, the student receives feedback on the completeness of the questioning and an explanation of the essential items. The use of this software was compared to the traditional format of student role play in a randomized trial.

METHODS

The central component of ViPATalk is a chatbot based on the AI language AIML, which generates an appropriate answer based on keywords in the student's question. To enable a realistic use, the student can enter the question via microphone (speech-to-text) and the answer generated by the chatbot is presented as a short video sequence, where the avatar is generated from a real image. Here, the transition between the sequences is seamless, resulting in a continuous movement of the avatar during the conversation.

RESULTS

The learning success by practicing with ViPATalk was tested in an anamnestic interview with actors as simulated patients. The completeness of the conversation was evaluated with regard to numerous aspects and also certain behaviors during the conversation. These results were compared with those after practicing using peer role play.

CONCLUSIONS

It was found that practicing with ViPATalk was mostly equivalent to the students' role play. In the subsequent survey of the students, the wish was expressed that the ViPATalk software should also be used as an online tool for self-study and that there should be more cases for practicing.

Learning technology in health professions education: Realising an (un)imagined future

CONTEXT

Technology is being introduced, used and studied in almost all areas of health professions education (HPE), often with a claim of making HPE better in one way or another. However, it remains unclear if technology has driven real change in HPE. In this article, we seek to develop an understanding of the transformative capacity of learning technology in HPE.

METHODS AND OUTCOMES

We first consider the wider scholarship highlighting the intersection between technology and pedagogy, articulating what is meant by transformation and the role of learning technology in driving educational transformation. We then undertake a synthesis of the current high visibility HPE-focused research. We sampled the literature in two ways-for the five highest impact factor health professional education journals over the past decade and for all PubMed indexed journals for the last 3 years-and categorised the extant research against the Substitution, Augmentation, Modification, Redefinition model. We found that the majority of research we sampled focussed on substituting or augmenting learning through technology, with relatively few studies using technology to modify or redefine what HPE is through the use of technology. Of more concern was the lack of theoretical justification for pedagogical improvement, including transformation, underpinning the majority of studies.

CONCLUSIONS

While all kinds of technology use in learning have their place, the next step for HPE is the robust use of technology aiming to lead transformation. This should be guided by transformational educational theory and aligned with pedagogical context. We challenge HPE practitioners and scholars to work thoughtfully and with intent to enable transformation in education for future health professionals.

Artificial Intelligence Supporting the Training of Communication Skills in the Education of Health Care Professions: Scoping Review

BACKGROUND

Communication is a crucial element of every health care profession, rendering communication skills training in all health care professions as being of great importance. Technological advances such as artificial intelligence (AI) and particularly machine learning (ML) may support this cause: it may provide students with an opportunity for easily accessible and readily available communication training.

OBJECTIVE

This scoping review aimed to summarize the status quo regarding the use of AI or ML in the acquisition of communication skills in academic health care professions.

METHODS

We conducted a comprehensive literature search across the PubMed, Scopus, Cochrane Library, Web of Science Core Collection, and CINAHL databases to identify articles that covered the use of AI or ML in communication skills training of undergraduate students pursuing health care profession education. Using an inductive approach, the included studies were organized into distinct categories. The specific characteristics of the studies, methods and techniques used by AI or ML applications, and main outcomes of the studies were evaluated. Furthermore, supporting and hindering factors in the use of AI and ML for communication skills training of health care professionals were outlined.

RESULTS

The titles and abstracts of 385 studies were identified, of which 29 (7.5%) underwent full-text review. Of the 29 studies, based on the inclusion and exclusion criteria, 12 (3.1%) were included. The studies were organized into 3 distinct categories: studies using AI and ML for text analysis and information extraction, studies using AI and ML and virtual reality, and studies using AI and ML and the simulation of virtual patients, each within the academic training of the communication skills of health care professionals. Within these thematic domains, AI was also used for the provision of feedback. The motivation of the involved agents played a major role in the implementation process. Reported barriers to the use of AI and ML in communication skills training revolved around the lack of authenticity and limited natural flow of language exhibited by the AI- and ML-based virtual patient systems. Furthermore, the use of educational AI- and ML-based systems in communication skills training for health care professionals is currently limited to only a few cases, topics, and clinical domains.

CONCLUSIONS

The use of AI and ML in communication skills training for health care professionals is clearly a growing and promising field with a potential to render training more cost-effective and less time-consuming. Furthermore, it may serve learners as an individualized and readily available exercise method. However, in most cases, the outlined applications and technical solutions are limited in terms of access, possible scenarios, the natural flow of a conversation, and authenticity. These issues still stand in the way of any widespread implementation ambitions.

Artificial intelligence in virtual standardized patients: Combining natural language understanding and rule based dialogue management to improve conversational fidelity

INTRODUCTION

Advances in natural language understanding have facilitated the development of Virtual Standardized Patients (VSPs) that may soon rival human patients in conversational ability. We describe herein the development of an artificial intelligence (AI) system for VSPs enabling students to practice their history taking skills.

METHODS

Our system consists of (1) Automated Speech Recognition (ASR), (2) hybrid AI for question identification, (3) classifier to choose between the two systems, and (4) automated speech generation. We analyzed the accuracy of the ASR, the two AI systems, the classifier, and student feedback with 620 first year medical students from 2018 to 2021.

RESULTS

System accuracy improved from ∼75% in 2018 to ∼90% in 2021 as refinements in algorithms and additional training data were utilized. Student feedback was positive, and most students felt that practicing with the VSPs was a worthwhile experience.

CONCLUSION

We have developed a novel hybrid dialogue system that enables artificially intelligent VSPs to correctly answer student questions at levels comparable with human SPs. This system allows trainees to practice and refine their history-taking skills before interacting with human patients.

Knowledge and attitudes of medical students in Lebanon toward artificial intelligence: A national survey study

Purpose

This study assesses the knowledge and attitudes of medical students in Lebanon toward Artificial Intelligence (AI) in medical education. It also explores the students' perspectives regarding the role of AI in medical education as a subject in the curriculum and a teaching tool.

Methods

This is a cross-sectional study using an online survey consisting of close-ended questions. The survey targets medical students at all medical levels across the 7 medical schools in Lebanon.

Results

A total of 206 medical students responded. When assessing AI knowledge sources (81.1%) got their information from the media as compared to (9.7%) from medical school curriculum. However, Students who learned the basics of AI as part of the medical school curriculum were more knowledge about AI than their peers who did not. Students in their clinical years appear to be more knowledgeable about AI in medicine. The advancements in AI affected the choice of specialty of around a quarter of the students (26.8%). Finally, only a quarter of students (26.5%) want to be assessed by AI, even though the majority (57.7%) reported that assessment by AI is more objective.

Conclusions

Education about AI should be incorporated in the medical school curriculum to improve the knowledge and attitudes of medical students. Improving AI knowledge in medical students will in turn increase acceptance of AI as a tool in medical education, thus unlocking its potential in revolutionizing medical education.

A scoping review: virtual patients for communication skills in medical undergraduates

BACKGROUND

Communication is an essential competence for medical students. Virtual patients (VP), computerized educational tools where users take the role of doctor, are increasingly used. Despite the wide range of VP utilization, evidence-based practical guidance on supporting development of communication skills for medical students remains unclear. We focused this scoping review on VP affordance for student learning especially important in the current environment of constrained patient access.

METHODS

This scoping review followed Arksey & O'Malley's methodology. We tested and used a search strategy involving six databases, resulting in 5,262 citations. Two reviewers independently screened titles, full texts (n= 158) and finally performed data extraction on fifty-five included articles. To support consideration of educational affordance the authors employed a pragmatic framework (derived from activity theory) to map included studies on VP structure, curricular alignment, mediation of VP activity, and socio-cultural context.

RESULTS

Findings suggest that not only the VP itself, but also its contextualization and associated curricular activities influence outcomes. The VP was trialled in the highest proportion of papers as a one-off intervention (19 studies), for an average duration of 44.9 minutes (range 10-120min), mainly in senior medical students (n=23), notably the largest group of studies did not have VP activities with explicit curricular integration (47%). There was relatively little repeated practice, low levels of feedback, self-reflection, and assessment. Students viewed VPs overall, citing authenticity and ease of use as important features. Resource implications are often omitted, and costings would facilitate a more complete understanding of implications of VP use.

CONCLUSION

Students should be provided with maximal opportunity to draw out the VPs' full potential through repeated practice, without time-constraint and with curricular alignment. Feedback delivery enabling reflection and mastery is also key. The authors recommend educators to explicitly balance computerized authenticity with instructional design integrated within the curriculum.

History-taking level and its influencing factors among nursing undergraduates based on the virtual standardized patient testing results: Cross sectional study

BACKGROUND

History-taking is an essential skill for nurses. In nursing education, it is necessary to objectively assess history-taking skills in a way that accurately reflects differences among students. Current history-taking evaluation methods lack objectivity, consistency and standardization, which makes it difficult to identify factors that influence history-taking skill. A virtual standardized patient (VSP) can provide history-taking practice with repeatability and consistency. It can make objective and standardized assessment possible by eliminating the subjectivity of different teachers and patients.

PURPOSE

To evaluate the history-taking skills of nursing undergraduates using a VSP, and to explore its independent influencing factors.

METHODS

A cross-sectional design was employed with a sample of convenience from 3 universities. All 174 nursing undergraduates had their history-taking level evaluated using one VSP via computer or mobile terminal. For each query raised by the students, the VSP could give a preset response. The responses were assigned corresponding scores according to their importance, with a total score of 100. The students' general demographics were documented, and their self-efficacy, critical thinking and communication skills were measured online with the Nursing History-taking Self-Efficacy Questionnaire, Yoon's Critical Thinking Disposition Instrument and the Supportive Communicative Scale. Univariate analysis, Pearson correlation analysis and multiple linear regression were performed.

RESULTS

The mean total history-taking score was 58.53 ± 14.32. History-taking scores were well discriminated among the students tested, with 10% of the students scoring above 80, 37% between 60 and 80, 22% between 50 and 60 and 31% scored below 50. The history-taking level of the nursing students was significantly associated with ethnicity, previous academic performance, and Supportive Communicative Scale scores.

CONCLUSIONS

Using the VSP as a history-taking assessment method is an effective way to achieve a relatively objective, standardized and consistent assessment of history-taking education. Ethnicity, previous academic performance and supportive communication skills independently influenced the students' history-taking level. Mature history-taking skills require not only solid theoretical knowledge but good communication skills.

Long experience with a web-based, interactive, conversational virtual patient case simulation for medical students' evaluation: comparison with oral examination

Virtual patients (VP) have been advocated as reliable tools for teaching and evaluating clinical skills and competence. We have developed an internet-based, OSCE-like, conversational VP system designed both for training and assessment of medical students. The system, that encompasses complete patient management from H&P to diagnostic procedures and treatment, has now been used regularly during the clerkship of internal medicine. The present article describes the system and compares assessments undertaken with the VP-system over the last five years, to traditional bed-side oral exams. All students practiced on their own exercise VP cases, while preparing for the final exam. A total of 586 students were evaluated simultaneously with both assessment modalities. The αCronbach of the VP exam averaged 0.86. No correlation was found between the grades obtained in the two exams, indicating that the VP exam evaluated different parameters than those assessed by the examiners in the oral examinations. We conclude that a VP system can be utilized as a valid and reliable examination tool. It is also most useful for independent training by students during their ward-based learning, as well as when not studying in classes, wards or clinics, when social distancing is required.

Virtual Reality Applications in Medicine During the COVID-19 Pandemic: Systematic Review (Preprint)

Background

Virtual reality can play an important role during the COVID-19 pandemic in the health care sector. This technology has the potential to supplement the traditional in-hospital medical training and treatment, and may increase access to training and therapies in various health care settings.

Objective

This systematic review aimed to describe the literature on health care–targeted virtual reality applications during the COVID-19 crisis.

Methods

We conducted a systematic search of the literature on the PsycINFO, Web of Science, and MEDLINE databases, according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidelines. The search string was as follows: “[(virtual reality)] AND [(COVID-19) OR (coronavirus) OR (SARS-CoV-2) OR (healthcare)].” Papers published in English after December 2019 in peer-reviewed journals were selected and subjected to the inclusion and exclusion criteria. We used the Mixed Methods Appraisal Tool to assess the quality of studies and the risk of bias.

Results

Thirty-nine studies met the inclusion criteria. Seventeen studies showed the usefulness of virtual reality during the COVID-19 crisis for reducing stress, anxiety, depression, and pain, and promoting physical activity. Twenty-two studies revealed that virtual reality was a helpful learning and training tool during the COVID-19 crisis in several areas, including emergency medicine, nursing, and pediatrics. This technology was also used as an educational tool for increasing public understanding of the COVID-19 pandemic. Different levels of immersion (ie, immersive and desktop virtual reality), types of head-mounted displays (ie, PC-based, mobile, and standalone), and content (ie, 360° videos and photos, virtual environments, virtual reality video games, and embodied virtual agents) have been successfully used. Virtual reality was helpful in both face-to-face and remote trials.

Conclusions

Virtual reality has been applied frequently in medicine during the COVID-19 pandemic, with positive effects for treating several health conditions and for medical education and training. Some barriers need to be overcome for the broader adoption of virtual reality in the health care panorama.

Trial Registration

International Platform of Registered Systematic Review and Meta-analysis Protocols (INPLASY) INPLASY202190108; https://inplasy.com/inplasy-2021-9-0108/

Virtual Standardized Patients for Mental Health Education

PURPOSE OF REVIEW

The training of psychiatrists and other mental health professionals requires education on a range of interpersonal, communication, and psychotherapy techniques. Classroom and workshop training must be augmented by experiential learning with feedback for skill implementation with fidelity. Virtual standardized patients (VSPs) are computerized conversational agents that can support experiential learning through standardized, consequence-free training environments at reduced costs.

RECENT FINDINGS

Research on mental health VSPs is rife with feasibility and acceptability pilot studies across various training populations and settings. Users have generally reported positive reactions to training with VSPs, though frustrations with some VSP speech recognition or VSP response relevance has been reported. Several studies have demonstrated a promising transfer of clinical skills from VSP training to human standardized patients and randomized trials supporting improved skill relative to reading or academic study are encouraging. As technology improves and natural language processing and accurate computer response generation for broad ranging conversational topics emerges, the field would benefit from research on the characteristics of effective VSPs for a range of purposes and trainee populations. Well-designed randomized evaluations of VSPs relative to best practices in education are needed, particularly regarding the impact of VSPs on clinical practice among actual patients.

Artificial Intelligence for Student Assessment: A Systematic Review

Artificial Intelligence (AI) is being implemented in more and more fields, including education. The main uses of AI in education are related to tutoring and assessment. This paper analyzes the use of AI for student assessment based on a systematic review. For this purpose, a search was carried out in two databases: Scopus and Web of Science. A total of 454 papers were found and, after analyzing them according to the PRISMA Statement, a total of 22 papers were selected. It is clear from the studies analyzed that, in most of them, the pedagogy underlying the educational action is not reflected. Similarly, formative evaluation seems to be the main use of AI. Another of the main functionalities of AI in assessment is for the automatic grading of students. Several studies analyze the differences between the use of AI and its non-use. We discuss the results and conclude the need for teacher training and further research to understand the possibilities of AI in educational assessment, mainly in other educational levels than higher education. Moreover, it is necessary to increase the wealth of research which focuses on educational aspects more than technical development around AI.

Pre-graduation medical training including virtual reality during COVID-19 pandemic: a report on students' perception

BACKGROUND

The Coronavirus Disease 19 (COVID-19) pandemic brought significant disruption to in-hospital medical training. Virtual reality simulating the clinical environment has the potential to overcome this issue and can be particularly useful to supplement the traditional in-hospital medical training during the COVID-19 pandemic, when hospital access is banned for medical students. The aim of this study was to assess medical students' perception on fully online training including simulated clinical scenarios during COVID-19 pandemic.

METHODS

From May to July 2020 when in-hospital training was not possible, 122 students attending the sixth year of the course of Medicine and Surgery underwent online training sessions including an online platform with simulated clinical scenarios (Body Interact™) of 21 patient-based cases. Each session focused on one case, lasted 2 h and was divided into three different parts: introduction, virtual patient-based training, and debriefing. In the same period, adjunctive online training with formal presentation and discussion of clinical cases was also given. At the completion of training, a survey was performed, and students filled in a 12-item anonymous questionnaire on a voluntary basis to rate the training quality. Results were reported as percentages or with numeric ratings from 1 to 4. Due to the study design, no sample size was calculated.

RESULTS

One hundred and fifteen students (94%) completed the questionnaire: 104 (90%) gave positive evaluation to virtual reality training and 107 (93%) appreciated the format in which online training was structured. The majority of participants considered the platform of virtual reality training realistic for the initial clinical assessment (77%), diagnostic activity (94%), and treatment options (81%). Furthermore, 97 (84%) considered the future use of this virtual reality training useful in addition to the apprenticeship at patient's bedside. Finally, 32 (28%) participants found the online access difficult due to technical issues.

CONCLUSIONS

During the COVID-19 pandemic, online medical training including simulated clinical scenarios avoided training interruption and the majority of participant students gave a positive response on the perceived quality of this training modality. During this time frame, a non-negligible proportion of students experienced difficulties in online access to this virtual reality platform.

Welches Potenzial haben virtuelle Realitäten in der klinischen und forensischen Psychiatrie? Ein Überblick über aktuelle Verfahren und Einsatzmöglichkeiten

Virtuelle Realitäten (VR) werden in der Diagnose und Behandlung von Patienten im klinischen Feld bereits seit 20 Jahren erfolgreich eingesetzt und weiterentwickelt. Seit etwas mehr als 5 Jahren gibt es nun auch erste Beispiele über die Anwendung von VR in psychiatrisch-forensischen Kontexten. Für die forensische Psychiatrie ist die Möglichkeit, realistische, sichere und kontrollierbare Diagnostik- und Lernumgebungen zu schaffen, der ausschlaggebende Vorteil der VR-Technologie. So können z. B. Straftäter in Szenarien behandelt oder begutachtet werden, welche im echten Leben risikoreich, unethisch oder ökologisch invalide wären. In diesem Artikel werden unterschiedliche aktuelle Studienbeispiele zu klinischer Behandlung und Diagnose von Patienten sowie der forensischen Prognose und Therapie von Straftätern vorgestellt. Damit zeigt der Überblick, dass VR mittlerweile auch in der forensischen Psychiatrie ein vielversprechendes Werkzeug sein kann, welches bereits etablierte Instrumente ergänzen oder erweitern kann. Auch in der Ausbildung von forensisch-psychiatrischem Fachpersonal können VR-Anwendungen eine Hilfe sein. Hier gibt es bereits erste vielversprechende Einsätze durch das Training mithilfe von virtuellen Patienten, jedoch benötigt es noch umfangreiche Forschungsarbeit auf diesem Feld, um sie im professionellen Alltag einsetzen zu können. Vor dem Einsatz von VR-Anwendungen sollten sich Forscher und Praktiker neben den Vorteilen auch mit den Nachteilen von VR auseinandersetzen und ein besonderes Augenmerk auf die ethischen Richtlinien werfen, welche in den letzten Jahren dazu erarbeitet wurden. Die stetige Weiterentwicklung und der immer breitere Einsatz von VR im klinischen und forensisch-psychiatrischen Feld zeigen, dass VR auch hier das Potenzial hat, ein etabliertes Forschungs- sowie Therapieinstrument zu werden.

Evaluation of a virtual agent to train medical students conducting psychiatric interviews for diagnosing major depressive disorders

BACKGROUND

A psychiatric diagnosis involves the physician's ability to create an empathic interaction with the patient in order to accurately extract semiology (i.e., clinical manifestations). Virtual patients (VPs) can be used to train these skills but need to be evaluated in terms of accuracy, and to be perceived positively by users.

METHODS

We recruited 35 medical students who interacted in a 35-min psychiatric interview with a VP simulating major depressive disorders. Semiology extraction, verbal and non-verbal empathy were measured objectively during the interaction. The students were then debriefed to collect their experience with the VP.

RESULTS

The VP was able to simulate the conduction of a psychiatric interview realistically, and was effective to discriminate students depending on their psychiatric knowledge. Results suggest that students managed to keep an emotional distance during the interview and show the added value of emotion recognition software to measure empathy in psychiatry training. Students provided positive feedback regarding pedagogic usefulness, realism and enjoyment in the interaction.

LIMITATIONS

Our sample was relatively small. As a first prototype, the measures taken by the VP would need improvement (subtler empathic questions, levels of difficulty). The face-tracking technique might induce errors in detecting non-verbal empathy.

CONCLUSION

This study is the first to simulate a realistic psychiatric interview and to measure both skills needed by future psychiatrists: semiology extraction and empathic communication. Results provide evidence that VPs are acceptable by medical students, and highlight their relevance to complement existing training and evaluation tools in the field of affective disorders.