RE-AIMing COVID-19 online learning for medical students: a massive open online course evaluation

RE-AIM applied to a primary care workforce training for rural providers and nurses: the Department of Veterans Affairs' Rural Women's Health Mini-Residency

Introduction

Application of the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework to evaluate workforce education and training programs targeting clinical health care staff has received relatively little attention. This paper aims to contribute to this area with RE-AIM findings from a women's health-focused workforce training program implemented by the U.S. Department of Veterans Affairs (VA). Over the past two decades, the rapid expansion of the women Veteran population in VA has necessitated a quick response to meet clinical demand. To address this health care need, the VA Offices of Rural Health (ORH) and Women's Health (OWH) partnered to deploy a primary care workforce development initiative for Rural Providers and Nurses-the Rural Women's Health Mini-Residency (Rural WH-MR)-to train VA clinicians in rural locations in skills for the care of women Veterans. Here we assess the applicability of RE-AIM as an evaluation framework in this context.

Methods

We evaluated the Rural WH-MR, relying on a primarily quantitative approach, rooted in RE-AIM. It included longitudinal and cross-sectional measurements from multiple quantitative and qualitative data sources to develop selected metrics. Data collection instruments consisted of pre-, post-, and follow-up training surveys, course evaluations, existing VA databases, and implementation reports. We developed metrics for and assessed each RE-AIM component by combining data from multiple instruments and then triangulating findings.

Results

Results from the Rural WH-MR program for fiscal years 2018-2020 indicate that RE-AIM provides an instructive evaluation framework for a rural workforce training program, particularly in eliciting clarity between measures of Reach vs. Adoption and focusing attention on both provider- and patient-level outcomes.

Discussion

We describe evaluation metric development and barriers to and facilitators of utilizing RE-AIM as an evaluation framework for a provider- and nurse-facing intervention such as this workforce training program. We also reflect upon RE-AIM benefits for highlighting process and outcomes indicators of a training program's success and lessons learned for evaluating rural workforce development innovations. Several of our observations have implications for training and evaluation approaches in rural areas with more limited access to health care services.

A RE-AIM framework analysis of a sports club health promotion MOOC: the PROSCeSS MOOC

Acquisition of health promotion (HP) skills and knowledge is essential for interventions development in this field. In sports clubs (SC), little HP training exists even though it was requested by SC actors. In response, the PROSCeSS (PROmotion de la Santé au sein du Clubs SportifS) MOOC (Massive Open Online Course) was developed to support actors of HP in the SC in the development of HP interventions. The present study evaluates the MOOC's effectiveness and learning process. The RE-AIM framework, measuring reach, effectiveness, adoption, implementation and maintenance was used to structure this study. Surveys were sent before and after the MOOC to the 2814 learners. Among the 502 (18%) respondents to the pre-survey, 80% reported belonging to a SC as a coach (35%) or manager (25%). The 14% of pre-survey respondents who completed the post-survey increased their HP knowledge score by 42% and their confidence to implement HP actions by 6%. Results present the strategies considered most important and feasible by the learners and the main barriers to the implementation of HP actions in the SC. This study shows that MOOC appears to be an appealing (93% of learners were satisfied) and effective solution (as long as it is followed) to develop HP knowledge and skills of SC actors in HP, meeting their needs and constraints. Although improvements should be made (especially in terms of promoting adoption), this type of educational format should be encouraged to support the HP potential of SC.

Teaching LGBTQ+ Health, a Web-Based Faculty Development Course: Program Evaluation Study Using the RE-AIM Framework

BACKGROUND

Many health professions faculty members lack training on fundamental lesbian, gay, bisexual, transgender, and queer (LGBTQ+) health topics. Faculty development is needed to address knowledge gaps, improve teaching, and prepare students to competently care for the growing LGBTQ+ population.

OBJECTIVE

We conducted a program evaluation of the massive open online course Teaching LGBTQ+ Health: A Faculty Development Course for Health Professions Educators from the Stanford School of Medicine. Our goal was to understand participant demographics, impact, and ongoing maintenance needs to inform decisions about updating the course.

METHODS

We evaluated the course for the period from March 27, 2021, to February 24, 2023, guided by the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework. We assessed impact using participation numbers, evidence of learning, and likelihood of practice change. Data included participant demographics, performance on a pre- and postcourse quiz, open-text entries throughout the course, continuing medical education (CME) credits awarded, and CME course evaluations. We analyzed demographics using descriptive statistics and pre- and postcourse quiz scores using a paired 2-tailed t test. We conducted a qualitative thematic analysis of open-text responses to prompts within the course and CME evaluation questions.

RESULTS

Results were reported using the 5 framework domains. Regarding Reach, 1782 learners participated in the course, and 1516 (85.07%) accessed it through a main course website. Of the different types of participants, most were physicians (423/1516, 27.9%) and from outside the sponsoring institution and target audience (1452/1516, 95.78%). Regarding Effectiveness, the median change in test scores for the 38.1% (679/1782) of participants who completed both the pre- and postcourse tests was 3 out of 10 points, or a 30% improvement (P<.001). Themes identified from CME evaluations included LGBTQ+ health as a distinct domain, inclusivity in practices, and teaching LGBTQ+ health strategies. A minority of participants (237/1782, 13.3%) earned CME credits. Regarding Adoption, themes identified among responses to prompts in the course included LGBTQ+ health concepts and instructional strategies. Most participants strongly agreed with numerous positive statements about the course content, presentation, and likelihood of practice change. Regarding Implementation, the course cost US $57,000 to build and was intramurally funded through grants and subsidies. The course faculty spent an estimated 600 hours on the project, and educational technologists spent another 712 hours. Regarding Maintenance, much of the course is evergreen, and ongoing oversight and quality assurance require minimal faculty time. New content will likely include modules on transgender health and gender-affirming care.

CONCLUSIONS

Teaching LGBTQ+ Health improved participants' knowledge of fundamental queer health topics. Overall participation has been modest to date. Most participants indicated an intention to change clinical or teaching practices. Maintenance costs are minimal. The web-based course will continue to be offered, and new content will likely be added.

The irreplaceable role of medical massive open online courses in China during the COVID-19 pandemic

BACKGROUND

Massive open online courses (MOOCs) have become innovative open-learning approach in medical education. This study aimed to evaluate the dynamic changes in the construction and application of medical MOOCs before and after the coronavirus disease 2019 (COVID-19) pandemic in China.

METHODS

The dynamic changes of usages about medical MOOCs before and after 2020 were mainly searched on the Smart Education of China Higher Education platform, and the detailed learning profiles and outcome indicators were further analyzed using 40 national first-class medical MOOCs from 'zhihuishu' platform.

RESULTS

A total of 2,405 medical MOOCs were exported from the Smart Education of China Higher Education platform, of which 1,313 (54.6%) were launched since 2020. The total and average numbers of participants of 141 national first-class medical MOOCs peaked during the initial spread of COVID-19 in 2020. We further analyzed the dynamic usage of MOOCs from 2018 to 2022 based on 40 national first-class medical MOOCs launched on the 'Zhihuishu' platform. The findings revealed that the number of registered learners (3,240 versus 2,654), questions and answers (27,005 versus 5,116) and students taking the final examination (2,782 versus 1,995) per semester were significantly higher since 2020 compared to these before 2020. Especially, the number of registered learners, registered schools, questions and answers, and students participating in online discussion, taking the unit quiz, taking final examinations and passing final examinations all peaked in the 2020 spring-summer semester. Pearson's correlation analysis found that the number of questions and answers and the number of learners who participated in online discussion were both positively correlated with the number of students who passed the final examination, and the correlation was especially strong since 2020. Moreover, the number of publications on medical MOOC research has soared since 2020 and has maintained a continuous upward trend.

CONCLUSIONS

High-quality medical MOOCs have been launched rapidly since the COVID-19 pandemic in China. The number of participants and online interactions of medical MOOCs peaked during the initial spread of COVID-19 in 2020. MOOCs are reliable and valid digital sources that facilitate medical higher education and play irreplaceable roles in emergency management.

Take a MOOC and then drop: A systematic review of MOOC engagement pattern and dropout factor

Massive Open Online Course (MOOC) plays an important role in education equity and lifelong learning without entrance barriers, time limitations, and geographical constraints. However, MOOC is criticized by researchers for its high dropout rate. The purpose of this paper is to summarize the engagement patterns of MOOC learners and factors affecting dropouts in higher education in the scholarly literature, published during the period of 2008 to 2021. Twenty-one available published studies in seven major academic databases had been reviewed and presented in the systematic literature review following the PRISMA methodology. The results showed that the engagement patterns of MOOC learners could be grouped to Start, Mid, and End according to the stage of MOOC learning. The factors affecting MOOC dropout could be grouped into categories of Course Attributes, Social Status, Cognitive Ability, Emotional Factor and Learning Behavior.

Effects of a sudden change in curriculum delivery mode in postgraduate clinical studies, following the COVID-19 pandemic

OBJECTIVE

To examine the effect of a change in the delivery mode of clinical neurology, a postgraduate subject at Macquarie University, following COVID-19 restrictions on face-to-face teaching.

METHODS

Participants were master of chiropractic students (n = 212) who completed 2019 or 2020 clinical neurology. The main outcome measure was a comparison of objective structured clinical examination (OSCE) marks between the 2019 and 2020 cohorts. The 2019 group underwent traditional, face-to-face learning, whereas the 2020 group were taught and examined remotely in session 1 but returned to campus in session 2. Descriptive analyses, between-group differences, and generalized linear models were performed.

RESULTS

Means for OSCE marks between the 2 groups were higher in the 2020 group in session 1 (p < .001). However, when students returned to campus in session 2, the means were significantly lower in the 2020 group compared with the 2019 group (p < .001). Generalized linear regression indicated that the web-based mode of delivery in 2020 might have had a significant impact on OSCE marks compared with their 2019 counterparts (p < .001).

CONCLUSION

The sudden change in the mode of delivery from face-to-face to remote learning and the change in testing methods in response to the global COVID-19 pandemic had a significant effect on clinical neurology student performance scores. The OSCE marks in the 2020 cohort for session 1 were higher than those in 2019. However, when the 2020 cohort returned to on-campus tutorials in session 2, their marks were lower than those of their 2019 counterparts.

Influencing factors for effective teaching evaluation of massively open online courses in the COVID-19 epidemics: An exploratory study based on grounded theory

Many factors affect the teaching of massively open online courses (MOOCs). In this study, to explore the factors that influence the effective teaching of MOOCs, a large number of relevant studies are analyzed. Based on grounded theory, semi-structured interviews were conducted with 30 students and teachers who used MOOCs for online teaching. The interview data were subjected to four research processes –open coding, axial coding, selective coding, and saturation testing– to explore the factors influencing MOOCs’ effective teaching and the interactions between them. The results demonstrate that: (1) Effective teachers, effective tuition, effective communication, active online learning, social support guarantees, and online course design have important positive effects on effective teaching, while only certain online learning behaviors will seriously affect the teaching effectiveness of MOOC, resulting in negative effects. (2) Effective communication is essential for effective teaching in MOOCs; effective teachers are the leading factor, thus teachers should take the initiative to study and understand the students to understand their various learning needs and difficulties. (3) Reasonable and effective classroom teaching design is key to improving MOOCs’ teaching efficiency. (4) E-learning is respected, cared for, and valued by society, including cognition, emotion, and learning platform support from family, school, teachers, and classmates, and has an important impact on students’ motivation and the effects of online learning. The results of this study further clarify factors influencing effective teaching of MOOCs, thus helping to enrich and supplement the theory of effective teaching and evaluation and providing theoretical guidance for teachers to effectively implement MOOC teaching.

Evaluation of an experiential clinical learning option during pandemic teaching suspensions

BACKGROUND

As students' direct patient contact was suspended because of COVID-19-related restrictions, we revised our clinical addiction medicine curriculum for students to learn about the different multidisciplinary clinical models delivered at our hospital and in community settings. Our aim was to provide an overview of clinical modalities and familiarize learners with clinician and patient experiential perspectives.

METHODS

We implemented a multi-pronged approach, offering an overview of clinical care programs through remote panels involving care providers at the clinics where students had previously been scheduled for in-person rotations. This included inpatient and office-based addiction services, addiction treatment program for adolescents and young adults, integrated addiction care and HIV primary care clinic, and opioid use urgent care clinic. Beyond having them join outpatient telehealth clinic visits, students also participated in an online panel involving patients in recovery to gain familiarity with their care perspectives; and joined a panel with recovery coaches to get further insights into patient challenges in clinical settings. Students further participated in remote opioid treatment trainings and observed clinical rounds of inpatient addiction consults and adolescent clinic team meetings.

RESULTS

With this revised curriculum, students learned about the variety of clinical modalities at the height of our hospital's COVID-19 pandemic burden. The evaluation suggested that students appreciated the authenticity of accounts from patients and providers about their challenges and satisfaction related to clinical care. While in a remote learning setting, students overall wished for more personal interaction with patients and providers. They also noted a lack of group cohesion and connection that they felt would otherwise have been met in an in-person program.

CONCLUSIONS

Remote learning allowed our program to connect trainees to the multidisciplinary field of addiction medicine despite the COVID-19 pandemic. In future program iterations, we will consider hybrid formats of in-person learning experiences with direct patient and faculty contact where possible, combined with online provider and patient panels possibly, in addition to virtual breakout formats to facilitate more personal student-patient and student-faculty interactions.

Program evaluation: An educator's portal into academic scholarship

Program evaluation is an "essential responsibility" but is often not seen as a scholarly pursuit. While Boyer expanded what qualifies as educational scholarship, many still need to engage in processes that are rigorous and of a requisite academic standard to be labelled as scholarly. Many medical educators may feel that scholarly program evaluation is a daunting task due to the competing interests of curricular change, remediation, and clinical care. This paper explores how educators can take their questions around outcomes and efficacy of our programs and efficiently engage in education scholarship. The authors outline how educators can examine whether training programs have a desired impact and outcomes, and then how they might leverage this process into education scholarship.

Evaluation in health professions education-Is measuring outcomes enough?

INTRODUCTION

In an effort to increase the rigour of evaluation in health professions education (HPE), a range of evaluation approaches are used. These largely focus on outcome evaluation as opposed to programme evaluation. We aim to review and critique the use of outcome evaluation models, using the Kirkpatrick Model as an example given its wide acceptance and use, and advocate for the use of programme evaluation models that help us understand how and why outcomes are occurring.

METHODS

We systematically searched OVID medline, Scopus, CINAHL and Pubmed, and hand searched six leading HPE journals to provide an overview of the use of the Kirkpatrick Model as well as a range of programme evaluation models in HPE. In addition to this, we synthesised the existing critiques of the Kirkpatrick Model as an example of outcome evaluation, to highlight the limitations of such models.

RESULTS

The use of the Kirkpatrick Model in HPE is widespread and increasing; however, studies focus on categorising outcomes, rather than explaining how and why they occur. The main criticisms of the model are as follows: it is outcomes focused and fails to consider factors that can impact training outcomes; it assumes positive casual linkages between the levels; there is an assumption that the higher-level outcomes are more important; and unintended impacts are not considered. The use of the Kirkpatrick Model by the MERSQI, BEME and WHO contribute to the myth that the Kirkpatrick Model is the gold standard for programme evaluation.

DISCUSSION

Moving forward, evaluations of HPE interventions must shift from focusing largely on measuring outcomes of interventions with little consideration for how and why these outcomes are occurring to programme evaluation that investigates what contributes to these outcomes. Other models that facilitate the evaluation of the complex processes that occur in HPE should be used instead of Kirkpatrick's.

Assessment of a massive open online course (MOOC) incorporating interactive simulation videos on residents' knowledge retention regarding mechanical ventilation

BACKGROUND

Understanding respiratory physiology and mechanical ventilation is a challenge for healthcare workers, particularly, medical residents. A team of French-speaking experts developed an innovative MOOC incorporating interactive simulation-based videos and serious games aiming at improving knowledge and skills in mechanical ventilation. Our objective was to evaluate the long-term knowledge retention regarding key concepts presented in this MOOC.

METHODS

French residents registered for the MOOC 2020's winter session were invited to participate in a two-step study. The first step consisted in evaluating students' pre-course knowledge of respiratory physiology and mechanical ventilation fusing a 20 five-item multiple choice questions test with a total score ranging from 0 to 100. For the second step, the same students answered the same test (after shuffling the questions) six months after the completion of the course. We assessed the impact of this MOOC on the students' knowledge retention by comparing pre-course and post-course scores.

RESULT

Of the 102 residents who agreed to participate in the study, 80 completed the course and their mean ± SD pre-course score was 76.0 ± 8.0. Fifty-one respondents also completed the second and their post-course score was significantly higher than the baseline one (83.1 ± 7.3 vs. 77.5 ± 7.6, p < 0.001). Scores of the first and second rounds did not differ upon comparing respondents' background specialty or number of years of residency. For the vast majority of individual questions (96%), the success rate was higher at the post-course than at the pre-course assessment.

CONCLUSION

An innovative MOOC incorporating simulation-based videos was effective in teaching medical residents basic mechanical ventilation knowledge and skills, especially in the field of respiratory physiology and ventilatory modes. We observed effective long-term knowledge retention with a higher score at the post-course assessment six months after the completion of the course compared with the pre-course score.

The Usefulness of a Massive Open Online Course about Postural and Technological Adaptations to Enhance Academic Performance and Empathy in Health Sciences Undergraduates

Massive open online courses (MOOCs) provide accessible and engaging information for Physical Therapy and Occupational Therapy students. The objective of this research was to determine the usefulness in improving academic performance and empathy in health sciences undergraduates, and to test a hypothetical model through structural equation analysis. This research was carried out using a descriptive and quasi-experimental design. It was conducted in a sample of 381 participants: 176 used a MOOC and 205 did not. The results of the Student’s t-test showed statistically significant differences in academic performance between the groups in favor of those students who had realized the MOOC. Participants carried out an evaluation rubric after taking MOOC. Statistically significant differences in empathy were also obtained between the pre (X = 62.06; SD = 4.41) and post (X = 73.77; SD = 9.93) tests. The hypothetical model tested via structural equation modeling was supported by the results. Motivation for the MOOC explained 50% of the variance. The MOOC (participation and realization) explained 58% of academic performance, 35% of cognitive empathy and 48% of affective empathy. The results suggest an association between higher realization and participation in a MOOC and higher levels of academic performance, and cognitive and affective empathy.

Use of MOOCs in Health Care Training: A Descriptive-Exploratory Case Study in the Setting of the COVID-19 Pandemic

This exploratory study is part of the training and innovation project (GID2016-16) of the National University of Distance Education (UNED) in Spain. The current socioeconomic and educational contexts derived from COVID-19 has led university institutions to develop methodological innovations in the teaching-learning process. Among these strengthening measures are the MOOCs, the most appropriate strategy to bring students closer to new digital platforms that favor the acquisition of knowledge. A methodological pluralism, combining quantitative and qualitative perspectives, has been used. The main results of the descriptive analysis compare the data of students enrolled in the MOOC, “Social and Health Care and Social Services in Social Work”, and other related bachelor’s and master’s degree courses during the four years analyzed. One of the conclusions is the emerging possibility of offering university studies that are more in line with the current teleworking market. The development of online methodologies favors the democratization of education, reaching the student body as a whole and universalizing content and learning. Among the main conclusions, it is worth highlighting the degree of satisfaction shown by the students who took advantage of the MOOC, and the training opportunities afforded by MOOC courses, during the time of COVID-19.