Use of Learning Analytics Data in Health Care-Related Educational Disciplines: Systematic Review

Early Bird or Night Owl: Insights into Dutch Students' Study Patterns using the Medical Faculty's E-learning Registrations

Phenomenon: Educational activities for students are typically arranged without consideration of their preferences or peak performance hours. Students might prefer to study at different times based on their chronotype, aiming to optimize their performance. While face-to-face activities during the academic schedule do not offer flexibility and cannot reflect students' natural learning rhythm, asynchronous e-learning facilitates studying at one's preferred time. Given their ubiquitous accessibility, students can use e-learning resources according to their individual needs and preferences. E-learning usage data hence serves as a valuable proxy for certain study behaviors, presenting research opportunities to explore students' study patterns. This retrospective study aims to investigate when and for how long undergraduate students used medical e-learning modules. Approach: We performed a cross-sectional analysis of e-learning usage at one medical faculty in the Netherlands. We used data from 562 undergraduate multimedia e-learning modules for pre-clinical students, covering various medical topics over a span of two academic years (2018/19 and 2019/20). We employed educational data mining approaches to process the data and subsequently identified patterns in access times and durations. Findings: We obtained data from 70,805 e-learning sessions with 116,569 module visits and 1,495,342 page views. On average, students used e-learning for 16.8 min daily and stopped using a module after 10.2 min, but access patterns varied widely. E-learning was used seven days a week with an hourly access pattern during business hours on weekdays. Across all other times, there was a smooth increase or decrease in e-learning usage. During the week, more students started e-learning sessions in the morning (34.5% vs. 19.1%) while fewer students started in the afternoon (42.6% vs. 50.8%) and the evening (19.4% vs. 27.0%). We identified 'early bird' and 'night owl' user groups that show distinct study patterns. Insights: This retrospective educational data mining study reveals new insights into the study patterns of a complete student cohort during and outside lecture hours. These findings underline the value of 24/7 accessible study material. In addition, our findings may serve as a guide for researchers and educationalists seeking to develop more individualized educational programs.

First steps of learning analytics in a blended learning general practice curriculum at Saarland University - a quantitative approach

Objectives

Medical education has been revolutionized by the growing importance of digital learning. Little is known about students' online study behaviour and its relationship with exam performance. This quantitative study analyses and describes students' digital learning behaviours in a blended learning curriculum for General practice at Saarland University, Germany. It also examines the relationship between digital learning behaviour and exam performance.

Methods

Cohort and individualized AMBOSS® user data from 195 students at Saarland University was analysed quantitatively. Performance in course-specific multiple-choice question sessions and user data of the integrated online learning activities were correlated with each other and with General practice exam grades. Anonymized data from 10,534 students from 35 other German universities served as the reference cohort. Differences in digital learning behaviour between the groups were calculated using Mann-Whitney-U-Test for non-normally distributed data.

Results

Students in the blended learning course used integrated content more frequently than the reference cohort (U=48777, p<0.001). The number of digital learning cards read correlated moderately with digital formative assessment performance (ρ=0.331, p=0.005 and ρ=0.217, p=0.034). Formative assessment scores and exam results correlated strongly in the summer semester cohort (ρ=0.505, p<0.001), and moderately in the winter semester cohort (ρ=0.381, p<0.001).

Conclusion

There is a difference in the usage of online learning activities when they are purposefully integrated into a curriculum. Digital learning activities including formative assessment may serve as valuable, constructively aligned exam preparation. This is relevant for medical educators when planning future blended learning curricula and portfolio systems, as it may save financial and human resources.

Frameworks for Integrating Learning Analytics With the Electronic Health Record

The information systems designed to support clinical care have evolved separately from those that support health professions education. This has resulted in a considerable digital divide between patient care and education, one that poorly serves practitioners and organizations, even as learning becomes ever more important to both. In this perspective, we advocate for the enhancement of existing health information systems so that they intentionally facilitate learning. We describe three well-regarded frameworks for learning that can point toward how health care information systems can best evolve to support learning. The Master Adaptive Learner model suggests ways that the individual practitioner can best organize their activities to ensure continual self-improvement. The PDSA cycle similarly proposes actions for improvement but at a health care organization's workflow level. Senge's Five Disciplines of the Learning Organization, a more general framework from the business literature, serves to further inform how disparate information and knowledge flows can be managed for continual improvement. Our main thesis holds that these types of learning frameworks should inform the design and integration of information systems serving the health professions. An underutilized mediator of educational improvement is the ubiquitous electronic health record. The authors list learning analytic opportunities, including potential modifications of learning management systems and the electronic health record, that would enhance health professions education and support the shared goal of delivering high-quality evidence-based health care.

Recent advances in Predictive Learning Analytics: A decade systematic review (2012-2022)

The last few years have witnessed an upsurge in the number of studies using Machine and Deep learning models to predict vital academic outcomes based on different kinds and sources of student-related data, with the goal of improving the learning process from all perspectives. This has led to the emergence of predictive modelling as a core practice in Learning Analytics and Educational Data Mining. The aim of this study is to review the most recent research body related to Predictive Analytics in Higher Education. Articles published during the last decade between 2012 and 2022 were systematically reviewed following PRISMA guidelines. We identified the outcomes frequently predicted in the literature as well as the learning features employed in the prediction and investigated their relationship. We also deeply analyzed the process of predictive modelling, including data collection sources and types, data preprocessing methods, Machine Learning models and their categorization, and key performance metrics. Lastly, we discussed the relevant gaps in the current literature and the future research directions in this area. This study is expected to serve as a comprehensive and up-to-date reference for interested researchers intended to quickly grasp the current progress in the Predictive Learning Analytics field. The review results can also inform educational stakeholders and decision-makers about future prospects and potential opportunities.

Implementation of competence by design in Canadian neurosurgery residency programs

OBJECTIVE

The Royal College of Physicians and Surgeons of Canada (RCPSC) recently redesigned the Canadian neurosurgery residency training curriculum by implementing a competency-based model of training known as Competence by Design (CBD) centered around the assessment of Entrustable Professional Activities (EPAs). This sequential explanatory mixed-methods study evaluated potential benefits and pitfalls of CBD in Canadian neurosurgery residency education.

METHODS

Two four-month interval surveys were distributed to all Canadian neurosurgery residents participating in CBD. The surveys assessed important educational components: CBD knowledge of key stakeholders, potential system barriers, and educational/psychological impacts on residents. Paired t-tests were done to assess changes over time. Based on longitudinal survey responses, semi-structured interviews were conducted to investigate in-depth residents' experience with CBD in neurosurgery. The qualitative analysis followed an explanatory approach, and a thematic analysis was performed.

RESULTS

Surveys had 82% average response rate (n = 25). Over time, most residents self-reported that they retrospectively understood concepts around CBD intentions (p = 0.02). Perceived benefits included faculty evaluations with more feedback that was clearer and more objective (53% and 51%). Pitfalls included the amount of time needed to navigate through EPAs (90%) and residents forgetting to initiate EPA forms (71%). There was no significant change over time. During interviews, five key themes were found. Potential solutions identified by residents to enhance their experience included learning analytics data availability, mobile app refinement, and dedicated time to integrate EPAs in the workflow.

CONCLUSION

This study was the first to assess resident-perceived benefits and pitfalls of the neurosurgery CBD training program in an educational framework context. In general, residents believed that theoretical principles behind CBD were valuable, but that technological ability and having enough time to request EPA assessments were significant barriers to success. Long-term studies are required to determine the definitive outcomes of CBD on residents' performance and ultimately, on patient care.

An exploration of student access to a learning management system-challenges and recommendations for educators and researchers

INTRODUCTION

This study examined student access to online resources of a faculty's learning management system (LMS). Issues relating to current e-learning resources usage were identified and formed the basis for recommendations to help assist stakeholders in teaching, learning and research.

METHODS

Learning analytics from four cohorts of undergraduate dental students were extracted from the database of a LMS spanning between 2012 and 2016. Individual datasets were combined into one master file, re-categorised, filtered and analysed based on cohort, year of study, course and nature of online resource.

RESULTS

A total of 157,293 access events were documented. The proportion of administrative to learning data varied across cohorts, with oldest cohort having the highest ratio (82:18) in their final year and most recent cohort having a ratio of 33:67 in their 4th year demonstrating a higher proportion to learning. Seven Learning domains were identified in the access data: access to problem-based learning resources was the highest and next was fixed prosthodontics videos. The prosthodontics discipline had the highest access across the curriculum while some others had very limited or even no learning access events.

CONCLUSION

A number of limitations have been identified with the analytics and learning resources in this LMS and engagement with learning resource provision. More detailed data capture of access use and unique identifiers to resources as well as keyword tagging of the resources are required to allow accurate mapping and support of students learning. Moreover, motivation or nudging of students behaviour to more actively engage with learning content needs exploration.

Development and Evaluation of an Innovative Web-based Training, Learning, and Sharing (i-TLS) Platform for Social Workers: Hong Kong Jockey Club SMART Family-Link Project (Preprint)

Background

Information and communication technology (ICT) use may enhance social work practice and continuous professional development. Under the Hong Kong Jockey Club SMART Family-Link Project, we developed an innovative web-based training, learning, and sharing platform (i-TLS) to support not only ICT and other learning needs of Hong Kong social workers but also their practice.

Objective

We developed i-TLS with 3 major components (i-Training, i-Learning, and i-Sharing) and assessed its acceptability and impact on facilitating ICT use in family services.

Methods

We described the i-TLS development based on a 4-phase model and evaluated i-TLS using the platform database, Google Analytics, a self-administered survey, and individual phone interviews 1 year after launching.

Results

i-TLS was launched in 12 nongovernmental organizations on July 1, 2019. The COVID-19 outbreak in December 2019 limited face-to-face services, which galvanized digital transformation in social work practice. By July 31, 2020, 313 social workers had registered with i-TLS. Approximately 79.6% (249/313) of users accessed i-TLS at least once in the past 28 days, averaging 3.2 (SD 1.35) platform visits per day and viewing 4.8 (SD 1.42) pages per visit. i-Training provided 41 mini-modules on applying ICT to family services, with 730 enrollments. Approximately 70% (511/730) of users completed the mini-modules and obtained digital mini-certificates. i-Learning provided 112 items of learning resources centered on ICT use in family services, with nearly 4000 page views. i-Sharing had 25 discussion threads with 59 posts. Approximately 53.7% (168/313) of users completed the 1-year evaluation survey, including 7.1% (12/168) who were phone interviewed. The mean i-TLS satisfaction score (out of 10) increased from light (4.99, SD 1.54) to occasional (6.15, SD 1.34) and frequent (6.31, SD 2.29) users. Frequent users showed higher scores (out of 10) than light users for an increase in knowledge (5.84, SD 1.34 vs 4.09, SD 1.74; P<.001), self-efficacy (5.23, SD 1.92 vs 3.96, SD 1.77; P=.02), and knowledge application (6.46, SD 1.33 vs 1.91, SD 1.40; P<.001). Interviewees reported increased ICT use in services and considered i-TLS an acceptable and supportive tool for learning and practice, especially during the pandemic.

Conclusions

i-TLS is acceptable to social workers and enhances their learning and use of ICT in family services. This was achieved through access to self-directed and collaborative learning and sharing of experiences within their practice. Further research on enhancing web-based platforms is needed to expand participation and capacity building among social workers and other health and social care professionals.

The relation of online learning analytics, approaches to learning and academic achievement in a clinical skills course

INTRODUCTION

This study examined relationships of students' access of e-learning (learning analytics) for a healthcare psychomotor skills course, approaches to learning (R-SPQ-2F psychometric) and academic achievement (summative examination results). An understanding of the relationships may help in supporting students learning.

METHODS

Two consecutive cohorts of 5th year dental students were asked to complete the R-SPQ-2F questionnaire and permission to access data of online videos and associated quizzes and two summative written examination results were examined. The summative assessments were an OSCA and a written-paper question both in prosthodontics. Multiple linear regression and correlation analysis were performed.

RESULTS

Students (n=98) performed a total of 10470 video access events and 7714 attempts in online quizzes. Deep learning approach was the strongest predictor variable (β=0.270; P=.004) on written-paper question result. While video and quiz access were moderately correlated (r=0.600; P<.001) to each other, video access was not a significant predictor to either of the examination results. Quiz access was negatively associated with academic achievement for the written-paper question results (β=-0.349; P<.001).

CONCLUSIONS

Only deep approach to learning appeared to be relevant for the written-paper question examination grades. Conversely, the number of video and quiz access did not relate to either examination results. This suggests e-learning access and examinations do not appear to relate under the conditions explored. Other attributes of learning management access may need to be explored to determine if access to learning management systems may be useful in offering remedial support to students.

Using learning management software data to compare students' actual and self-reported viewing of video lectures

PURPOSE

Student engagement is typically measured by class attendance and questionnaires. Learning management systems (LMS) provide additional sources of objective data that can evaluate student performance. This study compared dental students' self-report of course video viewing with LMS data.

METHODS

Dental students in a 2018 flipped pediatric dentistry course were given a questionnaire after course completion that captured self-reported course video viewing. Student data and time spent on videos were extracted from the LMS and matched to the questionnaire. McNemar's test evaluated differences between self-reported and matching LMS data.

RESULTS

Of the 109 enrolled students, 99 (91%) completed the questionnaire. Most students self-reported that they watched more than half of the videos (71%), re-watched at least once (89%), and watched at times other than 8 a.m.-5 p.m. (80%). Of the 104 students with data from the LMS, 40% of students watched more than half of the videos, 49% re-watched at least one video, and 60% watched between 8 a.m.-5 p.m. LMS data showed 14 (13%) students did not watch any videos. Significant differences were found between self-reported video viewing and matched LMS video data in (1) percentage of videos watched, (2) number of times videos were re-watched, and (3) most frequent time for watching the videos (p < 0.001 for all).

CONCLUSION

Technology is a tool that helps instructors more accurately track students' study habits compared to observation or self-report. When evaluating time spent watching videos via an LMS, students tend to overestimate the amount of viewing time.

Web-based formative assessment through clinical cases: role in pathophysiology teaching

BACKGROUND

Active learning strategies such as formative assessment through clinical cases may help to get a deeper learning. We have studied the effect of this kind of online formative assessment in pathophysiology teaching.

METHODS

Seven brief clinical cases were used to give formative assessment in the first semester of a pathophysiology course. To evaluate its effect on learning, we analyzed the proportion of students that passed the end of semester exam with a score above 60 over 100. We also analyzed the effect of the intervention according to the students' previous academic performance.

RESULTS

Ninety-six students participated in the study and sat the exam. Sixty-five of them passed it. Students that passed the exam had a higher previous academic performance and had done a higher number of exercises of formative assessment, both in univariate and multivariate analysis. The participants were divided in three groups, according to their previous academic performance. In the intermediate group, the number of cases done by the students who passed the exam was significantly higher than in those who did not pass it (median: 4 versus 0; P = 0.009).

CONCLUSION

Formative assessment through web-based clinical cases was followed by an improvement of the academic results in pathophysiology, mainly in students with intermediate performance.