Test-Only Learning via Virtual Patients to Improve Surgical Illness Scripts of Preclinical Medical Students as a Preparation for Clinical Clerkships: An Experimental Study

OBJECTIVE

The aim was to determine whether preclinical medical students can develop their illness scripts to a level comparable to that of clerkship students through test-only learning using repeated formative online testlets.

METHODS

In this experimental study, participants were 52 preclinical and 53 clinical medical students. The intervention group consisted of preclinical medical students, and the control group consisted of clinical medical students. The intervention group responded to online testlets containing feedback, an innovative formative assessment method called ContExtended Questions, on general surgery for 8 days by spending no more than 30 minutes each day. The control group completed the general surgery clerkship. The performances were assessed using 20 Key-Feature Question items. The intervention group was assessed twice: immediately after the intervention (the immediate test), and again 1 month later (the delayed test). The control group was assessed once, immediately after the clerkship. All performance tests were identical.

RESULTS

The preclinical students had a significantly higher mean score on the immediate test (83.1 ± 9.6) compared to the clinical students (75.4 ± 8.9), P < .001. The effect size (Cohen's d) was .83. However, the mean score in the delayed test (76.9 ± 13.6) was not significantly different from clinical students' mean score (75.4 ± 8.9), P > .05.

CONCLUSIONS

Test-only learning as a spaced repetition of online formative testlets is effective in preparing preclinical medical students to the clinical clerkship. Through using this approach in preclinical period, they can prepare themselves for the clinical environment to optimize the benefits derived from clerkships.

Judgments of Learning Following Retrieval Practice Produce Minimal Reactivity Effect on Learning of Education-Related Materials

Testing (i.e., retrieval practice) is one of the most powerful strategies to boost learning. A recent study observed an incidental finding that making judgments of learning (JOLs) following retrieval practice further enhanced learning of education-related texts to a medium extent (Cohen's d = 0.44) by comparison with retrieval practice itself, suggesting that making JOLs may serve as an easy-to-implement educational intervention to improve the benefits of testing. Three experiments (one pre-registered) were conducted to test the replicability of Ariel et al.'s incidental finding and to further determine whether making JOLs following retrieval practice reactively enhances the benefits of testing for text learning. The three experiments consistently provided Bayesian evidence supporting no reactivity effect of JOLs following retrieval practice, regardless of whether the replication experiments were conducted in a laboratory (Experiment 1) or online (Experiments 2 and 3), whether the stimuli were presented in the same language (Experiments 2 and 3) or not (Experiment 1), and whether participants were recruited from the sample pool (Experiment 2) or not (Experiments 1 and 3) as in the original study. These null findings imply that making JOLs cannot be utilized as a practical strategy to enhance the benefits of testing for learning of educationally related materials. Possible explanations for the null reactivity effect of JOLs following retrieval practice are discussed.

Use of practice tests with immediate feedback in an undergraduate molecular biology course

Multiple studies have shown that testing contributes to learning at all educational levels. In this observational classroom study, we report the use of a learning tool developed for a Genetics and Molecular Biology course at the college level. An interactive set of practice exams that included 136 multiple choice questions (MCQ) or matching queries was developed in the open-source Moodle platform. All MCQ questions contained four answer choices and configured for immediate feedback upon answering. Feedback consisted of providing the right answer and a short explanation of the learning objective examined. The interactive material was tested and refined for several semesters. Usefulness of this tool was assessed in two distinct settings: (1) during a face-to-face semester (Fall 2019) by comparing the grades in a final departmental exam between students who used the tool and those who did not, and (2) during an online semester (Fall 2020) by analyzing the grades in the first and last attempts on study sessions and students' performance in monthly exams. We found that when solving practice tests, students obtained a significantly higher scores in the last attempt compared with their first attempt, and that students who used the material performed better than those who did not. In all cases, answering the practice exams was optional, but students made full use of them preferentially during the online semester. This classroom research exemplifies the documented effectiveness of practice tests enhanced with feedback in biological sciences education through an open-source learning platform.

Impact of test-enhanced learning as a study strategy: An exploratory study with first-year dental students

PURPOSE

Research has demonstrated that learners who practice self-testing have superior long-term retention compared to those rereading the material alone, a phenomenon called test-enhanced learning. This testing effect can be leveraged by spacing out the testing practice over time, a technique called spaced repetition. In 2017, we provided dental students at the school with access to Osmosis, a web-based platform that supports test-enhanced learning and spaced repetition through flashcards. This exploratory study examined students' adoption of self-testing with flashcards and its impact on learning performance in basic sciences.

METHODS

Participants were 143 first-year predoctoral students at a dental school in the US. The platform analytics revealed the number of flashcards students answered throughout the first academic year (2019-2020). Regression analyses examined how self-testing with flashcards impacted students' exam scores in basic sciences. Analysis of variance (ANOVA) tests examined the difference in students' exam performance among the non, minimal, occasional, and regular flashcard users who answered 0, 1-99, 100-499, and over 500 flashcards, respectively.

RESULTS

Students answered 82,766 flashcards during the year. Additionally, they created 17,973 flashcards using the platform's flashcard authoring tool. Regression analyses showed that self-testing with flashcards correlated positively with students' exam performance in anatomy, biochemistry, nutrition, and physiology. ANOVA results revealed a statistically significant difference in students' exam performance in anatomy, biochemistry, and nutrition among the four groups.

CONCLUSIONS

This study is the first in dental education to examine students' self-testing on the Osmosis platform. Results revealed that there was widespread adoption of self-testing with flashcards. The study provided additional evidence to support the value of self-testing for dental students. It has practical implications of how test-enhanced learning can be incorporated into dental education to support student learning. The study contributed to the test-enhanced learning literature in dental education, an area that has been underexplored.

Test-enhanced learning improves learner attendance during a laparoscopic box trainer simulation program

Background

Laparoscopy is the gold standard approach for many surgical procedures, but it is a complex skill to learn. Laparoscopic simulation training may help, but it is unclear how to best engage trainees in these programs. Test‐enhanced learning (TEL) uses regular, well‐defined assessments of performance throughout the training phase of learning.

Aim

The aim of this study was to assess the effects of TEL on a laparoscopic simulation program involving a cohort of medical student volunteers.

Materials and methods

A prospective cohort study was performed with a convenience sample of 40 medical students. Students were recruited to participate in a ten‐week laparoscopic simulation program. Twenty students participated in a laparoscopic surgical program with TEL (‘TEL group’), and 20 students participated in a standard laparoscopic simulation program (‘control group’).

Results

Attendance in the TEL group was significantly higher than in the standard group (71 vs 51.5%, P = 0.03). There was no difference between groups in mean time scores. Four themes were identified in qualitative data drawn from student surveys – personal traits and motivators, training context, clear goals and feedback enabling understanding of one's own performance.

Conclusion

Testing laparoscopic skills throughout a learning program, in conjunction with individualised feedback and tracking of learning trajectory, increases trainee attendance. Laparoscopic simulation training programs are encouraged to reflect on the pedagogic framework in which their procedural skills training operates.

Twelve tips on guiding preparation for both high-stakes exams and long-term learning

High-stakes exams including admissions, licensing, and maintenance of certification examinations are commonplace in health professions education. Although exam scores and performance can often serve gate-keeping purposes, the broader goal of health professions education is to foster deep, self-directed, meaningful, motivated learning. Establishing strong support structures that emphasize deep learning and understanding rather than exam scores can be beneficial to preparing learners who have the knowledge base to be excellent practitioners. This article offers guidance that can be used by academic support centres, medical educators, learning specialists, and faculty advisors, or even test-takers, to help learners to balance score achievement and knowledge development, while simultaneously cultivating more efficient and motivated studying and increasingly self-regulated learning. This series of tips details considerations for building academic success supports, fostering a growth mindset, planning efficient and effective studying efforts, utilizing test-enhanced learning strategies, exam-taking skills practice, and other support structures that can help strengthen learning experiences overall.

Where and how to learn: The interactive benefits of contextual variation, restudying, and retrieval practice for learning

Students are often advised to do all of their studying in one good place, but restudying to-be-learned material in a new context can enhance subsequent recall. We examined whether there are similar benefits for testing. In Experiment 1 (n = 106), participants studied a 36-word list and 48 hr later-when back in the same or a new context-either restudied or recalled the list without feedback. After another 48 hr, all participants free-recalled the list in a new context. Experiment 2 (n = 203) differed by having the testing-condition participants restudy the list before being tested. Across both experiments, testing in a new context reduced recall, which carried over to the final test, whereas restudying in a new context did not impair (and in Experiment 2, significantly enhanced) recall. These findings reveal critical interactions between contextual-variation and retrieval-practice effects, which we interpret as consistent with a distribution-of-memory-strengths framework.

A Web-Based Virtual Microscopy Platform for Improving Academic Performance in Histology and Pathology Laboratory Courses: A Pilot Study

Virtual microscopy (VM) has been utilized to improve students' learning experience in microscope laboratory sessions, but minimal attention has been given to determining how to use VM more effectively. The study examined the influence of VM on academic performance and teacher and student perceptions and compared laboratory test scores before and after VM incorporation. A total of 662 third-year students studying histology and 651 fourth-year students studying pathology were divided into two groups. The light microscopy (LM) group used a light microscope in 2014 and 2015, while the LM + VM group used the VM platform and a light microscope in 2016 and 2017. Four factors positively predict laboratory scores (R square, 0.323; P < 0.001): (i) the pathology course and test-enhanced learning, (ii) the VM platform and experience, (iii) medical students and lecture scores, and (iv) female students. The LM + VM group exhibited less score variability on laboratory examinations relative to their mean than the LM group. The LM + VM group was also associated with fewer failing grades (F grade; odds ratio, 0.336; P < 0.001) and higher scores (A grade; odds ratio, 2.084; P < 0.001) after controlling for sex, school, course, and lecture grades. The positive effect of the VM platform on laboratory test grades was associated with prior experience using the VM platform and was synergistic with more interim tests. Both teachers and students agreed that the VM platform enhanced laboratory learning. The incorporation of the VM platform in the context of test-enhanced learning may help more students to master microscopic laboratory content.

Do Individual Differences in Cognition and Personality Predict Retrieval Practice Activities on MOOCs?

Online quizzes building upon the principles of retrieval practice can have beneficial effects on learning, especially long-term retention. However, it is unexplored how interindividual differences in relevant background characteristics relate to retrieval practice activities in e-learning. Thus, this study sought to probe for this research question on a massive open online course (MOOC) platform where students have the optional possibility to quiz themselves on the to-be-learned materials. Altogether 105 students were assessed with a cognitive task tapping on reasoning, and two self-assessed personality measures capturing need for cognition (NFC), and grittiness (GRIT-S). Between-group analyses revealed that cognitively high performing individuals were more likely to use the optional quizzes on the platform. Moreover, within-group analyses (n = 56) including those students using the optional quizzes on the platform showed that reasoning significantly predicted quiz performance, and quiz processing speed. NFC and GRIT-S were unrelated to each of the aforementioned retrieval practice activities.

Desirable Difficulties in Spatial Learning: Testing Enhances Subsequent Learning of Spatial Information

Previous work has shown that testing can enhance learning and retention of subsequently studied new information. The present study investigated this forward testing effect in spatial memory. In two experiments, participants studied four successively presented 3 × 3 arrays, each composed of the same nine objects. They were asked to memorize the locations of the objects which differed across the four arrays. Following presentation of Arrays 1-3, memory for the object locations of the respective array was tested (testing condition), or the array was re-presented for additional study (restudy condition). Thereafter, Array 4 was presented and tested in both the testing and the restudy condition. In Experiment 1, testing was self-paced, whereas in Experiment 2, testing time was controlled by the experimenter. Consistent across the two experiments, testing was found to enhance location memory for Array 4, relative to restudying. Furthermore, testing also reduced the number of confusion errors (i.e., the tendency to misplace objects to locations on which they had appeared previously) made during recall of Array 4, suggesting that testing reduced the interference potential of prior information. The results indicate that testing can enhance subsequent learning of spatial information by reducing the build-up of proactive interference from previously studied information.

Test-Enhanced Learning in Competence-Based Predoctoral Orthodontics: A Four-Year Study

Dental educators intend to promote integration of knowledge, skills, and values toward professional competence. Studies report that retrieval, in the form of testing, results in better learning with retention than traditional studying. The aim of this study was to evaluate test-enhanced experiences on demonstrations of competence in diagnosis and management of malocclusion and skeletal problems. The study participants were all third-year dental students (2011 N=88, 2012 N=74, 2013 N=91, 2014 N=85) at New York University College of Dentistry. The 2013 and 2014 groups received the test-enhanced method emphasizing formative assessments with written and dialogic delayed feedback, while the 2011 and 2012 groups received the traditional approach emphasizing lectures and classroom exercises. The students received six two-hour sessions, spaced one week apart. At the final session, a summative assessment consisting of the same four cases was administered. Students constructed a problem list, treatment objectives, and a treatment plan for each case, scored according to the same criteria. Grades were based on the number of cases without critical errors: A=0 critical errors on four cases, A-=0 critical errors on three cases, B+=0 critical errors on two cases, B=0 critical errors on one case, F=critical errors on four cases. Performance grades were categorized as high quality (B+, A-, A) and low quality (F, B). The results showed that the test-enhanced groups demonstrated statistically significant benefits at 95% confidence intervals compared to the traditional groups when comparing low- and high-quality grades. These performance trends support the continued use of the test-enhanced approach.

Test-enhanced learning: analysis of an experience with undergraduate nursing students

BACKGROUND

This study is based on the evidence that tests can be used as an educational tool to enhance learning, not just as an evaluation tool. There is a growing body of research that shows that participating in repeated testing improves learning, a phenomenon defined as Test-Enhanced Learning. The aim of the present study was to analyse the effect of the use of a test enhanced learning program integrated into a general psychology course for undergraduate nursing students and its interaction with the students' test anxiety.

METHODS

161 undergraduate nursing students attending a General Psychology course followed an educational program based on Test-Enhanced Learning methodology. Students were divided into two groups, an experimental group (TEL group) and a control group (Re-study group). TEL students took a multiple-choice test on the lecture topics. The Re-study group just read study material. Testing and re-study occurred at intervals of about a week after each lesson. TEL students received feedback immediately after each test. About two weeks after the end of the lessons, all the students took a final cumulative test on all the topics. Statistical analysis was used to analyse students' performances. After the administration of the cumulative unit test, all the students took a graded examination.

RESULTS

Students in the TEL group performed better than the controls, both in the final cumulative test and in a graded examination. TEL participants experienced better final cumulative test results than students not tested (M TEL = 23.11, M Re-study = 20.47, t(109.86) = -2.57, p < 0.05, r = 0.24). Test-Enhanced Learning program participation has a positive impact on exam performance (βG_Step1 = 0.46, p < 0.001). Finally, the analysis performed shows a slight moderating effect of test anxiety on Test-Enhanced Learning (βGxTA_Step3 = 0.15, p < 0.05).

DISCUSSION AND CONCLUSIONS

Test-Enhanced Learning can be an effective tool for promoting and enhancing learning. In fact, taking tests after studying produced better long-term retention and then better final test performance than re-reading without testing. Both students in the TEL group and the Re-study group with a high test anxiety level perform less well than colleagues with lower test anxiety. Nevertheless, students with higher test anxiety may obtain more benefits from participating in a Test-Enhanced Learning process than people with lower test anxiety. Further studies on larger and more representative samples are necessary in order to investigate the effect of test anxiety on Test-Enhanced Learning.

Test-Enhanced Learning in an Immunology and Infectious Disease Medicinal Chemistry/Pharmacology Course

Objective. To develop a series of active-learning modules that would improve pharmacy students' performance on summative assessments. Design. A series of optional online active-learning modules containing questions with multiple formats for topics in a first-year (P1) course was created using a test-enhanced learning approach. A subset of module questions was modified and included on summative assessments. Assessment. Student performance on module questions improved with repeated attempts and was predictive of student performance on summative assessments. Performance on examination questions was higher for students with access to modules than for those without access to modules. Module use appeared to have the most impact on low performing students. Conclusion. Test-enhanced learning modules with immediate feedback provide pharmacy students with a learning tool that improves student performance on summative assessments and also may improve metacognitive and test-taking skills.

Human structure in six and one-half weeks: one approach to providing foundational anatomical competency in an era of compressed medical school anatomy curricula

The University of Oklahoma College of Medicine reduced gross anatomy from a full semester, 130-hour course to a six and one-half week, 105-hour course as part of a new integrated systems-based pre-clinical curriculum. In addition to the reduction in contact hours, content from embryology, histology, and radiology were added into the course. The new curriculum incorporated best practices in the area of regular assessments, feedback, clinical application, multiple teaching modalities, and professionalism. A comparison of the components of the traditional and integrated curriculum, along with end of course evaluations and student performance revealed that the new curriculum was just as effective, if not more effective. This article also provides important lessons learned.

Characterization of medical students recall of factual knowledge using learning objects and repeated testing in a novel e-learning system

BACKGROUND

Spaced-repetition and test-enhanced learning are two methodologies that boost knowledge retention. ALERT STUDENT is a platform that allows creation and distribution of Learning Objects named flashcards, and provides insight into student judgments-of-learning through a metric called 'recall accuracy'. This study aims to understand how the spaced-repetition and test-enhanced learning features provided by the platform affect recall accuracy, and to characterize the effect that students, flashcards and repetitions exert on this measurement.

METHODS

Three spaced laboratory sessions (s0, s1 and s2), were conducted with n=96 medical students. The intervention employed a study task, and a quiz task that consisted in mentally answering open-ended questions about each flashcard and grading recall accuracy. Students were randomized into study-quiz and quiz groups. On s0 both groups performed the quiz task. On s1 and s2, the study-quiz group performed the study task followed by the quiz task, whereas the quiz group only performed the quiz task. We measured differences in recall accuracy between groups/sessions, its variance components, and the G-coefficients for the flashcard component.

RESULTS

At s0 there were no differences in recall accuracy between groups. The experiment group achieved a significant increase in recall accuracy that was superior to the quiz group in s1 and s2. In the study-quiz group, increases in recall accuracy were mainly due to the session, followed by flashcard factors and student factors. In the quiz group, increases in recall accuracy were mainly accounted by flashcard factors, followed by student and session factors. The flashcard G-coefficient indicated an agreement on recall accuracy of 91% in the quiz group, and of 47% in the study-quiz group.

CONCLUSIONS

Recall accuracy is an easily collectible measurement that increases the educational value of Learning Objects and open-ended questions. This metric seems to vary in a way consistent with knowledge retention, but further investigation is necessary to ascertain the nature of such relationship. Recall accuracy has educational implications to students and educators, and may contribute to deliver tailored learning experiences, assess the effectiveness of instruction, and facilitate research comparing blended-learning interventions.

Effectiveness of test-enhanced learning in continuing health sciences education: a randomized controlled trial

BACKGROUND

Increasing concerns over the effectiveness and quality of continuing medical education (CME) programs has encouraged educators to use theoretically driven empirical research to improve the educational value of these activities. Within cognitive psychology, theories of learning and knowledge acquisition, such as test-enhanced learning (TEL), may be used to enhance the effectiveness of CME protocols and delivery. The present study examined whether the pedagogical benefits of testing can be observed in practicing physicians.

METHODS

A total of 83 physicians were recruited following an education session on constipation management and were randomized to either (a) the test condition (n = 43), where learners wrote a short test consisting of 10 short-answer questions (SAQs), or (b) the study condition (n = 40), where learners studied the same information. Four weeks later, 56 (68%) physicians completed a final test with 10 new SAQs, with 27 being from the initial test condition and 29 belonging to the initial study condition.

RESULTS

Performance on the final SAQ test was equivalent for both test (42.5%) and study-only (41.2%) conditions (p = .71).

DISCUSSION

The null findings in the present study are inconsistent with previous research showing the pedagogical benefits of testing relative to studying. Given that most TEL research focuses on novice learners, who lack strong associative memory networks, it is possible that TEL is specific to novices and not generalizable to experts. Alternative explanations focus on the importance of repeated, distributed testing with feedback.