Impact of Question Bank Use for In-Training Examination Preparation by OBGYN Residents - A Multicenter Study

OBJECTIVE

To examine the impact of access to and utilization of a commercially available question bank (TrueLearn) for in-training examination (ITE) preparation in Obstetrics and Gynecology (OBGYN).

DESIGN

This was a retrospective cohort study examining the impact of TrueLearn usage on ITE examination performance outcomes. Produced by the educational arm of the American College of Obstetricians and Gynecologists, the Council on Resident Education in Obstetrics and Gynecology (CREOG) exam is a multiple-choice test given to all residents annually. Residency programs participating in this study provided residency program mean CREOG scores from the year prior (2015), and the first (2016) and second (2017) years of TrueLearn usage. Programs also contributed resident-specific CREOG scores for each resident for 2016 and 2017. This data was combined with each resident's TrueLearn usage data that was provided by TrueLearn with residency program consent. The CREOG scores consisted of the CREOG score standardized to all program years, the CREOG score standardized to the same program year (PGY) and the total percent (%) correct. TrueLearn usage data included number of practice questions completed, number of practice tests taken, average number of days between successive tests, and percent correct of answered practice questions.

SETTING

OBGYN Residency Training Programs.

PARTICIPANTS

OBGYN residency programs that purchased and utilized TrueLearn for the 2016 CREOG examination were eligible for participation (n = 14). Ten residency programs participated, which consisted of 212 residents in 2016 and 218 residents in 2017.

RESULTS

TrueLearn was used by 78.8% (167/212) of the residents in 2016 and 84.9% (185/218) of the residents in 2017. No significant difference was seen in the average CREOG scores available on a per- program level before versus after the first year of implementation either using the CREOG score standardized to all PGYs (mean difference 1.0; p = 0.58) or standardized to the same PGY (mean difference 3.1; p = 0.25). Using resident-level data, there was no significant difference in mean CREOG score standardized to all PGYs between users and non-users of TrueLearn in 2016 (mean, 199.4 vs 196.7; p = 0.41) or 2017 (mean, 198.2 vs 203.4; p = 0.19). The percent of practice questions answered correctly on TrueLearn was positively correlated with the CREOG score standardized to all PGYs (r = 0.47 for 2016 and r = 0.60 for 2017), as well as with the CREOG total percent correct (r = 0.47 for 2016 and r = 0.61 for 2017). Based on a simple linear regression, for every 500 practice questions completed, the CREOG score significantly increased for PGY-2 residents by an average (±SE) of 7.3 ± 2.8 points (p = 0.013); the average increase was 0.7 ± 2.5 (p = 0.79) for PGY-3 residents and 5.8 ± 3.3 points (p = 0.09) for PGY-4 residents.

CONCLUSIONS

Adoption of an online question bank did not result in higher mean CREOG scores at participating institutions. However, performance on the TrueLearn questions correlated with ITE performance, supporting predictive validity and the use of this question bank as a formative assessment for resident education and exam preparation.

How to Prepare for the American Board of Surgery In-Training Examination (ABSITE): A Systematic Review

INTRODUCTION

Performance on ABSITE is an important factor when monitoring resident progress. It predicts future performance and has lasting effects. Understanding the highest-yield preparation strategies can help residents in their study efforts and optimize performance.

METHODS

A literature search was conducted searching PubMed, EMBASE and JAMA Network in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Searches were performed for the terms "ABSITE" and "American Board of Surgery In-Training Examination". Only studies discussing individual study habits from May 2011 to May 2021 were included.

RESULTS

19 studies were included in qualitative synthesis. Year-round clinical study failed to show significant correlation to ABSITE performance although year-round ABSITE review was more consistently correlated. During a dedicated study period, increased time and increased total practice questions completed are associated with improved performance. The correlation of individual resources such as ABSITE review books, textbooks, audio podcasts and ABSITE preparatory courses to improved ABSITE performance was not proven.

CONCLUSIONS

Residents should optimize study strategies based on methods that have consistently shown to improve performance. Recommendations for best preparation strategies are provided.

Formative Feedback with In-Class Question Bank Utilization Improves Resident Satisfaction with General Surgery Didactics

BACKGROUND

Formative feedback provides low-stakes opportunities for educational improvement. To enrich our basic science didactics, formative feedback measures were incorporated into our didactics using mobile devices.

MATERIALS AND METHODS

Lecture changes included institutional paid access to a commercial question bank, a 5-item in-class pre-didactic quiz curated from the question bank and taken on the resident's mobile device, and group discussion of quiz topics. An anonymous survey was sent to participating residents.

RESULTS

Overall response rate was 71% among residents. All reported that the new lecture format was a valuable addition to the basic science curriculum (100% Agree/Strongly Agree), and formative assessments provided valuable feedback about the progress of their learning (Strongly Agree = 42%, Agree =58%). All residents reported that in-class use of their mobile device for quizzes was convenient, with majority (84%) preferring it over paper printouts. Residents were more motivated to study before lecture (Strongly Agree = 42%, Agree =42%), with majority also reporting the new format helped identify weaknesses in their knowledgebase (Strongly Agree = 58%, Agree =33%). While majority of residents agreed that quizzes motivated them to study more after lecture, a large portion disagreed (42%). Majority of senior residents reported that the process of composing quizzes prior to lecture enriched their own learning (57%) and helped them find gaps in their knowledge (71%).

CONCLUSION

Incorporating a commercial question bank within didactics gives general surgery residents formative feedback and encourages learning outside the classroom, leading to improved satisfaction with basic science didactics.

Spaced Education With ABSITE Quest Resulting in Improved American Board of Surgery In-Training Examination Performance

OBJECTIVE

The American Board of Surgery In-Training Examination (ABSITE) is an annual exam taken by general surgery residents as a cognitive assessment of the knowledge gained throughout each year of training. Several question banks are available for ABSITE preparation. However, ABSITE Quest (AQ) utilizes a method called spaced education which has been demonstrated to help with retaining information longer and improve exam performance. This study hypothesizes that using this method will help residents improve their ABSITE performance.

DESIGN

Retrospective survey data was collected from residents who participated in AQ, including postgraduate year (PGY) level, as well as 2019 and 2020 ABSITE percentiles. AQ user data was used to match respondent's total number of questions completed and daily engagement level to the survey data. Paired, single-tailed student's t test was used to evaluate the significance of ABSITE percentile change between 2019 and 2020 among AQ users.

SETTING

ChristianaCare, Newark, DE, United States. Nonclinical.

PARTICIPANTS

All ABSITE Quest users were surveyed (n = 390), of which 104 responded. 21 responses were from PGY1 residents and were excluded, resulting in a total of 83 responses.

RESULTS

The mean percentile difference of AQ users from 2019 to 2020 was +15.8 (p < 0.00001). When categorizing by the total number of questions completed, high users demonstrated a mean percentile difference of +15.3 (p = 0.00002), average users had a difference of +19.1 (p = 0.00029), and low users showed a percentile difference of +1.2 (p = 0.45244). When categorizing by daily engagement level, high users demonstrated a mean percentile difference of +17.9 (p < 0.00001), low users had a mean percentile difference of +15.3 (p = 0.00124), and minimal users showed a mean percentile change of -5.7.

CONCLUSIONS

The use of the spaced education method with ABSITE Quest, especially in users with a greater number of questions completed and high levels of daily engagement, correlated with a significant improvement on ABSITE performance.

Correlation of Medical College Admission Test Scores and Self-assessment Materials with the United States Medical Licensing Examination Step 1 Performance

Purpose

Candidates' performance on the United States Medical Licensing Examination (USMLE) Step 1 examination had been correlated with the Medical College Admission Test (MCAT). However, in 2015, a new MCAT format was released and its correlation with Step 1 remains to be fully analyzed. Preparation for Step 1 typically involves purchasing and perusing practice tests from the National Board of Medical Examiners (NBME) and UWorld; however, their predictive value to performance on Step 1 remains to be ascertained, especially with the release of five new NBME practice tests. Additionally, there is a need for accurately predicting Step 1 scores to self-evaluate study progress and reduce student anxiety.

Rationale

Program directors rank USMLE Step 1 scores as the number one criterion in selecting interviewees for residency. Step 1 scores are more important than Step 2 scores, Dean’s letter, or other letters of recommendation in determining the overall ranking of a candidate after interviews.

Hypotheses

The authors hypothesized that the new MCAT scores correlated positively with Step 1 scores and that the new NBME practice tests were more predictive of performance on Step 1 as compared to old NBME tests. 

Methods

Linear regression analysis followed by either analysis of variance (ANOVA) or Student's t-tests were used to analyze 399 responses. Data obtained was used to update an existing Step 1 score predictor, which was then validated.

Results

A positive correlation between the MCAT (average score: 510.1 ± 6.3) and Step 1 scores (average score: 246.1 ± 14.2) was observed. While new NBME practice tests were more predictive of Step 1 scores than old NBME tests, UWorld test scores were the most predictive. Students who practiced with the new NBME practice tests scored significantly higher than students who did not use them. However, students using any of the UWorld practice tests did significantly better than students who practiced using only NBME practice tests but not UWorld practice tests. Ironically, NBME16,the second-most correlativetest to Step 1 performance, is no longer available for purchase. Overall, taking six or more practice tests significantly enhanced Step 1 scores; the optimal number of tests was found to be between six and nine. The predicted score by an updated Step 1 score predictor was within 3.8 points or 1.6% of the actual Step 1 score.

Conclusions

We believe this study will aid in the selection and purchase of appropriate self-assessment tests as preparatory material for the USMLE Step 1 examination. It will also introduce them to an existing Step 1 score predictor that will help determine their readiness for Step 1.

Formative student-authored question bank: perceptions, question quality and association with summative performance

Purpose of the study

There are few studies on the value of authoring questions as a study method, the quality of the questions produced by students and student perceptions of student-authored question banks. Here we evaluate PeerWise, a widely used and free online resource that allows students to author, answer and discuss multiple-choice questions.

Study design

We introduced two undergraduate medical student cohorts to PeerWise (n=603). We looked at their patterns of PeerWise usage; identified associations between student engagement and summative exam performance; and used focus groups to assess student perceptions of the value of PeerWise for learning. We undertook item analysis to assess question difficulty and quality.

Results

Over two academic years, the two cohorts wrote 4671 questions, answered questions 606 658 times and posted 7735 comments. Question writing frequency correlated most strongly with summative performance (Spearman’s rank: 0.24, p=<0.001). Student focus groups found that: (1) students valued curriculum specificity; and (2) students were concerned about student-authored question quality. Only two questions of the 300 ’most-answered' questions analysed had an unacceptable discriminatory value (point-biserial correlation <0.2).

Conclusions

Item analysis suggested acceptable question quality despite student concerns. Quantitative and qualitative methods indicated that PeerWise is a valuable study tool.

Stepping up of MCQs' quality through a multi-stage reviewing process

BACKGROUND

Assessment is a crucial and complex process. Thus quality assurance of assessment Methods is essential when assessment is used for the purposes of certification.

AIM

To evaluate the effect of continuous well-structured process of the question bank revision, enlightened by item analysis, in improving the quality of Multiple-Choice Questions (MCQs).

SETTING

The Family Medicine (FM) training certification exam for the Egyptian Board accredited for International Membership of the Royal College of General Practitioners (MRCGP[INT]).

METHOD

The results of the item analysis of Applied Knowledge Tests (AKTs) of two academic years (2009, 2013) were included in the study. The AKT test consisted of two papers, each of 100 MCQS, and blue printed against the FM training curriculum. A total of 226 candidates sat for the two exams; 102 in 2009 and 124 in 2013.

RESULTS

There were more MCQs with moderate difficulty and higher discrimination in 2013. Significant improvement was found in the discrimination index (DI) values in 2013 (p < 0.001). and questions with a high facility and classified as easy decreased from 40.5 to 28.7%. The average number of functioning distractors per item increased from 1.99 in 2009 to 2.19 in 2013 (p = 0.015).

CONCLUSION

Revision of well-constructed MCQs on a regular basis, and in a structured manner, improved the quality of the MCQs and consequently improved the validity of the examination.

Constructing a question bank based on script concordance approach as a novel assessment methodology in surgical education

BACKGROUND

Script Concordance Test (SCT) is a new assessment tool that reliably assesses clinical reasoning skills. Previous descriptions of developing SCT-question banks were merely subjective. This study addresses two gaps in the literature: 1) conducting the first phase of a multistep validation process of SCT in Plastic Surgery, and 2) providing an objective methodology to construct a question bank based on SCT.

METHODS

After developing a test blueprint, 52 test items were written. Five validation questions were developed and a validation survey was established online. Seven reviewers were asked to answer this survey. They were recruited from two countries, Saudi Arabia and Canada, to improve the test's external validity. Their ratings were transformed into percentages. Analysis was performed to compare reviewers' ratings by looking at correlations, ranges, means, medians, and overall scores.

RESULTS

Scores of reviewers' ratings were between 76% and 95% (mean 86% ± 5). We found poor correlations between reviewers (Pearson's: +0.38 to -0.22). Ratings of individual validation questions ranged between 0 and 4 (on a scale 1-5). Means and medians of these ranges were computed for each test item (mean: 0.8 to 2.4; median: 1 to 3). A subset of test items comprising 27 items was generated based on a set of inclusion and exclusion criteria.

CONCLUSION

This study proposes an objective methodology for validation of SCT-question bank. Analysis of validation survey is done from all angles, i.e., reviewers, validation questions, and test items. Finally, a subset of test items is generated based on a set of criteria.