Progress test utopia

The progress test as a structuring initiative for programmatic assessment

BACKGROUND

The Progress Test is an individual assessment applied to all students at the same time and on a regular basis. The test was structured in the medical undergraduate education of a conglomerate of schools to structure a programmatic assessment integrated into teaching. This paper presents the results of four serial applications of the progress test and the feedback method to students.

METHODS

This assessment comprises 120 items offered online by means of a personal password. Items are authored by faculty, peer-reviewed, and approved by a committee of experts. The items are classified by five major areas, by topics used by the National Board of Medical Examiners and by medical specialties related to a national Unified Health System. The correction uses the Item Response Theory with analysis by the "Rasch" model that considers the difficulty of the item.

RESULTS

Student participation increased along the four editions of the tests, considering the number of enrollments. The median performances increased in the comparisons among the sequential years in all tests, except for test1 - the first test offered to schools. Between subsequent years of education, 2nd-1st; 4th-3rd and 5th-4th there was an increase in median scores from progress tests 2 through 4. The final year of undergraduate showed a limited increase compared to the 5th year. There is a consistent increase in the median, although with fluctuations between the observed intervals.

CONCLUSION

The progress test promoted the establishment of regular feedback among students, teachers and coordinators and paved the road to engagement much needed to construct an institutional programmatic assessment.

Feedback, fairness, and validity: effects of disclosing and reusing multiple-choice questions in medical schools

BACKGROUND

Disclosure of items used in multiple-choice-question (MCQ) exams may decrease student anxiety and improve transparency, feedback, and test-enhanced learning but potentially compromises the reliability and fairness of exams if items are eventually reused. Evidence regarding whether disclosure and reuse of test items change item psychometrics is scarce and inconclusive.

METHODS

We retrospectively analysed difficulty and discrimination coefficients of 10,148 MCQ items used between fall 2017 and fall 2019 in a large European medical school in which items were disclosed from fall 2017 onwards. We categorised items as 'new'; 'reused, not disclosed'; or 'reused, disclosed'. For reused items, we calculated the difference from their first ever use, that is, when they were new. Differences between categories and terms were analysed with one-way analyses of variance and independent-samples t tests.

RESULTS

The proportion of reused, disclosed items grew from 0% to 48.4%; mean difficulty coefficients increased from 0.70 to 0.76; that is, items became easier, P < .001, η_p² = 0.011. On average, reused, disclosed items were significantly easier (M = 0.83) than reused, not disclosed items (M = 0.71) and entirely new items (M = 0.66), P < .001, η_p² = 0.087. Mean discrimination coefficients increased from 0.21 to 0.23; that is, item became slightly more discriminating, P = .002, η_p² = 0.002.

CONCLUSIONS

Disclosing test items provides the opportunity to enhance feedback and transparency in MCQ exams but potentially at the expense of decreased item reliability. Discrimination was positively affected. Our study may help weigh advantages and disadvantages of using previously disclosed items.

Development of a shared item repository for progress testing in veterinary education

Introduction

Progress testing in education is an assessment principle for the measurement of students' progress over time, e.g., from start to graduation. Progress testing offers valid longitudinal formative measurement of the growth in the cognitive skills of the individual students within the subjects of the test as well as a tool for educators to monitor potential educational gaps and mismatches within the curriculum in relation to the basic veterinary learning outcomes.

Methods

Six veterinary educational establishments in Denmark, Finland, Germany (Hannover), the Netherlands, Norway, and Sweden established in cooperation with the European Association of Establishments for Veterinary Education (EAEVE) a common veterinary item repository that can be used for progress testing in European Veterinary Education Establishments (VEEs), linear as well as computer adaptive, covering the EAEVE veterinary subjects and theoretical "Day One Competencies." First, a blueprint was created, suitable item formats were identified, and a quality assurance process for reviewing and approving items was established. The items were trialed to create a database of validated and calibrated items, and the responses were subsequently psychometrically analyzed according to Modern Test Theory.

Results

In total, 1,836 items were submitted of which 1,342 were approved by the reviewers for trial testing. 1,119 students from all study years and all partners VEEs participated in one or more of six item trials, and 1,948 responses were collected. Responses were analyzed using Rasch Modeling (analysis of item-fit, differential item function, item-response characteristics). A total of 821 calibrated items of various difficulty levels matching the veterinary students' abilities and covering the veterinary knowledge domains have been banked.

Discussion

The item bank is now ready to be used for formative progress testing in European veterinary education. This paper presents and discusses possible pitfalls, problems, and solutions when establishing an international veterinary progress test.

Can trauma leagues contribute to better cognitive performance and technical skills of medical students? The experience of the Unicamp trauma league

PURPOSE

Trauma leagues (TLs) are extracurricular programs that offer medical students supervised exposure to trauma and acute care surgery, mentorship, and participation in other academic activities. TLs are fully approved medical schools, and currently, over 100 TLs exist in Brazil. We hypothesized that the performance/competence of medical students who participated in TLs was superior compared to non-participants. This study evaluated and compared the cognitive performance and technical skills of the two groups.

METHODS

This retrospective cohort study evaluated the performance of TL medical students to non-TL alumni from 2005 to 2017, using the students' academic performance coefficient, Clinical Competence Assessment, and Progress Test results. SigmaPlot 12.0 software was used to perform statistical analyses, including Mann-Whitney comparison tests and the Kruskal-Wallis test to confirm the data.

RESULTS

Of the 1366 medical students who graduated from a Brazilian university, 966 were included, with 17.9% having participated in TL. Compared to non-TL participants, TL students demonstrated better cognitive performance according to the performance coefficient (p = 0.017) and Progress Test result (p < 0.001), and higher achievement in the Clinical Competence Assessment (p < 0.001).

CONCLUSION

The academic performance of TL students was superior to that of non-TL students at the University of Campinas (Unicamp), suggesting a positive impact of TL in the preparation of future doctors. The study findings suggest that participation in TL at Unicamp was beneficial in preparing better doctors and should be considered by medical schools worldwide.

EVIDENCE LEVEL

II (Retrospective cohort).

Relationship between medical programme progress test performance and surgical clinical attachment timing and performance

Purpose: Progress tests (PTs) assess applied knowledge, promote knowledge integration, and facilitate retention. Clinical attachments catalyse learning through an appropriate learning context. The relationship between PT results and clinical attachment sequence and performance are under-explored. Aims: (1) Determine the effect of Year 4 general surgical attachment (GSA) completion and sequence on overall PT performance, and for surgically coded items; (2) Determine the association between PT results in the first 2 years and GSA assessment outcomes.Materials and methods: All students enrolled in the medical programme, who started Year 2 between January 2013 and January 2016, were included; with follow up until December 2018. A linear mixed model was applied to study the effect of undertaking a GSA on subsequent PT results. Logistic regressions were used to explore the effect of past PT performance on the likelihood of a student receiving a distinction grade in the GSA.Results: 965 students were included, representing 2191 PT items (363 surgical items). Sequenced exposure to the GSA in Year 4 was associated with increased performance on surgically coded PT items, but not overall performance on the PT, with the difference decreasing over the year. PT performance in Years 2-3 was associated with an increased likelihood of being awarded a GSA distinction grade (OR 1.62, p < 0.001), with overall PT performance a better predictor than performance on surgically coded items.Conclusions: Exposure to a surgical attachment improves PT results in surgically coded PT items, although with a diminishing effect over time, implying clinical exposure may accelerate subject specific learning. Timing of the GSA did not influence end of year performance in the PT. There is some evidence that students who perform well on PTs in preclinical years are more likely to receive a distinction grade in a surgical attachment than those with lower PT scores.

Cognitive diagnostic modelling in healthcare professions education: an eye-opener

Criticisms about psychometric paradigms currently used in healthcare professions education include claims of reductionism, objectification, and poor compliance with assumptions. Nevertheless, perhaps the most crucial criticism comes from learners' difficulty in interpreting and making meaningful use of summative scores and the potentially detrimental impact these scores have on learners. The term "post-psychometric era" has become popular, despite persisting calls for the sensible use of modern psychometrics. In recent years, cognitive diagnostic modelling has emerged as a new psychometric paradigm capable of providing meaningful diagnostic feedback. Cognitive diagnostic modelling allows the classification of examinees in multiple cognitive attributes. This measurement is obtained by modelling these attributes as categorical, discrete latent variables. Furthermore, items can reflect more than one latent variable simultaneously. The interactions between latent variables can be modelled with flexibility, allowing a unique perspective on complex cognitive processes. These characteristic features of cognitive diagnostic modelling enable diagnostic classification over a large number of constructs of interest, preventing the necessity of providing numerical scores as feedback to test takers. This paper provides an overview of cognitive diagnostic modelling, including an introduction to its foundations and illustrating potential applications, to help teachers be involved in developing and evaluating assessment tools used in healthcare professions education. Cognitive diagnosis may represent a revolutionary new psychometric paradigm, overcoming the known limitations found in frequently used psychometric approaches, offering the possibility of robust qualitative feedback and better alignment with competency-based curricula and modern programmatic assessment frameworks.

Rethinking Assessment Design: Evidence-Informed Strategies to Boost Educational Impact in the Anatomical Sciences

University assessment is in the midst of transformation. Assessments are no longer designed solely to determine that students can remember and regurgitate lecture content, nor in order to rank students to aid with some future selection process. Instead, assessments are expected to drive, support, and enhance learning and to contribute to student self-assessment and development of skills and attributes for a lifetime of learning. While traditional purposes of certifying achievement and determining readiness to progress remain important, these new expectations for assessment can create tensions in assessment design, selection, and deployment. With the recognition of these tensions, three contemporary approaches to assessment in medical education are described. These approaches include careful consideration of the educational impact of assessment-before, during (test or recall enhanced learning) and after assessments; development of student (and staff) assessment literacy; and planning of cohesive systems of assessment (with a range of assessment tools) designed to assess the various competencies demanded of future graduates. These approaches purposefully straddle the cross purposes of assessment in modern health professions education. The implications of these models are explored within the context of medical education and then linked with contemporary work in the anatomical sciences in order to highlight current synergies and potential future innovations when using evidence-informed strategies to boost the educational impact of assessments.

The Correlation Between Students' Progress Testing Scores and Their Performance in a Residency Selection Process

Brazil is currently seeing an increased number of medical schools, leading to high competition for medical residency vacancies. Public managers have thus considered Progress Testing scores potentially useful as part of the final decision in the medical residency selection process. We analyzed whether there is a correlation between students' Progress Testing scores and their performances in medical residency selection. We examined four subsequent cohorts of students who attempted Progress Testing yearly and compared their accumulated scores with their medical residency selection scores from Botucatu Medical School, Universidade Estadual Paulista. We included 212 students who finished the 6-year medical course in 2013, 2014, 2015, and 2016. The comparison between the area under the Progress Testing curve and the medical residency selection score was performed using a Pearson correlation, with a p value set at < 0.05. We found a positive association between the two scores (p < 0.05 for the 4 years). Next, the students were grouped according to their performance in Progress Testing: above one, within one, and below one standard deviation. A chi-square test was used to compare the rates of approval with the second step of the medical residency selection process. Approval rates were 91.7%, 69.2%, and 42.1%, respectively (p < 0.05). We conclude that, in fact, there is a correlation between students' performance on these measures. This is partially explained by the fact that both instruments measure cognitive competencies and knowledge. These data may support national policy changes for medical residency selection.