Progress testing: critical analysis and suggested practices

Re-examining single-moment-in-time high-stakes examinations in specialist training: A critical narrative review

In this critical narrative review, we challenge the belief that single-moment-in-time high-stakes examinations (SMITHSEx) are an essential component of contemporary specialist training. We explore the arguments both for and against SMITHSEx, examine potential alternatives, and discuss the barriers to change.SMITHSEx are viewed as the "gold standard" assessment of competence but focus excessively on knowledge assessment rather than capturing essential competencies required for safe and competent workplace performance. Contrary to popular belief, regulatory bodies do not mandate SMITHSEx in specialist training. Though acting as significant drivers of learning and professional identity formation, these attributes are not exclusive to SMITHSEx.Skills such as crisis management, procedural skills, professionalism, communication, collaboration, lifelong learning, reflection on practice, and judgement are often overlooked by SMITHSEx. Their inherent design raises questions about the validity and objectivity of SMITHSEx as a measure of workplace competence. They have a detrimental impact on trainee well-being, contributing to burnout and differential attainment.Alternatives to SMITHSEx include continuous low-stakes assessments throughout training, ongoing evaluation of competence in the workplace, and competency-based medical education (CBME) concepts. These aim to provide a more comprehensive and context-specific assessment of trainees' competence while also improving trainee welfare.Specialist training colleges should evolve from exam providers to holistic education sources. Assessments should emphasise essential practical knowledge over trivia, align with clinical practice, aid learning, and be part of a diverse toolkit. Eliminating SMITHSEx from specialist training will foster a competency-based approach, benefiting future medical professionals' well-being and success.

Assessment of medical students' Surgery knowledge based on Progress Test

Progress Testing (PT) is an assessment tool whose use has grown throughout Brazil in the last decade. PT makes it possible to assess the students' knowledge gain throughout the undergraduate course and, for their interpretations to be valid, their items (questions) must have adequate quality from the point of view of content validity and reliability of results. In this study, we analyzed the psychometric characteristics of the items and the performance of students in the content area of surgery from 2017 to 2023. For the analyses, we used the assumptions of Classical Test Theory, Bloom's taxonomy and Cronbach's alpha reliability coefficient. The items were easy (average difficulty index between 0.3-0.4), with fair to good discrimination (discrimination index between 0.3-0.4) and with a predominance of medium to high taxonomy. Reliability remained substantial over the years (>0.6). Students' knowledge gain in surgery was found to be progressive and more important from the 3rd year of the undergraduate course, reaching approximately 70-75% in the 6th year. This measurements framework can be replicated in other contexts for a better understanding of student learning and for qualification of evaluation processes.

Longitudinal Assessment to Evaluate Continued Certification and Lifelong Learning in Healthcare Professionals: A Scoping Review

The balance of lifelong learning with assessment for continued certification is a challenge faced by healthcare professions. The value of single-point-in-time assessments has been questioned, and a shift to longitudinal assessments (LA) has been undertaken to assess lifelong learning over-time. This scoping review was conducted to inform healthcare certifying organizations who are considering LA as an assessment tool of competence and lifelong learning in healthcare professionals. A search of 6 databases and grey literature yielded 957 articles. After screening and removal of duplicates, 14 articles were included. Most articles were background studies informing the underpinnings of LA in the form of progress testing, pilot studies, and process of implementation. Progress testing is used in educational settings. Pilot studies reported satisfaction with LA's ease of use, online format, and provision of lifelong learning. Implementation processes reveal that key aspects of success include stakeholder participation, phased rollout, and a publicly available content outline. Initial outcomes data affirm that LA addresses knowledge gaps, and results in improved performance on maintenance of certification exams. Future research is needed to substantiate validity evidence of LA and its correlation with high-stakes exam performance when assessing lifelong learning and continued competence of healthcare professionals over time.

Relationship between medical programme progress test performance and surgical clinical attachment timing and performance

Purpose: Progress tests (PTs) assess applied knowledge, promote knowledge integration, and facilitate retention. Clinical attachments catalyse learning through an appropriate learning context. The relationship between PT results and clinical attachment sequence and performance are under-explored. Aims: (1) Determine the effect of Year 4 general surgical attachment (GSA) completion and sequence on overall PT performance, and for surgically coded items; (2) Determine the association between PT results in the first 2 years and GSA assessment outcomes.Materials and methods: All students enrolled in the medical programme, who started Year 2 between January 2013 and January 2016, were included; with follow up until December 2018. A linear mixed model was applied to study the effect of undertaking a GSA on subsequent PT results. Logistic regressions were used to explore the effect of past PT performance on the likelihood of a student receiving a distinction grade in the GSA.Results: 965 students were included, representing 2191 PT items (363 surgical items). Sequenced exposure to the GSA in Year 4 was associated with increased performance on surgically coded PT items, but not overall performance on the PT, with the difference decreasing over the year. PT performance in Years 2-3 was associated with an increased likelihood of being awarded a GSA distinction grade (OR 1.62, p < 0.001), with overall PT performance a better predictor than performance on surgically coded items.Conclusions: Exposure to a surgical attachment improves PT results in surgically coded PT items, although with a diminishing effect over time, implying clinical exposure may accelerate subject specific learning. Timing of the GSA did not influence end of year performance in the PT. There is some evidence that students who perform well on PTs in preclinical years are more likely to receive a distinction grade in a surgical attachment than those with lower PT scores.

Progress testing: An educational perspective exploring the rationale for progress testing and its introduction into a Diagnostic Radiography curriculum

INTRODUCTION

In March 2020, the first diagnostic radiography degree apprenticeship programme in England was launched at the authors' institution. As part of the programme development and design, the programme development team explored and then implemented progress testing into a strand of the programme. The objective of this educational perspective is to scrutinise the literature around the use of progress testing in higher education programmes, namely medicine, to explain how and why this decision was reached.

METHODS

The initial search strategy was developed using the electronic databases CINHAL Complete and SCOPUS. Key words included 'progress test' and 'medicine' or 'health' or 'education' or 'higher education'. Eliminating articles that were not relevant, and also identifying and adding additional articles by key authors and experts resulted in thirty-three key articles being considered for review.

RESULTS

The thirty-three articles were a mixture of review articles, empirical research, case studies and conference presentations. Five key themes were identified which are discussed in this article; the evolution of progress testing; advantages of progress testing, disadvantages of progress testing, developing a test framework and academic progression and student feedback.

DISCUSSION

Progress testing is now well established in pre-registration medical programmes globally. The advantages of progress testing and the use of frequent look rapid remediation appear to be undisputed. Key disadvantages with progress testing were identified as it being an administrative heavy assessment process as well as a perceived bias towards male students undertaking this type of assessment.

CONCLUSION

Now this assessment practice is established within medicine, it seems reasonable to explore its use in other areas of healthcare, such as radiography.

10 years of experience in adopting, implementing and evaluating progress testing for Saudi medical students

Objectives

The progress test (PT) is a comprehensive examination that is designed to match the knowledge acquisition necessary at graduation and monitors progress during the entire period of an undergraduate program. Qassim College of Medicine (QCM) began using the multi-institutional PT in the Kingdom of Saudi Arabia (KSA). This study aimed to determine if the PT can be utilized to assess the progress of medical students at different Saudi medical colleges with different educational approaches, as well as whether this testing modality could be accepted by other colleges.

Methods

Beside the establishment of a PT committee, comprehensive blueprinting was crafted to sample 200 A-type multiple choice questions (MCQs) from different disciplines. The PT is a paper-and-pencil model and is answered in a 4-h period. All PT items followed a uniform design.

Results

In total, 13 rounds of the progress test have been conducted. The number of participating colleges increased from three (with 285 students) in the first test (May 2012) to more than 20 (with >6000 students) in the ninth round (February 2017). The average % scores for first-year students ranged from 3.0% to 7.9% while the average scores for fifth-year students ranged from 34.0% to 43.0%.

Conclusion

The conduction of this meticulously crafted test to evaluate knowledge achievement at medical graduation is a fruitful tool and helps to provide constructive feedback for test-takers and other stakeholders relating to their relative positions among other fellows at the national level.

Written-Based Progress Testing: A Scoping Review

PURPOSE

Progress testing is an increasingly popular form of assessment in which a comprehensive test is administered to learners repeatedly over time. To inform potential users, this scoping review aimed to document barriers, facilitators, and potential outcomes of the use of written progress tests in higher education.

METHOD

The authors followed Arksey and O'Malley's scoping review methodology to identify and summarize the literature on progress testing. They searched 6 databases (Academic Search Complete, CINAHL, ERIC, Education Source, MEDLINE, and PsycINFO) on 2 occasions (May 22, 2018, and April 21, 2020) and included articles written in English or French and pertaining to written progress tests in higher education. Two authors screened articles for the inclusion criteria (90% agreement), then data extraction was performed by pairs of authors. Using a snowball approach, the authors also screened additional articles identified from the included reference lists. They completed a thematic analysis through an iterative process.

RESULTS

A total of 104 articles were included. The majority of progress tests used a multiple-choice and/or true-or-false question format (95, 91.3%) and were administered 4 times a year (38, 36.5%). The most documented source of validity evidence was internal consistency (38, 36.5%). Four major themes were identified: (1) barriers and challenges to the implementation of progress testing (e.g., need for additional resources); (2) established collaboration as a facilitator of progress testing implementation; (3) factors that increase the acceptance of progress testing (e.g., formative use); and (4) outcomes and consequences of progress test use (e.g., progress testing contributes to an increase in knowledge).

CONCLUSIONS

Progress testing appears to have a positive impact on learning, and there is significant validity evidence to support its use. Although progress testing is resource- and time-intensive, strategies such as collaboration with other institutions may facilitate its use.

Factors associated with student performance on the medical residency test

OBJECTIVE

To determine whether the scores of the Progress test, the Skills and Attitude test, and the medical internship are correlated with the medical residency exam performance of students who started medical school at the Federal University of São Paulo in 2009.

METHODS

The scores of 684 Progress tests from years 1-6 of medical school, 111 Skills and Attitude exams (5th year), 228 performance coefficients for the 5th and 6th years of internship, and 211 scores on the medical residency exam were analyzed longitudinally. Correlations between scores were assessed by Pearson's correlation. Factors associated with medical residency scores were analyzed by linear regression.

RESULTS

Scores of Progress tests from years 1-6 and the Skills and Attitude test showed at least one moderate and significant correlation with each other. The theoretical exam and final exam scores in the medical residency had a moderate correlation with performance in the internship. The score of the theoretical medical residency exam was associated with performance in internship year 6 (β=0.833; p<0.001), and the final medical residency exam score was associated with the Skills and Attitude score (β=0.587; p<0.001), 5th-year internship score, (β=0.060; p=0.025), and 6th-year Progress test score (β=0.038; p=0.061).

CONCLUSIONS

The scores of these tests showed significant correlations. The medical residency exam scores were positively associated with the student's performance in the internship and on the Skills test, with a tendency for the final medical residency exam score to be associated with the 6th-year Progress test.

Motivations of assessment item writers in medical programs: a qualitative study

BACKGROUND

The challenge of generating sufficient quality items for medical student examinations is a common experience for medical program coordinators. Faculty development strategies are commonly used, but there is little research on the factors influencing medical educators to engage in item writing. To assist with designing evidence-based strategies to improve engagement, we conducted an interview study informed by self-determination theory (SDT) to understand educators' motivations to write items.

METHODS

We conducted 11 semi-structured interviews with educators in an established medical program. Interviews were transcribed verbatim and underwent open coding and thematic analysis.

RESULTS

Major themes included; responsibility for item writing and item writer motivations, barriers and enablers; perceptions of the level of content expertise required to write items; and differences in the writing process between clinicians and non-clinicians.

CONCLUSIONS

Our findings suggest that flexible item writing training, strengthening of peer review processes and institutional improvements such as improved communication of expectations, allocation of time for item writing and pairing new writers with experienced writers for mentorship could enhance writer engagement.

Introducing Summative Progress Testing in Radiology Residency: Little Change in Residents' Test Results After Transitioning from Formative Progress Testing

INTRODUCTION

Educational effects of transitioning from formative to summative progress testing are unclear. Our purpose was to investigate whether such transitioning in radiology residency is associated with a change in progress test results.

METHODS

We investigated a national cohort of radiology residents (N > 300) who were semi-annually assessed through a mandatory progress test. Until 2014, this test was purely formative for all residents, but in 2014/2015, it was transitioned (as part of a national radiology residency program revision) to include a summative pass requirement for new residents. In 7 posttransitioning tests in 2015-2019, including summatively and formatively tested residents who followed the revised and pre-transitioning residency program, respectively, we assessed residents' relative test scores and percentage of residents that reached pass standards.

RESULTS

Due to our educational setting, most posttransitioning tests had no residents in the summative condition in postgraduate year 4-5, nor residents in the formative condition in year 0.5-2. Across the 7 tests, relative test scores in postgraduate year 1-3 of the summative resident group and year 3.5-4.5 of the formative group differed significantly (p < 0.01 and p < 0.05, respectively, Kruskal-Wallis test). However, scores fluctuated without consistent time trends and without consistent differences between both resident groups. Percentage of residents reaching the pass standard did not differ significantly across tests or between groups.

DISCUSSION

Transitioning from formative to summative progress testing was associated with overall steady test results of the whole resident group in 4 post-transitioning years. We do not exclude that transitioning may have positive educational effects for resident subgroups.

Assessing the growth in clinical skills using a progress clinical skills examination

BACKGROUND

This study evaluates the generalizability of an eight-station progress clinical skills examination and assesses the growth in performance for six clinical skills domains among first- and second-year medical students over four time points during the academic year.

METHODS

We conducted a generalizability study for longitudinal and cross-sectional comparisons and assessed growth in six clinical skill domains via repeated measures ANOVA over the first and second year of medical school.

RESULTS

The generalizability of the examination domain scores was low but consistent with previous studies of data gathering and communication skills. Variations in case difficulty across administrations of the examination made it difficult to assess longitudinal growth. It was possible to compare students at different training levels and the interaction of training level and growth. Second-year students outperformed first-year students, but first-year students' clinical skills performance grew faster than second-year students narrowing the gap in clinical skills over the students' first year of medical school.

CONCLUSIONS

Case specificity limits the ability to assess longitudinal growth in clinical skills through progress testing. Providing students with early clinical skills training and authentic clinical experiences appears to result in the rapid growth of clinical skills during the first year of medical school.

Relationships between Bloom's taxonomy, judges' estimation of item difficulty and psychometric properties of items from a progress test: a prospective observational study

BACKGROUND

Progress tests are longitudinal assessments of students' knowledge based on successive tests. Calibration of the test difficulty is challenging, especially because of the tendency of item-writers to overestimate students' performance. The relationships between the levels of Bloom's taxonomy, the ability of test judges to predict the difficulty of test items and the real psychometric properties of test items have been insufficiently studied.

OBJECTIVE

To investigate the psychometric properties of items according to their classification in Bloom's taxonomy and judges' estimates, through an adaptation of the Angoff method.

DESIGN AND SETTING

Prospective observational study using secondary data from students' performance in a progress test applied to ten medical schools, mainly in the state of São Paulo, Brazil.

METHODS

We compared the expected and real difficulty of items used in a progress test. The items were classified according to Bloom's taxonomy. Psychometric properties were assessed based on their taxonomy and fields of knowledge.

RESULTS

There was a 54% match between the panel of experts' expectations and the real difficulty of items. Items that were expected to be easy had mean difficulty that was significantly lower than that of items that were expected to be medium (P < 0.05) or difficult (P < 0.01). Items with high-level taxonomy had higher discrimination indices than low-level items (P = 0.026). We did not find any significant differences between the fields in terms of difficulty and discrimination.

CONCLUSIONS

Our study demonstrated that items with high-level taxonomy performed better in discrimination indices and that a panel of experts may develop coherent reasoning regarding the difficulty of items.

Formative assessment scores in tutorial sessions correlates with OSCE and progress testing scores in a PBL medical curriculum

BACKGROUND

Effective assessments programs are a challenge in problem-based learning (PBL). One of the main principles of this educational setting is the Formative Assessment (FA). We hypothesized that students' performance assessed by FA in tutorial sessions in a PBL curriculum is related to other summative assessments.

OBJECTIVE

To investigate the correlation among FA in tutorial sessions with grades obtained in Objective Structured Clinical Evaluation (OSCE) and Progress Testing (PT) to better understand the assessment process in PBL medical teaching approach and to predict student's future performance.

DESIGN

An observational cross-sectional study was conducted comparing FA, OSCE and PT scores from 4th to 8th semester medical students. Correlation analyses were performed using pooled and separate data from the 4th and 8th semesters.

RESULTS

From the 5th to 8th semester, OSCE scores were smaller compared to the FA, while PT scores were lower in all stages. In the pooled data, the correlation analysis showed a significant positive relationship between grades on FA and OSCE, FA and PT and OSCE and PT. A significant correlation among the three assessments strategies was also detected in the 8^th semester, but not in the 4th semester.

CONCLUSIONS

Assessment strategies in PBL approach, including FA, OSCE and PT, have positive correlations, which increases as the medical course becomes more complex.

The Correlation Between Students' Progress Testing Scores and Their Performance in a Residency Selection Process

Brazil is currently seeing an increased number of medical schools, leading to high competition for medical residency vacancies. Public managers have thus considered Progress Testing scores potentially useful as part of the final decision in the medical residency selection process. We analyzed whether there is a correlation between students' Progress Testing scores and their performances in medical residency selection. We examined four subsequent cohorts of students who attempted Progress Testing yearly and compared their accumulated scores with their medical residency selection scores from Botucatu Medical School, Universidade Estadual Paulista. We included 212 students who finished the 6-year medical course in 2013, 2014, 2015, and 2016. The comparison between the area under the Progress Testing curve and the medical residency selection score was performed using a Pearson correlation, with a p value set at < 0.05. We found a positive association between the two scores (p < 0.05 for the 4 years). Next, the students were grouped according to their performance in Progress Testing: above one, within one, and below one standard deviation. A chi-square test was used to compare the rates of approval with the second step of the medical residency selection process. Approval rates were 91.7%, 69.2%, and 42.1%, respectively (p < 0.05). We conclude that, in fact, there is a correlation between students' performance on these measures. This is partially explained by the fact that both instruments measure cognitive competencies and knowledge. These data may support national policy changes for medical residency selection.

Dynamic Measurement in Health Professions Education: Rationale, Application, and Possibilities

Dynamic measurement modeling (DMM) is a psychometric paradigm that uses longitudinal data to estimate individual students' growth in measured skills over the course of an educational program (i.e., growth scores). DMM represents a more formal way of assessing learning progress across the health professions education continuum. In this article, the authors provide justification for this approach in health professions education and demonstrate its proof-of-concept use with three time points of United States Medical Licensing Examination Step exams to generate growth scores for 454 current and recent medical learners. The authors demonstrate that learners vary substantially on their growth scores, and those growth scores exhibit psychometric reliability. In addition, growth scores significantly and positively correlated with indicators of medical learner readiness (e.g., undergraduate grade point average and Medical College Admission Test scores). These growth scores were also capable of significantly and positively correlating with future ratings of clinical competencies during internship as assessed through a survey sent to their program directors at the end of the first postgraduate year (e.g., patient care, interpersonal skills). These preliminary findings of reliability and validity for DMM growth scores provide initial evidence for further investigation into the suitability of a dynamic measurement paradigm in health professions education.

Plus ça change, plus c'est pareil: Making a continued case for the use of MCQs in medical education

Despite the increased emphasis on the use of workplace-based assessment in competency-based education models, there is still an important role for the use of multiple choice questions (MCQs) in the assessment of health professionals. The challenge, however, is to ensure that MCQs are developed in a way to allow educators to derive meaningful information about examinees' abilities. As educators' needs for high-quality test items have evolved so has our approach to developing MCQs. This evolution has been reflected in a number of ways including: the use of different stimulus formats; the creation of novel response formats; the development of new approaches to problem conceptualization; and the incorporation of technology. The purpose of this narrative review is to provide the reader with an overview of how our understanding of the use of MCQs in the assessment of health professionals has evolved to better measure clinical reasoning and to improve both efficiency and item quality.

Do students achieve the desired learning goals using open-book formative assessments?

OBJECTIVES

The present study aimed to examine whether medical students benefit from an open-book online formative assessment as a preparation for a practical course.

METHODS

A between-subjects experimental design was used: participants - a whole cohort of second-year medical students (N=232) - were randomly assigned to either a formative assessment that covered the topic of a subsequent practical course (treatment condition) or a formative assessment that did not cover the topic of the subsequent course (control condition). Course-script-knowledge, as well as additional in-depth-knowledge, was assessed.

RESULTS

Students in the treatment condition had better course-script knowledge, both at the beginning, t₍₂₁₂₎ = 4.96, p < .01, d = 0.72., and in the end of the practical course , t₍₂₀₈₎ = 4.80, p < .01, d = 0.68. Analyses of covariance show that this effect is stronger for those students who understood the feedback that was presented within the formative assessment, F_{(1, 213)}=10.17, p<.01. Additionally, the gain of in-depth-knowledge was significantly higher for students in the treatment condition compared to students in the control condition, t₍₂₀₈₎ = 3.68., p < .05, d = 0.72 (0.51).

CONCLUSIONS

Students benefit from a formative assessment that is related to and takes place before a subsequent practical course. They have a better understanding of the topic and gain more in-depth-knowledge that goes beyond the content of the script. Moreover, the study points out the importance of feedback pages in formative assessments.

Resident and Faculty Attitudes Toward the Dutch Radiology Progress Test as It Transitions from a Formative to a Summative Measure of Licensure Eligibility

BACKGROUND

Progress testing, a regularly administered comprehensive test of a complete knowledge domain, usually serves to provide learners feedback and has a formative nature.

OBJECTIVE

Our study aimed to investigate the acceptability of introducing a summative component in the postgraduate Dutch Radiology Progress Test (DRPT) among residents and program directors in a competency-based training program.

METHODS

A 15-item questionnaire with 3 items on acceptability of summative postgraduate knowledge testing, 7 on acceptability of the summative DRPT regulations, 4 on self-reported educational effects, and 1 open comment item was distributed nationally among 349 residents and 81 radiology program directors.

RESULTS

The questionnaire was filled out by 330 residents (95%) and 48 (59%) program directors. Summative postgraduate knowledge testing was regarded as acceptable by both groups, but more so by program directors than residents. The transition toward summative assessment in the DRPT was received neutrally to slightly positively by residents, while program directors regarded it as an improvement and estimated the summative criteria to be lighter and less stressful than did residents. The residents' self-reported educational effects of summative assessment in the DRPT were limited, whereas program directors expected a greater end-of-training knowledge improvement than residents.

CONCLUSIONS

Both residents and program directors support summative postgraduate knowledge testing, although it is more accepted by program directors. Residents receive summative radiological progress testing neutrally to slightly positively, while program directors generally value it more positively than residents. Directors should be aware of these different perspectives when introducing or developing summative progress testing in residency programs.

Enhanced Requirements for Assessment in a Competency-Based, Time-Variable Medical Education System

Competency-based, time-variable medical education has reshaped the perceptions and practices of teachers, curriculum designers, faculty developers, clinician educators, and program administrators. This increasingly popular approach highlights the fact that learning among different individuals varies in duration, foundation, and goal. Time variability places particular demands on the assessment data that are so necessary for making decisions about learner progress. These decisions may be formative (e.g., feedback for improvement) or summative (e.g., decisions about advancing a student). This article identifies challenges to collecting assessment data and to making assessment decisions in a time-variable system. These challenges include managing assessment data, defining and making valid assessment decisions, innovating in assessment, and modeling the considerable complexity of assessment in real-world settings and richly interconnected social systems. There are hopeful signs of creativity in assessment both from researchers and practitioners, but the transition from a traditional to a competency-based medical education system will likely continue to create much controversy and offer opportunities for originality and innovation in assessment.

Time-Variable Training in Medicine: Theoretical Considerations

The introduction of competency-based medical education has shifted thinking from a fixed-time model to one stressing attained competencies, independent of the time needed to arrive at those competencies. In this article, the authors explore theoretical and conceptual issues related to time variability in medical training, starting with the Carroll model from the 1960s that put time in the equation of learning. They discuss mastery learning, deliberate practice, and learning curves.While such behaviorist theories apply well to structured courses and highly structured training settings, learning in the clinical workplace is not well captured in such theories or in the model that Carroll proposed. Important in clinical training are self-regulation and motivation; neurocognitive perspectives of time and learning; professional identity formation; and entrustment as an objective of training-all of which may be viewed from the perspective of the time needed to complete training. The authors conclude that, in approaching time variability, the Carroll equation is too simplistic in its application to the breadth of medical training. The equation may be expanded to include variables that determine effective workplace learning, but future work will need to examine the validity of these additional factors.

Percepção e desempenho de graduandos de enfermagem em avaliação de metodologias ativas

Resumo Objetivo: Conhecer o desempenho e a percepção dos estudantes de enfermagem na prova de progressão e verificar a existência de relação entre o desempenho e a série em curso e entre o desempenho e o grau de dificuldade da prova. Métodos: Trata-se de um estudo descritivo de abordagem quantitativa, utilizou-se um instrumento constituído por questões sócio demográficas, classificação do grau de dificuldade da prova e duas perguntas sobre vantagens e desvantagens da prova de progressão. Resultados: A população constituiu-se de 78 estudantes. Apresentaram idade média de 24,4 anos e predomínio do sexo feminino, 89,7%. Os estudantes que consideraram a prova fácil tiveram desempenho maior (p=0.036), os estudantes do segundo ano destacaram-se em relação ao primeiro (p=0,014). Consideraram vantagem: conteúdo da prova 32,0%, progressão e desempenho 25,7%, questões de múltipla escolha 23,1%, e preparo para concursos 7,7%. As desvantagens, 39,7% prova muito extensa, 26,9% ausência de conhecimento, 15,3% temática e estrutura inadequadas e 6,4%pouco tempo para realização da prova. Conclusão: A Prova de Progressão avalia o desempenho do estudante longitudinalmente, identifica potencialidades e fragilidades curriculares, além de avaliar a instituição. Compreende um dos instrumentos avaliativos no uso de metodologias ativas de ensino e deve ser incentivada quanto à sua aplicabilidade nos cursos de graduação.

Fourteen years of progress testing in radiology residency training: experiences from The Netherlands

Objectives

To describe the development of the Dutch Radiology Progress Test (DRPT) for knowledge testing in radiology residency training in The Netherlands from its start in 2003 up to 2016.

Methods

We reviewed all DRPTs conducted since 2003. We assessed key changes and events in the test throughout the years, as well as resident participation and dispensation for the DRPT, test reliability and discriminative power of test items.

Results

The DRPT has been conducted semi-annually since 2003, except for 2015 when one digital DRPT failed. Key changes in these years were improvements in test analysis and feedback, test digitalization (2013) and inclusion of test items on nuclear medicine (2016). From 2003 to 2016, resident dispensation rates increased (Pearson’s correlation coefficient 0.74, P-value <0.01) to maximally 16 %. Cronbach´s alpha for test reliability varied between 0.83 and 0.93. The percentage of DRPT test items with negative item-rest-correlations, indicating relatively poor discriminative power, varied between 4 % and 11 %.

Conclusions

Progress testing has proven feasible and sustainable in Dutch radiology residency training, keeping up with innovations in the radiological profession. Test reliability and discriminative power of test items have remained fair over the years, while resident dispensation rates have increased.

Key Points

• Progress testing allows for monitoring knowledge development from novice to senior trainee.

• In postgraduate medical training, progress testing is used infrequently.

• Progress testing is feasible and sustainable in radiology residency training.

Toward a better judgment of item relevance in progress testing

BACKGROUND

Items must be relevant to ensure item quality and test validity. Since "item relevance" has not been operationalized yet, we developed a rubric to define it. This study explores the influence of this rubric on the assessment of item relevance and on inter-rater agreement.

METHODS

Members of the item review committee (RC) and students, teachers, and alumni (STA) reassessed the relevance of 50 previously used progress test (PT) items and decided about their inclusion using a 5-criteria rubric. Data were analyzed at item level using paired samples t-tests, Intraclass Correlation Coefficients (ICC), and linear regression analysis, and at rater level in a generalizability analysis per group.

RESULTS

The proportion of items that the RC judged relevant enough to be included decreased substantially from 1.00 to 0.72 (p < 0.001). Agreement between the RC and STA was high, with an ICC of >0.7 across items. The relation between inclusion and relevance was strong (correlation = 0.89, p < 0.001), and did not differ between RC and STA. To achieve an acceptable inter-rater reliability for relevance and inclusion, 6 members must serve on the RC.

CONCLUSIONS

Use of the rubric results in a stricter evaluation of items' appropriateness for inclusion in the PT and facilitates agreement between the RC and other stakeholders. Hence, it may help increase the acceptability and validity of the PT.