Estimating and comparing the reliability of a suite of workplace-based assessments: an obstetrics and gynaecology setting

Use of Generalizability Theory for Exploring Reliability of and Sources of Variance in Assessment of Technical Skills: A Systematic Review and Meta-Analysis

PURPOSE

Competency-based education relies on the validity and reliability of assessment scores. Generalizability (G) theory is well suited to explore the reliability of assessment tools in medical education but has only been applied to a limited extent. This study aimed to systematically review the literature using G-theory to explore the reliability of structured assessment of medical and surgical technical skills and to assess the relative contributions of different factors to variance.

METHOD

In June 2020, 11 databases, including PubMed, were searched from inception through May 31, 2020. Eligible studies included the use of G-theory to explore reliability in the context of assessment of medical and surgical technical skills. Descriptive information on study, assessment context, assessment protocol, participants being assessed, and G-analyses was extracted. Data were used to map G-theory and explore variance components analyses. A meta-analysis was conducted to synthesize the extracted data on the sources of variance and reliability.

RESULTS

Forty-four studies were included; of these, 39 had sufficient data for meta-analysis. The total pool included 35,284 unique assessments of 31,496 unique performances of 4,154 participants. Person variance had a pooled effect of 44.2% (95% confidence interval [CI], 36.8%-51.5%). Only assessment tool type (Objective Structured Assessment of Technical Skills-type vs task-based checklist-type) had a significant effect on person variance. The pooled reliability (G-coefficient) was 0.65 (95% CI, .59-.70). Most studies included decision studies (39, 88.6%) and generally seemed to have higher ratios of performances to assessors to achieve a sufficiently reliable assessment.

CONCLUSIONS

G-theory is increasingly being used to examine reliability of technical skills assessment in medical education, but more rigor in reporting is warranted. Contextual factors can potentially affect variance components and thereby reliability estimates and should be considered, especially in high-stakes assessment. Reliability analysis should be a best practice when developing assessment of technical skills.

Workplace-based assessments in postgraduate medical education: A hermeneutic review

OBJECTIVES

Since their introduction, workplace-based assessments (WBAs) have proliferated throughout postgraduate medical education. Previous reviews have identified mixed findings regarding WBAs' effectiveness, but have not considered the importance of user-tool-context interactions. The present review was conducted to address this gap by generating a thematic overview of factors important to the acceptability, effectiveness and utility of WBAs in postgraduate medical education.

METHOD

This review utilised a hermeneutic cycle for analysis of the literature. Four databases were searched to identify articles pertaining to WBAs in postgraduate medical education from the United Kingdom, Canada, Australia, New Zealand, the Netherlands and Scandinavian countries. Over the course of three rounds, 30 published articles were thematically analysed in an iterative fashion to deeply engage with the literature in order to answer three scoping questions concerning acceptability, effectiveness and assessment training. As each round was coded, themes were refined and questions added until saturation was reached.

RESULTS

Stakeholders value WBAs for permitting assessment of trainees' performance in an authentic context. Negative perceptions of WBAs stem from misuse due to low assessment literacy, disagreement with definitions and frameworks, and inadequate summative use of WBAs. Effectiveness is influenced by user (eg, engagement and assessment literacy) and tool attributes (eg, definitions and scales), but most fundamentally by user-tool-context interactions, particularly trainee-assessor relationships. Assessors' assessment literacy must be combined with cultural and administrative factors in organisations and the broader medical discipline.

CONCLUSIONS

The pivotal determinants of WBAs' effectiveness and utility are the user-tool-context interactions. From the identified themes, we present 12 lessons learned regarding users, tools and contexts to maximise WBA utility, including the separation of formative and summative WBA assessors, use of maximally useful scales, and instituting measures to reduce competitive demands.

Mini-Clinical Evaluation Exercise in the Era of Milestones and Entrustable Professional Activities in Obstetrics and Gynaecology: Resume or Reform?

OBJECTIVE

The Accreditation Council for Graduate Medical Education (ACGME) milestones and the core Entrustable Professional Activities (EPAs) provide guiding frameworks and requirements for assessing residents' progress. The Mini-Clinical Evaluation Exercise (Mini-CEX) is a formative assessment tool used to provide direct observation after an ambulatory or clinical encounter. This study aimed to investigate the feasibility and reliability of the Mini-CEX in the authors' obstetrics and gynaecology (OB/GYN) residency program and its ability to measure residents' progress and competencies in the frameworks of ACGME milestones and EPAs.

METHODS

OB/GYN residents' 5-academic-year Mini-CEX performance was analyzed retrospectively to measure reliability and feasibility. Additionally, realistic evaluation was conducted to assess the usefulness of Mini-CEX in the frameworks of ACGME milestones and EPAs.

RESULTS

A total of 395 Mini-CEX evaluations for 49 OB/GYN residents were analyzed. Mini-CEX evaluation data significantly discriminated among residents' training levels (P < 0.003). Residents had an average of 8.1 evaluations per resident completed; 10% of second-year residents and 28% of third-year residents were evaluated 10 or more times per year, whereas no postgraduate year 1 or postgraduate year 4 residents achieved this number. Mini-CEX data could contribute to all 6 primary measurement domains of OB/GYN milestones and 8 of 10 EPAs required for first-year residents.

CONCLUSION

The Mini-CEX demonstrated potential for measuring residents' clinical competencies in their ACGME milestones. Faculty time commitment was the main challenge. Reform is necessary for the current feedback structure in Mini-CEX, faculty development, and operational guidelines that help residency programs match residents' clinical competency ratings with ACGME milestones and EPAs.

Assessors' and trainees' perceptions of workplace-based assessments in histopathology

AIM

Workplace-based assessments (WBAs) are an integral part of specialist medical and dental training in the UK. Histopathology represents a very different training environment to the surgical specialities so it is important to know that the current WBAs are fit for purpose. A literature search revealed very little evidence regarding the perception of WBAs in histopathology. The aim of this study was therefore to assess the current perception of WBAs in histopathology by assessors and trainees.

METHODS

A questionnaire was sent to all histopathology assessors and trainees on the Royal College of Pathologists database. Data were analysed using basic descriptive statistics. Free text comments were analysed by thematic analysis.

RESULTS

The questionnaire was completed by 242/2175 assessors (response rate 11.1%) and 91/519 trainees (response rate 17.5%). There was no significantly different perception of WBAs between trainers and trainees (p>0.05). Key findings included respondents stating that they strongly disagreed or disagreed WBAs identify diagnostic competence (68% of trainees; 62% of assessors), identify trainees in difficulty (69% of trainees; 64% of assessors) or accurately reflect trainee progress (62% of trainees; 59% of assessors). Analysis of free text comments identified three main issues related to WBAs: their utilisation, purpose and design.

CONCLUSION

There was an overall negative perception of WBAs, particularly their perceived regulatory nature. It appears that the current WBAs do not always meet the criteria of a useful test in that they are perceived as being applied variably, produce results with questionable validity and have little apparent educational value.

Clinical assessment of transthoracic echocardiography skills: a generalizability study

BACKGROUND

Transthoracic echocardiography (TTE) is a widely used cardiac imaging technique that all cardiologists should be able to perform competently. Traditionally, TTE competence has been assessed by unstructured observation or in test situations separated from daily clinical practice. An instrument for assessment of clinical TTE technical proficiency including a global rating score and a checklist score has previously shown reliability and validity in a standardised setting. As clinical test situations typically have several sources of error giving rise to variance in scores, a more thorough examination of the generalizability of the assessment instrument is needed.

METHODS

Nine physicians performed a TTE scan on the same three patients. Then, two raters rated all 27 TTE scans using the TTE technical assessment instrument in a fully crossed, all random generalizability study. Estimated variance components were calculated for both the global rating and checklist scores. Finally, dependability (phi) coefficients were also calculated for both outcomes in a decision study.

RESULTS

For global rating scores, 66.6% of score variance can be ascribed to true differences in performance. For checklist scores this was 88.8%. The difference was primarily due to physician-rater interaction. Four random cases rated by one random rater resulted in a phi value of 0.81 for global ratings and two random cases rated by one random rater showed a phi value of 0.92 for checklist scores.

CONCLUSIONS

Using the TTE checklist as opposed to the TTE global rating score had the effect of minimising the largest source of error variance in test scores. Two cases rated by one rater using the TTE checklist are sufficiently reliable for high stakes examinations. As global rating is less time consuming it could be considered performing four global rating assessments in addition to the checklist assessments to account for both reliability and content validity of the assessment.

Workplace-based assessment for vocational registration of international medical graduates

INTRODUCTION

Medical regulatory authorities need efficient and effective methods of ensuring the competence of immigrating international medical graduates (IMGs). Not all IMGs who apply for specialist vocational registration will have directly comparable qualifications to those usually accepted. As general licensure examinations are inappropriate for these doctors, workplace-based assessment (WBA) techniques would appear to provide a solution. However, there is little published data on such outcomes.

METHODS

All cases of WBA (n = 81) used for vocational registration of IMGs in New Zealand between 2008 and 2013 were collated and analyzed.

RESULTS

The successful completion rate of IMGs through the pathway was 87%. The majority (64%) undertook the year of supervised practice and the final assessment in a provincial center. For those unsuccessful in the pathway, inadequate clinical knowledge was the most common deficit found, followed by poor clinical reasoning.

DISCUSSION

A WBA approach for assessing readiness of IMGs for vocational registration is feasible. The constructivist theoretical perspective of WBA has particular advantages in assessing the standard of practice for experienced practitioners working in narrow scopes than traditional methods of assessment. The majority of IMGs undertook both the clinical year and the assessment in provincial hospitals, thus providing a workforce for underserved areas.