Review of the validity of DOPS as an assessment tool for the procedural skills of surgical trainees

INTRODUCTION

Surgical trainees at all stages are mandated to use workplace-based assessments (WBAs) to gain feedback from their trainers. Direct observation of procedural skills (DOPS) is a tool to assess hands-on surgical skills. This review of the literature seeks to ascertain how valid DOPS are as an assessment tool for the procedural skills of surgical trainees according to the American Psychology Association (APA) validity framework.

METHODS

Relevant literature was identified through a structured search of Medline, CINAHL and Web of Science databases, with further papers included on citation review. Following this, papers meeting the inclusion and exclusion criteria were included in the final review specifying those investigating WBAs including DOPS, assessments in practice rather than simulation and specifically for postgraduate surgical trainees.

FINDINGS

Sixteen papers were included in the final analysis. Extracted data from the returned papers were assessed for evidence of validity in each of the five domains on the APA framework: validity based on consequence, response process, reliability, content and relationship to other variables. There are studies that show good parameters for reliability and validity for specific DOPS used in endoscopy and otolaryngology. However, there is confusion over the purpose of DOPS among trainers and trainees as to whether they should be used formatively or summatively. Recent changes to the surgical curriculum have sought to address this, and further work into the impact of this needs to be done.

Evaluating the perceptions of workplace-based assessments in surgical training: a systematic review

INTRODUCTION

Workplace-based assessments (WBAs) are intended to maximise learning opportunities in surgical training. There is speculation as to whether mandatory assessments in this form contribute to a tick-box culture. The objective of this review was to investigate surgical trainees' attitudes towards WBAs.

METHODS

This systematic review of qualitative studies was performed in accordance with the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) statement. The literature was searched on the Medline®, Embase™, PubMed and Web of Science™ databases on 22 March 2022.

RESULTS

Sixteen studies were included in the review, mostly carried out on users of the Intercollegiate Surgical Curriculum Programme portfolio in the UK. Trainees felt that WBAs were educationally useful, providing opportunity for feedback, but this was overshadowed by a pressure to reach a set annual quota for WBAs and achieve high scores. Other themes included inaccurate recording of WBAs, the role of WBAs as formative or summative assessments, engagement and accessibility of trainers, and lack of time to complete WBAs.

CONCLUSIONS

Negative perceptions about WBAs were widespread among surgical trainees despite a recognition of their capacity to facilitate learning. This review supports the recent removal of the annual quota for WBAs in UK surgical training programmes.

Assessment-Seeking Strategies: Navigating the Decision to Initiate Workplace-Based Assessment

Phenomenon: Competency-based medical education (CBME) relies on workplace-based assessment (WBA) to generate formative feedback (assessment for learning-AfL) and make inferences about competence (assessment of learning-AoL). When approaches to CBME rely on residents to initiate WBA, learners experience tension between seeking WBA for learning and for establishing competence. How learners resolve this tension may lead to unintended consequences for both AfL and AoL. We sought to explore the factors that impact both decisions to seek and not to seek WBA and use the findings to build a model of assessment-seeking strategy used by residents. In building this model we consider how the link between WBA and promotion or progression within a program impacts an individual's assessment-seeking strategy. Approach: We conducted 20 semi-structured interviews with internal medicine residents at Queen's University about the factors that influence their decision to seek or avoid WBA. Using grounded theory methodology, we applied a constant comparative analysis to collect data iteratively and identify themes. A conceptual model was developed to describe the interaction of factors impacting the decision to seek and initiate WBA. Findings: Participants identified two main motivations when deciding to seek assessments: the need to fulfill program requirements and the desire to receive feedback for learning. Analysis suggested that these motivations are often at odds with each other. Participants also described several moderating factors that impact the decision to initiate assessments, irrespective of the primary underlying motivation. These included resident performance, assessor factors, training program expectations, and clinical context. A conceptual framework was developed to describe the factors that lead to strategic assessment-seeking behaviors. Insights: Faced with the dual purpose of WBA in CBME, resident behavior in initiating assessment is guided by specific assessment-seeking strategies. Strategies reflect individual underlying motivations, influenced by four moderating factors. These findings have broad implications for programmatic assessment in a CBME context including validity considerations for assessment data used in summative decision-making including readiness for unsupervised practice.

Assessing Operative Skill in the Competency-based Education Era: Lessons From the UK and Ireland

INTRODUCTION

Decisions regarding the operative competence of surgical residents in the United Kingdom and Ireland are informed by operative workplace-based assessments (WBAs) and operative number targets for index procedures. This review seeks to outline the validity evidence of these assessment methods.

METHODS

A review of the MEDLINE (Pubmed), EMBASE and Cochrane Library databases was undertaken in accordance with the Joanna Briggs Institute Protocol for Scoping Reviews (2020). Articles were included if they provided evidence of the validity of procedure-based assessments, direct observation of procedural skills, or indicative operative number targets. The educational impact of each article was evaluated using a modified Kirkpatrick model.

RESULTS

Twenty-eight articles outlining validity evidence of WBAs and operative number targets were synthesised by narrative review. Five studies documented users' views on current assessment methods (Kirkpatrick level 1). Two articles recorded changes in attitudes towards current operative assessments (level 2a). Ten studies documented the ability of current assessments to record improvements in operative competence (level 2b). Ten studies measured a change in behaviour as a result of the introduction of these assessments (level 3). One article studied the ability of operative assessments to predict clinical outcomes (level 4b).

CONCLUSIONS

Operative WBAs are reliable. Scores achieved correlate with both time spent in training and recorded operative experience. Trainers and residents have concerns regarding the subjectivity of these assessments and the opportunistic nature in which they are used. Operative number targets are not criterion-referenced, lack validity evidence, and may be set too low to ensure operative competence.

The usefulness of work-based assessments in higher surgical training: A systematic review

BACKGROUND

Work-based assessments including procedure based assessments, case based discussions, clinical evaluation exercises and direct observation of procedural skills are used in Higher Surgical Training Program. This systematic review aims to investigate the trainer and trainee's perception of the usefulness of workplace based assessments in Higher Surgical Training Programme.

MATERIALS AND METHODS

Embase, MEDLINE and PubMed databases were searched for relevant studies published up to Jan 15, 2021. The following search terms were used: procedure based assessments, case based discussions, clinical evaluation exercises, direct observation of procedural skills, Higher Surgical Training Program (with and without their abbreviations), surgical training, formative assessment. Usefulness was analysed according to van der Vleuten's utility formula, which is the product of educational impact, validity, reliability, acceptability, cost-effectiveness and feasibility.

RESULTS

23 studies were included; Six on procedure based assessments, two on case based discussions, one on mini clinical evaluation exercises, three on direct observation of procedural skills and eleven on multiple methods. Overall, procedure based assessments had positive Kirkpatrick level 1 or 2 impact were valid, reliable and acceptable. Mini clinical evaluation exercises showed positive Kirkpatrick level 1 satisfaction in trainees and trainers and were feasible. Case based discussions were shown to have both positive and negative Kirkpatrick levels 1 and 2 impact and were rated valid and reliable. Direct observation of procedural skills were valid with favourable Kirkpatrick level 1 impact with more usage and modification of the forms. Multiple methods as used in the intercollegiate surgical curriculum programme portfolio was found to be less encouraging in achieving positive higher level of educational impact. The barriers included lack of time, lack of faculty development and lack of engagement.

CONCLUSION

It is important that work-based assessments are used properly, so that trainees get the maximum benefit from them. None of the studies included in our systematic review demonstrated an impact on behaviour or society (Kirkpatrick level 3 and 4), towards which future research should be directed to.

The Usefulness of Procedure-Based Assessments in Postgraduate Surgical Training Within the Intercollegiate Surgical Curriculum Programme; A Scoping Review

BACKGROUND

Workplace-based assessments are a mandatory component of postgraduate surgical training within the United Kingdom and Ireland. Procedure-based assessments (PBAs) and direct observation of practical skills (DOPS) are integrated within the Intercollegiate Surgical Curriculum Programme online platform and aim to assess trainees' performance in practical surgical skills; no reviews have previously investigated their educational usefulness in postgraduate surgical training. Usefulness was defined by the 5 criteria detailed by Van der Vleuten for determining the usefulness of educational assessment tools: validity; reliability; acceptability to learners and faculty; impact on future learning and practice; and costs (to the individual trainee, the institution, and society at large).

METHODS

Scoping review methodology was used to examine the educational usefulness of PBA and DOPS assessments in postgraduate surgical training. A literature search of Ovid MEDLINE, EMBASE, and Web of Science databases was undertaken. The preferred reporting items for systematic reviews and meta-analyses standards for systematic reviews were followed.

RESULTS/DISCUSSION

Ten studies met the inclusion criteria, with 1368 trainees and trainers included. A variety of study methodologies were identified. Although there is some evidence for the validity and reliability of both PBA and DOPS assessments, further work is required in both these domains including on the number of assessments required to ensure satisfactory reliability. This is a research priority, especially if these assessments become a component of summative assessment of trainee competency. The literature indicates that these assessments are generally acceptable to learners and faculty but their acceptability is negatively impacted upon by uncertainty over whether these assessments constitute a formative or summative assessment of trainees. With regards to costs, correct use of PBAs does require allocated time and resources to ensure their correct use and this must be factored into trainer and trainee job plans.

CONCLUSIONS

PBA and DOPS assessments are educationally useful tools in postgraduate surgical training. Further research is required to determine the number of assessments required to ensure adequate reliability. To ensure the educational benefits of these assessments are not diminished, clarification from postgraduate training schemes is required regarding whether these assessments an assessment for learning or an assessment of learning.

Working with underperformance in surgical training: Educational strategies for supervisors

Underperformance is a significant challenge for surgical educators. This paper outlines some educational strategies to manage the very complex demands presented by underperformance during training in the workplace. Preventative approaches include: setting expectations with early conversations; conscious attention to the workplace curriculum; and building trainee evaluative judgement through assessment and feedback. Once underperformance has been identified, other approaches including: developing a joint action plan; reducing or increasing feedback opportunities; and ensuring the trainee remains motivated. While these strategies cannot prevent failure, they offer the trainee opportunities to succeed.