Development of a tool to support holistic generic assessment of clinical procedure skills

The Intraoperative Complications Assessment and Reporting with Universal Standards (ICARUS) Global Surgical Collaboration Project: Development of Criteria for Reporting Adverse Events During Surgical Procedures and Evaluating Their Impact on the Postoperative Course

BACKGROUND

Intraoperative adverse events (iAEs) are surgical and anesthesiologic complications. Despite the availability of grading criteria, iAEs are infrequently reported in the surgical literature and in cases for which iAEs are reported, these events are described with significant heterogeneity.

OBJECTIVE

To develop Intraoperative Complications Assessment and Reporting with Universal Standards (ICARUS) Global Surgical Collaboration criteria to standardize the assessment, reporting, and grading of iAEs. The ultimate aim is to improve our understanding of the nature and frequency of iAEs and our ability to counsel patients regarding surgical procedures.

DESIGN, SETTING, AND PARTICIPANTS

The present study involved the following steps: (1) collecting criteria for assessing, reporting, and grading of iAEs via a comprehensive umbrella review; (2) collecting additional criteria via a survey of a panel of experienced surgeons (first round of a modified Delphi survey); (3) creating a comprehensive list of reporting criteria; (4) combining criteria acquired in the first two steps; and (5) establishing a consensus on clinical and quality assessment utility as determined in the second round of the Delphi survey.

OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS

Panel inter-rater agreement and consistency were assessed as the overall percentage agreement and Cronbach's α.

RESULTS AND LIMITATIONS

The umbrella review led to nine common criteria for assessing, grading, and reporting iAEs, and review of iAE grading systems led to two additional criteria. In the first Delphi round, 35 surgeons responded and two criteria were added. In the second Delphi round, 13 common criteria met the threshold for final guideline inclusion. All 13 criteria achieved the consensus minimum of 70%, with agreement on the usefulness of the criteria for clinical and quality improvement ranging from 74% to 100%. The mean inter-rater agreement was 89.0% for clinical improvement and 88.6% for quality improvement.

CONCLUSIONS

The ICARUS Global Collaboration criteria might aid in identifying important criteria when reporting iAEs, which will support all those involved in patient care and scientific publishing.

PATIENT SUMMARY

We consulted a panel of experienced surgeons to develop a set of guidelines for academic surgeons to follow when publishing surgical studies. The surgeon panel proposed a list of 13 criteria that may improve global understanding of complications during specific procedures and thus improve the ability to counsel patients on surgical risk.

A Protocol for the Development of the Intraoperative Complications Assessment and Reporting With Universal Standards Criteria: The ICARUS Project

Introduction

Perioperative complications, especially intraoperative adverse events (iAEs), carry significant potential for long-term sequelae in a patient's postoperative course. These events represent a substantial gap in contemporary surgical literature, with only a fraction of publications reporting intraoperative complications as outcomes of interest. To date, there is no universal standard for comprehensively reporting intraoperative complications in surgical practice and literature beyond the systems developed for grading individual events. We aim to establish a set of best-practice criteria for iAE reporting known as the Intraoperative Complication Assessment and Reporting with Universal Standards (ICARUS) Guidelines.

Methods and analysis

We will generate the ICARUS reporting guidelines using the EQUATOR Network development framework and the SQUIRE Guidelines. The initial step involves an umbrella review and meta-analysis of systemic reviews (SRs) assessing the perioperative adverse events of common surgeries. Measures for assessing, collecting, grading, and reporting the iAEs will be merged into a comprehensive list of criteria. Using a modified Delphi methodology, a team of expert surgeons (≥ 200 inpatient procedures/years) will contribute to and evaluate the proposed reporting guidelines. The panel will evaluate both the clinical usefulness and quality assessment and improvement utility of each criterion using a 5-point Likert. We expect multiple survey rounds until consensus regarding the utility of the guidelines is reached.

Dissemination

We plan to share then validate the newly developed guidelines within each surgical field. Dissemination will involve publicly shared guidelines, simultaneous journal publications, conference presentations, encouragement for journal endorsement, and application for inclusion in the Equator Network database. The study team plans to continue collecting feedback for future extension of the intraoperative reporting guidelines.

Highlights

Intraoperative adverse events are underreported and lack homogeneity in surgical literatureWe aim to use a modified Delphi methodology to develop the consensus-based, intraoperative complications assessment and reporting with universal standards (ICARUS) guidelinesWe will disseminate the ICARUS guidelines through journal publications and presentations at national and international meetings; journals and professional organizations will be encouraged to endorse the ICARUS guidelines.

Development and Pilot Validation of an Instrument Assessing Sensorimotor Skills for Percutaneous Gastral Puncture

Introduction

The acquisition of sensorimotor skills, so-called "technical skills", plays an essential part in the professional and continuing educational training of medical and nursing staff. Facilities turn to simulator training to promote the safe and accurate performance of endoscopic examinations. Thus, this study aimed to develop and pilot-test a corresponding assessment instrument to monitor necessary sensorimotor or "technical" skills of the examiner for a safe percutaneous endoscopic gastrostomy (AS-PEG).

Materials and Methods

Instrument development and pilot validation involved four stages: identification of potential items and initial draft of the AS-PEG; expert panel with 11 experts (content validity index [CVI] calculated); empirical validation using a quasi-experimental intervention on simulators; revision of the pilot AS-PEG taking expert assessment, and empirical testing into consideration.

Results

The initial instrument yielded 13 categories and 44 items describing the PEG procedure. Experts rated 30 out of 44 items (68%) extremely or very important for the safety of the puncture of the stomach. Initial item-CVIs ranged from 0.00 to 1.00; scale-CVI was 0.61. Twenty-four trainees (7 physicians, 17 nurses) participated in the pilot simulation study. On average, 8:25 min were required for PEG placement (min-max 5:59-13:38 min, SD = 1:43). The revised AS-PEG version was reduced to 14 items with a range of the item CVI from 0.8 to 1.0, and a scale-CVI of 0.90.

Conclusion

The AS-PEG instrument facilitates the evaluation of sensorimotor skills during percutaneous gastric puncture procedures within the context of PEG placement, across professions and without relating to the number of procedures previously performed. The instrument is economical and shows satisfying content validity.

Global Assessment of Urological Endoscopic Skills (GAUES): development and validation of a novel assessment tool to evaluate endourological skills

OBJECTIVE

To develop and evaluate an assessment tool for endourological skills during simulation including cystoscopy, ureteroscopy (URS) and transurethral resection (TUR) procedures.

METHODS

We designed a Global Assessment of Urological Endoscopic Skills (GAUES) tool, comprised of nine endourology task-specific and two global-rating skills items. The tool was developed through two rounds of the Delphi process. The GAUES tool was used to assess acquisition of URS and TUR skills of novices (Year 2 core surgical trainees, CT2) and intermediate level trainees (residents at the start of the UK higher surgical training programme in Urology, Speciality Trainee Year 3, ST3) at the Urology Simulation Boot Camp (USBC) between 2016 and 2018. Validity was evaluated by comparing scores between trainees with different levels of urological experience. Inter-rater reliability was also assessed.

RESULTS

We evaluated 130 residents, 52% of trainees were at an intermediate stage of training and 39% were novices. In all, 9% of the anonymous forms were missing demographics. The completion rate of the GAUES tool during the USBC for URS and TUR was 85% and 89%, respectively. Our analysis demonstrated a significant difference in all domains between intermediates and novices at assessment in URS, except for one domain more suited to clinical assessment (P = 0.226). There was excellent intraclass correlation (ICC) overall between the two experts' judgements, ICC = 0.841 (95% confidence interval 0.767-0.893; P < 0.001, n = 88).

CONCLUSIONS

We have developed the novel GAUES tool for cystoscopic, URS and TUR skills. Overall, we demonstrated good face, content and construct validity and excellent reliability, suggesting that the GAUES tool can be useful for endourological skills assessment.

Intraoperative Adverse Incident Classification (EAUiaiC) by the European Association of Urology ad hoc Complications Guidelines Panel

BACKGROUND

A surgical adverse incident (AI) is defined as any deviation from the normal operative course. Current complication-grading systems mostly focus on postoperative events.

OBJECTIVE

To propose an intraoperative AI classification (EAUiaiC) to facilitate reporting.

DESIGN, SETTING, AND PARTICIPANTS

The classification was developed using a modified Delphi process in which experts answered two rounds of survey questionnaires organised by the European Association of Urology ad hoc Complications Guidelines Panel. Experts evaluated AI terminology using a 5-point Likert scale for clarity, exhaustiveness, hierarchical order, mutual exclusivity, clinical utility, and quality improvement.

OUTCOME MEASURES AND STATISTICAL ANALYSIS

We considered ≥70% agreement for inclusion or exclusion. The resultant EAUiaiC was evaluated using ten sample clinical scenarios. Numerical and graphical statistical techniques were used to report the results.

RESULTS AND LIMITATIONS

In total, 343 respondents participated. The proposed EAUiaiC system comprises eight AI grades ranging from grade 0 (no deviation and no consequence to the patient) to grade 5B (wrong surgery site or intraoperative death). In the validation stage, EAUiaiC was rated highly favourably in terms of relevance and reliability (consistency) by 126 experts. Ratings for self-reported ease of use were at acceptable levels.

CONCLUSIONS

We propose a novel intraoperative AI classification (EAUiaiC) for use for urological procedures. Both the initial assessment of feasibility and the subsequent assessment of reliability suggest that it is a simple and effective tool for classifying intraoperative complications.

PATIENT SUMMARY

Complications in surgery are common. It is helpful to classify complications in a uniform and objective manner so that surgeons can easily compare outcomes and learn from complications. Here we propose a new classification system for complications that occur during urological surgical procedures. An abstract of this work was presented at the 2018 congress of the European Association of Urology.

Twelve tips for efficient procedural simulation

Procedural simulation (PS) is increasingly being used worldwide in healthcare for training caregivers in psychomotor competencies. It has been demonstrated to improve learners' confidence and competence in technical procedures, with consequent positive impacts on patient outcomes and safety. Several frameworks can guide healthcare educators in using PS as an educational tool. However, no theory-informed practical framework exists to guide them in including PS in their training programs. We present 12 practical tips for efficient PS training that translates educational concepts from theory to practice, based on the existing literature. In doing this, we aim to help healthcare educators to adequately incorporate and use PS both for optimal learning and for transfer into professional practice.

Establishing assessment criteria for clinical reasoning in orthopedic manual physical therapy: a consensus-building study

OBJECTIVES

Clinical reasoning (CR) represents one of the core components of clinical competence in Orthopaedic Manual Physical Therapy (OMPT). While education standards have been developed to guide curricular design, assessment of CR has not yet been standardized. Without theory-informed and rigorously developed measures, the certification of OMPTs lacks credibility and is less defensible. The purpose of this study was to use a theory-informed approach to generate assessment criteria for developing new assessment tools to evaluate CR in OMPT.

METHODS

A list of assessment criteria was generated based on international education standards and multiple theoretical perspectives. A modified Delphi method was used to gain expert consensus on the importance of these assessment criteria for the assessment of CR in OMPT. The OMPTs from 22 countries with experience in assessing CR were invited to participate in three rounds of online questionnaires to rate their level of agreement with these criteria. Responses were tabulated to analyze degree of consensus and internal consistency.

RESULTS

Representatives from almost half of the OMPT member organizations (MO) participated in three rounds of the Delphi. High levels of agreement were found among respondents regarding the importance and feasibility of most assessment criteria. There was high internal consistency among items within the proposed item subgroupings.

DISCUSSION

A list of assessment criteria has been established that will serve as a framework for developing new assessment tools for CR assessment in OMPT. These criteria will be important for guiding the design of certification processes in OMPT as well as other episodes of CR assessment throughout OMPT training.

Development and face validation of strategies for improving consultation skills

While formative workplace based assessment can improve learners' skills, it often does not because the procedures used do not facilitate feedback which is sufficiently specific to scaffold improvement. Provision of pre-formulated strategies to address predicted learning needs has potential to improve the quality and automate the provision of written feedback. To systematically develop, validate and maximise the utility of a comprehensive list of strategies for improvement of consultation skills through a process involving both medical students and their clinical primary and secondary care tutors. Modified Delphi study with tutors, modified nominal group study with students with moderation of outputs by consensus round table discussion by the authors. 35 hospital and 21 GP tutors participated in the Delphi study and contributed 153 new or modified strategies. After review of these and the 205 original strategies, 265 strategies entered the nominal group study to which 46 year four and five students contributed, resulting in the final list of 249 validated strategies. We have developed a valid and comprehensive set of strategies which are considered useful by medical students. This list can be immediately applied by any school which uses the Calgary Cambridge Framework to inform the content of formative feedback on consultation skills. We consider that the list could also be mapped to alternative skills frameworks and so be utilised by schools which do not use the Calgary Cambridge Framework.

Assessing fitness-to-practice of overseas-trained health practitioners by Australian registration & accreditation bodies

BACKGROUND

Assessment of fitness-to-practice of health professionals trained overseas and who wish to practice in Australia is undertaken by a range of organisations. These organisations conduct assessments using a range of methods. However there is very little published about how these organisations conduct their assessments. The purpose of the current paper is to investigate the methods of assessment used by these organisations and the issues associated with conducting these assessments.

METHODS

A series of semi-structured interviews was undertaken with a variety of organisations who undertake assessments of overseas-trained health professionals who wish to practice in Australia. Content analysis of the interviews was used to identify themes and patterns.

RESULTS

Four themes were generated from the content analysis of the interviews: (1) assessing; (2) process; (3) examiners; and (4) cost-efficiency. The themes were interconnected and each theme also had a number of sub-themes.

CONCLUSIONS

The organisations who participated in the present study used a range of assessment methods to assess overseas trained health professionals. These organisations also highlighted a number of issues, particularly related to examiners and process issues, pre- and post-assessment. Organisations demonstrated an appreciation for ongoing review of their assessment processes and incorporating evidence from the literature to inform their processes and assessment development.

Model of practical skill performance as an instrument for supervision and formative assessment

There are still weaknesses in the practical skills of newly graduated nurses. There is also an escalating pressure on existing clinical placements due to increasing student numbers and structural changes in health services. Innovative educational practices and the use of tools that might support learning are sparsely researched in the field of clinical education for nursing students. This paper reports on an action research study that promoted and investigated use of The Model of Practical Skill Performance as a learning tool during nursing students' clinical placement. Clinical supervisors and two cohorts of nursing students placed in a hospital setting shared their experiences on the use of the model in six focus group interviews. Data was also generated through the supervisors' reflective logs. The model was viewed as highly applicable in the planning of learning situations as well as during practice, performance and formative assessment of practical skills learning. It provided a common language about practical skills and enhanced the participants' understanding of professionalism in practical nursing skill. In conclusion, the model helped to highlight the complexity in mastering practical skills, afforded help in sequencing a learning process that supported the novice, and contributed to a more nuanced feedback by supervisors.

Optimizing the utility of communication OSCEs: omit station-specific checklists and provide students with narrative feedback

OBJECTIVE

To evaluate how the utility (reliability, validity, acceptability, feasibility, cost and educational impact) of a communication-OSCE was influenced by whether or not station-specific (StSp) checklists were used together with a generic instrument and whether or not narrative feedback was provided to students.

METHODS

At ten stations, faculty members rated standardized patient-student interactions using the common ground (CG) instrument (at all stations) and StSp-checklists. Both raters and patients provided written feedback. The impact of changing the design on the various utility parameters was assessed: reliability by means of a generalizability study, cost using the Reznick model and the other utility parameters by means of a survey.

RESULTS

Use of the generic instrument (CG) proved more reliable (G coefficient=0.67) than using the StSp-checklists (G=0.47) or both (G=0.65) while there was a high correlation between both scale scores (Pearsons'r=0.86). The cost was 6.5% higher when StSp-checklists were used and 5% higher when narrative feedback was provided.

CONCLUSION

The utility of a communication OSCE can be enhanced by omitting StSp-checklists and by providing narrative feedback to students.

PRACTICE IMPLICATIONS

The same generic assessment scale can be used in all stations of a communication OSCE. Providing feedback to students is promising but it increases the costs.

Checking the checklist: a content analysis of expert- and evidence-based case-specific checklist items

OBJECTIVES

Research on objective structured clinical examinations (OSCEs) is extensive. However, relatively little has been written on the development of case-specific checklists on history taking and physical examination. Background information on the development of these checklists is a key element of the assessment of their content validity. Usually, expert panels are involved in the development of checklists. The objective of this study is to compare expert-based items on OSCE checklists with evidence-based items identified in the literature.

METHODS

Evidence-based items covering both history taking and physical examination for specific clinical problems and diseases were identified in the literature. Items on nine expert-based checklists for OSCE examination stations were evaluated by comparing them with items identified in the literature. The data were grouped into three categories: (i) expert-based items; (ii) evidence-based items, and (iii) evidence-based items with a specific measure of their relevance.

RESULTS

Out of 227 expert-based items, 58 (26%) were not found in the literature. Of 388 evidence-based items found in the literature, 219 (56%) were not included in the expert-based checklists. Of these 219 items, 82 (37%) had a specific measure of importance, such as an odds ratio for a diagnosis, making that diagnosis more or less probable.

CONCLUSIONS

Expert-based, case-specific checklist items developed for OSCE stations do not coincide with evidence-based items identified in the literature. Further research is needed to ascertain what this inconsistency means for test validity.