Deliberate practice and acquisition of expert performance: a general overview

Program directors' perceptions of importance of pediatric procedural skills and resident preparedness

Background

The Royal College of Physicians and Surgeons of Canada (RCPSC) objectives for training in pediatrics include 26 procedural skills, 11 of which are included in the final in-training evaluation report (FITER). The importance of each procedure for practice and the preparedness of pediatric residency graduates to perform these procedures are not known.

Methods

A questionnaire was distributed to all pediatric residency program directors and members of the RCPSC Specialty Committee in Pediatrics (N = 21) in October 2010, requesting them to rate the perceived importance and preparedness of graduating pediatric residents in all procedural skills on a 5 point Likert scale, as well as the presence of a curriculum and documentation for each procedure. Mean importance and preparedness were calculated for each procedure.

Results

Response rate was 16/21 (76 %). Perceived preparedness was significantly lower than importance for the majority of procedures (p < 0.05). Ten procedures had a high mean importance rating (>3) but a low mean preparedness rating (<3). Presence of a curriculum and documentation for procedures varied across centers, and their presence was correlated with both perceived importance and preparedness (p < 0.0001).

Conclusions

Many procedures in which pediatric residents are required to be competent by the RCPSC are felt to be important. Residents are not felt to be adequately prepared in several of the required procedures by the time of graduation. Procedures with high ratings of importance but low preparedness ratings should be targeted for curricular interventions.

Electronic supplementary material

The online version of this article (doi:10.1186/s13104-015-1499-8) contains supplementary material, which is available to authorized users.

Evaluation of a Standardized Program for Training Practicing Anesthesiologists in Ultrasound-Guided Regional Anesthesia Skills

OBJECTIVES

Practicing anesthesiologists have generally not received formal training in ultrasound-guided perineural catheter insertion. We designed this study to determine the efficacy of a standardized teaching program in this population.

METHODS

Anesthesiologists in practice for 10 years or more were recruited and enrolled to participate in a 1-day program: lectures and live-model ultrasound scanning (morning) and faculty-led iterative practice and mannequin-based simulation (afternoon). Participants were assessed and recorded while performing ultrasound-guided perineural catheter insertion at baseline, at midday (interval), and after the program (final). Videos were scored by 2 blinded reviewers using a composite tool and global rating scale. Participants were surveyed every 3 months for 1 year to report the number of procedures, efficacy of teaching methods, and implementation obstacles.

RESULTS

Thirty-two participants were enrolled and completed the program; 31 of 32 (97%) completed the 1-year follow-up. Final scores [median (10th-90th percentiles)] were 21.5 (14.5-28.0) of 30 points compared to 14.0 (9.0-20.0) at interval (P < .001 versus final) and 12.0 (8.5-17.5) at baseline (P < .001 versus final), with no difference between interval and baseline. The global rating scale showed an identical pattern. Twelve of 26 participants without previous experience performed at least 1 perineural catheter insertion after training (P < .001). However, there were no differences in the monthly average number of procedures or complications after the course when compared to baseline.

CONCLUSIONS

Practicing anesthesiologists without previous training in ultrasound-guided regional anesthesia can acquire perineural catheter insertion skills after a 1-day standardized course, but changing clinical practice remains a challenge.

Introducing simulation-based education to healthcare professionals: exploring the challenge of integrating theory into educational practice

BACKGROUND

Introducing simulation-based education to the curricular programme of healthcare professionals can be challenging. This study explored the early experiences of healthcare professionals in the use of simulation. This was in the context of the Kuwait-Scotland transformational health innovation network programme.

METHODS

Two cohorts of healthcare professionals undertook a simulation module as part of faculty development programme in Kuwait. Participants' initial perceptions of simulators were gathered using a structured questionnaire in the clinical skills centre. Their subsequent ability to demonstrate the application of simulation was evaluated through analyses of the video-recordings of teaching sessions they undertook and written reflections of their experiences of using simulation.

RESULTS

In theory, participants were able to identify simulators' classification and fidelity. They also recognised some of the challenges of using simulators. In their teaching sessions, most participants focused on using part-task trainers to teach procedural skills. In their written reflections, they did not articulate a justification for their choice of simulator or its limitations.

CONCLUSION

This study demonstrated a theory-to-practice gap in the early use of simulation by healthcare educators. The findings highlight the need for deliberate practice and adequate mentorship for educators to develop confidence and competence in the use of simulation as part of their educational practice.

Neuronal bases of structural coherence in contemporary dance observation

The neuronal processes underlying dance observation have been the focus of an increasing number of brain imaging studies over the past decade. However, the existing literature mainly dealt with effects of motor and visual expertise, whereas the neural and cognitive mechanisms that underlie the interpretation of dance choreographies remained unexplored. Hence, much attention has been given to the action observation network (AON) whereas the role of other potentially relevant neuro-cognitive mechanisms such as mentalizing (theory of mind) or language (narrative comprehension) in dance understanding is yet to be elucidated. We report the results of an fMRI study where the structural coherence of short contemporary dance choreographies was manipulated parametrically using the same taped movement material. Our participants were all trained dancers. The whole-brain analysis argues that the interpretation of structurally coherent dance phrases involves a subpart (superior parietal) of the AON as well as mentalizing regions in the dorsomedial prefrontal cortex. An ROI analysis based on a similar study using linguistic materials (Pallier et al., 2011) suggests that structural processing in language and dance might share certain neural mechanisms.

Patient Safety Risks of Basic Urological Procedures Performed by Junior and Senior Residents

OBJECTIVE

To investigate the current performance of urological residents regarding basic urological procedures in relation to patient safety issues and the identification of specific training needs.

DESIGN

Observational data of 146 urethrocystoscopies (UCSs), 27 transrectal ultrasounds of the prostate (TRUSs), 38 transrectal ultrasound-guided prostatic biopsies (TRUSPs), and 30 transurethral resections of bladder tumor (TURBTs) were collected. Performance was evaluated using scoring lists including details on completeness of procedural steps, level of independence, time, and the incidence of unintended events. The causal factors contributing to the unintended events were identified by 2 expert urologists and classified according to the recognized PRISMA method.

SETTING

This study was performed in 5 teaching hospitals in the Netherlands.

PARTICIPANTS

We included 11 junior residents and 5 senior residents in urology in the final study cohort.

RESULTS

Senior residents showed a lower degree of completeness in material usage than junior residents did during UCS (p < 0.01) and in preparation, material usage, and procedure during TRUSP (all p < 0.05). In UCS and TURBT, senior residents received significantly less feedback than junior residents did (both p < 0.01). Incidence of unintended events for junior vs senior residents was 11% and 4% in UCS, 0% and 7% in transrectal ultrasound of the prostate, 36% and 62% in TRUSP, and 41% and 23% in TURBT, respectively. Overall, unintended events were mainly caused by human factors, in particular, verification and skills-based issues.

CONCLUSION

Present performance of basic urological procedures involves a high percentage of unintended events, especially in TRUSP and TURBT, which are mainly caused by human factors and are a potential threat for patient safety. Junior residents are less independent but more thorough in the performance of UCS and TRUSP than senior residents are. Targeted skills training including assessment should be implemented before privileges for independent practice are granted to reduce the incidence of unintended events and optimize patient safety.

Predicting plasticity: acute context-dependent changes to vocal performance predict long-term age-dependent changes

Understanding the factors that predict and guide variation in behavioral change can lend insight into mechanisms of motor plasticity and individual differences in behavior. The performance of adult birdsong changes with age in a manner that is similar to rapid context-dependent changes to song. To reveal mechanisms of vocal plasticity, we analyzed the degree to which variation in the direction and magnitude of age-dependent changes to Bengalese finch song could be predicted by variation in context-dependent changes. Using a repeated-measures design, we found that variation in age-dependent changes to the timing, sequencing, and structure of vocal elements ("syllables") was significantly predicted by variation in context-dependent changes. In particular, the degree to which the duration of intersyllable gaps, syllable sequencing at branch points, and fundamental frequency of syllables within spontaneous [undirected (UD)] songs changed over time was correlated with the degree to which these features changed from UD song to female-directed (FD) song in young-adult finches (FDyoung). As such, the structure of some temporal features of UD songs converged over time onto the structure of FDyoung songs. This convergence suggested that the FDyoung song could serve as a stable target for vocal motor plasticity. Consequently, we analyzed the stability of FD song and found that the temporal structure of FD song changed significantly over time in a manner similar to UD song. Because FD song is considered a state of heightened performance, these data suggest that age-dependent changes could reflect practice-related improvements in vocal motor performance.

From Generating in the Lab to Tutoring Systems in Classrooms

Work in cognitive and educational psychology examines a variety of phenomena related to the learning and retrieval of information. Indeed, Alice Healy, our honoree, and her colleagues have conducted a large body of groundbreaking research on this topic. In this article we discuss how 3 learning principles (the generation effect, deliberate practice and feedback, and antidotes to disengagement) discussed in Healy, Schneider, and Bourne (2012) have influenced the design of 2 intelligent tutoring systems that attempt to incorporate principles of skill and knowledge acquisition. Specifically, this article describes iSTART-2 and the Writing Pal, which provide students with instruction and practice using comprehension and writing strategies. iSTART-2 provides students with training to use effective comprehension strategies while self-explaining complex text. The Writing Pal provides students with instruction and practice to use basic writing strategies when writing persuasive essays. Underlying these systems are the assumptions that students should be provided with initial instruction that breaks down the tasks into component skills and that deliberate practice should include active generation with meaningful feedback, all while remaining engaging. The implementation of these assumptions is complicated by the ill-defined natures of comprehension and writing and supported by the use of various natural language processing techniques. We argue that there is value in attempting to integrate empirically supported learning principles into educational activities, even when there is imperfect alignment between them. Examples from the design of iSTART-2 and Writing Pal guide this argument.

Validity: applying current concepts and standards to gynecologic surgery performance assessments

Validity is critical for meaningful assessment of surgical competency. According to the Standards for Educational and Psychological Testing, validation involves the integration of data from well-defined classifications of evidence. In the authoritative framework, data from all classifications support construct validity claims. The two aims of this study were to develop a categorization method for validity evidence published in support of surgery performance assessments and to summarize the results of applying this methodology to the gynecologic surgery literature. This was a critical analysis of published observations reported as validity evidence in studies with a construct validity claim. Medline and Embase databases were searched using keywords: "surgery" and "construct validity". Parameters included English-language articles published from 2000 to 2012. Gynecologic studies were analyzed for definitions of construct validity and nonstandard terminology. Categorization criteria were developed and applied by the researchers to all observations. Two independent evaluators examined reported observations for compliance with guidelines provided by the Standards. Inter-rater agreement was calculated using weighted kappa. The initial search returned 167 articles. Twenty-five articles were left for inclusion in our analysis. Eighteen (72 %) articles defined construct validity as the ability to discriminate between expert and novice levels of proficiency. Within the sample, 80 discrete observations of reported validity evidence were identified and categorized according to standard classifications. Nearly 30 % of all published observations intended to demonstrate differences in performance by level of proficiency, 25 % described a scoring model, and 14 % demonstrated support of assessment content. Not one article contained a statistical correlation between assessment scores and objective outcomes from the authentic surgical environment. Medians for level of rigor ranged from 0 to 1 across all forms of evidence. Weighted kappa values ranged 0.60-0.91. Validity claims in gynecologic surgical assessment over-rely on generalizability evidence. No test-criterion evidence was observed. Increased awareness of current standards and systematic argument development is needed for gynecologic performance assessments.

Learn, see, practice, prove, do, maintain: an evidence-based pedagogical framework for procedural skill training in medicine

Acquisition of competency in procedural skills is a fundamental goal of medical training. In this Perspective, the authors propose an evidence-based pedagogical framework for procedural skill training. The framework was developed based on a review of the literature using a critical synthesis approach and builds on earlier models of procedural skill training in medicine. The authors begin by describing the fundamentals of procedural skill development. Then, a six-step pedagogical framework for procedural skills training is presented: Learn, See, Practice, Prove, Do, and Maintain. In this framework, procedural skill training begins with the learner acquiring requisite cognitive knowledge through didactic education (Learn) and observation of the procedure (See). The learner then progresses to the stage of psychomotor skill acquisition and is allowed to deliberately practice the procedure on a simulator (Practice). Simulation-based mastery learning is employed to allow the trainee to prove competency prior to performing the procedure on a patient (Prove). Once competency is demonstrated on a simulator, the trainee is allowed to perform the procedure on patients with direct supervision, until he or she can be entrusted to perform the procedure independently (Do). Maintenance of the skill is ensured through continued clinical practice, supplemented by simulation-based training as needed (Maintain). Evidence in support of each component of the framework is presented. Implementation of the proposed framework presents a paradigm shift in procedural skill training. However, the authors believe that adoption of the framework will improve procedural skill training and patient safety.

Learning curves in health professions education

Learning curves, which graphically show the relationship between learning effort and achievement, are common in published education research but are not often used in day-to-day educational activities. The purpose of this article is to describe the generation and analysis of learning curves and their applicability to health professions education. The authors argue that the time is right for a closer look at using learning curves-given their desirable properties-to inform both self-directed instruction by individuals and education management by instructors.A typical learning curve is made up of a measure of learning (y-axis), a measure of effort (x-axis), and a mathematical linking function. At the individual level, learning curves make manifest a single person's progress towards competence including his/her rate of learning, the inflection point where learning becomes more effortful, and the remaining distance to mastery attainment. At the group level, overlaid learning curves show the full variation of a group of learners' paths through a given learning domain. Specifically, they make overt the difference between time-based and competency-based approaches to instruction. Additionally, instructors can use learning curve information to more accurately target educational resources to those who most require them.The learning curve approach requires a fine-grained collection of data that will not be possible in all educational settings; however, the increased use of an assessment paradigm that explicitly includes effort and its link to individual achievement could result in increased learner engagement and more effective instructional design.

Job Demands, Job Resources, and Flexible Competence

Building upon previous research that focused on the relationships between job demands, job resources, and employee psychological well-being, this longitudinal research makes a unique contribution by relating job demands and job resources to teachers’ professional development (TPD) at work and flexible competence, the latter being a key factor in teachers’ career development. This study was carried out among 211 teachers working in primary and secondary education in the Netherlands. TPD at work appeared to be related to flexible competence and proved to be a mediator between job resources, on the one hand, and flexible competence, on the other hand. Job resources positively enhanced TPD at work and in turn were related to flexible competence. Moreover, a direct negative relationship between job demands and flexible competence was found.

Conducting Elite Performance Training

Training to excellence in the conduct of surgical procedures has many similarities to the acquisition and mastery of technical skills in elite-level music and sports. By using coaching techniques and strategies gleaned from analysis of professional music ensembles and athletic training, surgical educators can set conditions that increase the success rate of training to elite performance. This article describes techniques and strategies used in both music and athletic coaching, and it discusses how they can be applied and integrated into surgical simulation and education.

How faculty members experience workplace-based assessment rater training: a qualitative study

CONTEXT

Direct observation of clinical skills is a common approach in workplace-based assessment (WBA). Despite widespread use of the mini-clinical evaluation exercise (mini-CEX), faculty development efforts are typically required to improve assessment quality. Little consensus exists regarding the most effective training methods, and few studies explore faculty members' reactions to rater training.

OBJECTIVES

This study was conducted to qualitatively explore the experiences of faculty staff with two rater training approaches - performance dimension training (PDT) and a modified approach to frame of reference training (FoRT) - to elucidate how such faculty development can be optimally designed.

METHODS

In a qualitative study of a multifaceted intervention using complex intervention principles, 45 out-patient resident faculty preceptors from 26 US internal medicine residency programmes participated in a rater training faculty development programme. All participants were interviewed individually and in focus groups during and after the programme to elicit how the training influenced their approach to assessment. A constructivist grounded theory approach was used to analyse the data.

RESULTS

Many participants perceived that rater training positively influenced their approach to direct observation and feedback, their ability to use entrustment as the standard for assessment, and their own clinical skills. However, barriers to implementation and change included: (i) a preference for holistic assessment over frameworks; (ii) challenges in defining competence; (iii) difficulty in changing one's approach to assessment, and (iv) concerns about institutional culture and buy-in.

CONCLUSIONS

Rater training using PDT and a modified approach to FoRT can provide faculty staff with assessment skills that are congruent with principles of criterion-referenced assessment and entrustment, and foundational principles of competency-based education, while providing them with opportunities to reflect on their own clinical skills. However, multiple challenges to incorporating new forms of training exist. Ongoing efforts to improve WBA are needed to address institutional and cultural contexts, and systems of care delivery.

Assessment of Movement Patterns during Intubation between Novice and Experienced Providers Using Mobile Sensors: A Preliminary, Proof of Concept Study

BACKGROUND

There are likely marked differences in endotracheal intubation (ETI) techniques between novice and experienced providers. We performed a proof of concept study to determine if portable motion technology could identify the motion components of ETI between novice and experienced providers.

METHODS

We recruited a sample of novice and experienced providers to perform ETIs on a cadaver. Their movements during ETI were recorded with inertial measurement units (IMUs) on the left wrist. The signals were assessed visually between novice and experienced providers to identify areas of differences at key steps during ETI. We then calculated spectral smoothness (SS), a quantitative measure inversely related to movement variability, for all ETI attempts.

RESULTS

We enrolled five novice and five experienced providers. When visually inspecting the data, we noted maximum variability when inserting the blade of the laryngoscope into the mouth and while visualizing the glottic opening. Novice providers also had greater overall variability in their movement patterns (SS novice 6.4 versus SS experienced 26.6).

CONCLUSION

Portable IMUs can be used to detect differences in movement patterns between novice and experienced providers in cadavers. Future ETI educational efforts may utilize portable IMUs to help accelerate the learning curve of novice providers.

Barriers and Facilitators to Effective Feedback: A Qualitative Analysis of Data From Multispecialty Resident Focus Groups

BACKGROUND

Despite the importance of feedback, the literature suggests that there is inadequate feedback in graduate medical education.

OBJECTIVE

We explored barriers and facilitators that residents in anesthesiology, emergency medicine, obstetrics and gynecology, and surgery experience with giving and receiving feedback during their clinical training.

METHODS

Residents from 3 geographically diverse teaching institutions were recruited to participate in focus groups in 2012. Open-ended questions prompted residents to describe their experiences with giving and receiving feedback, and discuss facilitators and barriers. Data were transcribed and analyzed using the constant comparative method associated with a grounded theory approach.

RESULTS

A total of 19 residents participated in 1 of 3 focus groups. Five major themes related to feedback were identified: teacher factors, learner factors, feedback process, feedback content, and educational context. Unapproachable attendings, time pressures due to clinical work, and discomfort with giving negative feedback were cited as major barriers in the feedback process. Learner engagement in the process was a major facilitator in the feedback process.

CONCLUSIONS

Residents provided insights for improving the feedback process based on their dual roles as teachers and learners. Time pressures in the learning environment may be mitigated by efforts to improve the quality of teacher-learner relationships. Forms for collecting written feedback should be augmented by faculty development to ensure meaningful use. Efforts to improve residents' comfort with giving feedback and encouraging learners to engage in the feedback process may foster an environment conducive to increasing feedback.

Preparedness and education in airway management

A prepared airway practitioner performs safe airway management, displaying skill, knowledge, and a full awareness of human factors, within a culture of safety. The education of prepared practitioners should include deliberate practice and distributed learning and should aim for expertise rather than mere competence. Translational outcomes from improved education and training can significantly decrease patient morbidity and mortality.

The Benefits of Simulation-Based Education

Simulation-based education is an accepted teaching methodology within many disciplines, but has yet to be fully integrated into the clinical education models of speech-language pathology and audiology. Simulation-based education is an innovative opportunity to enhance clinical education in the fields of speech-language pathology and audiology. The approach provides a number of benefits that positively impact critical stakeholders, including training programs, faculty, students, clinical supervisors, and, most importantly, the clients requiring care. The benefits of simulation-based education will be reviewed relative to the existing literature both within and outside the professions.

Trends in technical and team simulations: Challenging the status Quo of surgical training

Traditional opportunistic and experiential surgical training can expose patients to excess risk of harm, particularly when new technology or surgical approaches are applied to highly complex operations. Declining birth rates, decreasing academic regionalization of complex neonatal procedures, duty hour restrictions, and the increasing need for highly specialized surgical skills all serve to further degrade the educational opportunities for surgical trainees. Even more concerning, practicing pediatric surgeons are also struggling with declines in neonatal surgical volumes, with fewer opportunities to maintain established cognitive, technical, and nontechnical skills in the operating room. In an effort to balance patient safety with a deeply rooted commitment to surgical education, surgical educators have developed a number of innovative educational strategies to achieve these goals.

Do you see what I see? A randomised pilot study to evaluate the effectiveness and efficiency of simulation-based training with videolaryngoscopy for neonatal intubation

Introduction

Direct laryngoscopy (DL) and airway intubation are critical for neonatal resuscitation. A challenge in teaching DL is that the instructor cannot assess the learners’ airway view. Videolaryngoscopy (VL), which allows display of a patient's airway on a monitor, enables the instructor to view the airway during the procedure. This pilot study compared deliberate practice using either VL with instruction (I-VL) or traditional DL. We hypothesised that I-VL would improve the efficiency and effectiveness of neonatal intubation (NI) training.

Methods

Participants (students, paediatric interns and neonatal fellows) were randomised to I-VL or DL. Baseline technical skills were assessed using a skills checklist and global skills assessment. Following educational sessions, deliberate practice was performed on mannequins using the Storz C-MAC. With I-VL, the instructor could guide training using a real-time airway monitor view. With DL, feedback was based solely on technique or direct visual confirmation, but the instructor and learner views were not concurrent. During summative assessment, procedural skills checklists were used to evaluate intubation ability on a neonatal airway trainer. The duration of attempts was recorded, and recorded airway views were blindly reviewed for airway grade. ‘Effectiveness’ reflected achievement of the minimum passing score (MPS). ‘Efficiency’ was the duration of training for learners achieving the MPS.

Results

58 learners were randomised. Baseline demographics were similar. All participants had a significant improvement in knowledge, skills and comfort/confidence following training. There were no significant differences between randomised groups in efficiency or effectiveness, but trends towards improvement in each were noted. Fellows were more likely to achieve ‘competency’ postinstruction compared to non-fellows (p<0.001).

Conclusions

This educational intervention to teach NI increased the learner's knowledge, technical skills and confidence in procedural performance in both groups. I-VL did not improve training effectiveness. The small sample size and participant diversity may have limited findings, and future work is indicated.

Building bridges between doctors and patients: the design and pilot evaluation of a training session in argumentation for chronic pain experts

BACKGROUND

Shared decision-making requires doctors to be competent in exchanging views with patients to identify the appropriate course of action. In this paper we focus on the potential of a course in argumentation as a promising way to empower doctors in presenting their viewpoints and addressing those of patients. Argumentation is the communication process in which the speaker, through the use of reasons, aims to convince the interlocutor of the acceptability of a viewpoint. The value of argumentation skills for doctors has been addressed in the literature. Yet, there is no research on what a course on argumentation might look like. In this paper, we present the content and format of a training session in argumentation for doctors and discuss some insights gained from a pilot study that examined doctors' perceived strengths and limitations vis-à-vis this training.

METHODS

The training session (eight hours) combined different aspects from prominent theories of argumentation and was designed to strengthen doctors' argumentative discussion skills. A convenient, self-selected sample of 17 doctors who were experts in the field of chronic pain participated in the training and evaluated it via a feedback form and semi-structured interviews.

RESULTS

The participants found that the training session gave a structure to types of communication they use to interact with their patients, and taught them techniques that can increase their effectiveness. Moreover, it provided tools to help address some of the challenges of modern doctor-patient interactions, including dealing with patients' unrealistic expectations and medically inaccurate beliefs, and reaching agreement when there are differences of opinion.

CONCLUSIONS

This study enriches the research in the field of medical education. In line with the findings of studies that explore the value of argumentation in different fields, argumentative discussion skills can be applied by doctors to express their views and to account for the views of patients without patronizing the interaction. In this paper, we provide a basis to reflect on the value of argumentation in enhancing patients' right to autonomy and self-determination in interactions with their doctors.

Development of an instrument to measure deliberate practice in professional nurses

PURPOSE

This paper describes the development of the Deliberate Practice in Nursing Questionnaire (DPNQ) and the reliability and validity characteristics of the instrument.

METHODS

A cross-sectional, descriptive study assessed the DPNQ in a sample of critical care registered nurses (RN). It was conducted at one large Midwestern teaching hospital. A medical intensive care unit (ICU), a surgical ICU, and a trauma/burn ICU participated. Instrument construction involved item development based on a literature review, an existing deliberate practice questionnaire and existing parameters of deliberate practice in nursing. Content reliability and validity were established by expert panel review and survey testing. Probit analysis of survey data was used to develop a composite score for the DPNQ.

RESULTS

Expert panel review revealed an inter-rater agreement (80% reliability) of .92-.96 and a content validity index of 0.94. The final DPNQ consists of 24 items with six subcategories and a composite score of 96. Cronbach's alpha coefficient for the DPNQ in this study was .660 (standardized, .703). The instrument was further validated with the Nurse Competence Scale. Deliberate practice was significantly, positively correlated with competence (rs=.366, p=001).

CONCLUSIONS

Findings from the expert panel provided guidance for development and revision of the DPNQ. Survey testing of the instrument revealed a promising measure of deliberate practice with good reliability and validity characteristics. Identification of a relationship between deliberate practice and competence confirms existing evidence in other domains, providing further validation. Understanding deliberate practice provides a unique way to examine nursing expertise.

The use of high-fidelity manikins for advanced life support training--A systematic review and meta-analysis

OBJECTIVES

The objective of this study was to evaluate the effectiveness of high versus low fidelity manikins in the context of advanced life support training for improving knowledge, skill performance at course conclusion, skill performance between course conclusion and one year, skill performance at one year, skill performance in actual resuscitations, and patient outcomes.

METHODS

A systematic search of Pubmed, Embase and Cochrane databases was conducted through January 31, 2014. We included two-group non-randomized and randomized studies in any language comparing high versus low fidelity manikins for advanced life support training. Reviewers worked in duplicate to extract data on learners, study design, and outcomes. The GRADE (Grades of Recommendation, Assessment, Development and Evaluation) approach was used to evaluate the overall quality of evidence for each outcome.

RESULTS

3840 papers were identified from the literature search of which 14 were included (13 randomized controlled trials; 1 non-randomized controlled trial). Meta-analysis of studies reporting skill performance at course conclusion demonstrated a moderate benefit for high fidelity manikins when compared with low fidelity manikins [Standardized Mean Difference 0.59; 95% CI 0.13-1.05]. Studies measuring skill performance at one year, skill performance between course conclusion and one year, and knowledge demonstrated no significant benefit for high fidelity manikins.

CONCLUSION

The use of high fidelity manikins for advanced life support training is associated with moderate benefits for improving skills performance at course conclusion. Future research should define the optimal means of tailoring fidelity to enhance short and long term educational goals and clinical outcomes.

Robotics and surgery: A sustainable relationship?

Robotic surgery is increasingly being employed to overcome the disadvantages associated with use of conventional techniques such as laparoscopy. However, despite significant promise, there are some clear disadvantages and robust evidence base supporting the use of robotic assistance remains lacking. In this paper, the advantages and drivers for robotics will be discussed, its drawbacks and its future role in surgery.

Reliable and valid assessment of point-of-care ultrasonography

OBJECTIVE

To explore the reliability and validity of the Objective Structured Assessment of Ultrasound Skills (OSAUS) scale for point-of-care ultrasonography (POC US) performance.

BACKGROUND

POC US is increasingly used by clinicians and is an essential part of the management of acute surgical conditions. However, the quality of performance is highly operator-dependent. Therefore, reliable and valid assessment of trainees' ultrasonography competence is needed to ensure patient safety.

METHODS

Twenty-four physicians, representing novices, intermediates, and experts in POC US, scanned 4 different surgical patient cases in a controlled set-up. All ultrasound examinations were video-recorded and assessed by 2 blinded radiologists using OSAUS. Reliability was examined using generalizability theory. Construct validity was examined by comparing performance scores between the groups and by correlating physicians' OSAUS scores with diagnostic accuracy.

RESULTS

The generalizability coefficient was high (0.81) and a D-study demonstrated that 1 assessor and 5 cases would result in similar reliability. The construct validity of the OSAUS scale was supported by a significant difference in the mean scores between the novice group (17.0; SD 8.4) and the intermediate group (30.0; SD 10.1), P = 0.007, as well as between the intermediate group and the expert group (72.9; SD 4.4), P = 0.04, and by a high correlation between OSAUS scores and diagnostic accuracy (Spearman ρ correlation coefficient = 0.76; P < 0.001).

CONCLUSIONS

This study demonstrates high reliability as well as evidence of construct validity of the OSAUS scale for assessment of POC US competence. Hence, the OSAUS scale may be suitable for both in-training as well as end-of-training assessment.

Creating an environment for patient safety and teamwork training in the operating theatre: A quasi-experimental study

INTRODUCTION

Positive safety and a teamwork climate in the training environment may be a precursor for successful teamwork training. This pilot project aimed to implement and test whether a new interdisciplinary and team-based approach would result in a positive training climate in the operating theatre.

METHOD

A 3-day educational module for training the complete surgical team of specialist nursing students and residents in safe teamwork skills in an authentic operative theatre, named Co-Op, was implemented in a university hospital. Participants' (n=22) perceptions of the 'safety climate' and the 'teamwork climate', together with their 'readiness for inter-professional learning', were measured to examine if the Co-Op module produced a positive training environment compared with the perceptions of a control group (n=11) attending the conventional curriculum.

RESULTS

The participants' perceptions of 'safety climate' and 'teamwork climate' and their 'readiness for inter-professional learning' scores were significantly higher following the Co-Op module compared with their perceptions following the conventional curriculum, and compared with the control group's perceptions following the conventional curriculum.

CONCLUSION

The Co-Op module improved 'safety climate' and 'teamwork climate' in the operating theatre, which suggests that a deliberate and designed educational intervention can shape a learning environment as a model for the establishment of a safety culture.

A procedural skills OSCE: assessing technical and non-technical skills of internal medicine residents

Internists are required to perform a number of procedures that require mastery of technical and non-technical skills, however, formal assessment of these skills is often lacking. The purpose of this study was to develop, implement, and gather validity evidence for a procedural skills objective structured clinical examination (PS-OSCE) for internal medicine (IM) residents to assess their technical and non-technical skills when performing procedures. Thirty-five first to third-year IM residents participated in a 5-station PS-OSCE, which combined partial task models, standardized patients, and allied health professionals. Formal blueprinting was performed and content experts were used to develop the cases and rating instruments. Examiners underwent a frame-of-reference training session to prepare them for their rater role. Scores were compared by levels of training, experience, and to evaluation data from a non-procedural OSCE (IM-OSCE). Reliability was calculated using Generalizability analyses. Reliabilities for the technical and non-technical scores were 0.68 and 0.76, respectively. Third-year residents scored significantly higher than first-year residents on the technical (73.5 vs. 62.2%) and non-technical (83.2 vs. 75.1%) components of the PS-OSCE (p < 0.05). Residents who had performed the procedures more frequently scored higher on three of the five stations (p < 0.05). There was a moderate disattenuated correlation (r = 0.77) between the IM-OSCE and the technical component of the PS-OSCE scores. The PS-OSCE is a feasible method for assessing multiple competencies related to performing procedures and this study provides validity evidence to support its use as an in-training examination.

A pilot study to assess the construct and face validity of the Northwestern Objective Microanastomosis Assessment Tool

OBJECT

Microsurgical skills remain an integral component of neurosurgical education. There is a need for an objective scale to assess microsurgical skills. The objective of this study was to assess the face and construct validity of a bench training microanastomosis module and an objective assessment scale, i.e., the Northwestern Objective Microanastomosis Assessment Tool (NOMAT).

METHODS

Medical students, neurosurgical residents, and postdoctoral research fellows at Northwestern University were enrolled in the study. Trainees were divided into 3 groups based on microsurgical experience: 1) experienced, 2) exposed, and 3) novices. Each trainee completed two end-to-end microanastomoses using a 1-mm and a 3-mm synthetic vessel. Two cameras were installed to capture procedural footage. One neurosurgeon blindly graded the performance of trainees using both objective and subjective methods to assess construct validity. Two neurosurgeons reviewed the contents of the simulation module to assess face validity.

RESULTS

Twenty-one trainees participated in the study, including 6 experienced, 6 exposed, and 9 novices. The mean NOMAT score for experienced trainees on the 1-mm module was 47.3/70 compared with 26.0/70 and 25.8/70 for exposed and novice trainees, respectively (p = 0.02). Using subjective grading, experienced trainees performed significantly better on the 1-mm module (64.2/100) compared with exposed or novice trainees (23.3/100 and 25.0/100, respectively; p = 0.02). No statistical difference between groups was noted for the 3-mm module with both NOMAT and subjective grading. Experienced trainees took less time to perform both tasks compared with the others.

CONCLUSIONS

Face and construct validities of the microanastomosis module were established. The scale and the microanastomosis module could help assess the microsurgical skills of neurosurgical trainees and serve as a basis for the creation of a microsurgical curriculum.

Using criteria-based interview models for assessing clinical expertise to select physiotherapists at major multisport games

BACKGROUND

Benchmarking is an established means of identifying levels of specialist practice and competence-based interviews are a tool used to facilitate this. The London 2012 Olympic and Paralympic Games (LOCOG) provided an opportunity to introduce a pragmatic approach to assess expert behaviour in large numbers of volunteer physiotherapists.

AIM

To test inter-tester repeatability of an assessment matrix used to score clinical expertise in a sporting context, followed by reporting on the findings for physiotherapy selection.

METHODS

Four volunteers were videoed answering an identical sport-specific clinical scenario. Recordings were initially scored by two experienced assessors (gold standard). Subsequently, nine assessors scored the same videos then compared with the gold standard. 602 physiotherapists were assessed during volunteer interviews for LOCOG. Scores were compared to those based on self-reported clinical experience. Cross-tabulation was used to determine levels of agreement for the initial scoring matrix and the relationship between the two scoring systems was analysed.

RESULTS

Levels of agreement ranged from 22% to 88%. A good correlation (r=0.754 p>0.001) was found between self-reported clinical experience and scenario scores. Marginal data points in the correlational analysis indicated that only 52% of participants attained the same score across assessments.

CONCLUSIONS

A tool to facilitate assessment for large scale selection could be used for Host Nation physiotherapist selection for Major Games. There is a need to train assessors in this environment, as well as provide detailed matrices, specific to each context being assessed, to ensure that observations can be made on domain-specific and general, non-technical aspects.

Assessment of lumbar puncture skill in experts and nonexperts using checklists and quantitative tracking of needle trajectories: implications for competency-based medical education

CONSTRUCT: With the current shift toward competency-based education, rigorous assessment tools are needed for procedurally based tasks.

BACKGROUND

Multiple tools exist to evaluate procedural skills, each with specific weaknesses.

APPROACH

We sought to determine if quantitative needle tracking could be used as a measure of lumbar puncture (LP) performance and added discriminatory value to a dichotomous checklist. Thirty-two medical students were divided into 2 groups. One group was asked to practice an LP once (single practice [SP]) and the other 5 times (multiple practice [MP]). Experts (attending ER physicians, senior ER residents, and a junior anesthesia resident) were used as comparators. Medical students were assessed again at 1 month to assess skill retention. Groups were assessed performing an LP with an electromagnetic tracking device that allows the needle's 3-dimensional movements to be captured and analyzed, and a dichotomous checklist.

RESULTS

Quantitative needle metrics as assessed by electromagnetic tracking showed a decreasing trend in needle movement distance with practice and with experience. The SP group made significantly more checklist mistakes initially as compared to the MP group (1.2 vs. 0.3, p <.05). At 1 month, there was a significant increase in both groups' mistakes (SP 3.4 vs. MP 1.3, p =.01). No correlation existed between individuals' needle motion and checklist mistakes.

CONCLUSIONS

These findings suggest that quantitative needle tracking identifies students who struggle with needle insertion but are successful at completing the dichotomous checklist.

Deliberate practice as a framework for evaluating feedback in residency training

OBJECTIVE

Using the theory of deliberate practice, a key component of Ericsson's theory of expertise development, this study aims to evaluate the quality of written feedback given to learners.

METHODS

The authors created a feedback scoring system based on the key elements of deliberate practice and used it to assess the quality of written feedback provided to residents in 205 mini-CEX encounter forms. Scores were assigned to each feedback entry for identification of the following: Task, performance gap and action plan.

RESULTS

The scoring system allowed for reliable identification of the components that facilitate deliberate practice in written feedback provided to trainees. However, only one of these components was identified in 70% of the feedback entries. A specific task was identified in 56%, whereas specific performance gaps and action plans were identified in only 3.9% and 13.7% of encounters, respectively.

CONCLUSIONS

Scoring written feedback identified that tasks were often specifically described, but performance gaps and action plans were less frequently and specifically mentioned. Educators might improve feedback effectiveness by better articulating to trainees the gap between their performance and an expert standard, as well as by providing them with specific learning plans.

Preliminary evaluation of a novel thoracoscopic infant lobectomy simulator

PURPOSE

Thoracoscopic lobectomy in infants requires advanced minimally invasive skills. Simulation-based education has the potential to improve complex procedural skills without exposing the patient to undue risks. The study purposes were (1) to create a size-appropriate infant lobectomy simulator and (2) to evaluate validity evidence to support or refute its use in surgical education.

MATERIALS AND METHODS

In this Institutional Review Board-exempt study, a size-appropriate rib cage for a 3-month-old infant was created. Fetal bovine tissue completed the simulator. Thirty-three participants performed the simulated thoracoscopic lobectomy. Participants completed a self-report, 26-item instrument consisting of 25 4-point rating scales (from 1=not realistic to 4=highly realistic) and a one 4-point Global Rating Scale. Validity evidence relevant to test content and response processes was evaluated using the many-facet Rasch model, and evidence of internal structure (inter-item consistency) was estimated using Cronbach's alpha.

RESULTS

Experienced surgeons (observed average=3.6) had slightly higher overall rating than novice surgeons (observed average=3.4, P=.001). The highest combined observed averages were for the domain Physical Attributes (3.7), whereas the lowest ratings were for the domains Realism of Experience and Ability to Perform Tasks (3.4). The global rating was 2.9, consistent with "this simulator can be considered for use in infant lobectomy training, but could be improved slightly." Inter-item consistency for items used to evaluate the simulator's quality was high (α=0.90).

CONCLUSIONS

With ratings consistent with high physical attributes and realism, we successfully created an infant lobectomy simulator, and preliminary evidence relevant to test content, response processes, and internal structure was supported. Participants rated the model as realistic, relevant to clinical practice, and valuable as a learning tool. Minor improvements were suggested prior to its full implementation as an educational and testing tool.

Immediate Surgical Skills Feedback in the Operating Room Using "SurF" Cards

BACKGROUND

Ensuring residents develop operative skills requires application of the principles of guided learning, deliberate practice, and directed feedback.

OBJECTIVE

We sought to create and implement a tool to promote procedural "key" step review and immediate feedback on surgical skills, and examined faculty and resident satisfaction with surgical skills feedback.

METHODS

We created surgical skills feedback (SurF) cards for 8 gynecologic procedures. Faculty/fellows and residents completed prestudy surveys querying frequency of preoperative key step review and satisfaction with surgical skill feedback, a SurF card each time 1 of 8 procedures was performed, and poststudy surveys to evaluate for changes.

RESULTS

Prestudy surveys were completed by 31 faculty/fellows and 20 residents, with 55% (17 of 31) of the faculty/fellows and 5% (1 of 20) of the residents reporting key step review before surgery. All reported low satisfaction rates with feedback frequency, quality, and timeliness. After implementation of SurF cards, preoperative key step review occurred in 78% (82 of 105) of the procedures. Twenty-one faculty/fellows (68%) and 16 residents (80%) completed our poststudy survey. Faculty/fellows reported statistically similar key step review (n  =  15 [71%], P  =  .23), while residents reported that key step review had significantly improved (n  =  6 [38%], P  =  .01). Resident satisfaction with feedback frequency (5% to 50%, P  =  .002) and quality (15% to 50%, P  =  .02) increased significantly.

CONCLUSIONS

The SurF cards we developed facilitated procedural key step review, were associated with significantly improved resident satisfaction with surgical feedback, and could prove helpful with outcomes assessments, such as Accreditation Council for Graduate Medical Education-required documentation of Milestone attainment.

Effects of Postgraduate Medical Education "Boot Camps" on Clinical Skills, Knowledge, and Confidence: A Meta-Analysis

BACKGROUND

Throughout their medical education, learners face multiple transition periods associated with increased demands, producing stress and concern about the adequacy of their skills for their new role.

OBJECTIVE

We evaluated the effectiveness of boot camps in improving clinical skills, knowledge, and confidence during transitions into postgraduate or discipline-specific residency programs.

METHODS

Boot camps are in-training courses combining simulation-based practice with other educational methods to enhance learning and preparation for individuals entering new clinical roles. We performed a search of MEDLINE, CINAHL, PsycINFO, EMBASE, and ERIC using boot camp and comparable search terms. Inclusion criteria included studies that reported on medical education boot camps, involved learners entering new clinical roles in North American programs, and reported empirical data on the effectiveness of boot camps to improve clinical skills, knowledge, and/or confidence. A random effects model meta-analysis was performed to combined mean effect size differences (Cohen's d) across studies based on pretest/posttest or comparison group analyses.

RESULTS

The search returned 1096 articles, 15 of which met all inclusion criteria. Combined effect size estimates showed learners who completed boot camp courses had significantly "large" improvements in clinical skills (d  =  1.78; 95% CI 1.33-2.22; P < .001), knowledge (d  =  2.08; 95% CI 1.20-2.96; P < .001), and confidence (d  =  1.89; 95% CI 1.63-2.15; P < .001).

CONCLUSIONS

Boot camps were shown as an effective educational strategy to improve learners' clinical skills, knowledge, and confidence. Focus on pretest/posttest research designs limits the strength of these findings.

Cognitive complexity of the medical record is a risk factor for major adverse events

CONTEXT

Patients in tertiary care hospitals are more complex than in the past, but the implications of this are poorly understood as "patient complexity" has been difficult to quantify.

OBJECTIVE

We developed a tool, the Complexity Ruler, to quantify the amount of data (as bits) in the patient’s medical record. We designated the amount of data in the medical record as the cognitive complexity of the medical record (CCMR). We hypothesized that CCMR is a useful surrogate for true patient complexity and that higher CCMR correlates with risk of major adverse events.

DESIGN

The Complexity Ruler was validated by comparing the measured CCMR with physician rankings of patient complexity on specific inpatient services. It was tested in a case-control model of all patients with major adverse events at a tertiary care pediatric hospital from 2005 to 2006.

MAIN OUTCOME MEASURES

The main outcome measure was an externally reported major adverse event. We measured CCMR for 24 hours before the event, and we estimated lifetime CCMR.

RESULTS

Above empirically derived cutoffs, 24-hour and lifetime CCMR were risk factors for major adverse events (odds ratios, 5.3 and 6.5, respectively). In a multivariate analysis, CCMR alone was essentially as predictive of risk as a model that started with 30-plus clinical factors.

CONCLUSIONS

CCMR correlates with physician assessment of complexity and risk of adverse events. We hypothesize that increased CCMR increases the risk of physician cognitive overload. An automated version of the Complexity Ruler could allow identification of at-risk patients in real time.