The relationship between competence and performance: implications for assessing practice performance

The assessment of physiotherapy practice is a robust measure of entry-level physiotherapy standards: Reliability and validity evidence from a large, representative sample

The Assessment of Physiotherapy Practice (APP) is a 20-item assessment instrument used to assess entry-level physiotherapy practice in Australia, New Zealand and other international locations. Initial APP reliability and validity evidence supported a unidimensional or single latent factor as the best representation of entry-level physiotherapy practice performance. However, there remains inconsistency in how the APP is interpreted and operationalised across Australian and New Zealand universities offering entry-level physiotherapy programs. In essence, the presumption that the psychometric integrity of the APP generalises across people, time, and contexts remains largely untested. This multi-site, archival replication study utilised APP assessment data from 8,979 clinical placement assessments, across 19 Australian and New Zealand universities, graduating entry-level physiotherapy students (n=1865) in 2019. Structural representation of APP scores were examined via confirmatory factor analysis and penalised structural equation models. Factor analyses indicated a 2-factor representation, with four items (1-4) for the professional dimension and 16 items (5-20) for the clinical dimension, is the best approximation of entry-level physiotherapy performance. Measurement invariance analyses supported the robustness of this 2-factor representation over time and across diverse practice areas in both penultimate and final years of study. The findings provide strong evidence for the psychometric integrity of the APP, and the 2-factor alternative interpretation and operationalisation is recommended. To meet entry-level standards students should be assessed as competent across both professional and clinical dimensions of physiotherapy practice.

Establishing new education standards in burns - A scoping review

INTRODUCTION

This scoping review aimed to provide an overview of prospective new burn education requirements.

DATA SOURCE

Three independent researchers (AS, ZG, MK) screened the articles in medical databases (PubMed, Scopus, Google Scholar).

STUDY INCLUSION AND EXCLUSION CRITERIA

The inclusion filters were "education + burn" or "simulation + burns" and "Entrustable Professional Activity + burns". Titles, abstracts and full texts in English were filtered to choose original articles and reviews describing novel methods in education in burns.

RESULTS

The search strategy identified 2214 records. 24 studies were enrolled into the study.

CONCLUSIONS

Medical simulation in burns and Entrustable Professional Activities are promising tools in burn education. Skill-specific unified Entrustable Professional Activities should be established to improve burn team performance.

The interpretation-use argument- the essential ingredient for high quality assessment design and validation

There is increasing interest in health professions education (HPE) in applying argument-based validity approaches, such as Kane's, to assessment design. The critical first step in employing Kane's approach is to specify the interpretation-use argument (IUA). However, in the HPE literature, this step is often poorly articulated. This article provides guidance on developing the IUA using a worked example involving a workplace performance assessment tool. In developing the IUA, we have drawn inspiration from approaches used in the discipline of language assessment to situate the inferences, warrants and assumptions in the context of the assessment tool. The worked example makes use of Toulmin's model of informal logic/argumentation as a framework to structure the IUA and presents Toulmin diagrams for each inference such that the reader can connect the argument chain together. We also present several lessons learned so the reader can understand the issues we grappled with in developing the IUA. A well laid out IUA allows the argument to be critiqued by others and provides a framework to guide collection of validity evidence, and therefore is an essential ingredient in the work of assessment design and validation.

Entrustable professional activities, emergency medicine and medical education: a systematic review

INTRODUCTION

Entrustable professional activities (EPAs) are one way to assess competencies, and are designed to bridge the gap between theoretical competencies and real world clinical practice.

AIMS

This was a systematic review which aims to answer the question: "Which EPAs related to Emergency Medicine are described for medical schools?".

METHODS

We included original qualitative, interventional and observational studies (cross-sectional, case-control, and cohort studies) that described EPAs relevant to Emergency Medicine for Medical School. The search strategy was created using a combination of keywords and standardized index terms related to EPAs and Emergency Medicine.

RESULTS

The search strategy identified 991 citations. After screening the titles and abstracts, we identified 85 potentially relevant studies. After the full-text review, a total of 11 reports met the criteria for inclusion.

CONCLUSION

Recognizing a patient requiring urgent or emergent care and initiating evaluation and management is the most common EPA related to Emergency Medicine described at Medical Schools. Use of EPAs is associated with increased student satisfaction and improved competences. However, there is a lack of undergraduate EM specific EPAs being systematically developed and published, and this should be an area to be explored in future studies.

Expertise in surgical neuro-oncology. Results of a survey by the EANS neuro-oncology section

Introduction

Technical advances and the increasing role of interdisciplinary decision-making may warrant formal definitions of expertise in surgical neuro-oncology.

Research question

The EANS Neuro-oncology Section felt that a survey detailing the European neurosurgical perspective on the concept of expertise in surgical neuro-oncology might be helpful.

Material and methods

The EANS Neuro-oncology Section panel developed an online survey asking questions regarding criteria for expertise in neuro-oncological surgery and sent it to all individual EANS members.

Results

Our questionnaire was completed by 251 respondents (consultants: 80.1%) from 42 countries. 67.7% would accept a lifetime caseload of >200 cases and 86.7% an annual caseload of >50 as evidence of neuro-oncological surgical expertise. A majority felt that surgeons who do not treat children (56.2%), do not have experience with spinal fusion (78.1%) or peripheral nerve tumors (71.7%) may still be considered experts. Majorities believed that expertise requires the use of skull-base approaches (85.8%), intraoperative monitoring (83.4%), awake craniotomies (77.3%), and neuro-endoscopy (75.5%) as well as continuing education of at least 1/year (100.0%), a research background (80.0%) and teaching activities (78.7%), and formal interdisciplinary collaborations (e.g., tumor board: 93.0%). Academic vs. non-academic affiliation, career position, years of neurosurgical experience, country of practice, and primary clinical interest had a minor influence on the respondents' opinions.

Discussion and conclusion

Opinions among neurosurgeons regarding the characteristics and features of expertise in neuro-oncology vary surprisingly little. Large majorities favoring certain thresholds and qualitative criteria suggest a consensus definition might be possible.

Junior doctors receiving supervisor and peer support are more work-engaged professionals who express their voice for quality improvement

PURPOSE

In daily practice, junior doctors can contribute to quality improvement by providing innovative suggestions for change, referred to as voice behavior. Junior doctors are more likely to engage in voice behavior when they receive sufficient support from supervisors and peers. Such support has also been associated with less burnout and more work engagement. However, whether less burned-out and more work-engaged junior doctors demonstrate more voice behaviors in the face of sufficient supervisor and peer support is unclear. Therefore, we studied whether and how associations of supervisor and peer support with junior doctors' voice behaviors are mediated by burnout and work engagement.

MATERIALS & METHODS

Participants were 301 junior doctors that completed a web-based survey including validated questionnaires on supervisor and peer support, burnout, work engagement, and voice behavior.

RESULTS

Supervisor and peer support were associated with lower levels of burnout and higher levels of work engagement. Work engagement, but not burnout, mediated the associations of supervisor and peer support with voice behaviors.

CONCLUSIONS

Junior doctors who received more supervisor or peer support were more work-engaged and reported more voice behaviors. Thus, supervisor and peer support should be cultivated to facilitate junior doctors' roles as work-engaged professionals in quality improvement.

Family Medicine Resident Experience Toward Workplace-Based Assessment Form in Improving Clinical Teaching: An Exploratory Qualitative Study

Background

Workplace-Based Assessment (WPBA) has been widely utilized for assessing performance in training sites for both formative and summative purposes. Currently, with the recently updated duration of the family medicine (FM) training program in Saudi Arabia from four years to three years, the possible impact of such a change on assessment would need to be investigated. This objective was to explore the experiences of FM residents regarding the usage of WPBA as an assessment tool for improving clinical teaching at King Abdulaziz Hospital (KAH), Al Ahsa, Saudi Arabia.

Methods

The study involves an exploratory qualitative phenomenological approach targeting family medicine resident in KAH was used. Purposive sampling techniques were used. In this descriptive study, data was collected through the utilization of 1:1 semi-structured interviews guided by directive prompts. All recorded interviews were transcribed verbatim. An inductive analytical approach was applied for thematic analysis of transcripts.

Results

Fifteen participants were individually interviewed until data saturation was reached. The themes that emerged were organized into the categories of underlying principles of WPBA, the impact of the learning environment, associated opportunities and challenges, and making WPBA more effective. Participants expressed that the orientation provided by the program was insufficient, although the core principles were clear to them. They valued the senior peers' support and encouragement for the creation of a positive learning environment. However, time limit, workload, and a lack of optimum ideal implementation reduced the educational value and effectiveness of WPBA among senior residents.

Conclusion

The study examined residents' experiences with WPBA and concluded that low levels of satisfaction were attributed to implementation-related problems. Improvements should be made primarily in two areas: better use of available resources and more systematic prior planning. Revision and assignment of the selection process were suggested, in addition to the implementation of the new curriculum. The research will assist stakeholders in selecting and carrying out evaluation techniques that will enhance residents' abilities.

Should we screen aging physicians for cognitive decline?

OBJECTIVES

To synthesize evidence relevant for informed decisions concerning cognitive testing of older physicians.

METHODS

Relevant literature was systematically searched in Medline, EMBASE, PsycInfo, and ERIC, with key findings abstracted and synthesized.

RESULTS

Cognitive abilities of physicians may decline in an age range where they are still practicing. Physician competence and clinical performance may also decline with age. Cognitive scores are lower in physicians referred for assessment because of competency or performance concerns. Many physicians do not accurately self-assess and continue to practice despite declining quality of care; however, perceived cognitive decline, although not an accurate indicator of ability, may accelerate physicians' decision to retire. Physicians are reluctant to report colleagues' cognitive problems. Several issues should be considered in implementing cognitive screening. Most cognitive assessment tools lack normative data for physicians. Scientific evidence linking cognitive test results with physician performance is limited. There is no known level of cognitive decline at which a doctor is no longer fit to practice. Finally, relevant domains of cognitive ability vary across medical specialties.

CONCLUSION

Physician cognitive decline may impact clinical performance. If cognitive assessment of older physicians is to be implemented, it should consider challenges of cognitive test result interpretation.

Developing entrustable professional activities for general dentistry at the University of North Carolina

INTRODUCTION

As part of curriculum innovation, the University of North Carolina (UNC) Adams School of Dentistry identified core entrustable professional activities (EPAs) that graduates must demonstrate for practice readiness. This paper describes the development of the UNC EPAs and the perceptions of the general dentistry faculty.

METHODS

Upon establishing a blueprint of knowledge, skills, and attitudes of UNC graduates, using a distributed leadership approach, faculty teams developed EPAs focused on the patient care process. The American Dental Education Association Compendium of Clinical Competency Assessments and Commission on Dental Accreditation Standards informed the team's work. Perceptions of the assessment framework were examined using a questionnaire completed by 13 general dentistry faculty considering the importance, accuracy, and agreement of each EPA, associated domains of competence, and encounter management on a 6-point rating scale.

RESULTS

Distributed leadership was a useful strategy in EPA development to disperse decision-making and build ownership. Through multiple iterations, four EPAs (assessment, plan of care, collaborative care, and provision of care) with associated sub-EPAs emerged. EPAs included a description, required knowledge and skills, and rubrics for assessment. The general dentistry faculty reported a high level of importance, accuracy, and agreement with EPAs, domains of competence, and encounter management.

DISCUSSION

EPAs provide a standardized manner to describe the comprehensive work dentists perform, shifting away from individual competencies. The UNC EPAs provide the foundation for longitudinal measures of competence preparing graduates for independent practice. With limited EPAs frameworks available in dentistry, we aim to inform the development and implementation of EPAs across dental education.

Models and materials for teaching auricular framework carving: A systematic review

INTRODUCTION

The process of carving an auricular framework is technically challenging and unique to the patient. As such, there is a need for a robust and reliable training model for practicing and planning ear reconstruction. The aim of this study is to assess the best models and methods available to practice the carving of an auricular framework.

METHODS

A systematic review was undertaken in accordance with Preferred Reporting Items for Systematic Reviews and Meta Analyses guidelines using MEDLINE, Embase, and Cochrane databases. Terms such as "ear", "reconstruction" and "teaching" were searched.

RESULTS

A total of 354 articles were identified, and 13 studies met the inclusion criteria. Vegetables, animal tissue, synthetic materials, as well as more advanced methods such as 3D-printed moulds, were analysed. The similarity of these materials to human costal cartilage was investigated to determine the best or most suitable ones for the purpose. The methods used in the studies were also analysed. Due to heterogeneity of the studies, it was not possible to conduct a quantitative analysis.

CONCLUSION

This review identifies that for the junior surgeons at the skill acquisition phase of their training in auricular framework carving repetition using firstly a cheap synthetic material would be most useful, followed by animal cartilage or 3D printing using silicone. These materials bear the most resemblance to human costal cartilage, and by repeating the carvings, proficiency will improve. Those surgeons with an established ear reconstruction practise, wishing to undertake immediate pre-operatively simulation could benefit from cross-sectional imaging and 3D printing of a patient's non-affected ear to ensure a good match.

Competency-based assessment in neonatal simulation-based training

Simulation is a cornerstone of training in neonatal clinical care, allowing learners to practice skills in a safe and controlled environment. Competency-based assessment provides a systematic approach to evaluating technical and behavioral skills observed in the simulation environment to ensure the learner is prepared to safely perform the skill in a clinical setting. Accurate assessment of competency requires the creation of tools with evidence of validity and reliability. There has been considerable work on the use of competency-based assessment in the field of neonatology. In this chapter, we review neonatal simulation-based training, examine competency-based assessment tools, explore methods to gather evidence of the validity and reliability, and review an evidence-based approach to competency-based assessment using simulation.

Rating of physiotherapy student clinical performance in a paediatric setting: are assessors consistent in their rating of a simulated clinical student performance?

BACKGROUND

During workplace-based clinical placements, best practice assessment states students should expect consistency between assessors rating their performance. To assist clinical educators (CEs) to provide consistent assessment of physiotherapy student performance, nine paediatric vignettes depicting various standards of simulated student performance, as assessed by the Assessment of Physiotherapy Practice (APP), were developed. The APP defines adequate on the global rating scale (GRS) as the minimally acceptable standard for an entry-level physiotherapist. The project aimed to evaluate consistency of paediatric physiotherapy educators assessing simulated student performance using the APP GRS.

METHODS

Three paediatric scenarios representing neurodevelopment across three age ranges, infant, toddler and adolescent, were developed and scripted that depicted a 'not adequate', 'adequate' and 'good-excellent' performance based on the APP GRS. An expert panel (n = 9) conducted face and content validation. Once agreement was reached for all scripts, each video was filmed. A purposive sample of physiotherapists providing paediatric clinical education in Australia were invited to participate in the study. Thirty-five CEs, with minimum 3-years clinical experience and had supervised a student within the past year, were sent three videos at four-week intervals. Videos depicted the same clinical scenario, however performance varied with each video. Participants rated the performance on the four categories: 'not adequate', 'adequate', 'good' and 'excellent' Consistency among raters was assessed using percentage agreement to establish reliability.

RESULTS

The vignettes were assessed a combined total of 59 times. Across scenarios, percentage agreement at the not adequate level was 100%. In contrast, the adequate scenarios for the Infant, Toddler and Adolescent video failed to meet the 75% agreement level. However, when combining adequate or good-excellent, percentage agreement was > 86%. The study demonstrated strong consensus when comparing not adequate to adequate or better performance. Importantly, no performance scripted as not adequate was passed by any assessor.

CONCLUSIONS

Experienced educators demonstrate consistency in identifying not adequate from adequate or good-excellent performance when assessing a simulated student performance using the APP. Recommendation for practice: These validated video vignettes will be a valuable training tool to improve educator consistency when assessing student performance in paediatric physiotherapy.

Evaluation of the Workplace-Based Assessment Anaesthesia-Clinical Evaluation Exercise (A-CEX) and Its Role in the Royal College of Anaesthetists 2021 Curriculum

The workplace-based assessment (WPBA) Anaesthesia-Clinical Evaluation Exercise (A-CEX) is used in anaesthetic training in the Royal College of Anaesthetists 2021 curriculum. WBPAs are part of a multimodal approach to assess competencies, but can be limited by their granularity. They are an essential component of assessment and are used in both a formative and summative capacity. The A-CEX is a form of WBPA which evaluates knowledge, behaviours and skill of anaesthetists in training across a variety of 'real world' situations. An entrustment scale is assigned to the evaluation which has implications for future practice and ongoing supervision requirements. Despite being a key component in the curriculum the A-CEX has drawbacks. Its qualitative nature results in variation in feedback provided amongst assessors, which may have ongoing implications for clinical practice. Furthermore, the completion of an A-CEX can be viewed as a 'tick box' exercise and does not guarantee that learning has taken place.  Currently no direct evidence exists as to the benefit of the A-CEX in anaesthetic training, but extrapolated data from other studies may show validity. However, the assessment remains a key part of the 2021 curriculum, Future areas for consideration include education for those assessing trainees via A-CEX, altering the matrix of assessment to a less granular approach and a longitudinal study as to the utility of A-CEX in anaesthetics training.

Effect of Workplace-Based Assessment Utilization as a Formative Assessment for Learning Among Family Medicine Postgraduates at the Faculty of Medicine, Menoufia University: A Prospective Study

Background Workplace-based assessment (WBA) is a group of assessment approaches that assesses the trainees' performance through their observation and monitoring in real clinical settings and then provides them with constructive and relevant feedback. Many WBA tools are available, including the mini-clinical evaluation exercise (mini-CEX), direct observation of procedural skills (DOPS), case-based discussions, and multisource feedback (peers, seniors, and patients). A WBA can help medical students improve their clinical competencies and ensure that qualified physicians graduate. Methods This prospective study was done in the family medicine department at the Menoufia Faculty of Medicine in Egypt and passed through two phases. Phase I was introducing an orientation lecture for family medicine staff and a convenient sample of 21 family medicine postgraduates about WBA. Phase II was conducting a monthly mini-CEX and DOPS for the postgraduates. Finally, students' satisfaction with the WBA was assessed, and all collected data were analyzed via Statistical Package for Social Science (SPSS) version 23 (IBM Corp., Armonk, NY). Results A total of 105 feedback sheets were obtained. These feedback sheets were subdivided into 63 mini-CEX feedback sheets (21 sheets from each mini-CEX session for three sessions) and 42 DOPS feedback sheets (21 sheets from each DOPS session for two sessions), all of which were collected and analyzed. A significant improvement was detected in the mini-CEX and DOPS feedback scores of the postgraduates throughout the consecutive sessions (9.5 ± 2.7, 24.9 ± 2.5, 27.29 ± 1.5) (P < 0.001) for Mini-CEX and (6.1 ± 1.8 versus 9.0 ± 1.2) (P < 0.001) for DOPS. About 93% of the postgraduates recommended the application of WBA for their peers, and 86% of them requested to perform it again for other different clinical cases and procedures. Conclusion Workplace-based assessment in the form of Mini-CEX and DOPS revealed its ability to improve clinical knowledge and skills among family medicine postgraduates who became motivated to undergo it again in search of improving their clinical performance and reducing their stresses related to final summative and objective structured clinical examinations (OSCEs).

The Utility and Educational Impact of a Virtual Webinar to Deliver an International Undergraduate Cardiovascular Conference

Introduction

Conferences are an important avenue for dissemination of knowledge, research and provide networking opportunities for career development. The COVID-19 pandemic has prompted adoption of virtual platforms for delivery of these conferences. The aim of the study was to determine the utility and educational impact of a student-led virtual webinar to deliver an undergraduate cardiovascular conference compared to a traditional in-person conference.

Methods

We conducted a two-day virtual conference using the Zoom platform in June 2021. The conference consisted of cardiology subspecialty lectures, and workshops were conducted by a junior doctor, senior cardiology trainees and consultants. The conference also outlaid a virtual poster hall and oral presentation session while networking opportunities were encouraged using breakout rooms and poster hall chat function. A 38-item self-administered online questionnaire was designed and disseminated at the end of the conference to all attending delegates. All data analysis and data visualisation strategies were conducted on R statistical programming.

Results

Eight-hundred and forty students from 55 countries attended the event. Four hundred and ninety participants (58.5% response rate, 55.9% female) completed the questionnaire. Factors such as weekend conference (84.9%), student-led or organised (84.1%), environmental/sustainable (82.3%), appropriate level for me (81.5%) and comfort to present (80.8%) were deemed to be at least equal to traditional in-person conference. The conference also increased participants' interest, their core cardiology knowledge and improved their critical analysis and basic echocardiography skills [median 4 (IQR 3-5) for all parameters]. Overall, participants also found it easy to use the virtual platform [median 5 (IQR 5-5)] and easier to ask questions compared to in-person conferences [median 5 (IQR 4-5) vs median 4 (IQR 3-5), p < 0.001].

Conclusion

Our virtual conference provided opportunities to students that the COVID-19 pandemic would have otherwise affected; however, its utility and educational impact will need to be assessed within its individual context of delivery.

Measuring the Impact of a Faculty Development Program on Clinical Educators

INTRODUCTION

An Academy of Clinical Educators (ACE) was established at the University of Newcastle, to support and build capacity among existing and prospective medical educators. ACE established a Certificate of Clinical Teaching and Supervision (CCTS) program, the final assessment of which was a reflective piece on how the course has affected participants' practice as clinical teachers or supervisors and how changes are expected to impact learner achievement. We conducted a qualitative evaluation of these to explore the impact of the CCTS on participants' teaching.

METHODS

Thirty-one participants (of 90 completers to date) consented for their written reflections to undergo qualitative thematic analysis and completed a survey exploring their preparation for, and experience of the program, and application of skills learnt.

RESULTS

Most participants reported applying the skills gained through the CCTS to their teaching practice to a large (n=23; 72%) or very large (n=5; 16%) extent. Four themes emerged from the qualitative data, aligned with the topics of the CCTS: teaching structure; feedback; orientation; and assessment. Participants described application of more structured approaches to orientation, teaching and feedback, positive student responses, and self-reported satisfaction with adopted changes.

DISCUSSION

The CCTS has motivated change in the teaching practice of participants. Although evidence presented here is limited by the self-reported nature, descriptions of actual changes in practice were detailed and specific enough to suggest they could act as a proxy for objectively measured change in behaviour and outcome.

CONCLUSION

A faculty development program delivered to clinicians with a range of teaching and education-related roles, from varied clinical disciplines and professions, can promote improved, structured teaching and feedback.

A Survey on Changes to the Canadian Anatomical Pathology Certification Examination Due to Coronavirus Disease 2019 and Implications for Competency-Based Medical Education

The coronavirus disease 2019 pandemic resulted in a dramatic change in the Royal College of Physicians and Surgeons of Canada assessment process through elimination of the oral and practical components of the 2020 Anatomical Pathology examination. Our study sought to determine stakeholder opinions and experiences on these changes in the context of the 2019 implementation of competency-based medical education. Surveys were designed for residents and practicing pathologists. In total, 57 residents (estimated response rate 29%) and 185 pathologists (estimated response rate 19%) participated across Canada; 67% of pathologists disagreed with the 2020 Royal College examination changes, compared with 30% for residents (P = <.00001). When asked whether the Royal College examination should be eliminated, 95% of pathologists indicated they would be against this, compared to only 34% of residents (P = <.00001). Perceptions on changes to and importance of different components of assessment in competency-based medical education were similar between pathologists and residents, with participants perceiving assessment practices to have changed fairly little since its implementation, with the exception of more frequent feedback. Analysis of narrative comments identified several common themes around assessment, including the need for objectivity and standardization and the problem of failure-to-fail. However, residents identified numerous elements of their performance that can be assessed only through longitudinal evaluation. Pathologists, on the other hand, tended to view these aspects of performance as laden with bias. Our results will hopefully help guide future innovation in assessment by characterizing different stakeholder perspectives on key issues in medical education.

It does not have to be either or! Assessing competence in medicine should be a continuum between an analytic and a holistic approach

Assessing competence is a tremendous challenge in medical education. There are two contrasting approaches in competence assessment: an analytic approach that aims to precisely measure observable constituents and facets of competence and a holistic approach that focuses on a comprehensive assessment of competences in complex real situations reflecting actual performance. We would like to contribute to the existing discourse about medical competence and its assessment by proposing an approach that can provide orientation for the development of competence-based assessment concepts in undergraduate and postgraduate medical education. The approach follows Kane's framework of an "argument-based approach" to validity and is based on insights into task complexity, testing and learning theories as well as the importance of the learning environment. It describes a continuum from analytic to holistic approaches to assess the constituents and facets of competence to performance. We conclude that the complexity of a task should determine the selection of the assessment and suggest to use this approach to reorganize and adapt competence assessment.

Building Veterinarians beyond Veterinary School: Challenges and Opportunities for Continuing Professional Development in Veterinary Medicine

Continuing professional development (CPD) is an important mechanism for veterinarians to acquire, maintain, and enhance their capability to perform competently in their chosen practice area over their career. Although most licensing bodies require veterinarians to complete a minimum number of CPD hours each registration cycle, there are known issues with verifying that these activities are having the desired effects of ensuring professional competence and improving outcomes for patients, owners, and veterinarians. In this review, we summarize the literature across different health care professions to highlight three key challenges for veterinary CPD programs. These are (a) defining what it means to be professionally competent across different career stages from graduation to retirement, (b) delivering CPD activities that are effective in promoting evidence-based medicine and behavioral change in practice, and (c) developing reliable and sustainable systems to formally assess the continued professional competence of veterinarians. A common theme across all challenges was the importance of interacting with colleagues as an opportunity to receive external feedback on their professional strengths and weaknesses and to develop stronger support networks for managing common stressors in clinical practice. There was also a recognized need to develop more transparent outlines of the available and acceptable options for managing different animal health concerns as a first step toward identifying new opportunities for the veterinary profession to elevate the level of care provided. Based on these findings, we propose a new framework for defining, delivering, and evaluating CPD that promotes stronger collaboration between veterinarians to improve professional and personal well-being.

Assessing anaesthesiology and intensive care specialty physicians: An Italian language multisource feedback system

BACKGROUND

Physician professionalism, including anaesthesiologists and intensive care doctors, should be continuously assessed during training and subsequent clinical practice. Multi-source feedback (MSF) is an assessment system in which healthcare professionals are assessed on several constructs (e.g., communication, professionalism, etc.) by multiple people (medical colleagues, coworkers, patients, self) in their sphere of influence. MSF has gained widespread acceptance for both formative and summative assessment of professionalism for reflecting on how to improve clinical practice.

METHODS

Instrument development and psychometric analysis (feasibility, reliability, construct validity via exploratory factor analysis) for MSF questionnaires in a postgraduate specialty training in Anaesthesiology and intensive care in Italy. Sixty-four residents at the Università del Piemonte Orientale (Italy) Anesthesiology Residency Program. Main outcomes assessed were: development and psychometric testing of 4 questionnaires: self, medical colleague, coworker and patient assessment.

RESULTS

Overall 605 medical colleague questionnaires (mean of 9.3 ±1.9) and 543 coworker surveys (mean 8.4 ±1.4) were collected providing high mean ratings for all items (> 4.0 /5.0). The self-assessment item mean score ranged from 3.1 to 4.3. Patient questionnaires (n = 308) were returned from 31 residents (40%; mean 9.9 ± 6.2). Three items had high percentages of "unable to assess" (> 15%) in coworker questionnaires. Factor analyses resulted in a two-factor solution: clinical management with leadership and accountability accounting for at least 75% of the total variance for the medical colleague and coworker's survey with high internal consistency reliability (Cronbach's α > 0.9). Patient's questionnaires had a low return rate, a limited exploratory analysis was performed.

CONCLUSIONS

We provide a feasible and reliable Italian language MSF instrument with evidence of construct validity for the self, coworkers and medical colleague. Patient feedback was difficult to collect in our setting.

Case-Based Discussion in United Kingdom General Practice Training: A Critical Analysis

Case-based discussion (CbD) is a form of workplace-based assessment to assess the progress of learning in general practice trainees in the United Kingdom. We aim to identify the need and rationale behind CbD. The usefulness of CbD in the trainee’s learning will be analyzed with the help of well-recognized parameter such as utility equation. It will also be considered whether the assessment delivers what it is supposed to in actual practice. The current pitfalls will be identified with suggestions for potential improvements.

Entrustment Decision Making: Extending Miller's Pyramid

The iconic Miller's pyramid, proposed in 1989, characterizes 4 levels of assessment in medical education ("knows," "knows how," "shows how," "does"). The frame work has created a worldwide awareness of the need to have different assessment approaches for different expected outcomes of education and training. At the time, Miller stressed the innovative use of simulation techniques, geared at the third level ("shows how"); however, the "does" level, assessment in the workplace, remained a largely uncharted area. In the 30 years since Miller's conference address and seminal paper, much attention has been devoted to procedures and instrument development for workplace-based assessment. With the rise of competency-based medical education (CBME), the need for approaches to determine the competence of learners in the clinical workplace has intensified. The proposal to use entrustable professional activities as a framework of assessment and the related entrustment decision making for clinical responsibilities at designated levels of supervision of learners (e.g., direct, indirect, and no supervision) has become a recent critical innovation of CBME at the "does" level. Analysis of the entrustment concept reveals that trust in a learner to work without assistance or supervision encompasses more than the observation of "doing" in practice (the "does" level). It implies the readiness of educators to accept the inherent risks involved in health care tasks and the judgment that the learner has enough experience to act appropriately when facing unexpected challenges. Earning this qualification requires qualities beyond observed proficiency, which led the authors to propose adding the level "trusted" to the apex of Miller's pyramid.

Workplace-based assessments in postgraduate medical education: A hermeneutic review

OBJECTIVES

Since their introduction, workplace-based assessments (WBAs) have proliferated throughout postgraduate medical education. Previous reviews have identified mixed findings regarding WBAs' effectiveness, but have not considered the importance of user-tool-context interactions. The present review was conducted to address this gap by generating a thematic overview of factors important to the acceptability, effectiveness and utility of WBAs in postgraduate medical education.

METHOD

This review utilised a hermeneutic cycle for analysis of the literature. Four databases were searched to identify articles pertaining to WBAs in postgraduate medical education from the United Kingdom, Canada, Australia, New Zealand, the Netherlands and Scandinavian countries. Over the course of three rounds, 30 published articles were thematically analysed in an iterative fashion to deeply engage with the literature in order to answer three scoping questions concerning acceptability, effectiveness and assessment training. As each round was coded, themes were refined and questions added until saturation was reached.

RESULTS

Stakeholders value WBAs for permitting assessment of trainees' performance in an authentic context. Negative perceptions of WBAs stem from misuse due to low assessment literacy, disagreement with definitions and frameworks, and inadequate summative use of WBAs. Effectiveness is influenced by user (eg, engagement and assessment literacy) and tool attributes (eg, definitions and scales), but most fundamentally by user-tool-context interactions, particularly trainee-assessor relationships. Assessors' assessment literacy must be combined with cultural and administrative factors in organisations and the broader medical discipline.

CONCLUSIONS

The pivotal determinants of WBAs' effectiveness and utility are the user-tool-context interactions. From the identified themes, we present 12 lessons learned regarding users, tools and contexts to maximise WBA utility, including the separation of formative and summative WBA assessors, use of maximally useful scales, and instituting measures to reduce competitive demands.

Assessing competency using simulation: the SimZones approach

Introduction

Competency-based medical education (CBME) is a system of medical training that focuses on a structured approach to developing the clinical abilities of medical education graduates and practicing physicians. CBME requires a robust and multifaceted system of assessment in order to both measure and guide the progress of learners toward pre-established goals. Simulation has been proposed as one method for assessing competency in healthcare workers. However, a longitudinal framework for assessing competency using simulation has not been developed.

Methods

Conjecture mapping methodology was used to map Miller's framework for competency assessment-'knows', 'knows how', 'shows how', and 'does'-to the five SimZones described by Roussin and Weinstock. The SimZones describe a system of organising the development and delivery of simulation-based education and offer a foundation for both guiding and organising assessment in a simulation context.

Results

A conceptualised alignment of the SimZones with Miller's pyramid of assessment was developed, as well as a detailed conjecture map. SimZone 0 (auto-feedback) and SimZone 1 (foundational instruction) mapped to 'knows' and 'knows how'. SimZone 2 (acute care instruction) mapped to 'shows how'. SimZone 3 (team and system development) mapped to 'shows how'. SimZone 4 (real-life debriefing and development) mapped to 'does'.

Conclusion

The SimZones system of competency assessment offers a robust, flexible, and multifaceted system to guide both formative and summative assessment in CBME. The SimZones approach adds to the many methods of competency assessment available to educators. Adding SimZones to the vocabulary of CBME may be helpful for the full deployment of CBME.

Practices, preferences, and opinions of New Zealand veterinarians towards continuing professional development

Aims: To describe the current practices and preferences of New Zealand veterinarians towards continuing professional development (CPD), explore factors acting as perceived barriers to CPD engagement, and identify opportunities for reforming the current CPD regulations. Methods: A cross-sectional survey of all veterinarians registered with the Veterinary Council of New Zealand (VCNZ) was conducted in October 2019. Descriptive statistics were provided for all quantitative study variables and thematic analysis was performed on the free-text survey comments to identify key issues regarding the current CPD regulations in New Zealand. Results: Complete survey responses were provided by 222/3,484 (6.4%) registered veterinarians. Most respondents (153/222; 68.9%) were satisfied with the amount of CPD completed and spent a mean of $2,511 (median $2,000; min $0; max $20,000) on CPD activities each year. The most popular CPD formats were those involving collegial interaction including conference and skills training workshops. The choice of CPD activities was most commonly influenced by interest in the topics and the desire to become more competent in their work while the main barriers were difficulties fitting CPD around work and family commitments. There were 164 (73.9%) respondents who cited at least one occasion in the previous 12 months where participating in CPD caused them to modify their existing practices. Thematic analysis of the free-text comments revealed several major themes related to the administrative burden of recording CPD activities under the current regulatory framework, and the need to develop CPD plans tailored towards the unique circumstances and goals of individual veterinarians. Respondents also expressed concerns over the reliability of using CPD record audits for evaluating professional competency. Factors identified as being the most helpful for improving the current CPD framework were simplifying the CPD points categories and developing a mechanism to record and share CPD activities in real-time. Conclusions: While most respondents felt there was significant value in completing CPD, the current regulatory framework was perceived to be administratively burdensome and inflexible in allowing them to tailor CPD activities to match their unique employment situation, learning style and professional goals. Clinical relevance: Providing resources that could assist veterinarians to design and implement tailored CPD programmes may improve professional and personal outcomes. However, further research is needed to develop more effective mechanisms for identifying veterinarians who are not performing competently without placing excessive administrative burdens on those who are.

Delphi study to determine the key qualities consultant histopathologists look for in their trainees

AIMS

A Delphi study to triangulate and determine the relative importance of the key qualities of trainees identified from qualitative interviews that sought to understand how consultant histopathologists determine diagnostic competences in trainees.

METHODS

Twelve participants were purposively chosen for the Delphi to form an expert panel of relevant stakeholders. Participants were asked to score and rank the items presented to them.

RESULTS

A total of 22 out of 27 of the key qualities of trainees (items) reached 'consensus in' after round 2 suggesting participants were able to agree that the majority of the items identified in the qualitative interviews were important to diagnostic competence. Five items reached 'no consensus'. Participants did not suggest any additional items. Participants particularly valued qualities of reflection and professionalism and trainees who understood the process of reaching a diagnosis and how their pathological report could impact on patient care.

CONCLUSIONS

This study has triangulated findings from our qualitative interviews and show that consultants value a wide variety of qualities when determining diagnostic competence in their trainees. The judgement is complex and is therefore best assessed longitudinally and on a number of cases, so consultants can look for consistency of both approach to diagnosis and of trainee behaviour.

Objective Assessment and Standard Setting for Basic Flexible Ureterorenoscopy Skills Among Urology Trainees Using Simulation-Based Methods

Objective: To objectively assess the performance of graduating urology residents performing flexible ureterorenoscopy (fURS) using a simulation-based model and to set an entrustability standard or benchmark for use across the educational spectrum. Methods: Chief urology residents and attending endourologists performed a standardized fURS task (ureterorenoscopy and repositioning of stones) using a Boston Scientific^© Lithovue ureteroscope on a Cook Medical^© URS model. All performances were video-recorded and blindly scored by both endourology experts and crowd-workers (C-SATS) using the Ureteroscopic Global Rating Scale, plus an overall entrustability score. Validity evidence supporting the scores was collected and categorized. The Borderline Group (BG) method was used to set absolute performance standards for the expert and crowdsourced ratings. Results: A total of 44 participants (40 chief residents, 4 faculties) completed testing. Eighty-three percent of participants had performed >50 fURS cases at the time of the study. Only 47.7% (mean score 12.6/20) and 61.4% (mean score 12.4/20) of participants were deemed "entrustable" by experts and crowd-workers, respectively. The BG method produced entrustability benchmarks of 11.8/20 for experts and 11.4/20 for crowd-worker ratings, resulting in pass rates of 56.9% and 61.4%. Conclusion: Using absolute standard setting methods, benchmark scores were set to identify trainees who could safely carry out fURS in the simulated setting. Only 60% of residents in our cohort were rated as entrustable. These findings support the use of benchmarks to earlier identify trainees requiring remediation.

The impact of mindfulness-based interventions on doctors' well-being and performance: A systematic review

OBJECTIVES

The well-being of doctors is at risk, as evidenced by high burnout rates amongst doctors around the world. Alarmingly, burned-out doctors are more likely to exhibit low levels of professionalism and provide suboptimal patient care. Research suggests that burnout and the well-being of doctors can be improved by mindfulness-based interventions (MBIs). Furthermore, MBIs may improve doctors' performance (eg in empathy). However, there are no published systematic reviews that clarify the effects of MBIs on doctor well-being or performance to inform future research and professional development programmes. We therefore systematically reviewed and narratively synthesised findings on the impacts of MBIs on doctors' well-being and performance.

METHODS

We searched PubMed and PsycINFO from inception to 9 May 2018 and independently reviewed studies investigating the effects of MBIs on doctor well-being or performance. We systematically extracted data and assessed study quality according to the Medical Education Research Study Quality Instrument (MERSQI), and narratively reported study findings.

RESULTS

We retrieved a total of 934 articles, of which 24 studies met our criteria; these included randomised, (un)controlled or qualitative studies of average quality. Effects varied across MBIs with different training contents or formats: MBIs including essential mindfulness training elements, or employing group-based training, mostly showed positive effects on the well-being or performance of doctors across different educational and hospital settings. Doctors perceived both benefits (enhanced self- and other-understanding) and challenges (time limitations and feasibility) associated with MBIs. Findings were subject to the methodological limitations of studies (eg the use of self-selected participants, lack of placebo interventions, use of self-reported outcomes).

CONCLUSIONS

This review indicates that doctors can perceive positive impacts of MBIs on their well-being and performance. However, the evidence was subject to methodological limitations and does not yet support the standardisation of MBIs in professional development programmes. Rather, health care organisations could consider including group-based MBIs as voluntary modules for doctors with specific well-being needs or ambitions regarding professional development.

Medical student selection criteria and junior doctor workplace performance

BACKGROUND

Medical school selection decisions have consequences beyond graduation. With generally low attrition rates, most medical students become junior doctors. Universities are therefore not just selecting students into a medical course; they are choosing the future medical workforce. Understanding the relationship between selection criteria and outcomes beyond the successful completion of a medical degree may inform approaches to student selection.

METHODS

A retrospective data matching study was conducted involving 39 interns employed by a South Australian local health network in 2017 who had originally entered Flinders University's medical school through a graduate pathway. Student selection data were matched with internship workplace performance scores (measured by supervising consultants' reports across five clinical rotations using a standardised assessment). Correlational analyses then examined associations between these two sets of variables.

RESULTS

An overall selection rank (equal thirds of weighted Grade Point Average from a prior degree, a panel interview, and a national selection test) was moderately associated with all performance measures, accounting for up to 25% of variance. Both weighted Grade Point Average and the interview had multiple and mostly moderate correlations with performance. An increasing number of years taken to complete the course was associated with poorer workplace performance across multiple outcome measures (moderate to strong negative associations with 31 to 62% of shared variance), as was age to a lesser extent (7 to 14%). The national selection test contributed a single and small relationship accounting for 5% of variance with one outcome measure.

CONCLUSIONS

Selection into medicine is a critical assessment given that most students become doctors. This study found multiple associations between selection scores and junior doctor workplace performance measures in the internship year, with weighted Grade Point Average from a prior degree and an interview appearing more important than the national selection test. Future collaborative research should map desired workplace performance outcomes to initial student selection and explore the impact of changes to selection which focus on assessment of these domains. The association between slower course progression and poorer workplace performance should also be examined.

Reviewing Competence in Practice: Reform of Continuing Professional Development for Irish Pharmacists

There has been significant reform of the Continuing Professional Development (CPD) requirements for Irish pharmacists over the past five years. In 2015, a new system was established that includes quality assurance of practitioner engagement in CPD and quality assurance of practitioner competence. Pharmacists must now plan and document their learning activities in an electronic portfolio (ePortfolio) and they must participate in an ePortfolio Review process once every five-year period. A random sample is chosen each year to participate in a review of their practice for pharmacists in patient-facing roles. This paper provides an overview of the development and implementation of these quality assurance processes and it considers the outcomes that were observed in the first four years of implementation. By April 2019, almost 3000 pharmacists had participated in the ePortfolio Review process over the preceding three years, of which 96.2% demonstrated appropriate engagement in CPD. In the preceding two years, almost 200 pharmacists had participated in Practice Review, of which 97.5% have demonstrated the required level of competence across four competencies. All of the pharmacists who did not demonstrate the required level of competence in one or more competency area during Practice Review had previously demonstrated appropriate engagement in CPD through the ePortfolio Review process. This raises interesting questions regarding the use of engagement in continuing education (CE) or CPD as a surrogate measure for competence by professions.

A New Competency-Based Instrument to Assess Resident Knowledge and Self-Efficacy in Primary Palliative Care

CONTEXT

There is a need to improve both primary palliative care (PPC) education and its assessment in graduate medical education (GME). We developed an instrument based on published palliative care (PC) competencies to assess resident competency and educational interventions.

OBJECTIVES

To describe the development and psychometric properties of a novel, competency-based instrument to measure resident knowledge and self-efficacy in PPC.

METHODS

We created a 2-part instrument comprised of a knowledge test (KT) and a self-efficacy inventory (SEI) addressing 18 consensus, core PC resident competencies across 5 domains: pain and symptom management; communication; psychosocial, spiritual, and cultural aspects of care; terminal care and bereavement; and PC principles and practice. The instrument was distributed to 341 internal medicine residents during academic years 2015 to 2016 and 2016 to 2017. A standard item analysis was performed on the KT. Internal consistency (Cronbach α) and variable relationships (factor analysis) were measured for the SEI.

RESULTS

One hundred forty-four residents completed the survey (42% response). For 15 KT items, difficulty ranged from 0.17 to 0.98, with 7 items ranging 0.20 to 0.80 (typical optimum difficulty); discrimination ranged from 0.03 to 0.60 with 10 items ≥0.27 (good to very good discrimination). Cronbach α was 0.954 for 35 SEI items. Factor analysis of combined 2015 to 2016 items yielded 4 factors explaining the majority of variance for the entire set of variables.

CONCLUSION

Our instrument demonstrates promising psychometric properties and reliability in probing the constructs of PC and can be further utilized in PC GME research to assess learners and evaluate PPC educational interventions.

Primary care performance of alternatively licenced physicians in Ontario, Canada: a cross-sectional study using administrative data

OBJECTIVES

Medical Regulatory Authorities (MRAs) provide licences to physicians and monitor those physicians once in practice to support their continued competence. In response to physician shortages, many Canadian MRAs developed alternative licensure routes to allow physicians who do not meet traditional licensure criteria to obtain licences to practice. Many physicians have gained licensure through alternative routes, but the performance of these physicians in practice has not been previously examined. This study compared the performance of traditionally and alternatively licenced physicians in Ontario using quality indicators of primary care. The purpose of this study was to examine the practice performance of alternatively licenced physicians and provide evaluative evidence for alternative licensure policies.

DESIGN

A cross-sectional retrospective examination of Ontario health administrative data was conducted using Poisson regression analyses to compare the performance of traditionally and alternatively licenced physicians.

SETTING

Primary care in Ontario, Canada.

PARTICIPANTS

All family physicians who were licenced in Ontario between 2000 and 2012 and who had complete medical billing data in 2014 were included (n=11 419).

OUTCOME MEASURES

Primary care quality indicators were calculated for chronic disease management, preventive paediatric care, cancer screening and hospital readmission rates using Ontario health administrative data.

RESULTS

Alternatively licenced physicians performed similarly to traditionally licenced physicians in many primary care performance measures. Minimal differences were seen across groups in indicators of diabetic care, congestive heart failure care, asthma care and cancer screening rates. Larger differences were found in preventive care for children less than 2 years of age, particularly for alternatively licenced physicians who entered Ontario from another Canadian province.

CONCLUSIONS

Our findings demonstrate that alternatively licenced physicians perform similarly to traditionally licenced physicians across many indicators of primary care. Our study also demonstrates the utility of administrative data for examining physician performance and evaluating medical regulatory policies and programmes.

The evaluation of e-learning resources as an adjunct to otolaryngology teaching: a pilot study

BACKGROUND

The concept of e-Learning has been rapidly accepted as an important component of medical education and is especially adept at teaching clinical skills. However, their impact on learning, particularly in Otolaryngology Head and Neck Surgery (OHNS) medical school curriculum, has yet to be adequately explored. The aim of this pilot study is to develop interactive e-Learning resources and evaluate their impact in enhancing OHNS teaching in medical school.

METHODS

This pilot study is a randomized controlled trial assessing the effectiveness of e-Learning resources in enhancing the current traditional lecture and tutorial-based teaching of OHNS in medical school. Nineteen final-year medical students from the University of Sydney were recruited for this study, who were randomly allocated into intervention group with additional e-Learning resources (Group A) and control group (Group B). Student knowledge was assessed through objective structured clinical examinations (OSCE) with use of standardized forms for objective scoring. Assessors were blinded to student randomization status. A post-study questionnaire was distributed to assess student feedback on the e-Learning resources.

RESULTS

Eight students were allocated to Group A and 11 students to Group B. Group A performed significantly better than Group B in the overall examination scores (78.50 ± 13.88 v. 55.82 ± 8.23; P = < 0.01). With the minimum pass mark of 65%, the majority of students in Group A was able to pass the OSCE assessments, while the majority of students in Group B failed (87.50% v. 9.10%; P = 0.01). The post-test questionnaire on the e-Learning resources showed very favorable feedback from the students' perspective.

CONCLUSION

Results from our pilot study suggests that the use of interactive online e-Learning resources can be a valuable adjunct in supplementing OHNS teaching in medical school, as they are readily accessible and allow flexible on-demand learning. Future studies involving large numbers of medical students are needed to validate these results.

Experiential knowledge of risk and support factors for physician performance in Canada: a qualitative study

OBJECTIVE

To identify, understand and explain potential risk and protective factors that may influence individual and physician group performance, by accessing the experiential knowledge of physician-assessors at three medical regulatory authorities (MRAs) in Canada.

DESIGN

Qualitative analysis of physician-assessors' interview transcripts. Telephone or in-person interviews were audio-recorded on consent, and transcribed verbatim. Interview questions related to four topics: Definition/discussion of what makes a 'high-quality physician;' factors for individual physician performance; factors for group physician performance; and recommendations on how to support high-quality medical practice. A grounded-theory approach was used to analyse the data.

SETTING

Three provinces (Alberta, Manitoba, Ontario) in Canada.

PARTICIPANTS

Twenty-three (11 female, 12 male) physician-assessors from three MRAs in Canada (the College of Physicians & Surgeons of Alberta, the College of Physicians and Surgeons of Manitoba and the College of Physicians and Surgeons of Ontario).

RESULTS

Participants outlined various protective factors for individual physician performance, including: being engaged in continuous quality improvement; having a support network of colleagues; working in a defined scope of practice; maintaining engagement in medicine; receiving regular feedback; and maintaining work-life balance. Individual risk factors included being money-oriented; having a high-volume practice; and practising in isolation. Group protective factors incorporated having regular communication among the group; effective collaboration; a shared philosophy of care; a diversity of physician perspectives; and appropriate practice management procedures. Group risk factors included: a lack of or ineffective communication/collaboration among the group; a group that doesn't empower change; or having one disruptive or 'risky' physician in the group.

CONCLUSIONS

This is the first qualitative inquiry to explore the experiential knowledge of physician-assessors related to physician performance. By understanding the risk and support factors for both individual physicians and groups, MRAs will be better-equipped to tailor physician assessments and limited resources to support competence and enhance physician performance.

Rating of physiotherapy student clinical performance: is it possible to gain assessor consistency?

BACKGROUND

Reliable interpretation of the Assessment of Physiotherapy Practice (APP) tool is necessary for consistent assessment of physiotherapy students in the clinical setting. However, since the APP was implemented, no study has reassessed how consistently a student performance is evaluated against the threshold standards. Therefore, the primary aim of this study was to determine the consistency among physiotherapy educators when assessing a student performance using the APP tool.

METHODS

Physiotherapists (n = 153) from Australia with a minimum 3 years clinical experience and who had supervised a physiotherapy student within the past 12-months were recruited. Three levels of performance (not adequate, adequate, good/excellent) were scripted and filmed across outpatient musculoskeletal, neurorehabilitation, cardiorespiratory and inpatient musculoskeletal. In the initial phase of the study, scripts were written by academic staff and reviewed by an expert panel (n = 8) to ensure face and content validity as well as clinical relevance prior to filming. In the second phase of the study, pilot testing of the vignettes was performed by clinical academics (n = 16) from Australian universities to confirm the validity of each vignette. In the final phase, study participants reviewed one randomly allocated vignette, in their nominated clinical area and rated the student performance including a rationale for their decision. Participants were blinded to the performance level. Percentage agreement between participants was calculated for each vignette with an a priori percentage agreement of 75% considered acceptable.

RESULTS

Consensus among educators across all areas was observed when assessing a performance at either the 'not adequate' (97%) or the 'good/excellent' level (89%). When assessing a student at the 'adequate' level, consensus reduced to 43%. Similarly, consensus amongst the 'not adequate' and 'good/excellent' ranged from 83 to 100% across each clinical area; while agreement was between 33 and 46% for the 'adequate' level. Percent agreement between clinical educators was 89% when differentiating 'not adequate' from 'adequate' or better.

CONCLUSION

Consistency is achievable for 'not adequate' and 'good/excellent' performances, although, variability exists at an adequate level. Consistency remained when differentiating an 'adequate' or better from a 'not adequate' performance.

Effects of Instruction on Parent Competency During Infant Handling in a Neonatal Intensive Care Unit

PURPOSE

The primary purpose of this study was to investigate the effectiveness of 3 different methods for delivering instruction on infant handling to parents in the neonatal intensive care unit (NICU).

METHODS

Ninety-six parents in the NICU received instruction. Parents were taught the same 3 infant-handling techniques after random assignment to the (1) direct, (2) video, or (3) written-pictorial instructional groups. After baseline competency assessment, parents received instruction according to their group. A masked evaluator assessed parent performance, and parents rated instructional effectiveness.

RESULTS

All groups significantly improved handling performance. The direct and video groups performed 2 handling activities significantly better than the written-pictorial group. No significant differences were found between the direct and video groups. All groups perceived the instruction as effective.

CONCLUSIONS

Direct and video instructions are equally effective in teaching parents to perform simple whole motor tasks in the NICU, and parents welcome the instruction.

How Educational Theory Can Inform the Training and Practice of Plastic Surgeons

It is important to optimize our current learning and teaching models, particularly in a climate of decreased clinical exposure. With technical advancements and clinical care now more accountable, traditional methods of skill acquisition need to be revisited. The past decade has seen changes in plastic surgery curricula. There has also been a shift toward competency-based training programs reflecting the growing emphasis on outcomes-based surgical education. This review explores the role of educational theory in promoting effective learning in practical skills teaching. Key models of educational theory are presented and their application to plastic surgery training to an expert level are highlighted. These models include (1) learning within communities of practice (Lave and Wenger’s theory); (2) the role of the zone of proximal development and importance of the availability of expert assistance (Vygotsky’s theory); (3) skill acquisition and retention (Dreyfus’ and Dreyfus’, and Fitts’ and Posner’s theories); (4) development of expertise after repeated practice and regular reinforcement (Ericsson’s theory); and (5) the assessment of competence (Miller’s triangle). Future plastic surgeons need to possess a thorough understanding of the technical and nontechnical skills required to manage patients effectively. Surgical educators are therefore compelled to develop practical training programs that can teach each of these skills in a safe, learner-centric manner. It is hoped that new approaches to surgical skills training are designed in light of our understanding of educational theory to optimize the training of the next generation of plastic surgeons.

Competences management for improving performance in health organizations

Purpose The purpose of this paper is to describe the design and construction of a privilege mapping system (clinical and organizational competences) of the medical staff of the Niguarda Hospital in Milan, Italy. The second aim is to measure and assess the impact of implementing an evaluation process of clinical competences at the same hospital. Design/methodology/approach The paper retraces the development and implementation of the evaluation of the privilege system, highlighting the subjects involved, the phases and outputs. Moreover, a questionnaire was distributed to 50 heads of unit involved in the planning, building and implementation of competences mapping. Five areas were investigated: competences evaluation for professional development; the impact on work organization and professional roles; professional collaboration; its impact according to context (hospital or unit) and time scale (short or long term); and ability to evaluate clinical outcome. Findings Results reveal success factors for the development and implementation of a privilege mapping system. Furthermore, the survey revealed that clinical leaders are aware of the importance of competences evaluation. In particular, they consider it as a management tool useful for professional development, for identifying excellence and planning operational activities. Originality/value Literature and practical evidence recognize the need to assess the clinical and organizational competences in order to assign tasks and responsibilities. However, there are no studies that describe the construction of systems of evaluation of privileges, as it has never been investigated as professionals perceive these tools.