Multisource feedback in professionalism for anesthesia residents

Unprofessionalism in anesthesiology: A qualitative study on classifying unprofessional behavior in anesthesiology residency education

STUDY OBJECTIVE

This study aims to identify the domains that constitute behaviors perceived to be unprofessional in anesthesiology residency training programs.

DESIGN

Qualitative study.

SETTING

Anesthesiology residency training programs.

PATIENTS

Not applicable. The participants involved residents, fellows, and faculty members purposefully sampled in four US-based anesthesiology residency programs.

INTERVENTIONS

Participants were asked to submit examples of unprofessional behavior they witnessed in anesthesiology residents, fellows, or faculty members via a Qualtrics link.

MEASUREMENTS

Not applicable. The behavior examples were independently reviewed and categorized into themes using content analysis.

MAIN RESULTS

A total of 116 vignettes were collected, resulting in a final list of 111 vignettes after excluding those that did not describe behavior exhibited by anesthesiology faculty or trainees. Fifty-eight vignettes pertained to unprofessional behaviors observed in faculty members and 53 were observed in trainees (residents and fellows). Nine unprofessionalism themes emerged in the analysis. The most common themes were VERBAL, SUPERVISION, QUALITY, ENGAGEMENT, and TIME. As to the distribution of role group (faculty versus trainee) by theme, unprofessional behaviors falling into the categories of BIAS, GOSSIP, LEWD, and VERBAL were observed more in faculty; whereas themes with unprofessional behavior primarily attributed to trainees included ENGAGEMENT, QUALITY, TIME, and SUPERVISION.

CONCLUSION

By reviewing reported professionalism-related vignettes within residency training programs, we identified classification descriptors for defining unprofessional behavior specific to anesthesiology residency education. Findings from this study enrich the definition of professionalism as a multi-dimensional competency pertaining to anesthesiology graduate medical education. This framework may facilitate preventative intervention and timely remediation plans for unprofessional behavior in residents and faculty.

Patient involvement in assessment of postgraduate medical learners: A scoping review

CONTEXT

Competency-based assessment of learners may benefit from a more holistic, inclusive, approach for determining readiness for unsupervised practice. However, despite movements towards greater patient partnership in health care generally, inclusion of patients in postgraduate medical learners' assessment is largely absent.

METHODS

We conducted a scoping review to map the nature, extent and range of literature examining the inclusion (or exclusion) of patients within the assessment of postgraduate medical learners. Guided by Arskey and O'Malley's framework and informed by Levac et al. and Thomas et al., we searched two databases (MEDLINE® and Embase®) from inception until February 2021 using subheadings related to assessment, patients and postgraduate learners. Data analysis examined characteristics regarding the nature and factor influencing patient involvement in assessment.

RESULTS

We identified 41 papers spanning four decades. Some literature suggests patients are willing to be engaged in assessment, however choose not to engage when, for example, language barriers may exist. When stratified by specialty or clinical setting, the influence of factors such as gender, race, ethnicity or medical condition seems to remain consistent. Patients may participate in assessment as a stand-alone group or part of a multi-source feedback process. Patients generally provided high ratings but commented on the observed professional behaviours and communication skills in comparison with physicians who focused on medical expertise.

CONCLUSION

Factors that influence patient involvement in assessment are multifactorial including patients' willingness themselves, language and reading-comprehension challenges and available resources for training programmes to facilitate the integration of patient assessments. These barriers however are not insurmountable. While understudied, research examining patient involvement in assessment is increasing; however, our review suggests that the extent which the unique insights will be taken up in postgraduate medical education may be dependent on assessment systems readiness and, in particular, physician readiness to partner with patients in this way.

How psychiatrists’ attitudes towards multi-source feedback including patient feedback influenced the educational value: a qualitative study

Background:  Multi-source feedback (MSF) is well-established in psychiatric training. However, evidence on the educational impact is not definitive and there is scanty evidence exploring its value for professional development of psychiatry trainees in the United Kingdom (UK). Evidence suggests the MSF tool currently used is not suitable for specialist trainees. This qualitative research project explored psychiatric doctors’ attitudes towards MSF with patient feedback, to determine how this influenced the feedback’s educational usefulness. Methods: A qualitative study using a phenomenological approach based on a constructivist approach. Purposive sampling identified trainee psychiatrists who completed a more extensive MSF, including patient feedback, than they currently use. They discussed their results in supervised sessions to plan how to use the feedback. Semi-structured interviews were conducted separately with trainees and their supervisors following completion of MSF. The data was analysed thematically. The study was completed in 2020. Results: Seven trainees and five supervisors participated. Four themes were identified. Most had positive opinions about the educational usefulness of MSF, including patient feedback, and made changes to their behaviour following the feedback. Interviewees valued patient feedback and identified it as important in psychiatry. Most valued their patient feedback over their colleague feedback. The complexities of patient feedback in psychiatry and how this may influence the educational usefulness of the feedback were discussed in detail. Conclusions: Findings suggest a need to review the current system of MSF in psychiatry in order to maximise educational benefits. In particular, this research points to the benefits of psychiatric trainees engaging with patient feedback.

How Feedback Is Given Matters: A Cross-Sectional Survey of Patient Satisfaction Feedback Delivery and Physician Well-being

OBJECTIVE

To evaluate how variation in the way patient satisfaction feedback is delivered relates to physician well-being and perceptions of its impact on patient care, job satisfaction, and clinical decision making.

PARTICIPANTS AND METHODS

A cross-sectional electronic survey was sent to faculty physicians from a large academic medical center in March 29, 2019. Physicians reported their exposure to feedback (timing, performance relative to peers, or channel) and related perceptions. The Professional Fulfillment Index captured burnout and professional fulfillment. Associations between feedback characteristics and well-being or perceived impact were tested using analysis of variance or logistic regression adjusted for covariates.

RESULTS

Of 1016 survey respondents, 569 (56.0%) reported receiving patient satisfaction feedback. Among those receiving feedback, 303 (53.2%) did not believe that this feedback improved patient care. Compared with physicians who never received feedback, those who received any type of feedback had higher professional fulfillment scores (mean, 6.6±2.1 vs 6.3±2.0; P=.03) but also reported an unfavorable impact on clinical decision making (odds ratio [OR], 2.9; 95% CI, 1.8 to 4.7; P<.001). Physicians who received feedback that included one-on-one discussions (as opposed to feedback without this channel) held more positive perceptions of the feedback's impact on patient care (OR, 2.0; 95% CI, 1.3 to 3.0; P=.003), whereas perceptions were less positive in physicians whose feedback included comparisons to named colleagues (OR, 0.5; 95% CI, 0.3 to 0.8; P=.003).

CONCLUSION

Providing patient satisfaction feedback to physicians was associated with mixed results, and physician perceptions of the impact of feedback depended on the characteristics of feedback delivery. Our findings suggest that feedback is viewed most constructively by physicians when delivered through one-on-one discussions and without comparison to peers.

A Training Intervention through a 360° Multisource Feedback Model

Physicians and other health sciences professionals need continuous training, not only in technical aspects of their activity but also in nontechnical, transversal competencies with a cost-efficient impact on the proper functioning of healthcare. The objective of this paper is to analyze the behavioral change among health professionals at a large public hospital following a training intervention on a set of core nontechnical competencies: Teamwork, Adaptability-Flexibility, Commitment-Engagement, Results Orientation, and Leadership Skills for Supervisors. The 360° Multisource Feedback (MSF) model was applied using three sources of information: supervisors, co-workers, and the workers themselves (self-assessment). A quasi-experimental pretest–post-test single-group design with two points in time was utilized. The training intervention improved the scores of only one of the trained competencies—the “Results Orientation” competency—although the scores were slightly inflated. Moreover, significant discrepancies were detected between the three sources, with supervisors awarding the highest scores. The magnitude of behavioral change was related to certain sociodemographic and organizational variables. The study was not immune to the ceiling effect, despite control measures aimed at avoiding it. The empirical evidence suggests that the 360° MSF model must be maintained over time to enhance and reinforce an evaluation culture for better patient care.

Blood Pressure Management by Anesthesia Professionals: Evaluating Clinician Skill From Electronic Medical Records

BACKGROUND

Avoiding intraoperative hypotension might serve as a measure of clinician skill. We, therefore, estimated the range of hypotension in patients of nurse anesthetists, and whether observed differences were associated with a composite of serious complications.

METHODS

First, we developed a multivariable model to predict the amount of hypotension, defined as minutes of mean arterial pressure (MAP) <65 mm Hg, for noncardiac surgical cases from baseline characteristics excluding nurse anesthetist. Second, we compared observed and predicted amounts of hypotension for each case and summarized "excess" amounts across providers. Third, we estimated the extent to which hypotension on an individual case level was independently associated with a composite of serious complications. Finally, we assessed the range of actual and excess minutes of MAP <65 mm Hg on a provider level, and the extent to which these pressure exposures were associated with complications.

RESULTS

We considered 110,391 hours of anesthesia by 99 nurse anesthetists. A total of 69% of 25,702 included cases had at least 1 minute of MAP <65 mm Hg, with a median (quartiles) of 4 (0-15) minutes on the case level. We were unable to explain much variance of intraoperative hypotension from baseline patient characteristics. However, cases in the highest 2 quartiles (>10 and >24 min/case more than predicted) were an estimated 27% (95% confidence interval [CI], 1.1-1.4) and 31% (95% CI, 1.2-1.5) more likely to experience complications compared to those with 0 excess minutes (both P < .001). There was little variation of the average excess minutes <65 mm Hg across the nurse anesthetists, with median (quartiles) of 1.6 (1.2-1.9) min/h. There was no association in confounder-adjusted models on the nurse anesthetist level between average excess hypotension and complications, either for continuous exposure (P = .09) or as quintiles (P = .30).

CONCLUSIONS

Hypotension is associated with complications on a case basis. But the average amount of hypotension for nurse anesthetists over hundreds of cases differed only slightly and was insufficient to explain meaningful differences in complications. Avoiding hypotension is a worthy clinical goal, but does not appear to be a useful metric of performance because the range of average amounts per clinician is not meaningfully associated with patient outcomes, at least among nurse anesthetists in 1 tertiary center.

Evaluating the effectiveness of undergraduate clinical education programs

Medical schools should use a variety of measures to evaluate the effectiveness of their clinical curricula. Both outcome measures and process measures should be included, and these can be organized according to the four-level training evaluation model developed by Donald Kirkpatrick. Managing evaluation data requires the institution to employ deliberate strategies to monitor signals in real-time and aggregate data so that informed decisions can be made. Future steps in program evaluation includes increased emphasis on patient outcomes and multi-source feedback, as well as better integration of existing data sources.

Anesthesia assessment in the era of competences: state of the art

Introduction: Anesthesiology requires procedure fulfillment, problem, and real-time crisis resolution, problem, and complications forecast, among others; therefore, the evaluation of its learning should center around how students achieve competence rather than solely focusing on knowledge acquisition. Literature  shows that despite the existence of numerous evaluation strategies, these are still underrated in most cases due to unawareness. Objective: The present article aims to explain the process of competency-based anesthesiology assessment, in addition to suggesting a brief description of the learning domains evaluated, theories of knowledge, instruments, and assessment systems in the area; and finally, to show some of the most relevant results regarding assessment systems in Colombia. Methodology: The results obtained in “Characteristics of the evaluation systems used by anesthesiology residency programs stakeholders in the educational process, a fact that motivated the publishing of this discussion around the topic of competency-based assessment in anesthesiology. Following a bibliography search with the keywords through PubMed, OVID, ERIC, DIALNET, and REDALYC, 110 articles were reviewed and 75 were established as relevant for the research’s theoretical  framework. Results and conclusion: Anesthesiology assessment should be conceived from the competency’s multidimensionality; it must be longitudinal and focused on the learning objectives.

Pilot project to assess and improve neurosurgery resident and staff perception of feedback to residents for self-improvement goal formation

OBJECTIVE

The Accreditation Council for Graduate Medical Education (ACGME) has pushed for more frequent and comprehensive feedback for residents during their training, but there is scant evidence for how neurosurgery residents view the current feedback system as it applies to providing information for self-improvement and goal formation. The authors sought to assess neurosurgery resident and staff perceptions of the current resident feedback system in providing specific, meaningful, achievable, realistic, and timely (SMART) goals. The authors then created a pilot project to improve the most unfavorably viewed aspect of the feedback system.

METHODS

The authors conducted an anonymous survey of neurosurgery residents and staff at an academic medical institution to assess SMART goals for resident feedback and used the results to create a pilot intervention to address the most unfavorably viewed aspect of the feedback system. The authors then conducted a postintervention survey to see if perceptions had improved for the target of the intervention.

RESULTS

Neurosurgery residents and staff completed an anonymous online survey, for which the results indicated that resident feedback was not occurring in a timely manner. The authors created a simple anonymous feedback form. The form was distributed monthly to neurosurgery residents, neurosurgical staff, and nurses, and the results were reported monthly to each resident for 6 months. A postintervention survey was then administered, and the results indicated that the opinions of the neurosurgery residents and staff on the timeliness of resident feedback had changed from a negative to a nonnegative opinion (p = 0.01).

CONCLUSIONS

The required ACGME feedback methods may not be providing adequate feedback for goal formation for self-improvement for neurosurgery residents. Simple interventions, such as anonymous feedback questionnaires, can improve neurosurgery resident and staff perception of feedback to residents for self-improvement and goal formation.

Remediating professionalism lapses in medical students and doctors: A systematic review

CONTEXT

A remediation intervention aims to facilitate the improvement of an individual whose competence in a particular skill has dropped below the level expected. Little is known regarding the effectiveness of remediation, especially in the area of professionalism. This review sought to identify and assess the effectiveness of interventions to remediate professionalism lapses in medical students and doctors.

METHODS

Databases Embase, MEDLINE, Education Resources Information Center and the British Education Index were searched in September 2017 and October 2018. Studies reporting interventions to remediate professionalism lapses in medical students and doctors were included. A standardised data extraction form incorporating a previously described behaviour change technique taxonomy was utilised. A narrative synthesis approach was adopted. Quality was assessed using the Critical Appraisal Skills Programme checklist.

RESULTS

A total of 19 studies on remediation interventions reported in 23 articles were identified. Of these, 13 were case studies, five were cohort studies and one was a qualitative study; 37% targeted doctors, 26% medical students, 16% residents and 21% involved mixed populations. Most interventions were multifaceted and addressed professionalism issues concomitantly with clinical skills, but some focused on specific areas (eg sexual boundaries and disruptive behaviours). Most used three or more behaviour change techniques. The included studies were predominantly of low quality as 13 of the 19 were case studies. It was difficult to assess the effectiveness of the interventions as the majority of studies did not carry out any evaluation.

CONCLUSIONS

The review identifies a paucity of evidence to guide best practice in the remediation of professionalism lapses in medical students and doctors. The literature tentatively suggests that remediating lapses in professionalism, as part of a wider programme of remediation, can facilitate participants to graduate from a programme of study, and pass medical licensing and mock oral board examinations. However, it is not clear from this literature whether these interventions are successful in remediating lapses in professionalism specifically. Further research is required to improve the design and evaluation of interventions to remediate professionalism lapses.

Learners and Luddites in the Twenty-first Century: Bringing Evidence-based Education to Anesthesiology

Anesthesiologists are both teachers and learners and alternate between these roles throughout their careers. However, few anesthesiologists have formal training in the methodologies and theories of education. Many anesthesiology educators often teach as they were taught and may not be taking advantage of current evidence in education to guide and optimize the way they teach and learn. This review describes the most up-to-date evidence in education for teaching knowledge, procedural skills, and professionalism. Methods such as active learning, spaced learning, interleaving, retrieval practice, e-learning, experiential learning, and the use of cognitive aids will be described. We made an effort to illustrate the best available evidence supporting educational practices while recognizing the inherent challenges in medical education research. Similar to implementing evidence in clinical practice in an attempt to improve patient outcomes, implementing an evidence-based approach to anesthesiology education may improve learning outcomes.

Just a Game: the Dangers of Quantifying Medical Student Professionalism

A medical student on her internal medicine clerkship says her numerical medical professionalism grade was "just a game." Building on this anecdote, we suggest there is good reason to believe that numerical summative assessments of medical student professionalism can, paradoxically, undermine medical student professionalism by sapping internal motivation and converting conversations about core professional values into just another hurdle to residency. We suggest better ways of supporting medical student professional development, including a portfolio comprised of written personal reflection and periodic 360° formative assessment in the context of longitudinal faculty coaching.

Professionalism in pediatric anesthesiology: Affirmation of a definition based on results of a nationally administered survey of pediatric anesthesiologists

BACKGROUND

Previously published work established the need for a specialty-specific definition of professionalism in pediatric anesthesiology. That work established a composite definition consisting of 11 domains and their component "defining themes" for professionalism in pediatric anesthesiology. As a next step toward assessing generalizability of our single-center findings, we sought to gain input from a national sample of pediatric anesthesiologists.

AIMS

The aim of this study was to establish the construct validity of our previously published multidimensional definition of professionalism in pediatric anesthesiology using a nationally representative sample of pediatric anesthesiologists.

METHODS

A survey was distributed via snowball sampling to the leaders of every pediatric anesthesiology fellowship program and pediatric anesthesia department or clinical division in the United States. Survey items were designed to validate individual component themes in the working definition. For affirmed items, the respondent was asked to rate the importance of the item. Respondents were also invited to suggest novel themes to be included in the definition.

RESULTS

A total of 216 pediatric anesthesiologists representing a variety of experience levels and practice settings responded to the survey. All 40 themes were strongly supported by the respondents, with the least supported theme receiving 71.6% approval. 92.8% of respondents indicated that the 11 domains previously identified formed a comprehensive list of domains for professionalism in pediatric anesthesiology. Four additional novel themes were suggested by respondents, including wellness/self-care/burnout prevention, political advocacy, justice within a practice organization, and respect for leadership/experienced partners. These are topics for future study. The survey responses also indicated a near-universal agreement that didactic lectures would be ineffective for teaching professionalism.

CONCLUSION

This national survey of pediatric anesthesiologists serves to confirm the construct validity of our prior working definition of professionalism in pediatric anesthesiology, and has uncovered several opportunities for further study. This definition can be used for both curriculum and policy development within the specialty.

Provision of Parent Feedback via the Communication Assessment Tool: Does It Improve Resident Communication Skills?

OBJECTIVE

To determine the impact of a curriculum that included parent feedback, via the Communication Assessment Tool (CAT), on resident communication skills.

METHODS

In a prospective, controlled study, categorical pediatric residents in continuity clinic were divided into control and intervention groups based on clinic day. Parent feedback was obtained for all residents at the beginning and end of the year using the CAT, a validated survey to assess physician communication. Intervention residents participated in learning conferences that reviewed communication best practices and received parental feedback via individual and group CAT scores. Scores were dichotomized as 5 (excellent) versus 1 to 4 (less than excellent) and reported as percentage of items rated excellent. Curriculum impact was assessed by comparing score changes between groups. Residents' scores in both arms were combined to assess changes from the beginning of the year to the end. Statistical testing was performed using generalized linear mixed-effects models.

RESULTS

All residents (N = 68) participated. Intervention (n = 38) and control (n = 30) residents received at least 10 CATs at the beginning and end of the year. The percentage of parents rating all items as excellent increased by similar percentages in intervention and control groups (60.9%-73.8% vs 61.1%-69.8; P = .38). When scores of residents in both arms were combined, improvement was found from the beginning to the end of the year for all CAT items (P < .001).

CONCLUSIONS

A curriculum including parent feedback from CATs did not significantly impact communication skills. However, communication skills improved over the year in intervention and control groups, suggesting that communication training occurs in multiple settings.

A Systematic Review of the Quality and Utility of Observer-Based Instruments for Assessing Medical Professionalism

BACKGROUND

Professionalism, which encompasses behavioral, ethical, and related domains, is a core competency of medical practice. While observer-based instruments to assess medical professionalism are available, information on their psychometric properties and utility is limited.

OBJECTIVE

We systematically reviewed the psychometric properties and utility of existing observer-based instruments for assessing professionalism in medical trainees.

METHODS

After selecting eligible studies, we employed the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) criteria to score study methodological quality. We identified eligible instruments and performed quality assessment of psychometric properties for each selected instrument. We scored the utility of each instrument based on the ability to distinguish performance levels over time, availability of objective scoring criteria, validity evidence in medical students and residents, and instrument length.

RESULTS

Ten instruments from 16 studies met criteria for consideration, with studies having acceptable methodological quality. Psychometric properties were variably assessed. Among 10 instruments, the Education Outcomes Service (EOS) group questionnaire and Professionalism Mini-Evaluation Exercise (P-MEX) possessed the best psychometric properties, with the P-MEX scoring higher on utility than the EOS group questionnaire.

CONCLUSIONS

We identified 2 instruments with best psychometric properties, with 1 also showing acceptable utility for assessing professionalism in trainees. The P-MEX may be an option for program directors to adopt as an observer-based instrument for formative assessment of medical professionalism. Further studies of the 2 instruments to aggregate additional validity evidence is recommended, particularly in the domain of content validity before they are used in specific cultural settings and in summative assessments.

The role of feedback in ameliorating burnout

PURPOSE OF REVIEW

Assessment of the current literature surrounding interventions directed toward the prevention of burnout in the field of medicine and particularly in anesthesiology.

RECENT FINDINGS

Recently, burnout has been noted to lead to medication errors and subsequently increased harm to our patients. On a personal level, burnout can lead to depression and even suicide amongst physicians. Strategies to prevent burnout amongst anesthesiologists that have been studied in the literature include multisource feedback, mentorship and early recognition.

SUMMARY

There remains no clear or definitive intervention to prevent burnout for physicians. However, changing our environment to embrace mentorship, the continual exchange of feedback and the fostering self-care could startlingly improve our work environment.

High-Quality Feedback Regarding Professionalism and Communication Skills in Otolaryngology Resident Education

Objective

The Accreditation Council for Graduate Medical Education (ACGME) requires competency-based education for residents and recommends 5 basic features of high-quality feedback. Our aim was to examine the incorporation of feedback in articles regarding professionalism and interpersonal/communication skills for otolaryngology residency training curriculum.

Data Sources

PubMed, Embase, ERIC, Cochrane Library, Web of Science, Scopus, and ClinicalTrials.gov .

Methods

We used studies identified during a systematic review of all indexed years through October 4, 2016.

Results

Eighteen studies were included in this review. Professionalism was discussed in 16, of which 15 (94%) examined aspects of feedback. Interpersonal/communication skills were the focus of 16 articles, of which 14 16 (88%) discussed aspects of feedback. Our assessment demonstrated that timeliness was addressed in 8 (44%) articles, specificity in 4 (22%), learner reaction and reflection in 4 (22%), action plans in 3 (20%), and balancing reinforcing/corrective feedback in 2 (13%). Two articles did not address feedback, and 6 did not address aspects of high-quality feedback. The ACGME-recommended feedback systems of ADAPT (ask, discuss, ask, plan together) and R2C2 (relationship, reactions, content, and coach) were not reported in any of the studies.

Conclusion

Feedback is an essential component of graduate medical education and is required by the ACGME milestones assessment system. However, the core feedback components recommended by the ACGME are rarely included in the otolaryngology resident education literature.