Using Natural Language Processing to Evaluate the Quality of Supervisor Narrative Comments in Competency-Based Medical Education

PURPOSE

Learner development and promotion rely heavily on narrative assessment comments, but narrative assessment quality is rarely evaluated in medical education. Educators have developed tools such as the Quality of Assessment for Learning (QuAL) tool to evaluate the quality of narrative assessment comments; however, scoring the comments generated in medical education assessment programs is time intensive. The authors developed a natural language processing (NLP) model for applying the QuAL score to narrative supervisor comments.

METHOD

Samples of 2,500 Entrustable Professional Activities assessments were randomly extracted and deidentified from the McMaster (1,250 comments) and Saskatchewan (1,250 comments) emergency medicine (EM) residency training programs during the 2019-2020 academic year. Comments were rated using the QuAL score by 25 EM faculty members and 25 EM residents. The results were used to develop and test an NLP model to predict the overall QuAL score and QuAL subscores.

RESULTS

All 50 raters completed the rating exercise. Approximately 50% of the comments had perfect agreement on the QuAL score, with the remaining resolved by the study authors. Creating a meaningful suggestion for improvement was the key differentiator between high- and moderate-quality feedback. The overall QuAL model predicted the exact human-rated score or 1 point above or below it in 87% of instances. Overall model performance was excellent, especially regarding the subtasks on suggestions for improvement and the link between resident performance and improvement suggestions, which achieved 85% and 82% balanced accuracies, respectively.

CONCLUSIONS

This model could save considerable time for programs that want to rate the quality of supervisor comments, with the potential to automatically score a large volume of comments. This model could be used to provide faculty with real-time feedback or as a tool to quantify and track the quality of assessment comments at faculty, rotation, program, or institution levels.

Implementing and optimizing a communication curriculum in medical teaching: Stakeholders’ perspectives

Objective

The relevance of communication in medical education is continuously increasing. At the Medical Faculty of Hamburg, the communication curriculum was further developed and optimized during this project. This article aims to describe the stakeholders’ perceived challenges and supporting factors in the implementation and optimization processes.

Methods

The initial communication curriculum and its development after a one-year optimization process were assessed with a curricular mapping. A SWOT analysis and group discussions were carried out to provide information on the need for optimization and on challenges the different stakeholders faced.

Results

The curricular mapping showed that the communication curriculum is comprehensive, coherent, integrated and longitudinal. In both the implementation and the project-related optimization processes, support from the dean, cooperation among all stakeholders and structural prerequisites were deemed the most critical factors for successfully integrating communication content into the curriculum.

Conclusion

The initiative and support of all stakeholders, including the dean, teachers and students, were crucial for the project’s success.

Practice implications

Although the implementation of a communication curriculum is recommended for all medical faculties, their actual implementation processes may differ. In a “top-down” and “bottom-up” approach, all stakeholders should be continuously involved in the process to ensure successful integration.

Downstream funding success of early career researchers for resubmitted versus new applications: A matched cohort

Background

Early career researchers face a hypercompetitive funding environment. To help identify effective intervention strategies for early career researchers, we examined whether first-time NIH R01 applicants who resubmitted their original, unfunded R01 application were more successful at obtaining any R01 funding within 3 and 5 years than original, unfunded applicants who submitted new NIH applications, and we examined whether underrepresented minority (URM) applicants differentially benefited from resubmission. Our observational study is consistent with an NIH working group’s recommendations to develop interventions to encourage resubmission.

Methods and findings

First-time applicants with US medical school academic faculty appointments who submitted an unfunded R01 application between 2000–2014 yielded 4,789 discussed and 7,019 not discussed applications. We then created comparable groups of first-time R01 applicants (resubmitted original R01 application or submitted new NIH applications) using optimal full matching that included applicant and application characteristics. Primary and subgroup analyses used generalized mixed models with obtaining any NIH R01 funding within 3 and 5 years as the two outcomes. A gamma sensitivity analysis was performed. URM applicants represented 11% and 12% of discussed and not discussed applications, respectively.

First-time R01 applicants resubmitting their original, unfunded R01 application were more successful obtaining R01 funding within 3 and 5 years than applicants submitting new applications—for both discussed and not discussed applications: discussed within 3 years (OR 4.17 [95 CI 3.53, 4.93]) and 5 years (3.33 [2.82–3.92]); and not discussed within 3 years (2.81 [2.52, 3.13]) and 5 years (2.47 [2.22–2.74]). URM applicants additionally benefited within 5 years for not discussed applications.

Conclusions

Encouraging early career researchers applying as faculty at a school of medicine to resubmit R01 applications is a promising potential modifiable factor and intervention strategy. First-time R01 applicants who resubmitted their original, unfunded R01 application had log-odds of obtaining downstream R01 funding within 3 and 5 years 2–4 times higher than applicants who did not resubmit their original application and submitted new NIH applications instead. Findings held for both discussed and not discussed applications.

Using Natural Language Processing to Automatically Assess Feedback Quality: Findings From 3 Surgical Residencies

PURPOSE

Learning is markedly improved with high-quality feedback, yet assuring the quality of feedback is difficult to achieve at scale. Natural language processing (NLP) algorithms may be useful in this context as they can automatically classify large volumes of narrative data. However, it is unknown if NLP models can accurately evaluate surgical trainee feedback. This study evaluated which NLP techniques best classify the quality of surgical trainee formative feedback recorded as part of a workplace assessment.

METHOD

During the 2016-2017 academic year, the SIMPL (Society for Improving Medical Professional Learning) app was used to record operative performance narrative feedback for residents at 3 university-based general surgery residency training programs. Feedback comments were collected for a sample of residents representing all 5 postgraduate year levels and coded for quality. In May 2019, the coded comments were then used to train NLP models to automatically classify the quality of feedback across 4 categories (effective, mediocre, ineffective, or other). Models included support vector machines (SVM), logistic regression, gradient boosted trees, naive Bayes, and random forests. The primary outcome was mean classification accuracy.

RESULTS

The authors manually coded the quality of 600 recorded feedback comments. Those data were used to train NLP models to automatically classify the quality of feedback across 4 categories. The NLP model using an SVM algorithm yielded a maximum mean accuracy of 0.64 (standard deviation, 0.01). When the classification task was modified to distinguish only high-quality vs low-quality feedback, maximum mean accuracy was 0.83, again with SVM.

CONCLUSIONS

To the authors' knowledge, this is the first study to examine the use of NLP for classifying feedback quality. SVM NLP models demonstrated the ability to automatically classify the quality of surgical trainee evaluations. Larger training datasets would likely further increase accuracy.

Reassessing career pathways of surgical leaders: An examination of surgical leaders' early accomplishments

BACKGROUND

The American College of Surgeon (ACS), American Surgical Association (ASA), Association of Women Surgeons (AWS), and Society of Black Academic Surgeons (SBAS) partnered to gain insight into whether inequities found in surgical society presidents may be present earlier.

METHODS

ACS, ASA, AWS, and SBAS presidents' CVs were assessed for demographics and scholastic achievements at the time of first faculty appointment. Regression analyses controlling for age were performed to determine relative differences across societies.

RESULTS

66 of the 68 presidents' CVs were received and assessed (97% response rate). 50% of AWS future presidents were hired as Instructors rather than Assistant professors, compared to 29.4% of SBAS, 25% of ASA and 29.4% of ACS. The future ACS, ASA, and SBAS presidents had more total publications than the AWS presidents, but similar numbers of 1st and Sr. author publications.

CONCLUSION

Gender inequities in academic surgeon hiring practices and perceived scholastic success may be present at first hire.

A Faculty Development Workshop for Planning and Implementing Interactive Virtual Case-Based Teaching

Introduction

The virtual learning environment has become increasingly important due to physical distance requirements put in place during the COVID-19 pandemic. The transition to a virtual format has been challenging for case-based teaching sessions, which involve substantial audience participation. We developed a faculty development workshop aimed at teaching health professions educators how to use various interactive virtual tools within videoconferencing platforms to facilitate virtual case-based sessions.

Methods

Two 90-minute workshops were piloted as a faculty development initiative. The facilitators demonstrated interactive teaching tools that could be used within virtual case-based sessions. Then, participants discussed how to incorporate these tools into case-based teaching sessions of different class sizes in small-group breakout sessions. Participants completed an online survey following each workshop to evaluate the sessions.

Results

A total of 18 and 26 subjects participated in the first and second workshops, respectively. Survey response rates were 100% (n = 18) and 65% (n = 17) for the first and second workshops, respectively. Both groups provided overall high ratings and reported that the workshop was clear, organized, and relevant. Participants were more familiar and comfortable with the use of various interactive tools for online teaching.

Discussion

Distance online teaching will be increasingly required for an undetermined time. Faculty development efforts are crucial to facilitate effective interactive teaching sessions that engage learners and maximize learning. This virtual teaching workshop is a simple and straightforward way to introduce a more interactive format to virtual case-based teaching in the health professions.

Augmenting physician examiner scoring in objective structured clinical examinations: including the standardized patient perspective

In Canada, high stakes objective structured clinical examinations (OSCEs) administered by the Medical Council of Canada have relied exclusively on physician examiners (PEs) for scoring. Prior research has looked at using SPs to replace PEs. This paper reports on two studies that implement and evaluate a standardized patient (SP) scoring tool to augment PE scoring. The unique aspect of this study is that it explores the benefits of combining SP and PE scores. SP focus groups developed rating scales for four dimensions they labelled: Listening, Communication, Empathy/Rapport, and Global Impression. In Study I, 43 SPs from one site of a national PE-scored OSCE rated 60 examinees with the initial SP rating scales. In Study II, 137 SPs used slightly revised rating scales with optional narrative comments to score 275 examinees at two sites. Examinees were blinded to SP scoring and SP ratings did not count. Separate PE and SP scoring was examined using descriptive statistics and correlations. Combinations of SP and PE scoring were assessed using pass-rates, reliability, and decision consistency and accuracy indices. In Study II, SP and PE comments were examined. SPs showed greater variability in their scoring, and rated examinees lower than PEs on common elements, resulting in slightly lower pass rates when combined. There was a moderate tendency for both SPs and PEs to make negative comments for the same examinee but for different reasons. We argue that SPs and PE assess performance from different perspectives, and that combining scores from both augments overall reliability of scores and pass/fail decisions. There is potential to provide examinees with feedback comments from each group.

Driving lesson or driving test? : A metaphor to help faculty separate feedback from assessment

Although there is consensus in the medical education world that feedback is an important and effective tool to support experiential workplace-based learning, learners tend to avoid the feedback associated with direct observation because they perceive it as a high-stakes evaluation with significant consequences for their future. The perceived dominance of the summative assessment paradigm throughout medical education reduces learners' willingness to seek feedback, and encourages supervisors to mix up feedback with provision of 'objective' grades or pass/fail marks. This eye-opener article argues that the provision and reception of effective feedback by clinical supervisors and their learners is dependent on both parties' awareness of the important distinction between feedback used in coaching towards growth and development (assessment for learning) and reaching a high-stakes judgement on the learner's competence and fitness for practice (assessment of learning). Using driving lessons and the driving test as a metaphor for feedback and assessment helps supervisors and learners to understand this crucial difference and to act upon it. It is the supervisor's responsibility to ensure that supervisor and learner achieve a clear mutual understanding of the purpose of each interaction (i.e. feedback or assessment). To allow supervisors to use the driving lesson-driving test metaphor for this purpose in their interactions with learners, it should be included in faculty development initiatives, along with a discussion of the key importance of separating feedback from assessment, to promote a feedback culture of growth and support programmatic assessment of competence.

Professional development for clerkship administrators: a 16-year overview of the clerkship administrator certificate program

BACKGROUND

Increasing accreditation requirements as well as transformations in medical school curricula necessitate administrative staff who are not only invested in the clerkship coordinator role but also view what they do as a career. To date, there has been a lack of professional development opportunities for clerkship administrators.

METHODS

In 2003, the Central Group on Educational Affairs of the Association of American Medical Colleges recognized a need for professional development for clerkship administrators. The Clerkship Administrator Certificate Program emerged from that decision and presented for the first time in 2004 in Omaha, Nebraska. This article provides an overview of the program, how it has been evaluated, and how it continues to evolve.

RESULTS

The program had two guiding principles: to offer professional development opportunities for clerkship administrators through interactive workshops and to ensure the program was feasible both in terms of completion and in cost. Over the past 16 years, the Clerkship Administrator Certificate Program workshops have been delivered to over 300 clerkship administrators. Of those, 206 have completed a project in order to receive their certificate. Projects have related to innovations in medical education (n = 41), grading (n = 26), professional development (n = 26), and patient care (n = 20) to name a few.

DISCUSSION

In order to meet the demands for presenting the workshops, a train-the-trainer model has been employed to expand the number of individuals presenting the workshops. Additional research needs to be done to determine influence of the program on future professional development endeavors.

Violence against trainees: urgent ethical challenges for medical educators and academic leaders in perinatal medicine

Objectives Violence against medical trainees confronts medical educators and academic leaders in perinatal medicine with urgent ethical challenges. Despite their evident importance, these ethical challenges have not received sufficient attention. The purpose of this paper is to provide an ethical framework to respond to these ethical challenges. Methods We used an existing critical appraisal tool to conduct a scholarly review, to identify publications on the ethical challenges of violence against trainees. We conducted web searches to identify reports of violence against trainees in Mexico. Drawing on professional ethics in perinatal medicine, we describe an ethical framework that is unique in the literature on violence against trainees in its appeal to the professional virtue of self-sacrifice and its justified limits. Results Our search identified no previous publications that address the ethical challenges of violence against trainees. We identified reports of violence and their limitations. The ethical framework is based on the professional virtue of self-sacrifice in professional ethics in perinatal medicine. This virtue creates the ethical obligation of trainees to accept reasonable risks of life and health but not unreasonable risks. Society has the ethical obligation to protect trainees from these unreasonable risks. Medical educators should protect personal safety. Academic leaders should develop and implement policies to provide such protection. Institutions of government should provide effective law enforcement and fair trials of those accused of violence against trainees. International societies should promulgate ethics statements that can be applied to violence against trainees. By protecting trainees, medical educators and academic leaders in perinatology will also protect pregnant, fetal, and neonatal patients. Conclusions This paper is the first to provide an ethical framework, based on the professional virtue of self-sacrifice and its justified limits, to guide medical educators and academic leaders in perinatal medicine who confront ethical challenges of violence against their trainees.

Didactic qualification of teaching staff in primary care medicine - a position paper of the Primary Care Committee of the Society for Medical Education

Having teaching staff with didactic qualifications in university teaching leads to a measurable improvement in academic skills among students. Previous recommendations on the type and scope of medical didactic qualification measures primarily apply to teaching staff at university and in-patient settings. The situation of primary care medicine, which often employs external lecturers and whose teaching takes place to a considerable extent in decentralized training facilities (teaching practices) is not adequately addressed. Taking into account a survey on the status quo at higher education institutions for General Practice in Germany, recommendations for minimum standards are made, based on national and international recommendations on the content and scope of medical didactic qualification measures. These recommendations include preliminary work by the Personnel and Organizational Development in Teaching (POiL) Committee of the Society for Medical Education (GMA), the MedicalTeachingNetwork (MDN), the Society of University Teaching Staff in General Medicine (GHA) as well as the experiences of the committee members, who hail from the field of general medicine, internal medicine and pediatrics amongst others.

Curriculum Changes and Trends 2010-2020: A Focused National Review Using the AAMC Curriculum Inventory and the LCME Annual Medical School Questionnaire Part II

Medical school curricula have evolved from 2010 to 2020. Numerous pressures and influences affect medical school curricula, including those from external sources, academic medical institutions, clinical teaching faculty, and undergraduate medical students. Using data from the AAMC Curriculum Inventory and the LCME Annual Medical School Questionnaire Part II, the nature of curriculum change is illuminated. Most medical schools are undertaking curriculum change, both in small cycles of continuous quality improvement and through significant change to curricular structure and content. Four topic areas are explored: cost consciousness, guns and firearms, nutrition, and opioids and addiction medicine. The authors examine how these topic areas are taught and assessed, where in the curriculum they are located, and how much time is dedicated to them in relation to the curriculum as a whole. When examining instructional methods overall, notable findings include (1) the decrease of lecture, although lecture remains the most used instructional method, (2) the increase of collaborative instructional methods, (3) the decrease of laboratory, and (4) the prevalence of clinical instructional methods in academic levels 3 and 4. Regarding assessment methods overall, notable findings include (1) the recent change of the USMLE Step 1 examination to a pass/fail reporting system, (2) a modest increase in narrative assessment, (3) the decline of practical labs, and (4) the predominance of institutionally developed written/computer-based examinations and participation. Among instructional and assessment methods, the most used methods tend to cluster by academic level. It is critical that faculty development evolves alongside curricula. Continued diversity in the use of instructional and assessment methods is necessary to adequately prepare tomorrow's physicians. Future research into the life cycle of a curriculum, as well optional curriculum content, is warranted.

Becoming outstanding educators: What do they say contributed to success?

Aspiring medical educators and their advisors often lack clarity about career paths. To provide guidance to faculty pursuing careers as educators, we sought to explore perceived factors that contributed to the career development of outstanding medical educators. Using a thematic analysis, investigators at two institutions interviewed 39 full or associate professor physician faculty with prominent roles as medical educators in 2016. The social cognitive career theory (SCCT) informed the interview guide. Investigators developed the codebook and performed iterative analysis using qualitative methods. Extensive team discussion generated the final themes. Eight themes emerged related to preparation, early successes, mentors, networks, faculty development, balance, work environment, and multiple identities. Preparation led to early successes, which served as "launch points," while mentors, networks, and faculty development programs served as career accelerators to open more opportunities, and a supportive work environment was an additional enabler of this pathway. Educators who reported balance between work and outside interests described boundary setting as well as selectively choosing new opportunities to establish boundaries in mid-career. Participants described multiple professional identities, and clinician and educator identities tended to merge and reinforce each other as careers progressed. This study revealed common themes describing trajectories of success among medical educators. These themes aligned with the SCCT, and typically replayed and spiraled over the course of the educators' careers. These findings resonate with other studies, lending credence to an approach to career development that can be shared with junior faculty who are exploring careers in medical education.

Developing a Novel 4-C Framework to Enhance Participation in Faculty Development

Phenomenon: Universities offer a variety of voluntary faculty development to ensure quality education, but face inconsistent faculty participation. Therefore, all Dutch universities require all faculty to obtain a teaching qualification certificate. Yet, like other medical centers, University Medical Center Utrecht continued to struggle with faculty nonparticipation. It has been postulated that clinician teachers may face unique challenges with responsibilities for patient care in addition to teaching and research, challenges that cannot be overcome by merely mandating faculty development or a teaching certificate. This project was conducted to gain insight into factors that hinder faculty participation and better understand what is needed to enhance faculty engagement in their professional development as teachers. Approach: UMC Utrecht has had a teaching certificate requirement for over 20 years. In 2015-2016, we conducted a local needs assessment, gathering faculty perspectives about the teaching certification process. To convey seriousness of purpose and promote commitment to change, we formally engaged key stakeholders from the outset, obtained grant funding for the needs assessment, and had an outside consultant lead the project. Faculty who were stalled or never started were questioned via semi-structured interviews. A focus group with those actively in the process of obtaining their certificate discussed perceived challenges in the process and recommended solutions. Faculty who obtained their teaching certificate completed an anonymous evaluation form. All evaluation comments and transcripts were thematically analyzed using open and axial coding. A literature review was performed to contextualize our findings and identify potential solutions. We compared our initial themes to these findings and found key challenge/solution categories, which we subsequently developed into a novel framework. Findings from the study and literature review were organized using this framework and shared with different stakeholders, all of whom engaged in problem-solving. Ideas and potential solutions were incorporated into a final report with recommendations for improving faculty support and provided to the institutional leadership. Findings: Of 23 faculty teachers approached, 8 (34.8%) agreed to be interviewed; 7 of 25 (28.0%) participated in the focus group; and 83 of 156 (53.2%) completed the evaluation. From the transcripts and evaluation comments, three themes emerged related to context and barriers: (a) skill development versus certification; (b) workplace priorities and culture, and (c) visibility and feasibility of the teacher's role. Triangulation of these themes with the literature revealed four challenge/solution categories - Competence, Context, Community, and Career. This 4-C framework facilitated communication of findings, structured the development of an action plan in response to the findings, and assured implementation of new initiatives for faculty support beyond competence development. Insights: Simply adopting requirements for faculty development may be insufficient and even invoke resistance. Improving faculty participation in faculty development and the quality of education requires institutional attention to not just faculty Competence needs, but also the factors of Context, Community, and Career that together comprise the culture experienced by faculty teachers. With institutional buy-in and commitment to change, the 4-C framework can help focus institutional attention on existing gaps in all four domains and guide the development of comprehensive solutions.

Standardized examinees: development of a new tool to evaluate factors influencing OSCE scores and to train examiners

Introduction: The Objective Structured Clinical Examination (OSCE) is an established format for practical clinical assessments at most medical schools and discussion is underway in Germany to make it part of future state medical exams. Examiner behavior that influences assessment results is described. Erroneous assessments of student performance can result, for instance, from systematic leniency, inconsistent grading, halo effects, and even a lack of differentiation between the tasks to be performed over the entire grading scale. The aim of this study was to develop a quality assurance tool that can monitor factors influencing grading in a real OSCE and enable targeted training of examiners. Material, Methods and Students: Twelve students at the Medical Faculty of the University of Heidelberg were each trained to perform a defined task for a particular surgical OSCE station. Definitions were set and operationalized for an excellent and a borderline performance. In a simulated OSCE during the first part of the study, the standardized student performances were assessed and graded by different examiners three times in succession; video recordings were made. Quantitative and qualitative analysis of the videos was also undertaken by the study coordinator. In the second part of the study, the videos were used to investigate the examiners' acceptance of standardized examinees and to analyze potential influences on scoring that stemmed from the examiners' experience. Results: In the first part of the study, the OSCE scores and subsequent video analysis showed that standardization for defined performance levels at different OSCE stations is generally possible. Individual deviations from the prescribed examinee responses were observed and occurred primarily with increased complexity of OSCE station content. In the second part of the study, inexperienced examiners assessed a borderline performance significantly lower than their experienced colleagues (13.50 vs. 15.15, p=0.035). No difference was seen in the evaluation of the excellent examinees. Both groups of examiners graded the item "ocial competence" - despite identical standardization - significantly lower for examinees with borderline performances than for excellent examinees (4.13 vs. 4.80, p<0.001). Conclusion: Standardization of examinees for previously defined performance levels is possible, making a new tool available in future not only for OSCE quality assurance, but also for training examiners. Detailed preparation of the OSCE checklists and intensive training of the examinees are essential. This new tool takes on a special importance if standardized OSCEs are integrated into state medical exams and, as such, become high-stakes assessments.

The Quality of Assessment of Learning (Qual) Score: Validity Evidence for a Scoring System Aimed at Rating Short, Workplace-Based Comments on Trainee Performance

Construct: This study seeks to determine validity evidence for the Quality of Assessment for Learning score (QuAL score), which was created to evaluate short qualitative comments that are related to specific scores entered into a workplace-based assessment, common within the competency-based medical education (CBME) context. Background: In the age of CBME, qualitative comments play an important role in clarifying the quantitative scores rendered by observers at the bedside. Currently there are few practical tools that evaluate mixed data (e.g. associated score-and-comment data), other than the comprehensive Completed Clinical Evaluation Report Rating tool (CCERR) that was originally derived to rate end-of-rotation reports. Approach: A multi-center, randomized cohort-based rating exercise was conducted to evaluate the rating properties of the QuAL score as compared to the CCERR. One group rated comments using the QuAL score, and the other group rated comments using the CCERR. A generalizability study (G-Study) and a decision study (D-study) were conducted to determine the number of meta-raters for a reliable rating (phi-coefficient target of >0.80). Both scores were correlated against rater's gestalt perceptions of utility for both faculty and residents reading the scores. Results: Twenty-five meta-raters from 20 sites participated in this rating exercise. The G-study revealed that the CCERR group (n = 13) rated the comments with a very high reliability (Phi = 0.97). Meanwhile, the QuAL group (n = 12) rated the comments with a similarly high reliability (Phi = 0.97). The QuAL score required only two raters to reach an acceptable target reliability of >0.80, while the CCERR required three. The QuAL score correlated with perceptions of utility (Meta-rater usefulness, Pearson's r = 0.69, p < 0.001; Perceived usefulness for trainee, r = 0.74, p < 0.001). The CCERR performed similarly, correlating with perceived faculty (r = 0.67, <0.001) and resident utility (0.79, <0.001). Conclusions: The QuAL score is reliable rating score that correlates well with perceptions of utility. The QuAL score may be useful for rating shorter comments generated by workplace-based assessments.

Inter-rater reliability in clinical assessments: do examiner pairings influence candidate ratings?

BACKGROUND

The reliability of clinical assessments is known to vary considerably with inter-rater reliability a key contributor. Many of the mechanisms that contribute to inter-rater reliability however remain largely unexplained and unclear. While research in other fields suggests personality of raters can impact ratings, studies looking at personality factors in clinical assessments are few. Many schools use the approach of pairing examiners in clinical assessments and asking them to come to an agreed score. Little is known however, about what occurs when these paired examiners interact to generate a score. Could personality factors have an impact?

METHODS

A fully-crossed design was employed with each participant examiner observing and scoring. A quasi-experimental research design used candidate's observed scores in a mock clinical assessment as the dependent variable. The independent variables were examiner numbers, demographics and personality with data collected by questionnaire. A purposeful sample of doctors who examine in the Final Medical examination at our institution was recruited.

RESULTS

Variability between scores given by examiner pairs (N = 6) was less than the variability with individual examiners (N = 12). 75% of examiners (N = 9) scored below average for neuroticism and 75% also scored high or very high for extroversion. Two-thirds scored high or very high for conscientiousness. The higher an examiner's personality score for extroversion, the lower the amount of change in his/her score when paired up with a co-examiner; reflecting possibly a more dominant role in the process of reaching a consensus score.

CONCLUSIONS

The reliability of clinical assessments using paired examiners is comparable to assessments with single examiners. Personality factors, such as extroversion, may influence the magnitude of change in score an individual examiner agrees to when paired up with another examiner. Further studies on personality factors and examiner behaviour are needed to test associations and determine if personality testing has a role in reducing examiner variability.

A review of U.S. Medical schools' promotion standards for educational excellence

Phenomenon: Given the growing number of medical science educators, an examination of institutions' promotion criteria related to educational excellence and scholarship is timely. This study investigates the extent to which medical schools' promotion criteria align with published standards for documenting and evaluating educational activities. Approach: This document analysis systematically analyzed promotion and tenure (P&T) guidelines from U.S. medical schools. Criteria and promotion expectations (related to context, quantity, quality, and engagement) were explored across five educational domains including: (i) teaching, (ii) curriculum/program development, (iii) mentoring/advising, (iv) educational leadership/administration, and (v) educational measurement and evaluation, in addition to research/scholarship and service. After independent review and data extraction, paired researchers compared findings and reached consensus on all discrepancies prior to final data submission. Descriptive statistics assessed the frequency of referenced promotion criteria. Findings: Promotion-related documents were retrieved from 120 (of 185) allopathic and osteopathic U.S. medical schools. Less than half of schools (43%; 52 of 120) documented a well-defined education-related pathway for advancement in academic rank. Across five education-specific domains, only 24% (12 of 50) of the investigated criteria were referenced by at least half of the schools. The least represented domain within P&T documents was "Educational Measurement and Evaluation." P&T documents for 47% of schools were rated as "below average" or "very vague" in their clarity/specificity. Insights: Less than 10% of U.S. medical schools have thoroughly embraced published recommendations for documenting and evaluating educational excellence. This raises concern for medical educators who may be evaluated for promotion based on vague or incomplete promotion criteria. With greater awareness of how educational excellence is currently documented and how promotion criteria can be improved, education-focused faculty can better recognize gaps in their own documentation practices, and more schools may be encouraged to embrace change and align with published recommendations.

Evaluation of a certifying training experience in research methodology and scientific writing at the Faculty of Medicine of Bejaia (Algeria)

INTRODUCTION

A Certificate of Specialization (C2S) in research methodology and scientific communication was established at the Faculty of Medicine of Bejaia (Algeria), for the benefit of university hospital teachers, in 2018. The objective of this study was to evaluate the impact of a clinical certifying-research training program on the acquisition of fundamental knowledge for the conduct of health research projects in its three conceptual, operational and editorial phases.

METHODS

This training took place during three face-to-face seminars (a total of 12 teaching days), in the form of lectures and workshops by eight lecturers, with a final exam and a thesis dissertation project. The data were collected through Pre- and post-tests which were distributed before and after each seminar while the questionnaire was administered by the end of the training in order to assess the whole course of this training. The knowledge assessment grids were composed of 20 items for each of the first two seminars and 12 items for the third seminar. According to the categories of the Likert scale, these items were weighted from 1 to 5 points, an overall score for the 52 items of 260 points.

RESULTS

A total of 38 candidates (selected from 140 applications) attended this training with an overall presenteeism rate of 93%. The differential scores ("pre-test" and "post-test") of progression of knowledge were successively 60%, 49% and 42% in the three seminars. Out of a total of 260 points, the overall learning score of all three seminars increased from an average of 119 points ± 8.66 to 180 points ± 15.87 (p <10-7), with a differential score of 51.6%.

CONCLUSION

The evaluation of the C2S clinical research program of the Bejaia Faculty of Medicine documented the significant evolution of knowledge of research methodology and scientific writing tools. The continuity of this training and its generalization to the Maghreb faculties of health sciences are highly recommended, for the improvement of scientific production in Algeria and the Great Maghreb.

Gender sensitivity in career mentoring - a project report from the Medical Faculty of Leipzig University

Objective: The elective subject "career management for medical students" is presented as an example of teaching gender sensitivity issues among medical studies at Leipzig University. The project report shows the interim results of promoting gender-sensitive teaching at the Medical Faculty of Leipzig University, as well as the elective's contribution to the development of gender sensitivity at the entire university. Method: Project Description and Results show the organization/procedure, participants and detailed contents of the elective since it began in Winter Term 2010/11. The research examines the elective's mandate at the Medical Faculty and beyond, i.e. by comparing with the efforts of other universities. Results: The elective is the first subject for credit within the clinical curriculum of medical studies at Leipzig University that connects the topics of gender sensitivity and career management. It creates a view of the specialties of medicine and research as they relate to gender, and also on the options of a medical career and touches the topic of gender equality. A faculty survey in the winter semester of 2011/12 reveals that nearly one third of the medical students want an extension of the curriculum around the topic of gender or even an independent subject "Gender Medicine". The elective is part of a cycle promoting gender equality at Leipzig University. Conclusion: The elective initiates and continues the implementation of gender-sensitive teaching at the Medical Faculty of Leipzig University. The management of the elective aims at the permanent establishment of the subject in the curriculum in order to encourage career ambitions early - especially for women.

Energizing the Conversation: How to Identify and Overcome Gender Inequalities in Academic Medicine

Gender inequality exists in advanced faculty and leadership positions at academic medical centers; however, despite growing awareness, how to best approach and rectify the issue is unknown. To energize the conversation on gender inequality at one academic medical center, chairs and women faculty were surveyed to identify barriers faced by women navigating their careers. A symposium with short talks to increase awareness, a panel with University leaders to discuss issues and successful strategies to overcome gaps, and focus groups to delve further into key areas that underlie inequity through an active café style format were planned and implemented. This multifaceted approach resulted in a wealth of knowledge. The symposium and panel highlighted important relevant issues and offered personal strategies for successful career advancement, while the focus group discussions further identified barriers and inspired ongoing efforts across departments and novel approaches to overcome three key issues (work-life integration, deliberate promotion of mentor/sponsor relationships, and overcoming unconscious bias) identified through the initial surveys. Compiled data were then disseminated to participants and University leaders to enhance awareness of available programs and prompt action in critical areas lacking support. Overall, the approach indicated that securing support from leaders and the academic community alike are pertinent to emphasize actions needed to overcome issues affecting women in academic medicine. Moreover, bringing leaders and faculty together for an informational session and brainstorming appears to energize the conversation. Such efforts can ultimately instill change and establish an inclusive environment where all members of the academic medicine community can thrive.

Academic Advocacy: Opportunities to Influence Health and Science Policy Under U.S. Lobbying Law

Medical school faculty and their colleagues in schools of nursing, public health, social work, and elsewhere often research issues of critical importance to health and science policy. When academics engage with government policymakers to advocate for change based on their research, however, they may find themselves engaged in "lobbying," thereby entering a complex environment of legal requirements and institutional policies that they may not fully understand. To promote academic advocacy, this article explains what is and is not legally permitted when it comes to engaging with policymakers and encourages academic institutions to facilitate permissible advocacy activities.U.S. law permits academic researchers to conduct certain types of policy-focused advocacy without running afoul of legal restrictions on lobbying. Academics acting in their personal capacities and with their own resources may freely engage with policymakers in any branch of government to provide their expertise and advocate for desired outcomes. When acting in their professional capacities, academics are free to engage in most advocacy activities directed to the executive and judicial branches, and they also may advocate to influence legislation and legislators within certain limits that are particularly relevant to academic work. In all cases, academics must take care to not use restricted funds for lobbying.Academic researchers have an important role to play in advancing evidence-based health and science policy. They should familiarize themselves with legal restrictions and opportunities to influence policy based on their research, and their institutions should actively support them in doing so.

A 360-degree assessment of teaching effectiveness using a structured-videorecorded observed teaching exercise for faculty development

BACKGROUND

Filming teaching sessions were reported in the medical literature in the 1980s and 1990s but appear to have been an underreported and/or underutilized teaching tool since that time. National faculty development programs, such as the Harvard Macy Institute (HMI) Program for Educators in Health Professions and the Stanford Faculty Development Center for Medical Teachers program, have attempted to bridge this gap in formal instruction in teaching skills through microteaching sessions involving videos for self- and peer-assessment and feedback.

OBJECTIVE

Current video-feedback faculty development initiatives are time intensive and impractical to implement broadly at an institutional level. Further, results of peer feedback have not been frequently reported in the literature at the institutional level. Our research aims to propose a convenient and effective process for incorporating video analysis into faculty devleopment programs.

DESIGN

Our work describes a novel technique using video-recorded, simulated teaching exercises to compile multi-dimensional feedback as an aid in faculty development programs that promote teaching-skill development. This research evaluated the effectiveness of a focused teaching practicum designed for faculty in multiple specialty departments with large numbers of older patients into a geriatrics-based faculty development program. Effectiveness of the practicum is evaluated using quantitative scoring and qualitative analysis of self-reflection as well as peer and trainee input.

RESULTS

VOTE sessions demonstrate an important exportable product which enable faculty to receive a detailed 360-degree assessment of their teaching.

CONCLUSION

This intervention can be easily replicated and revised, as needed, to fit into the educational curriculum at other academic medical centers.

Quality improvement in medical schools: vision meets culture

CONTEXT

Medical schools face growing pressures to develop quality improvement (QI) strategies to ensure the continuous quality of their education. To best achieve quality, both organisational processes and culture need to be oriented towards quality. Quality processes already exist at medical schools, at least externally driven by accreditation. However, the dominant culture in most medical schools is not typically oriented towards quality.

OBJECTIVES

This study explores whether QI practices are recognised as such in organisations not culturally QI-oriented. Specifically, it examines faculty members' perceptions about the degree of QI implementation in their medical education programmes. Understanding this perception will inform medical school leadership on how best to use resources for a successful execution of the school's QI vision.

METHODS

Leaders, clinical teachers and formal teachers at 16 of the 17 Canadian medical schools were invited to complete the 'Are We Making Progress?' questionnaire of the Malcolm Baldrige National Quality Award framework, the results of which have been broadly validated. The questionnaire measures the perceived level of QI implementation within organisations using 40 statements grouped under the framework's seven domains of performance excellence.

RESULTS

A total of 491 respondents from 11 (69%) schools completed the questionnaire; 173 (35%) identified as clinical teachers, 150 (31%) as formal teachers, and 168 (34%) as leaders. Perceived QI implementation levels were low across programmes (0.70-1.90 in Rasch person measures) and for each category of respondents. This was especially true for the domains of 'Strategy', 'Measurement/analysis/knowledge management' and 'Operations'. Leaders' perceptions of QI implementation were higher than those of teachers.

CONCLUSIONS

Medical schools' existing QI processes are not recognised as QI activities. For QI strategies to succeed, a programme's culture must embrace QI. In the execution of their QI visions, medical schools should spend resources on embedding quality in the organisation culture in addition to strengthening existing QI practices, especially in the domains listed above.

Jump-Starting Faculty Development in Quality Improvement and Patient Safety Education: A Team-Based Approach

PROBLEM

Quality improvement (QI) and patient safety (PS) are cornerstones of health care delivery. Accreditation organizations increasingly require that learners engage in QIPS. For many faculty, these are new domains. Additional faculty development is needed for them to teach and mentor trainees. Existing programs, such as the Association of American Medical Colleges Teaching for Quality (Te4Q) program, target individual faculty and thus accommodate only limited participants at a time, which is problematic for institutions that need to train many faculty to support their learners.

APPROACH

The authors invited diverse stakeholders from across the University of California, San Francisco (UCSF) School of Medicine and related health systems to participate in a team-based adaptation of the Te4Q program. The teams completed 5 projects based on previously identified priority areas to increase local capacity for QIPS teaching: (1) online modules for faculty new to QIPS, (2) a tool kit for graduate medical education programs, (3) a module for medical school clerkship directors, (4) guidelines for faculty to integrate early learners into QI projects, and (5) a "Teach-for-UCSF" certificate program in teaching QIPS.

OUTCOMES

Thirty-five faculty members participated in the initial Te4Q workshop in January 2015, and by fall 2016, all projects were implemented. These projects led to additional faculty development initiatives and a rapidly expanding number of faculty across campus with expertise in teaching QIPS.

NEXT STEPS

Further collaborations between faculty focused on QIPS in care delivery and those focused on QIPS education to promote QIPS teaching have resulted from these initial projects.

Alliance, Trust, and Loss: Experiences of Patients Cared for by Students in a Longitudinal Integrated Clerkship

PURPOSE

The longitudinal integrated clerkship (LIC) model, which allows medical students to participate in comprehensive care of a panel of patients over time, is rapidly expanding because of recognized benefits to students and faculty. This study aimed to determine how LIC student contact affected patients' experiences and self-described health outcomes.

METHOD

This qualitative case study used semistructured patient interviews to understand the impact of LIC learners at the University of Colorado School of Medicine on patients at Denver Health. Patients with at least 3 encounters with an LIC student and over age 18 were selected. Thirty patients were invited to participate in 2016-2017; 14 (47%) completed interviews before the thematic analysis reached saturation. Four researchers independently analyzed interview transcripts and reached consensus on emergent categories and themes.

RESULTS

Six broad themes were identified: beginnings of a relationship, caring demonstrated by student, growing to trust student, reaching a therapeutic alliance, improvement of patient outcomes due to student involvement, and a sense of loss after students completed the LIC program.

CONCLUSIONS

Patients deeply valued the therapeutic alliances built with LIC students involved in their care over time. These alliances led to improved patient experience, mitigation of perceived health system failures, and subjective improvement in health outcomes. Patients described a sense of loss at the end of the LIC when students were no longer involved in their care. Curricula that support students building longitudinal therapeutic relationships with their patients are an opportunity to improve patient experience while promoting students' professional development.

Effectiveness of a multicomponent safe surgery intervention on improving surgical quality in Tanzania's Lake Zone: protocol for a quasi-experimental study

INTRODUCTION

Effective, scalable strategies for improving surgical quality are urgently needed in low-income and middle-income countries; however, there is a dearth of evidence about what strategies are most effective. This study aims to evaluate the effectiveness of Safe Surgery 2020, a multicomponent intervention focused on strengthening five areas: leadership and teamwork, safe surgical and anaesthesia practices, sterilisation, data quality and infrastructure to improve surgical quality in Tanzania. We hypothesise that Safe Surgery 2020 will (1) increase adherence to surgical quality processes around safety, teamwork and communication and data quality in the short term and (2) reduce complications from surgical site infections, postoperative sepsis and maternal sepsis in the medium term.

METHODS AND ANALYSIS

Our design is a prospective, longitudinal, quasi-experimental study with 10 intervention and 10 control facilities in Tanzania's Lake Zone. Participants will be surgical providers, surgical patients and postnatal inpatients at study facilities. Trained Tanzanian medical data collectors will collect data over a 3-month preintervention and postintervention period. Adherence to safety as well as teamwork and communication processes will be measured through direct observation in the operating room. Surgical site infections, postoperative sepsis and maternal sepsis will be identified prospectively through daily surveillance and completeness of their patient files, retrospectively, through the chart review. We will use difference-in-differences to analyse the impact of the Safe Surgery 2020 intervention on surgical quality processes and complications. We will use interviews with leadership and surgical team members in intervention facilities to illuminate the factors that facilitate higher performance.

ETHICS AND DISSEMINATION

The study has received ethical approval from Harvard Medical School and Tanzania's National Institute for Medical Research. We will report results in peer-reviewed publications and conference presentations. If effective, the Safe Surgery 2020 intervention could be a promising approach to improve surgical quality in Tanzania's Lake Zone region and other similar contexts.

Assessing Preceptor Use of Cognitive Apprenticeship: Is the Maastricht Clinical Teaching Questionnaire (MCTQ) a Useful Approach?

Construct: The authors investigated the utility of the Maastricht Clinical Teaching Questionnaire (MCTQ) to assess preceptor use of cognitive apprenticeship teaching methods. Background: The MCTQ is an instrument first studied in medical and veterinary education; it was shown to produce valid and reliable data when evaluating how well preceptors engage in cognitive apprenticeship. Additional validity evidence, however, is needed to determine if the MCTQ is useful and applicable in other healthcare professions, such as pharmacy. Approach: Student pharmacists and pharmacy residents at a large academic medical center completed the MCTQ to evaluate pharmacist preceptors. A confirmatory factor analysis (CFA) was used to determine how consistent the data were with the 5-factor structure of the cognitive apprenticeship teaching methods (e.g., modeling, coaching, articulation, exploration, and safe learning environment). A structural equation model (SEM) was used to evaluate the relationship between the 5 factors. Preceptor and student perceptions were also collected to evaluate the response process and potential consequences of using the instrument. Results: Thirty-eight learners (2nd-year and 4th-year student pharmacists and pharmacy residents) submitted 157 evaluations. The CFA showed appreciable fit with the 5-factor structure of the cognitive apprenticeship framework after 2 modifications (removal of 1 item and moving 1 item to a different factor). The SEM had poor model fit compared to previous studies, and an exploratory analysis suggests that there is a complex relationship between the various teaching methods. Preceptors and students highly agreed that the instrument was valuable for providing feedback. Conclusion: The MCTQ has additional validity evidence that supports it as a viable instrument to assess preceptor engagement in teaching methods consistent with the cognitive apprenticeship framework. Additional research is warranted to explore the relationship between components in the cognitive apprenticeship framework to determine how preceptors can best focus their efforts to improve clinical teaching.

Qualities and Methods of Highly Effective Surgical Educators: A Grounded Theory Model

OBJECTIVE

To identify personal qualities and teaching methods of highly effective surgical educators using a novel research design.

DESIGN

In this qualitative study, surgical residents were sent an electronic survey soliciting nominations for faculty perceived as highly effective surgical educators. In-depth, semistructured interviews were conducted with surgeons receiving the most nominations. Grounded theory methodology identified themes for analysis.

SETTING

General, vascular, and plastic surgery residents and faculty at the University of Pennsylvania Health System.

PARTICIPANTS

A total of 77 surgical residents were surveyed. Data saturation occurred after 12 semistructured interviews with attending surgeons, corresponding to the top 15% of faculty.

RESULTS

Interviewees described both personal characteristics and specific teaching approaches that facilitated successful learning. These included providing exceptional surgical education as a mission, a strong influence from past mentors and role models, a love for the profession, and a low rate of self-professed burnout. Desirable teaching methods included promoting a culture of psychological safety (the perceived ability to take interpersonal risks within one's environment), progressive autonomy, accountability of trainees, and individualized teaching for the learner. Interviewees saw education as inseparable from clinical duties, and all surgeons believed providing exceptional patient care was the foundation of effective surgical teaching. The derived themes suggested that educators prefer "cognitive-based" approaches, focusing on learning processes rather than specific outcomes.

CONCLUSIONS

This study identified characteristics and educational styles of highly effective educators in a cohort of academic surgeons. This framework may inform the development of educational programs for residents and faculty in effective teaching methods.

Geriatric Psychiatry Teaching in Canadian Medical Schools: a Cross-Sectional Study

OBJECTIVE

With a growing geriatric population and limited geriatric psychiatrists in Canada, it is crucial to provide sufficient training in geriatric psychiatry during medical school. The authors examined how geriatric psychiatry education is delivered in Canadian medical schools during clerkship. Factors that could be associated with increased geriatric psychiatry teaching in medical schools were examined. The authors were also interested in comparing Canadian to US findings.

METHODS

A cross-sectional survey was distributed to the psychiatry medical education representatives attending the Canadian Organization of Undergraduate Psychiatry Educators (COUPE) semi-annual meeting in September 2017.

RESULTS

All 17 (100%) medical schools completed the survey. Fifteen of the 17 schools (88%) have geriatric psychiatry-specific learning objectives. Five schools (29%) offer a clinical component in geriatric psychiatry. One school has an award for clerks (6%), and no awards exist for faculty. The number of lecture hours in geriatric psychiatry is moderately correlated with the presence of a geriatric component to psychiatry clerkship (Spearman's rho = 0.67, p = 0.003) and the length of the geriatric portion of clerkship (Spearman's rho = 0.64, p value = 0.006). Lecture hours are also moderately correlated with the presence of a geriatric fellowship (Spearman's rho = 0.68, p value = 0.003).

CONCLUSIONS

Geriatric psychiatry clerkship education is inconsistent in Canada. There is virtually no recognition of excellence in teaching or undergraduate performance in this area in clerkship. Geriatric psychiatry may receive more frequent attention in Canadian medical schools than in US medical schools.

The American Society of Hematology (ASH) Medical Educators Institute: a Pilot Faculty Development Project for Hematology Educators

Clinician educators at academic medical centers often lack the community, mentorship, and faculty development to support their missions around education scholarship and teaching. Inadequate support for clinician educators can lead to professional dissatisfaction and slowed academic advancement. In 2014, ASH conducted a needs assessment of medical school hematology course directors, hematology-oncology fellowship program directors, and other ASH members identified as educators to determine this community's desire for faculty development in medical education. These data furthered the development of an annual faculty development program for hematology educators offering an interactive curriculum and support for an educational scholarly project. The needs assessment indicated that over 70% of respondents would be personally interested in a faculty development opportunity for hematology educators and only 11% had previously participated in such a program. A steering committee designed an intervention blending didactics, interactive small group exercises, webinars, mentorship for a scholarly project, 360-degree feedback for each participant, and a forum to discuss common career development goals. Of 42 applicants, 20 participants were chosen for the inaugural workshop. Following successful execution of the workshop, participants reported significant increase in confidence in the knowledge, skills, and attitudes targeted by the curriculum. A series of follow-up webinars have been developed to deliver additional content not covered during the workshop and to continue mentorship relationships. The curriculum will be further refined based on feedback from faculty and participants. Long-term outcome measurement will include tracking all participants' publications and presentations, time to promotion, and involvement in national medical education initiatives.

A Guide for Increasing Scholarship for Medical Educators

Disseminating scholarly work as a clinician educator is critical to furthering new knowledge in medical education, creating an evidence base for new practices, and increasing the likelihood of promotion. Knowing how to initiate scholarship and develop habits to support it, however, may not be clear. This perspective is designed to help readers choose and narrow their focus of scholarly interest, garner mentors, find potential project funding, and identify outside support through involvement with national organizations, collaborators, and faculty development programs. By incorporating these suggestions into their daily work, educators can find ways to connect their clinical and educational interests and make their daily work count toward scholarship.

Cross-Cultural Medical Care Training and Education: a National Survey of Pediatric Hematology/Oncology Fellows-in-Training and Fellowship Program Directors

Pediatric hematologists/oncologists face complex situations such as breaking bad news, treatment/clinical trials discussions, and end-of-life/hospice care. With increasing diversity in patient and physician populations, cultural competency and sensitivity training covering different aspects of pediatric hematology/oncology (PDHO) care can help improve health care delivery and reduce disparities. Though it is considered a required component of fellowship training, there is no clearly defined curriculum meant specifically for PDHO fellows-in-training (PDHO-F). A national online survey of 356 PDHO-F and 67 PDHO program directors (PDHO-PD) was conducted to assess the educational experience, perceptions about identifying barriers including one's own biases and trainee comfort in delivering culturally sensitive care in various PDHO relevant clinical situations. One hundred and eleven (31.2%) PDHO-F and 27 (40.3%) PDHO-PD responded. 30.6% of PDHO-F "strongly agreed/agreed" they received comprehensive cross-cultural communication (CCC) training. The top two teaching methods were faculty role modeling and informal teaching. Majority of CCC training is in medical school or residency and only 10.8% of PDHO-F reported that most of their CCC training was in fellowship. In most clinical situations, there was a modest direct correlation between the fellow's level of agreement that they received comprehensive CCC training and their comfort level. Comfort level with some clinical situations was also significantly different based on year of training. Fellowship training programs should have CCC curricula which use experiential learning models and lay the foundation for promoting cultural awareness, self-reflection, and better patient-physician partnerships which can eventually adapt to and surmount the challenges unique to the physician's chosen field of practice.

Shifting Academic Health Centers From a Culture of Community Service to Community Engagement and Integration

There is an increasing need for academic health centers (AHCs) to engage communities across their clinical, research, and educational missions. Although AHCs have a long-standing history of community service, a more comprehensive approach to working with communities is required to respond to shifts toward a population health paradigm, funder requirements for community engagement in research, and demands that medical education focus more on social and environmental determinants of health. Community engagement has been employed at many AHCs, though often in limited ways or relying heavily on students and faculty interested in serving communities. This limited involvement has been due, in part, to lack of infrastructure to support engagement, resource constraints, and the lack of a clear value proposition for long-term investments in community partnerships. However, there are compelling reasons for AHCs to take an enterprise-wide approach to working with communities. An enterprise-wide approach to community engagement will require reconsideration of communities, moving from viewing them as people or groups in need of service to seeing them as assets who can help AHCs better understand and address social determinants of health, enhance students' and trainees' ability to provide care, and increase the relevance and potential impact of research discoveries. To accomplish this, AHCs will need to establish the necessary infrastructure to support long-term community partnerships, adapt policies to support and reward engaged scholarship and teaching, and consider new ways of integrating community members in roles as advisors and collaborators across the AHC.

Assessment of Gender-Based Linguistic Differences in Physician Trainee Evaluations of Medical Faculty Using Automated Text Mining

Importance

Women are underrepresented at higher ranks in academic medicine. However, the factors contributing to this disparity have not been fully elucidated. Implicit bias and unconscious mental attitudes toward a person or group may be factors. Although academic medical centers use physician trainee evaluations of faculty to inform promotion decisions, little is known about gender bias in these evaluations. To date, no studies have examined narrative evaluations of medical faculty by physician trainees for differences based on gender.

Objective

To characterize gender-associated linguistic differences in narrative evaluations of medical faculty written by physician trainees.

Design, Setting, and Participants

This retrospective cohort study included all faculty teaching evaluations completed for the department of medicine faculty by medical students, residents, and fellows at a large academic center in Pennsylvania from July 1, 2015, through June 30, 2016. Data analysis was performed from June 1, 2018, through July 31, 2018.

Main Outcomes and Measures

Word use in faculty evaluations was quantified using automated text mining by converting free-text comments into unique 1- and 2-word phrases. Mixed-effects logistic regression analysis was performed to assess associations of faculty gender with frequencies of specific words and phrases present in a physician trainee evaluation.

Results

A total of 7326 unique evaluations were collected for 521 faculty (325 men [62.4%] and 196 women [37.6%]). The individual words art (odds ratio [OR], 7.78; 95% CI, 1.01-59.89), trials (OR, 4.43; 95% CI, 1.34-14.69), master (OR, 4.24; 95% CI, 1.69-10.63), and humor (OR, 2.32; 95% CI, 1.44-3.73) were significantly associated with evaluations of male faculty, whereas the words empathetic (OR, 4.34; 95% CI, 1.56-12.07), delight (OR, 4.26; 95% CI, 1.35-13.40), and warm (OR, 3.45; 95% CI, 1.83-6.49) were significantly associated with evaluations of female faculty. Two-word phrases associated with male faculty evaluations included run rounds (OR, 7.78; 95% CI, 1.01-59.84), big picture (OR, 7.15; 95% CI, 1.68-30.42), and master clinician (OR, 4.02; 95% CI, 1.21-13.36), whereas evaluations of female faculty were more likely to be associated with model physician (OR, 7.75; 95% CI, 1.70-35.39), just right (OR, 6.97; 95% CI, 1.51-32.30), and attention (to) detail (OR, 4.26; 95% CI, 1.36-13.40).

Conclusions and Relevance

The data showed quantifiable linguistic differences between free-text comments about male and female faculty in physician trainee evaluations. Further evaluation of these differences, particularly in association with ongoing gender disparities in faculty promotion and retention, may be warranted.

Cyberbullying in Academic Medicine: A Framework for Managing Social Media Attacks

Criticism, scathing comments, and harassment are becoming more common elements of social media discourse. Recent coordinated public attacks directed at higher education faculty illustrate these troubling trends. In several cases, these attacks have been politically motivated by participants who disagree with a faculty member's statements regarding sensitive subjects. Whereas most high-profile cases have included faculty teaching at the undergraduate level who use social media to promote scholarly discussion, medical school faculty may also be at risk, especially if their scholarly pursuits pertain to politically charged issues (e.g., race and diversity, firearms, vaccinations, the health of transgender populations). In today's digital environment of cellphone recordings, forwarded e-mails, and open-access manuscripts, any faculty member who discusses or engages in scholarship of politically sensitive issues on- or offline may be at risk. In this Invited Commentary, the authors discuss the multifaceted problem of cyberbullying of medical school faculty and provide recommendations to faculty and administrators about how to mitigate and manage these situations.

"All the ward's a stage": a qualitative study of the experience of direct observation of handoffs

Direct observation of clinical skills is central to assessment in a competency-based medical education model, yet little is known about how direct observation is experienced by trainees and observers. The objective of the study is to explore how direct observation was experienced by residents and faculty in the context of the I-PASS Handoff Study. In this multi-center qualitative study, we conducted focus groups and semi-structured interviews of residents and faculty members at eight tertiary pediatric centers in North America that implemented the I-PASS Handoff Bundle. We employed qualitative thematic analysis to interpret the data. Barriers to and strategies for direct observation were described relating to the observer, trainee, and clinical environment. Residents and faculty described a mutual awareness that residents change their performance of handoffs when observed, in contrast to their usual behavior in a clinical setting. Changes in handoff performance may depend on the nature of the observer or 'audience'. Direct observation also highlighted the importance of handoffs to participants, recognized as a clinical activity that warrants feedback and assessment. Dramaturgical theory can be used to understand our finding of 'front-stage' (observed) versus 'backstage' (unobserved) handoffs as distinct performances, tailored to an "audience". Educators must be cognizant of changes in performance of routine clinical activities when using direct observation to assess clinical competence.

A Decade of Teaching and Learning in Internal Medicine Ambulatory Education: A Scoping Review

BACKGROUND

Ambulatory training in internal medicine residency programs has historically been considered less robust than inpatient-focused training, which prompted a 2009 revision of the Accreditation Council for Graduate Medical Education (ACGME) Program Requirements in Internal Medicine. This revision was intended to create a balance between inpatient and outpatient training standards and to spur innovation in the ambulatory setting.

OBJECTIVE

We explored innovations in ambulatory education in internal medicine residency programs since the 2009 revision of the ACGME Program Requirements in Internal Medicine.

METHODS

The authors conducted a scoping review of the literature from 2008 to 2017, searching PubMed, ERIC, and Scopus databases. Articles related to improving educational quality of ambulatory components of US-based internal medicine residency programs were eligible for inclusion. Articles were screened for relevance and theme categorization and then divided into 6 themes: clinic redesign, curriculum development, evaluating resident practice/performance, teaching methods, program evaluation, and faculty development. Once a theme was assigned, data extraction and quality assessment using the Medical Education Research Study Quality Instrument (MERSQI) score were completed.

RESULTS

A total of 967 potentially relevant articles were discovered; of those, 182 were deemed relevant and underwent full review. Most articles fell into curriculum development and clinic redesign themes. The majority of included studies were from a single institution, used nonstandardized tools, and assessed outcomes at the satisfaction or knowledge/attitude/skills levels. Few studies showed behavioral changes or patient-level outcomes.

CONCLUSIONS

While a rich diversity of educational innovations have occurred since the 2009 revision of the ACGME Program Requirements in Internal Medicine, there is a significant need for multi-institution studies and higher-level assessment.