Assessment and remediation for physicians with suspected performance problems: an international survey

Remediation Programs for Regulated Health Care Professionals: A Scoping Review

PURPOSE

Clinical competence is essential for providing safe, competent care and is regularly assessed to ensure health care practitioners maintain competence. When deficiencies in competence are identified, practitioners may undergo remediation. However, there is limited evidence regarding the effectiveness of remediation programs. The purpose of this review is to examine the purpose, format, and outcomes of remediation programs for regulated health care practitioners.

METHODS

All six stages of the scoping review process as recommended by Levac et al were undertaken. A search was conducted within MEDLINE, Embase, CINAHL, ERIC, gray literature databases, and websites of Canadian provincial regulatory bodies. Emails were sent to Registrars of Canadian regulatory bodies to supplement data gathered from their websites.

RESULTS

A total of 14 programs were identified, primarily for physicians (n = 8). Reasons for remediation varied widely, with some programs identifying multiple reasons for referral such as deficiencies in recordkeeping (n = 7) and clinical skills (n = 6). Most programs (n = 9) were individualized to address specific deficiencies in competence. The process of remediation followed three stages: (1) assessment, (2) active remediation, and (3) reassessment. Most programs (n = 12) reported that remediation was effective in improving competence.

CONCLUSIONS

Regulatory bodies should consider implementing individualized remediation programs to ensure that clinicians' deficiencies in competence are addressed effectively. Further research is indicated, using reliable and valid outcome measures to assess competence immediately after remediation programs and beyond.

Emotion in remediation: A scoping review of the medical education literature

OBJECTIVES

Remediation can be crucial and high stakes for medical learners, and experts agree it is often not optimally conducted. Research from other fields indicates that explicit incorporation of emotion improves education because of emotion's documented impacts on learning. Because this could present an important opportunity for improving remediation, we aimed to investigate how the literature on remediation interventions in medical education discusses emotion.

METHODS

The authors used Arksey and O'Malley's framework to conduct a scoping literature review of records describing remediation interventions in medical education, using PubMed, CINAHL Complete, ERIC, Web of Science and APA PsycInfo databases, including all English-language publications through 1 May 2020 meeting search criteria. They included publications discussing remediation interventions either empirically or theoretically, pertaining to physicians or physician trainees of any level. Two independent reviewers used a standardised data extraction form to report descriptive information; they reviewed included records for the presence of mentions of emotion, described the mentions and analysed results thematically.

RESULTS

Of 1644 records, 199 met inclusion criteria and were reviewed in full. Of those, 112 (56%) mentioned emotion in some way; others focused solely on cognitive aspects of remediation. The mentions of emotion fell into three themes based on when the emotion was cited as present: during regular coursework or practice, upon referral for remediation and during remediation. One-quarter of records (50) indicated potential intentional incorporation of emotion into remediation programme design, but they were non-specific as to how emotions related to the learning process itself.

CONCLUSION

Even though emotion is omnipresent in remediation, medical educators frequently do not factor emotion into the design of remediation approaches and rarely explicitly utilise emotion to improve the learning process. Applications from other fields may help medical educators leverage emotion to improve learning in remediation, including strategies to frame and design remediation.

Optimising the delivery of remediation programmes for doctors: A realist review

CONTEXT

Medical underperformance puts patient safety at risk. Remediation, the process that seeks to 'remedy' underperformance and return a doctor to safe practice, is therefore a crucially important area of medical education. However, although remediation is used in health care systems globally, there is limited evidence for the particular models or strategies employed. The purpose of this study was to conduct a realist review to ascertain why, how, in what contexts, for whom and to what extent remediation programmes for practising doctors work to restore patient safety.

METHOD

We conducted a realist literature review consistent with RAMESES standards. We developed a programme theory of remediation by carrying out a systematic search of the literature and through regular engagement with a stakeholder group. We searched bibliographic databases (MEDLINE, EMBASE, PsycINFO, HMIC, CINAHL, ERIC, ASSIA and DARE) and conducted purposive supplementary searches. Relevant sections of text relating to the programme theory were extracted and synthesised using a realist logic of analysis to identify context-mechanism-outcome configurations (CMOcs).

RESULTS

A 141 records were included. The majority of the studies were from North America (64%). 29 CMOcs were identified. Remediation programmes are effective when a doctor's insight and motivation are developed and behaviour change reinforced. Insight can be developed by providing safe spaces, using advocacy to promote trust and framing feedback sensitively. Motivation can be enhanced by involving the doctor in remediation planning, correcting causal attribution, goal setting and destigmatising remediation. Sustained change can be achieved by practising new behaviours and skills, and through guided reflection.

CONCLUSION

Remediation can work when it creates environments that trigger behaviour change mechanisms. Our evidence synthesis provides detailed recommendations on tailoring implementation and design strategies to improve remediation interventions for doctors.

Remediation programmes for practising doctors to restore patient safety: the RESTORE realist review

Background

An underperforming doctor puts patient safety at risk. Remediation is an intervention intended to address underperformance and return a doctor to safe practice. Used in health-care systems all over the world, it has clear implications for both patient safety and doctor retention in the workforce. However, there is limited evidence underpinning remediation programmes, particularly a lack of knowledge as to why and how a remedial intervention may work to change a doctor’s practice.

Objectives

To (1) conduct a realist review of the literature to ascertain why, how, in what contexts, for whom and to what extent remediation programmes for practising doctors work to restore patient safety; and (2) provide recommendations on tailoring, implementation and design strategies to improve remediation interventions for doctors.

Design

A realist review of the literature underpinned by the Realist And MEta-narrative Evidence Syntheses: Evolving Standards quality and reporting standards.

Data sources

Searches of bibliographic databases were conducted in June 2018 using the following databases: EMBASE, MEDLINE, Cumulative Index to Nursing and Allied Health Literature, PsycINFO, Education Resources Information Center, Database of Abstracts of Reviews of Effects, Applied Social Sciences Index and Abstracts, and Health Management Information Consortium. Grey literature searches were conducted in June 2019 using the following: Google Scholar (Google Inc., Mountain View, CA, USA), OpenGrey, NHS England, North Grey Literature Collection, National Institute for Health and Care Excellence Evidence, Electronic Theses Online Service, Health Systems Evidence and Turning Research into Practice. Further relevant studies were identified via backward citation searching, searching the libraries of the core research team and through a stakeholder group.

Review methods

Realist review is a theory-orientated and explanatory approach to the synthesis of evidence that seeks to develop programme theories about how an intervention produces its effects. We developed a programme theory of remediation by convening a stakeholder group and undertaking a systematic search of the literature. We included all studies in the English language on the remediation of practising doctors, all study designs, all health-care settings and all outcome measures. We extracted relevant sections of text relating to the programme theory. Extracted data were then synthesised using a realist logic of analysis to identify context–mechanism–outcome configurations.

Results

A total of 141 records were included. Of the 141 studies included in the review, 64% related to North America and 14% were from the UK. The majority of studies (72%) were published between 2008 and 2018. A total of 33% of articles were commentaries, 30% were research papers, 25% were case studies and 12% were other types of articles. Among the research papers, 64% were quantitative, 19% were literature reviews, 14% were qualitative and 3% were mixed methods. A total of 40% of the articles were about junior doctors/residents, 31% were about practicing physicians, 17% were about a mixture of both (with some including medical students) and 12% were not applicable. A total of 40% of studies focused on remediating all areas of clinical practice, including medical knowledge, clinical skills and professionalism. A total of 27% of studies focused on professionalism only, 19% focused on knowledge and/or clinical skills and 14% did not specify. A total of 32% of studies described a remediation intervention, 16% outlined strategies for designing remediation programmes, 11% outlined remediation models and 41% were not applicable. Twenty-nine context–mechanism–outcome configurations were identified. Remediation programmes work when they develop doctors’ insight and motivation, and reinforce behaviour change. Strategies such as providing safe spaces, using advocacy to develop trust in the remediation process and carefully framing feedback create contexts in which psychological safety and professional dissonance lead to the development of insight. Involving the remediating doctor in remediation planning can provide a perceived sense of control in the process and this, alongside correcting causal attribution, goal-setting, destigmatising remediation and clarity of consequences, helps motivate doctors to change. Sustained change may be facilitated by practising new behaviours and skills and through guided reflection.

Limitations

Limitations were the low quality of included literature and limited number of UK-based studies.

Future work

Future work should use the recommendations to optimise the delivery of existing remediation programmes for doctors in the NHS.

Study registration

This study is registered as PROSPERO CRD42018088779.

Funding

This project was funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research; Vol. 9, No. 11. See the NIHR Journals Library website for further project information.

Education or regulation? Exploring our underlying conceptualisations of remediation for practising physicians

CONTEXT

The remediation of practising physicians is coming to the fore in several countries in response to increasing demands for physician accountability and quality improvement initiatives in health care. However, the profession continues to grapple with the concepts and processes of remediation, particularly for physicians in practice who struggle with performance issues related to clinical competence. This suggests that current conceptualisations of remediation might be contributing to this situation. This study therefore, explored how various administration-level stakeholders conceptualise remediation in order to construct a description of the nature and meaning of remediation.

METHODS

Semi-structured interviews were conducted using purposive sampling across the range of Canadian stakeholders involved in the remediation of practising physicians, including regulatory authorities, universities, national certifying bodies and medical organisations. With the use of constructivist grounded theory processes, analysis proceeded apace with data collection in an iterative process, with initial insights guiding subsequent interviews.

RESULTS

Participants often simultaneously held two different conceptualisations of remediation: (i) remediation as part of an educational continuum involving different degrees of support, and (ii) remediation as a regulatory process removing an individual's educational autonomy. Interviewees moved between these two conceptualisations but did not always appear to be aware of doing so. These conceptualisations each had different implications regarding the degree to which remediation can be incorporated into professional processes of maintaining competence.

CONCLUSIONS

Understanding that stakeholders frequently approach the complex issue of remediation with two different perspectives without conscious awareness of doing so may help to explain several challenges in the field, including the issues of what falls under the umbrella of remediation and who should be responsible for offering remediation support. Our findings suggest the need for conceptual clarity around remediation, both to ensure that we illuminate logistical dilemmas in enacting remediation and to address the stigma of 'de-professionalisation' that the provision of even minor educational supports (such as feedback) might invoke by association.

Importance of neuropsychological screening in physicians referred for performance concerns

Introduction

The literature suggests that 6–12% of practicing physicians are dyscompetent. Dyscompetence can manifest as failures in direct provision of care, but also issues with interpersonal and communications skills and professionalism. There is a growing literature suggesting the value of neurocognitive screening in physicians with clinical competency issues. The contribution of such screening in physicians with workplace behavioral issues is not as established. The aim of this exploratory study was to examine patterns of performance on a commonly used neuropsychological screening instrument. Performances differences, if present, could have implications for remediation and/or monitoring.

Methods

Published data on a computerized neurocognitive screening instrument (MicroCog) for normative physician samples, published data on physicians referred for clinical competency issues, and newly collected data on physicians with workplace behavioral issues were analyzed. A two-way analysis of variance (Sample X Index) and post-hoc paired comparisons were conducted. A second analysis was performed employing an aggregated estimate of normative physician performance.

Results

Results revealed a significant main effect for Sample and Index and a significant interaction effect. The second analysis of variance employing the pooled samples (Sample X Index) was conducted. The workplace behavior issues sample differed significantly from each of the samples. The Sample by Index interaction was significant.

Discussion

Significant differences in performance on a neurocognitive screening instrument were found between non-referred physicians and physicians with behavioral or medical/technical competency concerns. Those with workplace behavioral issues performed significantly better than those with medical/technical issues, but significantly worse than non-referred physicians. Using these findings, 2.0% of the normal sample versus 35.1% of the medical/technical sample, and 10.9% of the behavioral sample would fail the screen using typical, conservative cutoffs. Further study of the potential role of neurocognitive factors in physicians referred for behavioral comportment issues is warranted.

Remediation in Practicing Physicians: Current and Alternative Conceptualizations

Suboptimal performance in practicing physicians is a decades-old problem. The lack of a universally accepted definition of remediation, the paucity of research on best remediation practices, and the ongoing controversy regarding the institutional responsibility for enacting and overseeing the remediation of physicians suggest that it is not merely a difficult problem to solve but a problem that the community does not grapple with meaningfully. Undoubtedly, logistical and political considerations contribute to this state of affairs; however, other underlying conceptual issues may also play a role in the medical profession's difficulties in engaging with the challenges around remediation.Through a review of the medical education and other literatures, the authors examined current conceptualizations of both reme-diation itself and the individual being remediated, as well as how the culture of medicine influences these conceptions. The authors explored how conceptualizations of remediation and the surrounding culture might affect not only the medical community's ability to support but also its willingness to engage with physicians in need of remediation.Viewing remediation as a means of supporting practice change-rather than as a means of redressing gaps in knowledge and skill-might be a useful alternative conceptualization, providing a good place to start exploring new avenues of research. However, moving forward will require more than simply a reconceptualization of remediation; it will also necessitate a change in how the community views its struggling members and a change in the medical culture that currently positions professional autonomy as the foundational premise for individual practice improvement.

Rethinking Remediation: A Model to Support the Detailed Diagnosis of Clinicians' Performance Problems and the Development of Effective Remediation Plans

The successful remediation of clinicians demonstrating poor performance in the workplace is essential to ensure the provision of safe patient care. Clinicians may develop performance problems for numerous reasons, including health, personal factors, the workplace environment, or outdated knowledge/skills. Performance problems are often complex involving multifactorial issues, encompassing knowledge, skills, and professional behaviors. It is important that (where possible and appropriate) clinicians are supported through effective remediation to return them to safe clinical practice. A review of the literature demonstrated that research into remediation is in its infancy, with little known about the effectiveness of remediation programs currently. Current strategies for the development of remediation programs are mostly "intuitive"; a few draw upon established theories to inform their approach. Similarly, although it has been established that identification of the nature/scope of performance problems through assessment is an essential first step within remediation, the need for a more widespread "diagnosis" of why the problems exist is emerging. These reasons for poor performance, particularly in the context of experienced practicing clinicians, are likely to have an impact on the potential success of remediation and should be considered within the "diagnosis." A new model for diagnosing the performance problems of the clinicians has been developed, using behavioral change theories to explore known barriers to successful remediation, such as insight, motivation, attitude, self-efficacy, and the working environment, in addition to addressing known deficits regarding knowledge and skills. This novel approach is described in this article. An initial feasibility study has demonstrated the acceptability and practical implementation of our model.

Is the support that dental registrants in difficulty receive from postgraduate dental teams and other sources adequate?

Objective The aim of this research was to investigate the views of dental registrants in difficulty (DRiDS) on the support they received from postgraduate dental teams (PgDT) in Health Education England (HEE) and other sources. These data were complemented by the views of those appointed from the PgDT to support them on the service they provide.Method Qualitative data were collected by recording one-to-one semi structured telephone interviews, lasting approximately 30 minutes, with registrants in difficulty and supporters purposefully sampled from across England and Wales. Recordings were transcribed and the interview data analysed for recurring discourses and themes using thematic framework analysis.Setting and subjects All regional leads for DRiDs services in PgDT across the UK were asked to invite the DRiDs they were in contact with and the supporters they had appointed to contact the research team. Attempts were made to contact all who returned consent forms and six DRiDs and 11 supporters were eventually interviewed.Results Overall the DRiDs thought that the PgDT were very helpful. They were in many cases the only source of expert support and advice, particularly with regard to developing a personal development plan and collecting evidence about their practice to present to the regulator. There was a good match between the qualities that DRiDs wanted their supporters to have and the strengths supporters felt they brought to the role. The DRiDs had mixed views about the support provided by their indemnifiers and could not identify any other organisations that provided support once conditions had been imposed. Some had the support of peers; but both DRiDs and supporters felt there was a need for further support in addition to the educational support provided by PgDT and legal support provided by the indemnifier.Conclusion The DRiDS regarded the PgDT as their primary source of support and, in general, were very satisfied with the character and competence of the service.

UK Policy on Doctor Remediation: Trajectories and Challenges

Around the world, policy-makers, academics, and health service professionals have become increasingly aware of the importance of remediation, the process by which poor performance is "remedied," as part of the changing landscape of medical regulation. It is, therefore, an opportune time to critique the UK experience with remediation policy. This article frames, for the first time, the UK remediation policy as developing from a central policy aim that was articulated in the 1990s: to accelerate the identification of underperformance and, subsequently, remedy any problems identified as soon as possible. In pursuit of this aim, three policy trajectories have emerged: professionalizing and standardizing remediation provision; linking remediation with other forms of regulation, namely relicensure (known in the UK as medical revalidation); and fostering obligations for doctors to report themselves and others for remediation needs. The operationalization of policy along these trajectories, and the challenges that have arisen, has relevance for anyone seeking to understand or indeed improve remediation practices within any health care system. It is argued here that the UK serves as an example of the more general challenges posed by seeking to integrate remediation policy within broader frameworks of medical governance, in particular systems of relicensure, and the need to develop a solid evidence base for remediation practices.

Leadership development in a professional medical society using 360-degree survey feedback to assess emotional intelligence

BACKGROUND

The current research evaluated the potential utility of a 360-degree survey feedback program for measuring leadership quality in potential committee leaders of a professional medical association (PMA). Emotional intelligence as measured by the extent to which self-other agreement existed in the 360-degree survey ratings was explored as a key predictor of leadership quality in the potential leaders.

STUDY DESIGN

A non-experimental correlational survey design was implemented to assess the variation in leadership quality scores across the sample of potential leaders. A total of 63 of 86 (76%) of those invited to participate did so. All potential leaders received feedback from PMA Leadership, PMA Colleagues, and PMA Staff and were asked to complete self-ratings regarding their behavior.

RESULTS

Analyses of variance revealed a consistent pattern of results as Under-Estimators and Accurate Estimators-Favorable were rated significantly higher than Over-Estimators in several leadership behaviors.

CONCLUSIONS

Emotional intelligence as conceptualized in this study was positively related to overall performance ratings of potential leaders. The ever-increasing roles and potential responsibilities for PMAs suggest that these organizations should consider multisource performance reviews as these potential future PMA executives rise through their organizations to assume leadership positions with profound potential impact on healthcare. The current findings support the notion that potential leaders who demonstrated a humble pattern or an accurate pattern of self-rating scored significantly higher in their leadership, teamwork, and interpersonal/communication skills than those with an aggrandizing self-rating.

Long-term outcomes of a remedial education program for doctors with clinical performance deficits

INTRODUCTION

Medical regulatory authorities need reliable methods of assessing and remediating doctors where there are concerns over competence. There's a small but growing literature describing remediation programs and documenting their effectiveness. This article adds to that literature by describing a program associated with the Medical Council of New Zealand (MCNZ) and reporting outcomes for 24 consecutive doctors required to undergo remediation.

METHODS

Over the 18-month period covered in this study, 24 doctors were required by the MCNZ to enter remediation after a performance assessment. The data set used in this study was drawn from these 24 consecutive cases and included the nature of concerns, severity of concerns, results of remediation and outcome of a second assessment when such an assessment was ordered.

RESULTS

Of 24 doctors who underwent initial assessment, 5 failed to engage with remediation and withdrew from clinical work. A 12-month education remediation program was completed by all remaining 19 doctors. Of these, 13 were considered to be practicing at an acceptable standard at the end of remediation on the basis of sequential supervisor reports. Six doctors were required to have a second performance assessment. Of these, only 1 was considered to be functioning at an acceptable standard. Concurrent health concerns were common among this cohort of doctors.

DISCUSSION

Seventy-five percent of doctors who entered remedial education were considered to be practicing at an acceptable standard at the end of remediation. This accords well with international data. A small number of doctors appear to be unresponsive to remediation.

Physicians reentering clinical practice: characteristics and clinical abilities

INTRODUCTION

Limited information exists to describe physicians who return to practice after absences from patient care. The Center for Personalized Education for Physicians (CPEP) is an independent, not-for-profit organization that provides clinical competency assessment and educational programs for physicians, including those reentering practice. This article studies the medical licensure status, performance, and correlates between physician characteristics and performance on initial assessment.

METHODS

Sixty-two physicians who left practice voluntarily and without discipline or sanction and who were returning to practice in the same discipline as their previous practice participated in the CPEP reentry program. Physicians completed an objective clinical skills assessment including clinical interviews by specialty-matched board-certified physicians, simulated patient encounters, a documentation exercise, and a cognitive function screen. Physicians were rated from 1 (no or limited educational needs) to 4 (global, pervasive deficits). Performance scores were compared based on select physician characteristics.

RESULTS

Twenty-five (40.3%) participants were female; participants' average age was 53.7 years (female 48.1 years; male 57.5 years). Physicians left practice for family issues (30.6%), health issues (27.4%), retirement or nonmedical career change (17.7%), and change to medical administration (14.5%). Females were more likely than males to have left practice for child rearing (P < 0.0001). Approximately one-quarter (24.2%) of participants achieved a performance rating of 1 (best-performing group); 35.5% achieved a rating of 2; 33% achieved a rating of 3; 6.5% achieved a rating of 4 (worst-performing group). Years out of practice and increasing physician age predicted poorer performance (P = 0.0403, P = 0.0440). A large proportion of physicians presenting without an active license achieved active licensure; how many of these physicians actually returned to practice is not known.

DISCUSSION

Physicians who leave practice are a heterogeneous group. Most participants' performance warranted some formal education; few demonstrated global educational needs. The data from this study justify mandates that physicians demonstrate competence through an objective testing process prior to returning to practice. Emerging patterns regarding the performance of the reentering physician may help guide future policy.

Characteristics of Physicians Referred for a Competence Assessment: A Comparison of State Medical Board and Hospital Referred Physicians

This study compares key characteristics and performance of physicians referred to a clinical competence assessment and education program by state medical boards (boards) and hospitals. Physicians referred by boards (400) and by hospitals (102) completed a CPEP clinical competence assessment between July 2002 and June 2010. Key characteristics, self-reported specialty, and average performance rating for each group are reported and compared. Results show that, compared with hospital-referred physicians, board-referred physicians were more likely to be male (75.5% versus 88.3%), older (average age 54.1 versus 50.3 years), and less likely to be currently specialty board certified (80.4% versus 61.8%). On a scale of 1 (best) to 4 (worst), average performance was 2.62 for board referrals and 2.36 for hospital referrals. There were no significant differences between board and hospital referrals in the percentage of physicians who graduated from U.S. and Canadian medical schools. The most common specialties referred differed for boards and hospitals. Conclusion: Characteristics of physicians referred to a clinical competence program by boards and hospitals differ in important respects. The authors consider the potential reasons for these differences and whether boards and hospitals are dealing with different subsets of physicians with different types of performance problems. Further study is warranted.

Physicians reentering clinical practice: characteristics and clinical abilities

INTRODUCTION

Limited information exists to describe physicians who return to practice after absences from patient care. The Center for Personalized Education for Physicians (CPEP) is an independent, not-for-profit organization that provides clinical competency assessment and educational programs for physicians, including those reentering practice. This article studies the medical licensure status, performance, and correlates between physician characteristics and performance on initial assessment.

METHODS

Sixty-two physicians who left practice voluntarily and without discipline or sanction and who were returning to practice in the same discipline as their previous practice participated in the CPEP reentry program. Physicians completed an objective clinical skills assessment including clinical interviews by specialty-matched board-certified physicians, simulated patient encounters, a documentation exercise, and a cognitive function screen. Physicians were rated from 1 (no or limited educational needs) to 4 (global, pervasive deficits). Performance scores were compared based on select physician characteristics.

RESULTS

Twenty-five (40.3%) participants were female; participants' average age was 53.7 years (female 48.1 years; male 57.5 years). Physicians left practice for family issues (30.6%), health issues (27.4%), retirement or nonmedical career change (17.7%), and change to medical administration (14.5%). Females were more likely than males to have left practice for child rearing (P < 0.0001). Approximately one-quarter (24.2%) of participants achieved a performance rating of 1 (best-performing group); 35.5% achieved a rating of 2; 33% achieved a rating of 3; 6.5% achieved a rating of 4 (worst-performing group). Years out of practice and increasing physician age predicted poorer performance (P = 0.0403, P = 0.0440). A large proportion of physicians presenting without an active license achieved active licensure; how many of these physicians actually returned to practice is not known.

DISCUSSION

Physicians who leave practice are a heterogeneous group. Most participants' performance warranted some formal education; few demonstrated global educational needs. The data from this study justify mandates that physicians demonstrate competence through an objective testing process prior to returning to practice. Emerging patterns regarding the performance of the reentering physician may help guide future policy.

Resources and studies are required to build knowledge on assessment, service, and health care

The quality of continuing education in the health professions depends in part upon the success of educators in determining what clinicians need to know to improve practice. Studies are required to build knowledge of assessment, service, and the economies of health care education. All those interested in better systems of education and care must assure the availability of resources adequate to fulfill their primary obligations to improving practice.

Poorly performing physicians: does the Script Concordance Test detect bad clinical reasoning?

INTRODUCTION

Evaluation of poorly performing physicians is a worldwide concern for licensing bodies. The Collège des Médecins du Québec currently assesses the clinical competence of physicians previously identified with potential clinical competence difficulties through a day-long procedure called the Structured Oral Interview (SOI). Two peer physicians produce a qualitative report. In view of remediation activities and the potential for legal consequences, more information on the clinical reasoning process (CRP) and quantitative data on the quality of that process is needed. This study examines the Script Concordance Test (SCT), a tool that provides a standardized and objective measure of a specific dimension of CRP, clinical data interpretation (CDI), to determine whether it could be useful in that endeavor.

METHODS

Over a 2-year period, 20 family physicians took, in addition to the SOI, a 1-hour paper-and-pencil SCT. Three evaluators, blind as to the purpose of the experiment, retrospectively reviewed SOI reports and were asked to estimate clinical reasoning quality. Subjects were classified into 2 groups (below and above median of the score distribution) for the 2 assessment methods. Agreement between classifications is estimated with the use of the Kappa coefficient.

RESULTS

Intraclass correlation for SOI was 0.89. Cronbach alpha coefficient for the SCT was 0.90. Agreement between methods was found for 13 participants (Kappa: 0.30, P = 0.18), but 7 out of 20 participants were classified differently in both methods. All participants but 1 had SCT scores below 2 SD of panel mean, thus indicating serious deficiencies in CDI.

DISCUSSION

The finding that the majority of the referred group did so poorly on CDI tasks has great interest for assessment as well as for remediation. In remediation of prescribing skills, adding SCT to SOI is useful for assessment of cognitive reasoning in poorly performing physicians. The structured oral interview should be improved with more precise reporting by those who assess the clinical reasoning process of examinees, and caution is recommended in interpreting SCT scores; they reflect only a part of the reasoning process.