Assessment of practicing family physicians: comparison of observation in a multiple-station examination using standardized patients with observation of consultations in daily practice

A workers' health surveillance online training programme for occupational physicians

BACKGROUND

To support occupational physicians (OPs) in the implementation of workers' health surveillance (WHS), a training programme was developed.

AIMS

(i) To evaluate the effects of a WHS training programme for OPs on knowledge, self-efficacy and skills to implement WHS. (ii) To evaluate to what extent a WHS training programme is acceptable and feasible for implementation in practice.

METHODS

A single-blinded randomized controlled trial with waiting-list control group was used. The WHS training programme consisted of an e-learning and a 4.5-h online training session. OPs completed a knowledge test (0-8), self-efficacy questionnaires on knowledge and skills (6-60), and vignette assignments (0-16) to measure skills. OPs completed the questionnaires, either before and after the WHS training programme (intervention group), or before the training programme (control group) while receiving the training programme after the waiting period. All OPs completed questionnaires about the training's acceptability, and feasibility for implementation in practice. ANCOVA and Poisson regression analyses were conducted.

RESULTS

The self-efficacy score (M = 44.1 versus M = 37.2) (P < 0.001) and skills score (M = 9.6 versus M = 8.3) (P < 0.05) of OPs in the training group (N = 16) were higher than the control group (N = 23). No effect was found on knowledge. Evaluation of acceptability and feasibility showed that 21 (58%) OPs were very satisfied with the training part on initiating WHS, and 29 (85%) would recommend the WHS training programme to colleagues.

CONCLUSIONS

This WHS training programme has a positive effect on self-efficacy and skills of OPs to implement WHS, and may be acceptable and feasible to implement in practice.

Evaluation of Communication Skills Among Physicians: A Systematic Review of Existing Assessment Tools

OBJECTIVE

The importance of physician training in communication skills for motivating patients to adopt a healthy life-style and optimize clinical outcomes is increasingly recognized. This study inventoried and systematically reviewed the psychometric properties of, and the skills assessed by, existing assessment tools used to evaluate communication skills among physicians.

METHODS

This review was conducted in accordance with the PRISMA guidelines (PROSPERO: CRD42018091932). Four databases (PUBMED, EMBASE, PsychINFO, and SCOPUS) were searched up to December 2018, generating 3902 unique articles, which were screened by two authors. A total of 57 articles met the inclusion criteria and underwent full data extraction.

RESULTS

Forty-five different assessment tools were identified. Only 47% of the studies mentioned underlying theories or models for designing the tool. Fifteen communication skills were assessed across the tools, the five most prevalent were information giving (46%) or gathering (40%), eliciting patients' perspectives (44%), planning/goal setting (37%), and closing the session (32%). Most tools (93%) assessed communication skills using in-person role play exercises with standardized (61%) or real (32%) patients, but only 54% described the expertise of the raters who performed the evaluations. Overall, reporting of the psychometric properties of the assessment tools was poor-moderate (4.5 ± 1.3 out of 9).

CONCLUSIONS

Despite identifying several existing physician communication assessment tools, a high degree of heterogeneity between these tools, in terms of skills assessed and study quality, was observed, and most have been poorly validated. Research is needed to rigorously develop and validate accessible, convenient, "user-friendly," and easy to administer and score communication assessment tools.

Assessing the training needs of medical students in patient information gathering

BACKGROUND

Effective communication is at the heart of good medical practice but rates of error, patient complaints, and poor clinician job satisfaction are suggestive of room for improvement in this component of medical practice and education.

METHODS

We conducted semi-structured interviews with experienced clinicians (n = 19) and medical students (n = 20) to explore their experiences associated with teaching and learning clinical communication skills and identify targets for improvements to addressing these skills in medical curricula.

RESULTS

Interviews were thematically analysed and four key themes emerged; the importance of experience, the value of role-models, the structure of a consultation, and confidence.

CONCLUSIONS

The findings reinforce the need for improvement in teaching and learning communication skills in medicine, with particular opportunity to target approaches to teaching foundational skills which can establish a strong grounding before moving into more complex situations, thus preparing students for the flexibility required in medical interviewing. A second area of opportunity and need is in the engagement and training of clinicians as mentors and teachers, with the findings from both groups indicating that preparation for teaching and feedback is lacking. Medical programs can improve their teaching of communication skills and could learn from other fields s to identify applicable innovative approaches.

Do doctors' attachment styles and emotional intelligence influence patients' emotional expressions in primary care consultations? An exploratory study using multilevel analysis

OBJECTIVE

To investigate whether and how doctors' attachment styles and emotional intelligence (EI) might influence patients' emotional expressions in general practice consultations.

METHODS

Video recordings of 26 junior doctors consulting with 173 patients were coded using the Verona Coding Definition of Emotional Sequences (VR-CoDES). Doctors' attachment style was scored across two dimensions, avoidance and anxiety, using the Experiences in Close Relationships: Short Form questionnaire. EI was assessed with the Mayer-Salovey-Caruso Emotional Intelligence Test. Multilevel Poisson regressions modelled the probability of patients' expressing emotional distress, considering doctors' attachment styles and EI and demographic and contextual factors.

RESULTS

Both attachment styles and EI were significantly associated with frequency of patients' cues, with patient- and doctor-level explanatory variables accounting for 42% of the variance in patients' cues. The relative contribution of attachment styles and EI varied depending on whether patients' presenting complaints were physical or psychosocial in nature.

CONCLUSION

Doctors' attachment styles and levels of EI are associated with patients' emotional expressions in primary care consultations. Further research is needed to investigate how these two variables interact and influence provider responses and patient outcomes.

PRACTICE IMPLICATIONS

Understanding how doctors' psychological characteristics influence PPC may help to optimise undergraduate and postgraduate medical education.

Validating relationships among attachment, emotional intelligence and clinical communication

CONTEXT

In a previous study, we found that emotional intelligence (EI) mediates the negative influences of Year 1 medical students' attachment styles on their provider-patient communication (PPC). However, in that study, students were examined on a relatively straightforward PPC skill set and were not assessed on their abilities to elicit relevant clinical information from standardised patients. The influence of these psychological variables in more demanding and realistic clinical scenarios warrants investigation.

OBJECTIVES

This study aimed to validate previous research findings by exploring the mediating effect of EI on the relationship between medical students' attachment styles and their PPC across an ecologically valid PPC objective structured clinical examination (OSCE).

METHODS

Year 2 medical students completed measures of attachment (the Experiences in Close Relationships-Short Form [ECR-SF], a 12-item measure which provides attachment avoidance and attachment anxiety dimensional scores) and EI (the Mayer-Salovey-Caruso Emotional Intelligence Test [MSCEIT], a 141-item measure on the perception, use, understanding and management of emotions), prior to their summative PPC OSCE. Provider-patient communication was assessed using OSCE scores. Structural equation modelling (SEM) was used to validate our earlier model of the relationships between attachment style, EI and PPC.

RESULTS

A total of 296 of 382 (77.5%) students participated. Attachment avoidance was significantly negatively correlated with total EI scores (r = -0.23, p < 0.01); total EI was significantly positively correlated with OSCE scores (r = 0.32, p < 0.01). Parsimonious SEM confirmed that EI mediated the negative influence of attachment avoidance on OSCE scores. It significantly predicted 14% of the variance in OSCE scores, twice as much as the 7% observed in the previous study.

CONCLUSIONS

In more demanding and realistic clinical scenarios, EI makes a greater contribution towards effective PPC. Attachment is perceived to be stable from early adulthood, whereas EI can be developed using targeted educational interventions. The validation of this theoretical model of PPC in Year 2 medical students strengthens the potential educational implications of EI.

Differential growth in doctor-patient communications skills

CONTEXT

Although doctor-patient communication is considered a core competency for medical doctors, the effect of training has not been unequivocally established. Moreover, knowledge about the variance in the growth of different skills and whether certain patterns in growth can be detected could help us to develop more efficient programmes. We therefore investigated the growth in general practitioner (GP) trainees' doctor-patient communication skills in their first year and whether the growth was different for distinct categories of skills.

METHODS

Seventy-one first-year GP trainees were invited to participate in a study aimed at measuring their consultation skills at the beginning (baseline) and at the end of their first year (follow-up). Consultation skills were assessed with the MAAS-Global rating list for consultation skills.

RESULTS

Data on 29 general practitioner trainees were collected. MAAS-Global scores showed a significant growth on all items but one. Patient-oriented skills showed significantly more growth than task-oriented skills. Empathy as a separate skill seems to be mastered predominantly before the start of training.

CONCLUSIONS

Three patterns in the growth in skills were distinguished: (i) low baseline, relatively high follow-up, (ii) moderate baseline, moderate growth and (iii) high baseline, hardly any growth. Patient-oriented skills follow either pattern (i) or (iii), whereas task-oriented skills follow pattern (ii). These findings may help to define where the focus should lie in the training of doctor-patient communication skills.

Evidence-based competencies for improving communication skills in graduate medical education: a review with suggestions for implementation

Communicating with patients is arguably the most common and important activity in medical practice, but this activity receives relatively little emphasis in graduate medical education. We propose 12 evidence-based communication competencies that program directors can adopt as a framework for teaching and evaluating residents' communication skills. We review supporting evidence for these competencies and argue that communication should be treated like a procedural skill that must be taught and evaluated by observing real resident-patient interactions. We make practical suggestions for implementing these competencies by addressing three critical components of a competency-based approach to communication skills: patient safety, faculty development, and direct observation of residents. This approach to teaching and assessing communication skills provides a rationale for incorporating routine direct observation into graduate medical education programs and also for designing communication skills training that ensures graduating residents develop the skills needed to provide safe, effective patient care.

Validity considerations in the assessment of professionalism

During the last decade, interest in assessing professionalism in medical education has increased exponentially and has led to the development of many new assessment tools. Efforts to validate the scores produced by tools designed to assess professionalism have lagged well behind the development of these tools. This paper provides a structured framework for collecting evidence to support the validity of assessments of professionalism. The paper begins with a short history of the concept of validity in the context of psychological assessment. It then describes Michael Kane's approach to validity as a structured argument. The majority of the paper then focuses on how Kane's framework can be applied to assessments of professionalism. Examples are provided from the literature, and recommendations for future investigation are made in areas where the literature is deficient.

Association between nonverbal communication during clinical interactions and outcomes: a systematic review and meta-analysis

OBJECTIVE

To conduct a systematic review and meta-analysis of studies reporting associations between patients' and clinicians' nonverbal communication during real clinical interactions and clinically relevant outcomes.

METHODS

We searched 10 electronic databases, reference lists, and expert contacts for English-language studies examining associations between nonverbal communication measured through direct observation and either clinician or patient outcomes in adults. Data were systematically extracted and random effects meta-analyses were performed.

RESULTS

26 observational studies met inclusion criteria. Meta-analysis was performed for patient satisfaction, which was assessed in 65% of studies. Mental and physical health status were evaluated in 23% and 19% of included studies, respectively. Both clinician warmth and clinician listening were associated with greater patient satisfaction (p<0.001 both). Physician negativity was not related to patient satisfaction (p=0.505), but greater nurse negativity was associated with less patient satisfaction (p<0.001). Substantial differences in study design and nonverbal measures existed across studies.

CONCLUSION

Greater clinician warmth, less nurse negativity, and greater clinician listening were associated with greater patient satisfaction. Additional studies are needed to evaluate the impact of nonverbal communication on patients' mental and physical health.

PRACTICE IMPLICATIONS

Communication-based interventions that target clinician warmth and listening and nurse negativity may lead to greater patient satisfaction.

Two-dimensional structure of the MAAS-Global rating list for consultation skills of doctors

BACKGROUND

The MAAS-Global (MG) is widely used to assess doctor-patient communication skills. Reliability and validity have been investigated, but little is known about its dimensionality. Assuming physicians tend to adopt certain styles or preferences in their communication with patients, a multi-dimensional structure of the MG can be hypothesized.

AIM

This study investigates the dimensional structure of the MG and explores the validity of this structure by studying the relationship between potential MG sub-scales and general practice speciality trainees' personal characteristics.

METHODS

Communication skills of 68 first-year trainees in a two-station objective structured clinical examination were assessed. Exploratory factor analysis was conducted on the resulting MG item-scores. With t-tests and correlational analysis, the relationship between MG scores and trainees' personal characteristics was examined.

RESULTS

Two well-interpretable factors were found, representing patient-oriented and task-oriented communication skills. Being born in the Netherlands and empathy were positively associated with overall communication skills. Prior communication skills training was exclusively related to task-oriented communication skills. Empathy was associated with patient-oriented, but not with task-oriented communication skills.

CONCLUSION

The two-dimensional structure of the MG may be valuable in gaining a better understanding of factors influencing the acquisition of communication skills. This may be used to optimize teaching methods in communication skills training.

Identifying context factors explaining physician's low performance in communication assessment: an explorative study in general practice

Background

Communication is a key competence for health care professionals. Analysis of registrar and GP communication performance in daily practice, however, suggests a suboptimal application of communication skills. The influence of context factors could reveal why communication performance levels, on average, do not appear adequate. The context of daily practice may require different skills or specific ways of handling these skills, whereas communication skills are mostly treated as generic. So far no empirical analysis of the context has been made. Our aim was to identify context factors that could be related to GP communication.

Methods

A purposive sample of real-life videotaped GP consultations was analyzed (N = 17). As a frame of reference we chose the MAAS-Global, a widely used assessment instrument for medical communication. By inductive reasoning, we analyzed the GP behaviour in the consultation leading to poor item scores on the MAAS-Global. In these cases we looked for the presence of an intervening context factor, and how this might explain the actual GP communication behaviour.

Results

We reached saturation after having viewed 17 consultations. We identified 19 context factors that could potentially explain the deviation from generic recommendations on communication skills. These context factors can be categorized into doctor-related, patient-related, and consultation-related factors.

Conclusions

Several context factors seem to influence doctor-patient communication, requiring the GP to apply communication skills differently from recommendations on communication. From this study we conclude that there is a need to explicitly account for context factors in the assessment of GP (and GP registrar) communication performance. The next step is to validate our findings.

Effect of feedback from standardized patients on medical students' performance and perceptions of the neurological examination

BACKGROUND

Feedback can have a powerful influence on the performance of learners, and has traditionally been provided by faculty.

AIM

This study set out to explore whether feedback from a standardized patient (SP) can improve students' performance of the neurological examination.

METHODS

A randomized controlled design was used with final year medical students. The control group did not receive any feedback. The intervention groups received either written feedback or a combination of written and verbal feedback. A written test was given prior to the intervention to assess comparability of the three groups. Pretest and post-test scores on the neurological examination were compared. Attitudinal questionnaires were administered at the time of the posttest, and 6 months later.

RESULTS

Students receiving feedback from the SPs had significantly greater scores on the posttest than the control group. In the intervention groups, students receiving both verbal and written feedback scored significantly higher than those who received only written feedback. More positive perceptions of learning outcomes and the value of SP feedback were noted in the intervention groups.

CONCLUSION

SP feedback was associated with a significant increase in student scores on the neurological examination, as well as more favorable perceptions of the experience.

A pilot study assessing knowledge of clinical signs and physical examination skills in incoming medicine residents

BACKGROUND

Physical exam skills of medical trainees are declining, but most residencies do not offer systematic clinical skills teaching or assessment.

OBJECTIVE

To assess knowledge of clinical signs and physical exam performance among incoming internal medicine residents.

METHOD

For this study, 45 incoming residents completed a multiple choice question test to assess knowledge of clinical signs. A random selection of 20 underwent a faculty-observed objective structured clinical examination (OSCE) using patients with abnormal physical findings. Mean percentage scores were computed for the multiple choice question test, overall OSCE, and the 5 individual OSCE systems.

RESULTS

The mean scores were 58.4% (14.6 of 25; SD 11. 5) for the multiple choice question test and 54.7% (31.7 of 58; SD 11.0) for the overall OSCE. Mean OSCE scores by system were cardiovascular 30.0%, pulmonary 69.2%, abdominal 61.6%, neurologic 67.0%, and musculoskeletal 41.7%. Analysis of variance showed a difference in OSCE system scores (P < .001) with cardiovascular and musculoskeletal scores significantly lower than other systems.

CONCLUSION

Overall, physical exam knowledge and performance of new residents were unsatisfactory. There appears to be a pressing need for additional clinical skills training during medical school and residency training and we are planning a new clinical skills curriculum to address this deficiency.

Poorly performing physicians: does the Script Concordance Test detect bad clinical reasoning?

INTRODUCTION

Evaluation of poorly performing physicians is a worldwide concern for licensing bodies. The Collège des Médecins du Québec currently assesses the clinical competence of physicians previously identified with potential clinical competence difficulties through a day-long procedure called the Structured Oral Interview (SOI). Two peer physicians produce a qualitative report. In view of remediation activities and the potential for legal consequences, more information on the clinical reasoning process (CRP) and quantitative data on the quality of that process is needed. This study examines the Script Concordance Test (SCT), a tool that provides a standardized and objective measure of a specific dimension of CRP, clinical data interpretation (CDI), to determine whether it could be useful in that endeavor.

METHODS

Over a 2-year period, 20 family physicians took, in addition to the SOI, a 1-hour paper-and-pencil SCT. Three evaluators, blind as to the purpose of the experiment, retrospectively reviewed SOI reports and were asked to estimate clinical reasoning quality. Subjects were classified into 2 groups (below and above median of the score distribution) for the 2 assessment methods. Agreement between classifications is estimated with the use of the Kappa coefficient.

RESULTS

Intraclass correlation for SOI was 0.89. Cronbach alpha coefficient for the SCT was 0.90. Agreement between methods was found for 13 participants (Kappa: 0.30, P = 0.18), but 7 out of 20 participants were classified differently in both methods. All participants but 1 had SCT scores below 2 SD of panel mean, thus indicating serious deficiencies in CDI.

DISCUSSION

The finding that the majority of the referred group did so poorly on CDI tasks has great interest for assessment as well as for remediation. In remediation of prescribing skills, adding SCT to SOI is useful for assessment of cognitive reasoning in poorly performing physicians. The structured oral interview should be improved with more precise reporting by those who assess the clinical reasoning process of examinees, and caution is recommended in interpreting SCT scores; they reflect only a part of the reasoning process.

Shifting goals in medical communication. Determinants of goal detection and response formation

OBJECTIVES

Research and education of medical communication different perspective, making the extraction of clear recommendations from research that can be applied in education not always possible. In education, medical communication is encountered from a goal-oriented perspective, which is often lacking in quantitative research where the relationship between process variables and the content of medical practice is often ignored. The aim of this paper is to bring the worlds of research and education together by presenting a comprehensive model of determinants explaining the behavior of physicians in daily practice. A basic notion in this model is that medical communication is goal-oriented, problem-solving behavior. Goals in communication are not fixed, but permanently changing over time. Hence, communication abilities do not rely on behavioral skills only but also on perceptual skills in identifying goals. A number of determinants affect the cognitive processes of goal appraisal and response formation: knowledge, attitudes, social norms, self-efficacy, stressors and interfering goals.

CONCLUSION

Modeling medical communication as goal-oriented problem-solving behavior, and recognizing the complexity of goal appraisal and other key determinants of response formation may provide a common focus for both research and education in measuring, explaining and improving the HCP's behavior.

PRACTICE IMPLICATIONS

In education not only skill practicing but also reflection on the process and outcomes is important to understand how one acts in practice situations and should act in future situations. In research measurements should be expanded to take contextual and goal-oriented dimensions of the process of communication into account to make findings more relevant for education and practice.

The use of simulation in the development of individual cognitive expertise in emergency medicine

This consensus group from the 2008 Academic Emergency Medicine Consensus Conference, "The Science of Simulation in Healthcare: Defining and Developing Clinical Expertise," held in Washington, DC, May 28, 2008, focused on the use of simulation for the development of individual expertise in emergency medicine (EM). Methodologically sound qualitative and quantitative research will be needed to illuminate, refine, and test hypotheses in this area. The discussion focused around six primary topics: the use of simulation to study the behavior of experts, improving the overall competence of clinicians in the shortest time possible, optimizing teaching strategies within the simulation environment, using simulation to diagnose and remediate performance problems, and transferring learning to the real-world environment. Continued collaboration between academic communities that include medicine, cognitive psychology, and education will be required to answer these questions.

Assessment of family physicians' performance using patient charts: interrater reliability and concordance with chart-stimulated recall interview

Peer-assessment processes with chart review have been used for many years to assess the clinical performance of physicians. The Quebec medical licensing authority has been required by provincial law to assess the practicing Quebec physicians on a nonvoluntary basis. During the period from January 2001 to November 2004, 25 family physicians in active practice were randomly selected from a pool of about 300. For each physician, 25 to 40 patients' medical charts were randomly selected to evaluate the interrater reliability of peer-review assessment of medical charts and to compare ratings based on chart review with a chart-stimulated recall interview to those based on chart review alone. The concordance between chart review alone and that of chart review with chart-stimulated recall interview was 75% for chart keeping, 69% for clinical investigation, 81% for diagnostic accuracy, and 74% for treatment plan. Ratings based on chart review alone achieve moderate levels of reliability (Kappa = 0.44 to 0.56). It appears that some important information about quality of care is missed when only chart review is used.

L’évaluation des pratiques cliniques : quels modes de recueil des données et à partir de quelles sources d’information?

OBJECTIVE

The aim of our study is to present the methods used to observe clinical practices, to evaluate their advantages and limits, and to specify the context in which they are preformed.

METHODS

We conducted a bibliographical search on MEDLINE using key words, completed by an offline search. We selected articles published in French or English between 1985 and 2003.

RESULTS

Thirty articles, of which 3 were reviews, as well as an additional 5 reports and 9 books, were selected. We distinguished direct or objective approaches to clinical practices as well as indirect or declarative approaches. We also noted whether the approach was individual with data collected from individual practitioners, or collective, with data coming from databases. Most methods of collecting data used medical records, often in the context of internal or external clinical audits or peer reviews. The other sources of data were self-administered questionnaires, written case simulations and simulated patients. Data was also extracted from previously existing databases: claims databases, registries and ad hoc databases.

CONCLUSION

An objective individual approach insures the highest quality of information, but possible high costs can limit the use of this approach. It is necessary to continue researching the best means of collecting data on clinical practices, because of the obligation of each practitioner to evaluate individual clinical practices.

Viewpoint: power and communication: why simulation training ought to be complemented by experiential and humanist learning

The authors present an analysis of communication training for medical students using simulation patients, and its possible influence on later doctor-patient relationships. Many empirical studies have shown the various benefits of using simulation patients to teach communication skills, but theoretical sociology and humanistic reflection shed light on some fundamental differences between the student-doctor/actor-patient interactions practiced in simulation encounters and real doctor-patient relationships. In contrast to the usual power dynamics of a doctor-patient relation, those of simulation encounters are inverted and overwritten by an entirely different set of power relations, namely, those of the evaluator-student relationship. Since the power dynamics of real doctor-patient relations are generally overlooked, the altered dynamics of the simulation encounter are not readily perceived, and simulation encounters are thus often mistaken as accurate representations of clinical reality. Exclusive reliance on this pedagogic approach of simulation training may be encouraging students to become "simulation doctors" who act out a good relationship to their patients but have no authentic connection with them. The authors propose that liberal-arts learning and encounters with real patients should be used to cultivate students' abilities to create good doctor-patient relationships, as a compliment to the pedagogic benefits of simulation encounters.

Using standardized video cases for assessment of medical communication skills: reliability of an objective structured video examination by computer

OBJECTIVE

Using standardized video cases in a computerized objective structured video examination (OSVE) aims to measure cognitive scripts underlying overt communication behavior by questions on knowledge, understanding and performance. In this study the reliability of the OSVE assessment is analyzed using the generalizability theory.

METHODS

Third year undergraduate medical students from the Academic Medical Center of the University of Amsterdam answered short-essay questions on three video cases, respectively about history taking, breaking bad news, and decision making. Of 200 participants, 116 completed all three video cases. Students were assessed in three shifts, each using a set of parallel case editions. About half of all available exams were scored independently by two raters using a detailed rating manual derived from the other half. Analyzed were the reliability of the assessment, the inter-rater reliability, and interrelatedness of the three types of video cases and their parallel editions, by computing a generalizability coefficient G.

RESULTS

The test score showed a normal distribution. The students performed relatively well on the history taking type of video cases, and relatively poor on decision making and did relatively poor on the understanding ('knows why/when') type of questions. The reliability of the assessment was acceptable (G = 0.66). It can be improved by including up to seven cases in the OSVE. The inter-rater reliability was very good (G = 0.93). The parallel editions of the video cases appeared to be more alike (G = 0.60) than the three case types (G = 0.47).

DISCUSSION

The additional value of an OSVE is the differential picture that is obtained about covert cognitive scripts underlying overt communication behavior in different types of consultations, indicated by the differing levels of knowledge, understanding and performance. The validation of the OSVE score requires more research.

CONCLUSION AND PRACTICE IMPLICATIONS

A computerized OSVE has been successfully applied with third year undergraduate medical students. The test score meets psychometric criteria, enabling a proper discrimination between adequately and poorly performing students. The high inter-rater reliability indicates that a single rater is permitted.

Physician-patient encounters: the structure of performance in family and general office practice

INTRODUCTION

The College of Physicians and Surgeons of Ontario, the regulatory authority for physicians in Ontario, Canada, conducts peer assessments of physicians' practices as part of a broad quality assurance program. Outcomes are summarized as a single score and there is no differentiation between performance in various aspects of care. In this study we test the hypothesis that physician performance is multidimensional and that dimensions can be defined in terms of physician-patient encounters.

METHODS

Peer assessment data from 532 randomly selected family practitioners were analyzed using factor analysis to assess the dimensional structure of performance. Content validity was confirmed through consultation sessions with 130 physicians. Multiple-item measures were constructed for each dimension and reliability calculated. Analysis of variance determined the extent to which multiple-item measure scores would vary across peer assessment outcomes.

RESULTS

Six performance dimensions were confirmed: acute care, chronic conditions, continuity of care and referrals, well care and health maintenance, psychosocial care, and patient records.

DISCUSSION

Physician performance is multidimensional, including types of physician-patient encounters and variation across dimensions, as demonstrated by individual practice. A conceptual framework for multidimensional performance may inform the design of meaningful evaluation and educational recommendations to meet the individual performance of practicing physicians.

Summative assessment of medical students' communication skills and professional attitudes through observation in clinical practice

To establish medical students' professional competence for the medical profession, we designed a standardized observation procedure and the Amsterdam Attitude and Communication Scale (AACS) with nine five-point scale items, for summative assessment of their communication skills and professional attitudes. This study examines the reliability of the AACS assessment in clinical practice. In the Academic Medical Centre, Amsterdam, The Netherlands, the performance of 442 fifth year clinical students was judged six times in two settings: behaviour in clinical practice was judged independently twice by a doctor and a nurse; one videotaped patient interview was judged independently by a doctor and by a psychologist. The final mark was obtained by averaging ratings across all six assessments. Raters were 88 doctors, 29 nurses, and three psychologists.

MAIN OUTCOME MEASURES

Standard errors (SEs) for absolute judgements indicate measurement precision. Precision of AACS scores is considered sufficient with SEs smaller than 0.25. Multi-disciplinary assessment of students' clinical performance using the AACS is feasible and sufficiently precise (with an overall mean of 3.97 and standard deviation of 0.55, the absolute SE is 0.21). Judgements of behaviour in the clinic were more precise (SEs range from 0.11 to 0.16) than judgements of videotaped interviews (SEs are 0.25 and 0.29). The procedure is sufficiently precise if five or six assessments are combined.

Recognizing Biothreat Diseases

The UofL Center for the Deterrence of Biowarfare and Bioterrorism's (CDBB's) training involving standardized patients, who can make convincingly accurate representations of clinical signs and symptoms, and patient simulators has proven to be an effective means of enhancing bioterrorism preparedness. In addition to providing ready access to formative measures of preparedness, both of these teaching and learning tools also offer the option of summative evaluation of skills and knowledge acquired during training sessions. The use of moulage allows for very realistic representations of cutaneous anthrax and smallpox as well as other conditions such as recluse spider bites and chickenpox with which these biothreat infections can be confused. Exercises and drills expose deficiencies and permit essential reinforcement of skills developed in training. Unannounced exercises are especially useful in assessing public health preparedness.

Educating health professionals to respond to bioterrorism

In September 2003, a consortium of bioterrorism and health education experts from the University of Louisville, the University of Kentucky, the Kentucky Department for Public Health, and the Louisville Metro Health Department received funding from the Health Resources and Services Administration (HRSA) to develop a broadly based bioterrorism education program for health professionals in the Commonwealth of Kentucky and the surrounding region. This grant will fund a series of presentations tailored to the needs of professionals in medicine, dentistry, public health, nursing, behavioral medicine, allied health, pharmacy, veterinary medicine, and agriculture, providing coordinated training both on site and through distance learning technology. This article outlines the major grant-funded activities envisioned for the grant years 2003 through 2005, focusing on the use of standardized patients and computerized biosimulators, the transdisciplinary partnerships of the universities involved, and the essential collaboration provided by the state and local health departments.

Setting a standard for performance assessment of doctor-patient communication in general practice

CONTEXT

Continuing professional development (CPD) of general practitioners.

OBJECTIVE

Criterion-referenced standards for assessing performance in the real practice of general practitioners (GPs) should be available to identify learning needs or poor performers for CPD. The applicability of common standard setting procedures in authentic assessment has not been investigated.

METHODS

To set a standard for assessment of GP-patient communication with video observation of daily practice, we investigated 2 well known examples of 2 different standard setting approaches. An Angoff procedure was applied to 8 written cases. A borderline regression method was applied to videotaped consultations of 88 GPs. The procedures and outcomes were evaluated by the applicability of the procedure, the reliability of the standards and the credibility as perceived by the stakeholders, namely, the GPs.

RESULTS

Both methods are applicable and reliable; the obtained standards are credible according to the GPs.

CONCLUSIONS

Both modified methods can be used to set a standard for assessment in daily practice. The context in which the standard will be used - i.e. the specific purpose of the standard, the moment the standard must be available or if specific feedback must be given - is important because methods differ in practical aspects.

Family Medicine Residents' Performance with Detected Versus Undetected Simulated Patients Posing as Problem Drinkers

BACKGROUND

Simulated patients are commonly used to evaluate medical trainees. Unannounced simulated patients provide an accurate measure of physician performance.

PURPOSE

To determine the effects of detection of SPs on physician performance, and identify factors leading to detection.

METHODS

Fixty-six family medicine residents were each visited by two unannounced simulated patients presenting with alcohol-induced hypertension or insomnia. Residents were then surveyed on their detection of SPs.

RESULTS

SPs were detected on 45 out of 104 visits. Inner city clinics had higher detection rates than middle class clinics. Residents' checklist and global rating scores were substantially higher on detected than undetected visits, for both between-subject and within-subject comparisons. The most common reasons for detection concerned SP demographics and behaviour; the SP "did not act like a drinker" and was of a different social class than the typical clinic patient.

CONCLUSIONS

Multi-clinic studies involving residents experienced with SPs should ensure that the SP role and behavior conform to physician expectations and the demographics of the clinic. SP station testing does not accurately reflect physicians' actual clinical behavior and should not be relied on as the primary method of evaluation. The study also suggests that physicians' poor performance in identifying and managing alcohol problems is not entirely due to lack of skill, as they demonstrated greater clinical skills when they became aware that they were being evaluated. Physicians' clinical priorities, sense of responsibility and other attitudinal determinants of their behavior should be addressed when training physicians on the management of alcohol problems.

Comprehensive assessment of professional competence: the Rochester experiment

BACKGROUND

A required 2-week comprehensive assessment (CA) for 2nd-year medical students that integrates basic science, clinical skills, information management, and professionalism was implemented.

DESCRIPTION

The CA links standardized patients (SPs) with computer-based exercises, a teamwork exercise, and peer assessments; and culminates in student-generated learning plans.

EVALUATION

Scores assigned by SPs showed acceptable interrater reliability. Factor analyses defined meaningful subscales of the peer assessment and communication rating scales. Ratings of communication skills were correlated with information gathering, patient counseling, and peer assessments; these, in turn, were strongly correlated with the written exercises. Students found the CA fair, with some variability in opinion of the peer and written exercises. Useful learning plans and positive curricular changes were undertaken in response to the CA results.

CONCLUSION

A CA that integrates multiple domains of professional competence is feasible, useful to students, and fosters reflection and change. Preliminary data suggest that this format is reliable and valid.

Faculty and the observation of trainees' clinical skills: problems and opportunities

The clinical skills of medical interviewing, physical examination, and counseling remain vital to the effective care of patients, yet research continues to document serious deficiencies in clinical skills among students and residents. The most important method of evaluation is the direct observation of trainees performing these clinical skills. Standardized patients and other simulation technologies are important and reliable tools for teaching clinical skills and evaluating competence and will be incorporated in the near future as part of the United States Medical Licensing Examination. Standardized patients and simulation, however, cannot and should not replace the direct observation by faculty of trainees' clinical skills with actual patients. Faculty are in the best position to document improvement over time and to certify trainees have attained sophisticated levels of skill in medical interviewing, physical examination, and counseling. Unfortunately, current evidence suggests significant deficiencies in faculty direct observation evaluation skills. The author outlines the nature of the problems in clinical skills and their evaluation by faculty and ends with recommendations to improve the current state of faculty skills in evaluation.

Assessment of clinical skills in medical practice

The introduction of a clinical skills examination (CSE) to Step 2 of the U.S. Medical Licensing Examination (USMLE) has focused attention on the design and delivery of large-scale standardized tests of clinical skills and raised the question of the appropriateness of evaluation of these competencies across the span of a physician's career. This initiative coincides with growing pressure to periodically assess the continued competence of physicians in practice. The USMLE CSE is designed to certify that candidates have the basic clinical skills required for the safe and effective practice of medicine in the supervised environment of postgraduate training. These include history taking, physical examination, effective communication with patients and other members of the health care team, and clear and accurate documentation of diagnostic impressions and plans for further assessment. The USMLE CSE does not assess procedural skills. As physicians progress through training and enter practice, both knowledge base and requisite technical skills become more diverse. A variety of indirect and direct measures are available for evaluating physicians, but, at present, no single method permits high-stake inferences about clinical skills. Systematic and standardized assessments make a contribution to comprehensive evaluations, but they retain an element of assessing capacity rather than authentic performance in practice. Much work is needed to identify the optimal combination of methods to be employed in support of programs to ensure maintenance of competence of practicing physicians.

Predicting communication skills with a paper-and-pencil test

AIM

This study was conducted to investigate the value of a written knowledge test of communication skills for predicting scores on a performance test of communication skills.

METHOD

A paper-and-pencil test of knowledge about communication skills and a performance test of communication skills, consisting of four stations with standardised patients, were administered to students of two classes of the medical schools of Maastricht and Leiden, the Netherlands. The results on these tests were compared.

RESULTS

From the results of both instruments, the classes of the participating students could be recognised equally well: 60% correct qualifications of the classes by the knowledge test and 64% by the multiple station examination. Between the two tests an overall, disattenuated correlation of 0.60 was found (N=133, P < 0.01), suggesting moderate predictive value of the knowledge test for the performance test of communication skills. The correlation is stronger for students from Maastricht medical school than for their colleagues in Leiden. Correlation between the knowledge of communication skills test and other available test results of the participating Maastricht students is close to zero, suggesting that the test measures a distinct quality of students' competence.

DISCUSSION

The paper-and-pencil test of knowledge of communication skills has predictive value for the performance of these skills, but this value seems to be less pronounced than similar findings for clinical procedural skills. The stronger relationship between 'knowing how' and 'showing' in the Maastricht student group might be indicative of an effect of the training format.

Using simulated consultations to develop communications skills for neurology trainees

Communication skills are essential for clinical medicine yet, unlike in general practice, trainees in specialist and general medicine are not formally trained in them. We have used videotaped recording of simulated consultations to evaluate their acceptability and usefulness for training neurology specialist registrars. Twelve specialist registrars in neurology participated; their perceptions of the method were assessed using quantified scales and focus groups. All but one of the 12 trainees found the exercise useful both for improving clinical skills and for the imparting of information. The median visual analogue scores (0=useless, 100=very useful) for history taking and for imparting information were 91 and 90%, respectively. The median scores [and interquartile range (IQR)] of perceived usefulness for communication skills increased before to after (for use of video) from 68 (58-78) to 88 (80-92)% (P < 0.02), and (for use of simulated patients) from 51 (40-71) to 86 (79-89)% (P < 0.02). The focus groups provided additional qualitative data supporting the technique. We conclude that videotaped consultations with simulated patients are valued by most neurology trainees, both for improving their history-taking skills and for imparting information. The technique could be used more widely in neurology training, and may have a role in assessment.

Communication between nurses and simulated patients with cancer: evaluation of a communication training programme

In this paper the effect of a communication training programme on the instrumental and affective communication skills employed by ward nurses during the admittance interview with recently diagnosed cancer patients was investigated. The training focused on teaching nurses skills to discuss and handle patient emotions. For this purpose, 46 nurses participated in 92 videotaped admittance interviews with simulated patients. The study had a randomized pre-test-post-test design. Multi-level analysis was used to measure the effects of the training. The results revealed that the trained nurses significantly increased asking open-ended psychosocial questions, which indicates that they were actively exploring patients' feelings. Furthermore, the patients showed a significant increase in affective communication. In conclusion, the results of this study demonstrate that, although limited, training can induce favourable changes in the communication skills of nurses, and can even affect patient communication. Future studies should focus on the further evaluation of educational programmes to enhance communication skills.

Communication skills of nurses during interactions with simulated cancer patients

AIM

In this paper the balance of affective and instrumental communication employed by nurses during the admission interview with recently diagnosed cancer patients was investigated.

RATIONALE

The balance of affective and instrumental communication employed by nurses appears to be important, especially during the admission interview with cancer patients.

METHODS

For this purpose, admission interviews between 53 ward nurses and simulated cancer patients were videotaped and analysed using the Roter Interaction Analysis system, in which a distinction is made between instrumental and affective communication.

RESULTS

The results reveal that more than 60% of nurses' utterances were of an instrumental nature. Affective communication occurred, but was more related to global affect ratings like giving agreements and paraphrases than to discussing and exploring actively patients feelings by showing empathy, showing concern and optimism.

CONCLUSION

In future, nurses should be systematically provided with (continuing) training programmes, in which they learn how to communicate effectively in relation to patients' emotions and feelings, and how to integrate emotional care with practical and medical tasks.

Rhemulatologists' performance in dailiy practice

OBJECTIVE

To assess rheumatologists' performance for 8 rheumatologic conditions and to explore possible explanatory factors.

METHODS

After written informed consent was obtained, 27 rheumatologists (21% of all Dutch rheumatologists) practicing in 16 outpatient departments were each visited by 8 incognito "standardized patients" (SPs). The diagnoses of these 8 cases account for about 23% of all new referred patients in the Netherlands. Results for ordered lab tests as well as real radiographs with corresponding results from a radiologist were simulated. Information from the visits was obtained from the SPs, who completed predefined case-specific checklists, and by collecting data on resource utilization. Feedback was provided.

RESULTS

Altogether 254 encounters took place, of which 201 were first visits and 53 were followup visits. SPs were unmasked twice during a visit. There was considerable variation in resource utilization (lab tests and imaging) between cases and between rheumatologists. Mean costs per rheumatologist ranged from US $ 4.67 to $ 65.36 per visit for lab tests and from US $ 33.15 to $ 226.84 per visit for imaging tests. No significant correlations were seen between resource utilization costs and number of years of clinical experience or performance on checklist scores. Rheumatologists with longer experience had lower total item checklist scores (r = -0.47; P < 0.05).

CONCLUSION

A considerable variation in resource utilization was found among 27 Dutch rheumatologists. The information obtained is an excellent source for discussion on the appropriateness of care.

Quality assessment in general practice trainers

INTRODUCTION

General practice trainers hold a key position in general practice training, especially through their provision of a role model. Their own competence in general practice care is important in this regard. The purpose of the study was to evaluate whether a quality assessment programme could identify the strengths and weaknesses of GP trainers in four main domains of general practice care.

METHODS

The quality assessment programme comprised validated tests on four domains of general practice: general medical knowledge, knowledge of medical-technical skills, consultation skills and practice management. The criterion for the identification of relative strengths and weaknesses of GP trainers was a variation in the scores of trainers indicating higher and lower scores (strengths and weaknesses) within each domain.

RESULTS

GP trainers (n=105) were invited to participate in the study and 90% (n=94) did so. The variation in scores allowed the indication of strengths and weaknesses. Main strengths were: general medical knowledge of the digestive system; knowledge of medical skills relating to the skin; consultation skills concerning empathy; practice management with regard to accessibility. Main weaknesses were: general medical knowledge of the neurological system; knowledge of the medical/technical skills relating to the endocrine metabolic and nutritional system; consultation skills regarding shared decision making; practice management involving cooperation with staff and other care providers.

DISCUSSION

This first systematic evaluation of GP trainers identified their strengths and weaknesses. The weaknesses identified will be used in the improvement process as topics for collective improvement in the GP trainers' general curriculum and in individual learning plans.

Evaluation of a problem-based learning workshop using pre- and post-test objective structured clinical examinations and standardized patients

BACKGROUND

Osteoporosis is a health care issue in which family physicians play a major role. Although awareness of osteoporosis is high, recent studies suggest that application of recent advances in its treatment to the clinical setting may be low. We have developed a problem-based learning intervention for osteoporosis in which paired rheumatologists and family physicians developed nine problem-solving clinical scenarios. An educational matrix was used to link specific case scenarios with individual teaching objectives, developed via a previous needs assessment. Family physicians participated in the workshop, developing best practice responses to the clinical scenarios with a trained facilitator and content expert.

METHODS

To assess the impact of this intervention, family physicians participated in a pre- and post-test evaluation, using objective structured clinical examinations and standardized patients. Objective structured clinical examination stations tested knowledge, skills, and judgment relating to osteoporosis with respect to risk factors, use of appropriate investigations including bone mineral densitometry (BMD), strategies for the prevention of osteoporosis (both pharmacologic and nonpharmacologic), treatment options for established osteoporosis (bisphosphonates and hormone replacement therapy), and management of recent osteoporosis fracture. Participants were evaluated using a predetermined score generated by their responses to objective structured clinical examinations and standardized patients (max. score = 101). Evaluations were conducted anonymously, although participants had access to their own pre- and post-test results for personal feedback. The impact of the workshop was assessed by comparing pre- and post-test responses by group, by individual, and by station.

RESULTS

Participants demonstrated a significant improvement in their post-workshop scores. Of 40 participants, 26 showed improvement in score (> +10), 13 showed modest change (+1 to +10), and 1 showed a marked decrease (> -10). The greatest improvements were seen in the management of the male osteoporosis patient, determination of risk factors for osteoporosis, and the use and interpretation of bone mineral densitometry. Family physicians reported general satisfaction with the content and format of both the workshop and the evaluation process.

IMPLICATIONS

We conclude that this type of problem-based learning intervention workshop results in improved knowledge, skills, and judgment in the management of osteoporosis by family physicians as objectively assessed using a pre- and post-test format including objective structured clinical examinations and standardized patients.

Assessment in general practice: the predictive value of written-knowledge tests and a multiple-station examination for actual medical performance in daily practice

This study compares the predictive values of written-knowledge tests and a standardized multiple-station examination for the actual medical performance of general practitioners (GPs) in order to select effective assessment methods to be used in quality-improvement activities. A comprehensive assessment was performed in four phases. First, 100 GPs from the southern part of the Netherlands were assessed by a general medical knowledge test and by a knowledge test on technical skills. Second, in order to check for time-order effects, participants were randomly divided into two groups of 50 each, comparable on scores of both knowledge tests and on professional characteristics. Finally, both groups went through a multiple station examination using standardized patients and a practice video assessment of real surgery, but in opposite orders. Consultations were videotaped and assessed by well-trained peer observers. The drop-out rate was 10%. In both groups the predictive value of medical knowledge tests, ranging from 0.43 to 0.56 (Pearson correlation disattenuated), proved to be comparable with the predictive value of the multiple-station examination for actual performance (0.33-0.59). The overall explained variance of scores of the practice video assessment, measured by multiple regression analysis with performance scores as dependent variables and scores on the knowledge tests and the multiple-station examination as independent variables was moderate (19%). A time-order effect showed in only one direction: from practice video assessment to the multiple-station examination. The GP's professional characteristics did not contribute to the explanation of variation in performance. Medical knowledge tests can predict actual clinical performance to the same extent as a multiple-station examination. Compared with a station examination, a knowledge test may be a good alternative method for assessment the procedures of a large number of practising GPs.