Assessing practicing physicians in two settings using standardized patients

Competency-Based Education and Practice in Physical Therapy: It's Time to Act!

Competency-based education (CBE) is a concept, a philosophy, and an approach to educational design where learner progression occurs when competency is demonstrated. It assumes a set of standard defined performance outcomes for any level of professional practice-students, residents, or practicing physical therapists. Those outcomes are based on the health needs of society and guide the curricular design, implementation, and evaluation of health professions education programs. Lack of a CBE framework-with no required demonstration of competence throughout one's career-has the potential to lead to variation in physical therapists' skills and to unwarranted variation in practice, potentially hindering delivery of the highest quality of patient care. CBE requires a framework that includes a commonly understood language; standardized, defined performance outcomes at various stages of learner development; and a process to assess whether competence has been demonstrated. The purpose of this perspective article is to (1) highlight the need for a shared language, (2) provide an overview of CBE and the impetus for the change, (3) propose a shift toward CBE in physical therapy, and (4) discuss the need for the profession to adopt a mindset requiring purposeful practice across one's career to safely and most efficiently practice in a given area. Utilizing a CBE philosophy throughout one's career should ensure high-quality and safe patient care to all-patient care that can adapt to the changing scope of physical therapist practice as well as the health care needs of society. The physical therapy profession is at a point at which we must step up the transition to a competency-based system of physical therapist education.

FAMULATUR PLUS - A successful model for improving students' physical examination skills?

Introduction/Project description: Several studies have revealed insufficient physical examination skills among medical students, both with regard to the completeness of the physical examination and the accuracy of the techniques used. FAMULATUR PLUS was developed in response to these findings. As part of this practice-oriented instructional intervention, physical examination skills should be taught through examination seminars and problem-oriented learning approaches. In order to ensure practical relevance, all courses are integrated into a 30-day clinical traineeship in the surgery or internal medicine department of a hospital (FAMULATUR PLUS). Research question: Does participation in the FAMULATUR PLUS project lead to a more optimistic self-assessment of examination skills and/or improved performance of the physical examination? Methodology: A total of 49 medical students participated in the study. The inclusion criteria were as follows: enrollment in the clinical studies element of their degree program at the University of Ulm and completion of the university course in internal medicine examinations. Based on their personal preferences, students were assigned to either the intervention (surgery/internal medicine; n=24) or the control group (internal medicine; n=25). All students completed a self-assessment of their physical examination skills in the form of a questionnaire. However, practical examination skills were only assessed in the students in the intervention group. These students were asked to carry out a general physical examination of the simulation patient, which was recorded and evaluated in a standardized manner. In both instances, data collection was carried out prior to and after the intervention. Results: The scores arising from the student self-assessment in the intervention (IG) and control groups (CG) improves significantly in the pre-post comparison, with average scores increasing from 3.83 (±0.72; IG) and 3.54 (±0.37; CG) to 1.92 (±0.65; IG) and 3.23 (±0.73; CG). The general physical examination, which was only assessed among the students in the intervention group, was performed more completely after the instructional intervention than prior to it. Discussion: On the basis of the data collected, it can be deduced that the FAMULATUR PLUS course has a positive effect on the self-assessment of medical students with regard to their physical examination skills. The validity of this conclusion is limited by the small sample size. In addition, it remains unclear whether a more positive self-assessment correlates with an objective improvement in physical examination skills.

Community Pharmacist Practices in Hypertension Management

Objective:

To determine the current practices of community pharmacists in the management of hypertension.

Methods:

This was a cross-sectional, observational study of pharmacist practice using unannounced standardized patients (SPs) with hypertension visiting a random stratified sample of 101 community pharmacies in Edmonton, Alberta. Consent was not obtained from pharmacists.

Results:

Knowledge of current blood pressure guidelines and target values: Of the 101 community pharmacists who were visited by the SPs, 69% offered a general blood pressure target value (<120/80 mm Hg); only 7% stated the correct target blood pressure value for the SPs' particular scenario (<140/90 mm Hg). Only 14% requested enough patient history to properly determine target blood pressure. Review of medical history: Few pharmacists questioned the SPs about their medical history (7%), medication profile (16%), family history of cardiovascular disease (19%), previous elevated blood pressure readings (20%), or previous diagnosis of hypertension (22%). Accuracy/confirmation of blood pressure reading: 53% of pharmacists inquired about the conditions under which the blood pressure reading had been taken; 39% of pharmacists offered to retake the patient's blood pressure. Patient education: Pharmacists discussed how hypertension is diagnosed (76%), what hypertension is (46%), how to take a blood pressure reading properly (46%), and the impact of lifestyle measures on blood pressure (60%); they also gave supplemental educational materials (29%). Referral: 83% of pharmacists advised the SPs to make an appointment to see a physician.

Conclusions:

Pharmacists took reasonable steps to determine the accuracy of the blood pressure measurement, explain the diagnosis of hypertension, and refer to a physician. Major deficiencies were observed in assessment of target blood pressure and review of medical history. Pharmacists, alone or in collaboration with other health professionals, are urgently needed to play a major role in identifying, screening, and managing individuals with hypertension. Can Pharm J 2006;139(5):38–44.

Student performance of the general physical examination in internal medicine: an observational study

BACKGROUND

Many practicing physicians lack skills in physical examination. It is not known whether physical examination skills already show deficiencies after an early phase of clinical training. At the end of the internal medicine clerkship students are expected to be able to perform a general physical examination in every new patient encounter. In a previous study, the basic physical examination items that should standardly be performed were set by consensus. The aim of the current observational study was to assess whether medical students were able to correctly perform a general physical examination regarding completeness as well as technique at the end of the clerkship internal medicine.

METHODS

One hundred students who had just finished their clerkship internal medicine were asked to perform a general physical examination on a standardized patient as they had learned during the clerkship. They were recorded on camera. Frequency of performance of each component of the physical examination was counted. Adequacy of performance was determined as either correct or incorrect or not assessable using a checklist of short descriptions of each physical examination component. A reliability analysis was performed by calculation of the intra class correlation coefficient for total scores of five physical examinations rated by three trained physicians and for their agreement on performance of all items.

RESULTS

Approximately 40% of the agreed standard physical examination items were not performed by the students. Students put the most emphasis on examination of general parameters, heart, lungs and abdomen. Many components of the physical examination were not performed as was taught during precourses. Intra-class correlation was high for total scores of the physical examinations 0.91 (p <0.001) and for agreement on performance of the five physical examinations (0.79-0.92 p <0.001).

CONCLUSIONS

In conclusion, performance of the general physical examination was already below expectation at the end of the internal medicine clerkship. Possible causes and suggestions for improvement are discussed.

Realizing the promise and importance of performance-based assessment

Work-based assessment (WBA) is the assessment of trainees and physicians across the educational continuum of day-to-day competencies and practices in authentic, clinical environments. What distinguishes WBA from other assessment modalities is that it enables the evaluation of performance in context. In this perspective, we describe the growing importance, relevance, and evolution of WBA as it relates to competency-based medical education, supervision, and entrustment. Although a systematic review is beyond the purview of this perspective, we highlight specific methods and needed shifts to WBA that (a) consider patient outcomes, (b) use nonphysician assessors, and (c) assess the care provided to populations of patients. We briefly describe strategies for the effective implementation of WBA and identify outstanding research questions related to its use.

Making sense of work-based assessment: ask the right questions, in the right way, about the right things, of the right people

CONTEXT

Historically, assessments have often measured the measurable rather than the important. Over the last 30 years, however, we have witnessed a gradual shift of focus in medical education. We now attempt to teach and assess what matters most. In addition, the component parts of a competence must be marshalled together and integrated to deal with real workplace problems. Workplace-based assessment (WBA) is complex, and has relied on a number of recently developed methods and instruments, of which some involve checklists and others use judgements made on rating scales. Given that judgements are subjective, how can we optimise their validity and reliability?

METHODS

This paper gleans psychometric data from a range of evaluations in order to highlight features of judgement-based assessments that are associated with better validity and reliability. It offers some issues for discussion and research around WBA. It refers to literature in a selective way. It does not purport to represent a systematic review, but it does attempt to offer some serious analyses of why some observations occur in studies of WBA and what we need to do about them.

RESULTS AND DISCUSSION

Four general principles emerge: the response scale should be aligned to the reality map of the judges; judgements rather than objective observations should be sought; the assessment should focus on competencies that are central to the activity observed, and the assessors who are best-placed to judge performance should be asked to participate.

Good questions, good answers: construct alignment improves the performance of workplace-based assessment scales

CONTEXT

Assessment in the workplace is important, but many evaluations have shown that assessor agreement and discrimination are poor. Training discussions suggest that assessors find conventional scales invalid. We evaluate scales constructed to reflect developing clinical sophistication and independence in parallel with conventional scales.

METHODS

A valid scale should reduce assessor disagreement and increase assessor discrimination. We compare conventional and construct-aligned scales used in parallel to assess approximately 2000 medical trainees by each of three methods of workplace-based assessment (WBA): the mini-clinical evaluation exercise (mini-CEX); the acute care assessment tool (ACAT), and the case-based discussion (CBD). We evaluate how scores reflect assessor disagreement (V(j) and V(j*p) ) and assessor discrimination (V(p) ), and we model reliability using generalisability theory.

RESULTS

In all three cases the conventional scale gave a performance similar to that in previous evaluations, but the construct-aligned scales substantially reduced assessor disagreement and substantially increased assessor discrimination. Reliability modelling shows that, using the new scales, the number of assessors required to achieve a generalisability coefficient ≥0.70 fell from six to three for the mini-CEX, from eight to three for the CBD, from 10 to nine for 'on-take' ACAT, and from 30 to 12 for 'post-take' ACAT.

CONCLUSIONS

The results indicate that construct-aligned scales have greater utility, both because they are more reliable and because that reliability provides evidence of greater validity. There is also a wider implication: the disappointing reliability of existing WBA methods may reflect not assessors' differing assessments of performance, but, rather, different interpretations of poorly aligned scales. Scales aligned to the expertise of clinician-assessors and the developing independence of trainees may improve confidence in WBA.

Prospective observational study to evaluate NOTSS (Non-Technical Skills for Surgeons) for assessing trainees' non-technical performance in the operating theatre

BACKGROUND

Most surgical assessment has been aimed at technical proficiency. However, non-technical skills also affect patient safety and clinical effectiveness. The NOTSS (Non-Technical Skills for Surgeons) assessment instrument was developed specifically to assess the non-technical skills of individual surgeons in the operating theatre. This study evaluated NOTSS as a real-world assessment, with a mix of minimally trained assessors. The evaluation criteria were feasibility, validity and psychometric reliability.

METHODS

In a standard evaluation of NOTSS, 56 anaesthetists, 39 scrub nurses, two surgical care practitioners and three independent assessors provided 715 assessments of 404 surgical cases of 15 index procedures across six specialties performed by 85 surgical trainees.

RESULTS

The assessment was feasible, but important implementation challenges were highlighted. Most respondents considered the method valid, but with reservations about assessing cognition. The factor structure of scores, and their positive relationships with other measures of experience and performance, supported validity. Trainees' non-technical skill scores were relatively procedure-independent and achieved good reliability (generalizability coefficient 0·8 or more) when six to eight assessors observed one case each.

CONCLUSION

Minimally trained assessors, who are typically present in operating theatres, were sufficiently discriminating and consistent in their judgements of trainee surgeons' non-technical skills to provide reliable scores based on an achievable number of observations.

A pilot study assessing knowledge of clinical signs and physical examination skills in incoming medicine residents

BACKGROUND

Physical exam skills of medical trainees are declining, but most residencies do not offer systematic clinical skills teaching or assessment.

OBJECTIVE

To assess knowledge of clinical signs and physical exam performance among incoming internal medicine residents.

METHOD

For this study, 45 incoming residents completed a multiple choice question test to assess knowledge of clinical signs. A random selection of 20 underwent a faculty-observed objective structured clinical examination (OSCE) using patients with abnormal physical findings. Mean percentage scores were computed for the multiple choice question test, overall OSCE, and the 5 individual OSCE systems.

RESULTS

The mean scores were 58.4% (14.6 of 25; SD 11. 5) for the multiple choice question test and 54.7% (31.7 of 58; SD 11.0) for the overall OSCE. Mean OSCE scores by system were cardiovascular 30.0%, pulmonary 69.2%, abdominal 61.6%, neurologic 67.0%, and musculoskeletal 41.7%. Analysis of variance showed a difference in OSCE system scores (P < .001) with cardiovascular and musculoskeletal scores significantly lower than other systems.

CONCLUSION

Overall, physical exam knowledge and performance of new residents were unsatisfactory. There appears to be a pressing need for additional clinical skills training during medical school and residency training and we are planning a new clinical skills curriculum to address this deficiency.

Adoption of the Healthy Heart Kit by Alberta family physicians

OBJECTIVE

The Healthy Heart Kit (HHK) is a risk management and patient education kit for the prevention of cardiovascular disease (CVD) and the promotion of CV health. There are currently no published data examining predictors of HHK use by physicians. The main objective of this study was to examine the association between physicians' characteristics (socio-demographic, cognitive, and behavioural) and the use of the HHK.

METHODS

All registered family physicians in Alberta (n=3068) were invited to participate in the "Healthy Heart Kit" Study. Consenting physicians (n=153) received the Kit and were requested to use it for two months. At the end of this period, a questionnaire collected data on the frequency of Kit use by physicians, as well as socio-demographic, cognitive, and behavioural variables pertaining to the physicians.

RESULTS

The questionnaire was returned by 115 physicians (follow-up rate = 75%). On a scale ranging from 0 to 100, the mean score of Kit use was 61 [SD=26]. A multiple linear regression showed that "agreement with the Kit" and the degree of "confidence in using the Kit" was strongly associated with Kit use, explaining 46% of the variability for Kit use. Time since graduation was inversely associated with Kit use, and a trend was observed for smaller practices to be associated with lower use.

CONCLUSION

Given these findings, future research and practice should explore innovative strategies to gain initial agreement among physicians to employ such clinical tools. Participation of older physicians and solo-practitioners in this process should be emphasized.

Audio taping simulated patient encounters in community pharmacy to enhance the reliability of assessments

OBJECTIVES

To assess whether audio taping simulated patient interactions can improve the reliability of manually documented data and result in more accurate assessments.

METHODS

Over a 3-month period, 1340 simulated patient visits were made to community pharmacies. Following the encounters, data gathered by the simulated patient were relayed to a coordinator who completed a rating form. Data recorded on the forms were later compared to an audiotape of the interaction. Corrections were tallied and reasons for making them were coded.

RESULTS

Approximately 10% of cases required corrections, resulting in a 10%-20% modification in the pharmacy's total score. The difference between postcorrection and precorrection scores was significant.

CONCLUSIONS

Audio taping simulated patient visits enhances data integrity. Most corrections were required because of the simulated patients' poor recall abilities.

Long-term retention of smoking cessation counseling skills learned in the first year of medical school

BACKGROUND

Tobacco use is a significant cause of preventable morbidity and mortality in the United States, yet clinicians underutilize smoking cessation counseling. Medical schools are increasingly including training for smoking cessation skills in preclinical curricula. Information about long-term retention of these skills is needed.

OBJECTIVE

To assess retention of smoking cessation counseling skills learned in the first year of medical school.

DESIGN

Retrospective review of data collected for routine student and curriculum assessment.

PARTICIPANTS

Two cohorts of medical students at the University of Connecticut School of Medicine (total N = 112) in 1999-2001 and 2002-2004.

MEASUREMENTS AND MAIN RESULTS

Scores by standardized patients were compared from first and fourth-year assessments, based on checklist items corresponding to the 5 strategies recommended by the U.S. Public Health Service (Ask, Advise, Assess, Assist, Arrange). In study cases, 97% of first-year students "asked" about smoking and retained this skill in fourth year (p = .08). Ninety-four percent of first-year students "assessed" readiness to quit and retained this skill (p = .21). Ninety-six percent of first-year students "advised" smokers to quit and retained this skill (p = .18). Eighty-six percent of first year students "assisted" smokers in quitting and retained this skill (p = 0.10). Eighty-one percent of first year students "arranged" follow-up contact and performance of this strategy improved in the fourth year to 91% (p = .03).

CONCLUSIONS

Smoking cessation counseling skills demonstrated by first year medical students were, with brief formal reinforcement in the third year, well retained into the fourth year of medical school. It is appropriate to begin this training early in medical education.

Assessment of patient management skills and clinical skills of practising doctors using computer-based case simulations and standardised patients

CONTEXT

Standardised assessments of practising doctors are receiving growing support, but theoretical and logistical issues pose serious obstacles.

OBJECTIVES

To obtain reference performance levels from experienced doctors on computer-based case simulation (CCS) and standardised patient-based (SP) methods, and to evaluate the utility of these methods in diagnostic assessment.

SETTING AND PARTICIPANTS

The study was carried out at a military tertiary care facility and involved 54 residents and credentialed staff from the emergency medicine, general surgery and internal medicine departments.

MAIN OUTCOME MEASURES

Doctors completed 8 CCS and 8 SP cases targeted at doctors entering the profession. Standardised patient performances were compared to archived Year 4 medical student data.

RESULTS

While staff doctors and residents performed well on both CCS and SP cases, a wide range of scores was exhibited on all cases. There were no significant differences between the scores of participants from differing specialties or of varying experience. Among participants who completed both CCS and SP testing (n = 44), a moderate positive correlation between CCS and SP checklist scores was observed. There was a negative correlation between doctor experience and SP checklist scores. Whereas the time students spent with SPs varied little with clinical task, doctors appeared to spend more time on communication/counselling cases than on cases involving acute/chronic medical problems.

CONCLUSION

Computer-based case simulations and standardised patient-based assessments may be useful as part of a multimodal programme to evaluate practising doctors. Additional study is needed on SP standard setting and scoring methods. Establishing empirical likelihoods for a range of performances on assessments of this character should receive priority.

Schistosomiasis control in Ghana: case management and means for diagnosis and treatment within the health system

An essential component of integrated schistosomiasis control as promoted by WHO is adequate clinical care for patients presenting at health care facilities. We evaluated the functioning of the Ghanaian health system for diagnosis and treatment of schistosomiasis by interviewing health workers from 70 health care facilities in 4 geographical areas in April and May 2000. Results from presentation of 4 hypothetical cases and a subsequent interview demonstrated that patients presenting with symptoms related to schistosomiasis have a small chance of receiving adequate treatment: often health workers do not recognize the symptoms, especially those of Schistosoma mansoni; patients are frequently referred for a diagnostic test or treatment with a large risk of non-compliance; and praziquantel was not available in 78% of the health care facilities with reported schistosomiasis in their coverage area. The overall cost of treatment is considerable: [symbol: see text] 2.13 for S. haematobium and [symbol: see text] 1.81 for S. mansoni patients, with drug costs contributing approximately 40% of the total cost. To better meet WHO recommendations for passive case detection as part of integrated schistosomiasis control, the Ghanaian health system needs to emphasize training of health workers in schistosomiasis case recognition and case management and increase the availability of praziquantel. Experience from other West African countries indicate that this is feasible.

Faculty and the observation of trainees' clinical skills: problems and opportunities

The clinical skills of medical interviewing, physical examination, and counseling remain vital to the effective care of patients, yet research continues to document serious deficiencies in clinical skills among students and residents. The most important method of evaluation is the direct observation of trainees performing these clinical skills. Standardized patients and other simulation technologies are important and reliable tools for teaching clinical skills and evaluating competence and will be incorporated in the near future as part of the United States Medical Licensing Examination. Standardized patients and simulation, however, cannot and should not replace the direct observation by faculty of trainees' clinical skills with actual patients. Faculty are in the best position to document improvement over time and to certify trainees have attained sophisticated levels of skill in medical interviewing, physical examination, and counseling. Unfortunately, current evidence suggests significant deficiencies in faculty direct observation evaluation skills. The author outlines the nature of the problems in clinical skills and their evaluation by faculty and ends with recommendations to improve the current state of faculty skills in evaluation.

Measuring schistosomiasis case management of the health services in Ghana and Mali

The World Health Organization recommends passive case detection by regular health services as a minimum strategy for schistosomiasis morbidity control. To evaluate preparedness of the health systems in Ghana and Mali, we presented four clinical scenarios, two with blood in urine (main early symptom of Schistosoma haematobium) and two with (bloody) diarrhoea (main early symptom of S. mansoni), to health workers. We requested the health personnel for an initial diagnosis and case management strategy without providing information about our primary interest in schistosomiasis. The information was used to determine the chance that a person reporting with symptoms that might have been caused by schistosomiasis would receive praziquantel. All selected health workers participated. Their initial diagnosis was frequently S. haematobium for both scenarios with blood in urine. For the two scenarios with (bloody) diarrhoea, only few mentioned S. mansoni. At health centre level, case management in Mali mainly consisted of direct prescription of medication, whereas in Ghana health workers often referred to a hospital or requested a diagnostic test. The ultimate probability of prescribing praziquantel was relatively high for the scenarios with blood in urine, 60% in Ghana and 75% in Mali, but very low for both scenarios with (bloody) diarrhoea (<20%). Of those health care facilities that would prescribe praziquantel, 60% (Ghana) and 80% (Mali) had it in stock. In conclusion, the clinical scenario study showed that patients reporting with blood in urine will be treated with praziquantel at approximately half of the health care facilities, whereas of those presenting with (bloody) diarrhoea only few would receive treatment with praziquantel. Considering these facts, it is questionable if passive case detection is a sufficient basis for effective schistosomiasis morbidity control, especially for S. mansoni infection.

Cognitive, social and environmental sources of bias in clinical performance ratings

BACKGROUND

Global ratings based on observing convenience samples of clinical performance form the primary basis for appraising the clinical competence of medical students, residents, and practicing physicians. This review explores cognitive, social, and environmental factors that contribute unwanted sources of score variation (bias) to clinical performance evaluations.

SUMMARY

Raters have a 1 or 2-dimensional conception of clinical performance and do not recall details. Good news is reported more quickly and fully than bad news, leading to overly generous performance evaluations. Training has little impact on accuracy and reproducibility of clinical performance ratings.

CONCLUSIONS

Clinical performance evaluation systems should assure broad, systematic sampling of clinical situations; keep rating instruments short; encourage immediate feedback for teaching and learning purposes; encourage maintenance of written performance notes to support delayed clinical performance ratings; give raters feedback about their ratings; supplement formal with unobtrusive observation; make promotion decisions via group review; supplement traditional observation with other clinical skills measures (e.g., Objective Structured Clinical Examination); encourage rating of specific performances rather than global ratings; and establish the meaning of ratings in the manner used to set normal limits for clinical diagnostic investigations.

Assessment of clinical skills in medical practice

The introduction of a clinical skills examination (CSE) to Step 2 of the U.S. Medical Licensing Examination (USMLE) has focused attention on the design and delivery of large-scale standardized tests of clinical skills and raised the question of the appropriateness of evaluation of these competencies across the span of a physician's career. This initiative coincides with growing pressure to periodically assess the continued competence of physicians in practice. The USMLE CSE is designed to certify that candidates have the basic clinical skills required for the safe and effective practice of medicine in the supervised environment of postgraduate training. These include history taking, physical examination, effective communication with patients and other members of the health care team, and clear and accurate documentation of diagnostic impressions and plans for further assessment. The USMLE CSE does not assess procedural skills. As physicians progress through training and enter practice, both knowledge base and requisite technical skills become more diverse. A variety of indirect and direct measures are available for evaluating physicians, but, at present, no single method permits high-stake inferences about clinical skills. Systematic and standardized assessments make a contribution to comprehensive evaluations, but they retain an element of assessing capacity rather than authentic performance in practice. Much work is needed to identify the optimal combination of methods to be employed in support of programs to ensure maintenance of competence of practicing physicians.

Using standardised patients to measure physicians' practice: validation study using audio recordings

OBJECTIVE

To assess the validity of standardised patients to measure the quality of physicians' practice.

DESIGN

Validation study of standardised patients' assessments. Physicians saw unannounced standardised patients presenting with common outpatient conditions. The standardised patients covertly tape recorded their visit and completed a checklist of quality criteria immediately afterwards. Their assessments were compared against independent assessments of the recordings by a trained medical records abstractor.

SETTING

Four general internal medicine primary care clinics in California.

PARTICIPANTS

144 randomly selected consenting physicians.

MAIN OUTCOME MEASURES

Rates of agreement between the patients' assessments and independent assessment.

RESULTS

40 visits, one per standardised patient, were recorded. The overall rate of agreement between the standardised patients' checklists and the independent assessment of the audio transcripts was 91% (kappa=0.81). Disaggregating the data by medical condition, site, level of physicians' training, and domain (stage of the consultation) gave similar rates of agreement. Sensitivity of the standardised patients' assessments was 95%, and specificity was 85%. The area under the receiver operator characteristic curve was 90%.

CONCLUSIONS

Standardised patients' assessments seem to be a valid measure of the quality of physicians' care for a variety of common medical conditions in actual outpatient settings. Properly trained standardised patients compare well with independent assessment of recordings of the consultations and may justify their use as a "gold standard" in comparing the quality of care across sites or evaluating data obtained from other sources, such as medical records and clinical vignettes.

Using standardized patients to measure quality: evidence from the literature and a prospective study

BACKGROUND

Use of standardized patients for evaluating the clinical skills of medical students and medical trainees is commonplace. This has encouraged the use of standardized patients to evaluate the quality of physician practice in outpatient settings. However, there may be substantive differences between observing student performance and evaluating whether the provision of care meets defined quality criteria.

OBJECTIVES

This study had two primary objectives: (1) to review studies that use standardized patients to evaluate physician performance and (2) to ascertain directly whether standardized patients could be useful in assessing quality of outpatient care.

METHODS

A comprehensive literature review of studies that used standardized patients to assess physician performance was conducted. A prospective study that included 20 physicians at two outpatient settings and 27 actor patients assessed quality of care using eight clinical cases divided into five clinical domains, each of which had explicit criteria checklists based on widely accepted guidelines.

RESULTS

The literature review identified five important issues: developing scenarios, selecting explicit criteria, standardizing standardized patient training, creating subterfuges, and ensuring reliability and validity of measures. In the study, trained standardized patients were able to assess physician practice accurately for common medical conditions, using proven criteria linked to health outcomes. The detection rate was 3%. There was no performance variation between actors for seven of the eight cases.

CONCLUSIONS

Using standardized patients to measure the quality of care is practical and feasible. The major methodological challenge is incorporating observable evidence-based criteria into realistic scripts and objective checklists. The major logistical challenge is obtaining and maintaining undetected entry into physicians' offices.

History-taking and preventive medicine skills among primary care physicians: an assessment using standardized patients

BACKGROUND

The ability of primary care physicians to obtain important clinical information in initial encounters with new patients is a core competency that has received little attention in previous studies. This paper describes the history-taking and preventive screening skills of practicing primary care physicians in initial interactions with ambulatory patients, as determined by a large panel of standardized patients.

METHODS

Standardized patient cases with diverse presentations were developed and used to assess the clinical skills of 134 primary care physicians from five Northwest states. Scoring categories for each case identified the percentage and content of essential history items and preventive screening items performed. Physicians' scores were compared by training and practice characteristics.

RESULTS

Physicians asked 59% of essential history items. They frequently obtained appropriate information about presenting symptoms and medications, but they often missed important information about related symptoms and medical history. Physicians frequently screened for smoking and alcohol use, but rarely asked about recreational drug use. Although board-certified general internists performed more comprehensive histories than board-certified family practitioners in the same amount of time, both groups of providers missed a large number of items that should have been influential in developing diagnostic and treatment plans.

CONCLUSIONS

Primary care physicians may miss important patient information in their initial interactions with patients. Medical intake questionnaires or other approaches should be considered to ensure that more complete and accurate information is available to guide diagnostic and treatment plans.

The development of a 'Standardised Learner' in researching teaching behaviours

Standardised patients are increasingly used in medical education and the reliability and validity of their use as an assessment method are supported by published research. This paper details the application of this methodology to the educational interaction between a teacher and a learner using a standardised learner.In contrast with standardised patients, a standardised learner must use a level of medical knowledge and experience appropriate for the role assumed. Some aspects of reliability and validity have been considered and appear supportive of this methodology.The feasibility and value of this method for researching actual teacher behaviour are discussed.

Is the public being protected? Prevention of suboptimal medical practice through training programs and credentialing examinations

Governments have traditionally looked to the medical profession for leadership in health planning and have charged the profession with the responsibility of establishing and monitoring standards of medical practice. Training program accreditation and licensure/certification exams have been used as the primary methods of preventing unqualified individuals from entering medical practice. Despite the critical nature of the decision made at the time of licensure/certification, there is no information about the validity of these examinations for predicting subsequent practice and health outcome. In this article, the assumptions implicit in the current use of licensing/certifying examinations are identified, the relevant evidence is reviewed, and the implications of this evidence for current methods of measurement are discussed.