Development of an objective structured clinical examination (OSCE) for physical medicine and rehabilitation residents

Assessing Physical Medicine and Rehabilitation Residency Education Using the Neurological Assessment Competency Evaluation System

ABSTRACT

Competent neurologic examination and clinical skills are essential components in the care for patients in acute hospital and rehabilitation settings. To enhance the evaluation and education of Physical Medicine and Rehabilitation residents, the authors developed an educational objective structured clinical examination, the Neurological Exam Assessment Competency Evaluation System, and gathered 2 yrs of baseline data. The Neurological Exam Assessment Competency Evaluation System consisted of nine 9-min examination stations, seven with written clinical scenario with instructions for junior residents to complete the appropriate examination (stations: Altered Mental Status, Mild Traumatic Brain Injury, Dementia, Stroke, Falls, and the International Standards for Neurological Classification of Spinal Cord Injury Sensory and Motor Examinations). Examinees provided written responses to posed questions for the other two stations-Modified Ashworth Scale and brachial plexus. The assessment tools for this examination were designed for residency programs to evaluate the basic competencies as outlined by the Accreditation Council for Graduate Medical Education and Physical Medicine and Rehabilitation milestones. Based on the feedback received from the examinees and examiners, the Neurological Exam Assessment Competency Evaluation System can serve as an educational objective structured clinical examination for the improvement of trainee core competencies.

Reaching Consensus on Measuring Professional Behaviour in Physical Therapy Objective Structured Clinical Examinations

Purpose: We determined which professional behaviours (PBs) are important and feasible to measure in an objective structured clinical examination (OSCE) intended to assess the hands-on skills and knowledge of students in a Canadian physical therapy (PT) program. Methods: We used a modified Delphi technique to identify the criteria required to assess PBs in PT students during an OSCE. We conducted a focus group to better understand the results of the modified Delphi process. Results: Experienced local OSCE examiners participated in the modified Delphi panel, which consisted of two rounds of surveys: round 1 (n=12) and round 2 (n=10). A total of 31 PBs were reduced to 18 through the two rounds. Five of the panellists participated in the focus group, reduced the 18 PBs to 15, and then identified 4 as clinical skills. Participants categorized the remaining 11 as mixed PBs and clinical skills (1 item), PBs (4 items), or communication skills (6 items). Conclusion: This study provides preliminary evidence to support the feasibility and importance of evaluating 5 PB items in practical skills OSCEs for entry-to-practice PT students.

Expert panel consensus on assessment checklists for a rheumatology objective structured clinical examination

OBJECTIVE

While several regional fellowship groups conduct rheumatology objective structured clinical examinations (ROSCEs), none have been validated for use across programs. We aimed to establish agreement among subspecialty experts regarding checklist items for several ROSCE stations.

METHODS

We administered a 1-round survey to assess the importance of 173 assessment checklist items for 11 possible ROSCE stations. We e-mailed the survey to 127 rheumatology educators from across the US. Participants rated each item's importance on a 5-point Likert scale (1 = not important to 5 = very important). Consensus for high importance was predefined as a lower bound of the 95% confidence interval ≥4.0.

RESULTS

Twenty-five individuals (20%) completed the expert panel survey. A total of 133 of the 173 items (77%) met statistical cutoff for consensus to retain. Several items that had population means of ≥4.0 but did not meet the predetermined definition for consensus were rejected. The percentage of retained items for individual stations ranged from 24% to 100%; all items were retained for core elements of patient counseling and radiograph interpretation tasks. Only 24% of items were retained for a rehabilitation medicine station and 60% for a microscope use/synovial fluid analysis station.

CONCLUSION

This single-round expert panel survey established consensus on 133 items to assess on 11 proposed ROSCE stations. The method used in this study, which can engage a diverse geographic representation and employs rigorous statistical methods to establish checklist content agreement, can be used in any medical field.

Measuring professional behaviour in canadian physical therapy students' objective structured clinical examinations: an environmental scan

PURPOSE

To identify professional behaviours measured in objective structured clinical examinations (OSCEs) by Canadian university physical therapy (PT) programs.

METHOD

A cross-sectional telephone survey was conducted to review current practice and determine which OSCE items Canadian PT programs are using to measure PT students' professional behaviours. Telephone interviews using semi-structured questions were conducted with individual instructors responsible for courses that included an OSCE as part of the assessment component.

RESULTS

Nine PT programmes agreed to take part in the study, and all reported conducting at least one OSCE. The number and characteristics of OSCEs varied both within and across programs. Participants identified 31 professional behaviour items for use in an OSCE; these items clustered into four categories: communication (n=14), respect (n=10), patient safety (n=4), and physical therapists' characteristics (n=3).

CONCLUSIONS

All Canadian entry-level PT programmes surveyed assess professional behaviours in OSCE-type examinations; however, the content and style of assessment is variable. The local environment should be considered when determining what professional behaviours are appropriate to assess in the OSCE context in individual programmes.

Competency-based training: objective structured clinical exercises (OSCE) in marriage and family therapy

The field of marriage and family therapy (MFT) has recently engaged in the process of defining core competencies for the profession. Many MFT training programs are adapting their curriculum to develop more competency-based training strategies. The Objective Structured Clinical Examination (OSCE) is widely used in the medical profession to assess clinical competence. These examinations involve using simulated clinical situations as a tool in conducting summative evaluations of trainee competence. This article describes an adaptation of the OSCE procedures for competency-based training of MFT students. Instead of using the procedures as a summative examination as is typical in medical education, this article proposes how to use them as formative exercises in the development of student competence. The development of the OSCE is discussed, including "blueprinting," focused competencies, procedures, and feedback protocols. The article concludes with suggestions of how to continue the development of the OSCE for evaluation in MFT education.

Development, impact, and measurement of enhanced physical diagnosis skills

Evidence suggests that the quality and frequency of bedside clinical examination have declined. We undertook the study to (1) determine whether intensive instruction in physical examination enhances medical student skills and (2) develop a tool to evaluate those skills using a modified observed structured clinical examination (OSCE). This was a randomized, blinded, prospective, year-long study involving 3rd year students at the Albert Einstein College of Medicine. Students were randomized to receive intensive instruction in physical examination [study group (n = 46)] or usual instruction [control group (n = 75)] and evaluated by a modified OSCE. The OSCE consisted of 6 real patient stations: Head, ears, eyes, neck, throat; pulmonary; cardiovascular; gastrointestinal; neurology; musculoskeletal; and 2 computer imaging stations: genitourinary and dermatology. A faculty member present at each patient station evaluated student performance. Data were analyzed using t-tests for comparison of the mean scores between the two groups for each station and for average scores across stations. A total of 121 students were tested. The study group performed significantly better than the control group in the gastrointestinal station (p = 0.0004), the combined average score across the six real patient stations (p = 0.0001), and the combined average score across all eight stations (p = 0.0014). Intensive physical diagnosis instruction enhances physical examination skills of 3rd year medical students. The modified OSCE is a useful tool to evaluate these skills.

A national program director survey of the shift to competency-based education in ophthalmology

PURPOSE

To assess the resources, progress, and barriers for program director (PD) compliance with the Accreditation Council for Graduate Medical Education (ACGME) mandate.

DESIGN

Survey.

PARTICIPANTS

Ophthalmology PDs in the United States.

METHODS

A survey instrument was sent to all ophthalmology PDs in the United States.

MAIN OUTCOME MEASURES

The survey instrument assessed PD tenure, funding, current efforts, evaluation modalities, barriers, and desired resources to meet the ACGME competency mandate.

RESULTS

The survey was completed by 88 of 104 (85%) PDs. Significant disparities exist between PD financial compensation and actual time spent performing PD duties. This disparity and lack of understanding by faculty of the ACGME competencies were the most frequently cited barriers to success in complying with the ACGME mandate. The most commonly utilized assessment tools are the global or 360 degrees evaluations and clinical evaluation exercises.

CONCLUSIONS

Program directors surveyed believe they are not receiving adequate resources to allow them to comply with the ACGME mandate. Although some tools have been implemented by PDs for assessing the competencies, additional national and centralized resources would be helpful.

The Israel Center for Medical Simulation: a paradigm for cultural change in medical education

Simulation-based medical education (SBME) is a rapidly growing field, as is illustrated by the increased development of simulation centers worldwide. SBME is becoming a powerful force in addressing the need to increase patient safety through quality-care training. Recognizing the benefits of SBME, increasing numbers of bodies involved in medical and health care education and training are establishing simulation centers worldwide. The general model of most facilities focuses on a single simulation modality or a specific branch of medicine or health care, limiting their overall impact on patient safety and quality of care across the health care systems. MSR, the Israel Center for Medical Simulation, is a comprehensive, national, multimodality, multidisciplinary medical simulation center dedicated to enhancing hands-on medical education, performance assessment, patient safety, and quality of care by improving clinical and communication skills. The center uses an "error-driven" educational approach, which recognizes that errors provide an opportunity to create a unique beneficial learning experience. The authors present the Israeli experience as an alternative model, and describe the impact of the MSR model on the Israeli medical community during four years of activity. They also describe the opportunities this model has opened towards changing the culture of medical education and patient safety within Israel Although this model may require modification when implemented in other medical systems, it highlights important lessons regarding the power of SBME in triggering and bringing about cultural changes in traditional medical education.

Praktische Prüfungen in der Neurologie

Practically orientated examinations represent an important innovation in medical education. In the subject area of neurology, the "Objective Structured Clinical Examination" (OSCE) proved to be a viable form of examination for the subject of neurological anamnesis and a medical examination course at the University of Tuebingen in Germany. A total of 577 students in their sixth semester of study were examined in, to date, four OSCEs and achieved an average of 17.5 of possible 25 points in the subject of neurology. Improvement in point average was observable over the course of time, while the average performance of students across the total OSCE remained constant. Student evaluations regarding form and content of the neurological examinations yielded an average grade of B (in the German system 2.1). Simulation patients were rated with an average grade of A- (German system 1.7). Examination methods are presented together with experience concerning preparation, implementation, and developmental enhancement of neurological OSCE examinations. Implications of this novel form of examination for faculty development and regulating students' learning and study behaviour are discussed.

Teaching about Substance Abuse with Objective Structured Clinical Exams

BACKGROUND

Although residents commonly manage substance abuse disorders, optimal approaches to teaching these specialized interviewing and intervention skills are unknown.

OBJECTIVE

We developed a Substance Abuse Objective Structured Clinical Exam (OSCE) to teach addiction medicine competencies using immediate feedback. In this study we evaluated OSCE performance, examined associations between performance and self-assessed interest and competence in substance abuse, and assessed learning during the OSCE.

DESIGN

Five-station OSCE, including different substance abuse disorders and readiness to change stages, administered during postgraduate year-3 ambulatory rotations for 2 years.

PARTICIPANTS

One hundred and thirty-one internal and family medicine residents.

MEASUREMENTS

Faculty and standardized patients (SPs) assessed residents' general communication, assessment, management, and global skills using 4-point scales. Residents completed a pre-OSCE survey of experience, interest and competence in substance abuse, and a post-OSCE survey evaluating its educational value. Learning during the OSCE was also assessed by measuring performance improvement from the first to the final OSCE station.

RESULTS

Residents performed better (P<.001) in general communication (mean+/-SD across stations=3.12+/-0.35) than assessment (2.65+/-0.32) or management (2.58+/-0.44), and overall ratings were lowest in the contemplative alcohol abuse station (2.50+/-0.83). Performance was not associated with residents' self-assessed interest or competence. Perceived educational value of the OSCE was high, and feedback improved subsequent performance.

CONCLUSIONS

Although internal and family medicine residents require additional training in specialized substance abuse skills, immediate feedback provided during an OSCE helped teach needed skills for assessing and managing substance abuse disorders.

Managing the new mandate in resident education: a blueprint for translating a national mandate into local compliance

OBJECTIVE

The Accreditation Council for Graduate Medical Education (ACGME) has mandated that all residency programs implement an assessment process of 6 core competencies. Assessment of surgical competence is also included in the mandate. We describe our local efforts to meet this new mandate.

DESIGN

Systematic literature review.

METHODS

A systematic MEDLINE search (1996-2003) of the literature on residency assessment tools was performed. All relevant titles were reviewed by a content expert, abstracts were selected, and all appropriate full articles were reviewed. The Department of Ophthalmology at the University of Iowa formalized the competency review process by forming an ad hoc departmental task force for "Meeting the Competencies" composed of clinicians, technical staff, education specialists, the program director, the director of residency curriculum, the medical student director, and residents.

RESULTS

The task force reviewed the available literature, reviewed potential best practices, and reached consensus on an implementation plan. The following specific criteria for the assessment process were proposed: (1) there should be multiple assessments by multiple observers using multiple tools at multiple time points, (2) the tools should be reliable, reproducible, and valid; (3) the tools must be practical (i.e., feasible, convenient, low time commitment, easy to use, and inexpensive to implement and maintain); (4) the tools must produce qualitative and quantitative data, with direct linkage to improvement in educational outcomes in the future; (5) the assessment process must be linked to explicit and public learning objectives; and (6) the grading scale should be open and clearly defined, and the process should be judged as fair and accurate by both faculty and residents. The Meeting the Competencies task force reviewed all of the available tools from the literature and recommended a pilot implementation matrix matching specific tools to individual competencies. The 6 pilot tools include (1) written and oral examinations, (2) a 360 degrees global evaluation form (using multiple observers from different perspectives, including nurses, technicians, fellow residents, and patients, to provide a wider assessment), (3) a resident portfolio, (4) direct observation of operative performance and clinical examination, (5) a phone encounter tool, and (6) a journal club tool.

CONCLUSION

We propose a potential blueprint for meeting the challenge of assessing the new ACGME competencies in ophthalmology and translating the national mandate into local compliance.

Beyond fulfilling the core competencies: An objective structured clinical examination to assess communication and interpersonal skills in a surgical residency

OBJECTIVE

The Accreditation Council for Graduate Medical Education (ACGME) has challenged program directors to assess their residents' core competencies, including communication and interpersonal skills (CIS). We report our institution's experience using a series of standardized patient encounters in an objective structured clinical examination (OSCE) to evaluate CIS in surgical residents.

METHODS

Standardized patients rated the residents' ability to maintain a patient-centered approach across 6 challenging communication tasks. Residents received verbal feedback from the patients after each encounter and completed a survey indicating their experience and comfort with each task. Individual and group reports documented resident competency and provided aggregate information for curriculum review. Formal grades were not assigned.

RESULTS

Twenty-two residents in 2 surgical residency programs piloted the assessment. The Generalizability of the assessment was 0.81. Scores of second- and third-year residents were not significantly different. Residents found the program to be helpful and able to assess their skills.

CONCLUSIONS

The standardized patient-based OSCE is an effective method to assess communication and interpersonal skills and provides useful information for curriculum review.

Evaluation of pelvic and breast examination skills of interns in obstetrics and gynecology and internal medicine

OBJECTIVE

The purpose of this study was to evaluate the pelvic and breast examination skills of a group of interns who were entering obstetrics and gynecology and internal medicine and to determine whether previous experience predicts performance.

STUDY DESIGN

This was a prospective study that, with the use of an examination with a standardized format of 26 skills, assessed the performance of 10 interns who were entering internal medicine and 9 interns who were entering obstetrics and gynecology. chi(2) analysis and Fisher exact tests were used.

RESULTS

There was no significant difference in overall performance on the pelvic and breast examinations between the two groups. The obstetrics and gynecology interns performed significantly better (P<.05) on the three skills that assessed professional conduct. There was considerable variability within each group. There was no correlation between previous clinical experience and performance on the skills assessment, nor was there a correlation between perceived competence and actual performance.

CONCLUSIONS

Previous clinical experience does not predict performance in a standardized assessment of pelvic and breast examination skills nor does it account for the great variability in competence levels in interns who are entering obstetrics and gynecology and internal medicine.

The New competencies and their impact on resident training in ophthalmology

Changes in the health care environment and the evolving needs of external stakeholders have created a demand for a new set of competencies in residency training. This article reviews the forces that have shaped the development of the new competencies; defines the specific Accreditation Council for Graduate Medical Education competencies; describes the tools that might be used to measure these competencies; introduces current concepts and terms in the field of post-graduate medical education; and summarizes an implementation plan for ophthalmology. Meeting the new Accreditation Council for Graduate Medical Education competencies will require the development of a toolbox for measurement outcomes and a reinvention of the residency training process in ophthalmology. A coordinated, thoughtful, and standardized effort will be required to meet the challenge.

Correlation of standardized testing results with success on the 2001 American Board of Physical Medicine and Rehabilitation Part 1 Board Certificate Examination

OBJECTIVE

To determine if standardized testing results and other factors correlate with success on the 2001 American Board of Physical Medicine and Rehabilitation part 1 board certification examination (POE).

DESIGN

An anonymous two-part survey was distributed to 302 senior resident physicians in the American College of Graduate Medicine Education-accredited physical medicine and rehabilitation training programs in the United States deemed eligible for the 2001 POE.

RESULTS

A total of 94 of 302 distributed surveys (31.1%) were returned; 86 met the inclusion criteria (81 passes and five failures). A significant correlation was found between both quartile ranking on the Self Assessment Examination and United States Medical Licensing Examination (USMLE) passage on the first attempt with POE quartile rank (P < 0.01). Other factors, such as number of hours of faculty-led didactics per week, textbook use, and participation in formal board review courses did not correlate with POE quartile ranking.

CONCLUSION

Residents who were successful on previous standardized tests scored well on the POE. Quartile ranking on Self Assessment Examination and USMLE or National Board of Osteopathic Medical Examiners success was found to correlate significantly with POE quartile rank. This information may be helpful for future POE preparation and prospective candidate selection for physical medicine and rehabilitation.

Development of a musculoskeletal examination skills course for a physical medicine and rehabilitation residency program

This article describes the development of a musculoskeletal physical examination skills course for a physical medicine and rehabilitation residency training program. Course objectives, structure, and modification over time based on experience and resident feedback are discussed. The current course design is adaptable to meet the needs and resources of most residency programs.

One program's experience of OSCE vs. written board certification results: a pilot study

The objective structured clinical examination (OSCE) has been the focus of a lot debate with respect to reliability and validity. Much of the controversy surrounding these components lies in a lack of comparison with a "gold standard." Further work is needed to improve the evaluation of clinical skills to the point that a gold standard can truly be said to exist.

Functional independence measure prediction: an initial evaluation of residents' skills

The purpose of this preliminary study was to assess the ability of first-year PM&R residents to accurately predict functional outcome in an acute rehabilitation setting. Although statistically significant, the accuracy and precision of the residents' predictive skills were rather modest. A future focus on innovative methods to evaluate and develop these clinical skills appears warranted.

Further experience in development of an objective structured clinical examination for physical medicine and rehabilitation residents

Since the last report, two additional objective structured clinical examinations (PGY-2, PGY-3, and PGY-4, as well as incoming PGY-2) have been administered. As a result, our curriculum has been modified to strengthen physical examination skills, including specific workshops. Interrater reliability of evaluators has been tested for the first time to verify reliability, and refinements have been made in the standardized checklist grading system. The interrater grading of history-taking had good reliability (0.73-0.96), as did neurological and spine physical examination (0.84-0.88). The interrater grading reliability of small and large joint examination was more problematic (0.46-0.62) because of examiners' inability to have full visibility, evaluator's fatigue, and confusing evaluation scoring descriptions. We now use a two-point grading scale (correct or incorrect) for history but continue a three-point scale (correct, partially correct, or incorrect) for physical examination. The examination schedule is being modified to add more encounters, give time for trainee feedback, and further refinement of grading expectations for a more efficient and reliable scoring system.