Ability of the objective structured clinical examination to differentiate surgical residents, medical students, and physician assistant students

Improving the competency of medical students in clinical breast examination through a standardized simulation and multimedia-based curriculum

PURPOSE

Clinical breast examination (CBE) is an important step in the assessment of patients with breast-related complaints. We developed a standardized simulation and multimedia-based (SSMBI) curriculum using current evidenced-based recommendations. This study aimed to determine if SSMBI training resulted in better performance (examination scores and detection of abnormal findings) than the traditional teaching method.

METHODS

Novice fourth-year medical students were exposed to the SSMBI curriculum (n = 68) or traditional (n = 52) training. The traditional group was taught by a lecture and attending weekly clinics where they had hands-on experience. The SSMBI group underwent a structured lecture followed by an instructional video and dedicated simulated teaching. Both groups were assessed through a written knowledge exam and an objective structured clinical examination (OSCE). Student's t test and χ² tests were used to assess differences in CBE technique and knoweldge.

RESULTS

Students who underwent SSMBI training had significantly higher numbers of correct answers describing the different steps and justifications of CBE. OSCE performance was significantly higher in the SSMBI group. SSMBI-trained students were more likely to complete all the necessary CBE steps compared to traditionally-trained students (88.2% vs. 28.2%, p < 0.00001). The SSMBI group was also more systematic and more likely to perform adequate inspection, palpation, examination of the nipple-areolar complex, and identify and characterize a palpable lesion (p < 0.05).

CONCLUSIONS

Formal SSMBI training is an important asset when teaching medical students how to perform a CBE. Real clinical experience is still necessary to refine this skill and the physician-patient interaction.

Analysis of the level of clinical skills of physician assistants tested with simulated intensive care patients

RATIONALE, AIMS, AND OBJECTIVES

Since adequate staffing in intensive care units (ICUs) is an increasing problem worldwide, we investigated whether physician assistants (PAs) are able to substitute medical residents (MR) in ICUs with at least the same quality of clinical skills. In this study, we analysed the level of clinical skills of PAs in direct comparison with those who traditionally performed these tasks, ie, MR with 6 to 24 months of work experience in the ICU.

METHOD

Physician assistants and MRs in the ICUs were observed on their clinical skills by means of a simulated ICU comprising 2 scenarios on a human patient simulator with typical ICU cases. The level of clinical skills of PAs and MRs was videotaped and scored with predefined checklists by 2 independent intensivists per scenario. Percentage of the total score was calculated, and means were compared by Student t test.

RESULTS

A total of 11 PAs and 10 MRs participated in the study. Physician assistants and MRs scored equal (PA 66% ± 13% vs MR 68% ± 9%, P = .86) on their clinical performance in the simulated ICU setting.

CONCLUSION

This study showed equal performance of PAs and MRs on clinical skills in the simulated ICU setting.

Assessment of EFAST training for final year medical students in emergency medicine clerkship

Background

Extended Focused Assessment Sonography for Trauma (EFAST) is an important bedside tool for the management of multiple trauma patients. We aimed to evaluate the assessment of our EFAST education in the Emergency Medicine Clerkship (EMC) for final year medical students and the correlations of EFAST marks with other practical skill stations and the final multiple choice question (MCQ) exam marks.

Methods

Fifty-four final year medical students were trained on performing EFAST on human models during their 4-week clerkship. Students received an hour of didactic lecture, 4-hours practical sessions on human models, and completed a minimum of three EFAST examinations on trauma patients. Finally, the EFAST performance was evaluated on human models using a standard evaluation form during an Objective Structured Clinical Examination (OSCE). The marks of 51 students who completed the final exam were analyzed.

Results

The overall passing rate of the EFAST station was 88% (n: 45). EFAST station mark had significant weak correlations with other OSCE stations marks (p = 0.027, rho = 0.31), and with the final EMC mark (p = 0.032, rho = 0.3), but not with the final MCQ exam.

Conclusions

Final year medical students demonstrated effective EFAST learning as measured by their examination performance. One hour EFAST training and 4 -hours practice provide an acceptable level of skill for medical students. The EFAST final marks showed significant weak correlation with other OSCE station marks and final clerkship marks, but not with the final MCQ exam mark which assesses a different cognitive learning domain.

Comparing Surgical Clerkship Performance of Medical and Physician Assistant Students

OBJECTIVE

Third-year medical students (MS-III) and second-year physician assistant students (PA-S) have similar core clinical rotations during their education. Uniquely at our institution, both groups rotate together and are assessed by the same evaluation and grading standards. This study compares the performance of MS-III and PA-S during their combined surgical clerkship rotation.

DESIGN

A retrospective analysis on students' final clerkship grades, individual grades for clinical performance evaluation, objective structured clinical examination, faculty tutorials, and National Board of Medical Education (NBME) general surgery examination for academic years 2013 and 2014. Statistical analysis using the Statistical Package for Social Sciences (SPSS) was used. A p < 0.05 was considered statistically significant.

SETTING

The study took place in the Department of Surgery at the University of Oklahoma School of Community Medicine, Tulsa, Oklahoma.

PARTICIPANTS

All MS-III and PA-S that performed a clinical rotation in the Department of Surgery at the University of Oklahoma School of Community Medicine, Tulsa were included in the study. Fourth-year medical students on elective rotations were excluded. A total of 95 students were included, 51 MS-III and 44 PA-S.

RESULTS

The results indicated that MS-III had statistically significant higher clinical performance evaluation (p = 0.005), NBME (p < 0.001), and tutorial scores (p = 0.03) as compared to PA-S. However, there were no statistically significant findings between final grades and objective structured clinical examination scores. In comparing 2013 to 2014 MS-III classes, no statistically significant differences existed, but the 2014 PA-S class had statistically significant higher NBME (p = 0.022) and tutorial scores (p = 0.015) as compared to 2013 PA-S class.

CONCLUSION

MS-III performed better in tests evaluating medical knowledge, possibly a reflection of more in-depth different organ systems physiology and pathophysiology education that MS-III receive. No significant difference in clerkship performance was found. This could be attributed to similar clinical education both receiving as well as PA-S often having a history of prior professional health care experience. Over time of the study, PA-S performance seems to have improved. These observations might help to improve future curricula for MS-III as well as for PA-S.

Analysis of the level of general clinical skills of physician assistant students using an objective structured clinical examination

RATIONALE, AIMS AND OBJECTIVES

The physician assistant (PA) is trained to perform clinical tasks traditionally performed by medical doctors (MDs). Previous research showed no difference in the level of clinical skills of PAs compared with MDs in a specific niche, that is the specialty in which they are employed. However, MDs as well as PAs working within a specialty have to be able to recognize medical problems in the full scope of medicine. The objective is to examine PA students' level of general clinical skills across the breadth of clinical cases.

METHOD

A cross-sectional study was conducted. PA students and recently graduated MDs in the Netherlands were observed on their clinical skills by means of an objective structured clinical examination comprising five stations with common medical cases. The level of mastering history taking, physical examination, communication and clinical reasoning of PA students and MDs were described in means and standard deviation. Cohen's d was used to present effect sizes.

RESULTS

PA students and MDs score about equal on history taking (PA 5.8 ± 0.8 vs. MD 5.7 ± 0.7), physical examination (PA 4.8 ± 1.3 vs. MD 5.4 ± 0.8) and communication (PA: 8.2 ± 0.8 vs. MD: 8.6 ± 0.5) in the full scope of medicine. In the quality of the report, including the patient management plan, PA students scored a mean of 6.0 ± 0.6 and MDs 6.8 ± 0.6.

CONCLUSIONS

In this setting in the Netherlands, PA students and MDs score about equal in the appraisal of common cases in medical practice. The slightly lower scores of PA students' clinical reasoning in the full scope of clinical care may have raise attention to medical teams working with PAs and PA training programmes.

Designing and Implementing the Objective Structured Clinical Examination in Anesthesiology

Since its description in 1974, the Objective Structured Clinical Examination (OSCE) has gained popularity as an objective assessment tool of medical students, residents, and trainees. With the development of the anesthesiology residents’ milestones and the preparation for the Next Accreditation System, there is an increased interest in OSCE as an evaluation tool of the six core competencies and the corresponding milestones proposed by the Accreditation Council for Graduate Medical Education.

In this article the authors review the history of OSCE and its current application in medical education and in different medical and surgical specialties. They also review the use of OSCE by anesthesiology programs and certification boards in the United States and internationally. In addition, they discuss the psychometrics of test design and implementation with emphasis on reliability and validity measures as they relate to OSCE.

Training to Measure Ankle–Brachial Index at the Undergraduate Level

We examined the effectiveness of teaching ankle–brachial index (ABI) measurement to medical students. ABI was estimated in 28 lower limbs by an experienced vascular surgeon. After a 2-week training course, 5 fourth-year students repeated the estimations and their results were compared with that of the trainer’s. There was no difference in ABI values between trainees and trainer for subjects with mild-to-moderate peripheral arterial disease (PAD; 0.77 ± 0.22 vs 0.77 ± 0.19, respectively, P = .95). In the 4 normal limbs, ABI was 1.37 ± 0.12 and 1.16 ± 0.11, as measured by the trainer and the trainees, respectively ( P < .00001). In subjects with severe PAD, trainees tended to overestimate ABI ( P = .0002) in the beginning of the educational process, but this was no longer the case at a later stage of the training with no difference in ABI values between the 2 examiner groups ( P = .09). In conclusion, training of medical students in ABI measurement can be helpful toward accurate estimation of PAD and merits further practice.

Information-gathering patterns associated with higher rates of diagnostic error

Diagnostic errors are an important source of medical errors. Problematic information-gathering is a common cause of diagnostic errors among physicians and medical students. The objectives of this study were to (1) determine if medical students' information-gathering patterns formed clusters of similar strategies, and if so (2) to calculate the percentage of incorrect diagnoses in each cluster. A total of 141 2nd year medical students completed a computer case simulation. Each student's information-gathering pattern included the sequence of history, physical examination, and ancillary testing items chosen from a predefined list. We analyzed the patterns using an artificial neural network and compared percentages of incorrect diagnoses among clusters of information-gathering patterns. We input patterns into a 35 x 35 self organizing map. The network trained for 10,000 epochs. The number of students at each neuron formed a surface that was statistically smoothed into clusters. Each student was assigned to one cluster, the cluster that contributed the largest value to the smoothed function at the student's location in the grid. Seven clusters were identified. Percentage of incorrect diagnoses differed significantly among clusters (Range 0-42%, Chi (2) = 13.62, P = .034). Distance of each cluster from the worst performing cluster was used to rank clusters. This rank was compared to rank determined by percentage incorrect. We found a high positive correlation (Spearman Correlation = .893, P = .007). Clusters closest to the worst performing cluster had the highest percentages of incorrect diagnoses. Patterns of information-gathering were distinct and had different rates of diagnostic error.

Objective structured clinical examination (OSCE): review of literature and implications for nursing education

Objective structured clinical examination (OSCE) has been used to assess medical students since the mid 1970s, and in more recent years has been increasingly utilised to assess students from nursing and the allied health professions. This growing utilisation has led to considerable debate within the literature pertaining to the optimal use of OSCE as an assessment process. The purpose of this paper is to present a narrative review regarding some of the key issues affecting the utilisation of OSCE within the assessment of nursing students. The paper briefly reviews the historical development of OSCE within health professional assessment, and summarises some of its key strengths and limitations. It then offers a more 'in depth' consideration of the research literature pertaining to the reliability and validity of the OSCE process, which is then used as a basis for exploring some of the particular issues that need to be considered when OSCE is used to assess nursing students. Key issues identified include the need to carefully prepare and pilot new OSCE examinations and marking tools in order to ensure reliability and validity is optimised, and also the need to carefully consider the length, number and interdependence of OSCE stations to ensure that the potentially competing requirements of validity and reliability are balanced. The paper also recognises that whilst the evidence base regarding OSCE is extensive, the evidence base specific to nursing is more limited. There is therefore scope for further research in this area, as well as the need for careful debate regarding how national guidance may be a way of enhancing and standardising future OSCE examinations. The paper concludes that whilst caution must be applied in relying on OSCE as a sole means of practitioner assessment, used carefully it can make a helpful and meaningful contribution to health professional education.

Does Residency Training Improve Performance of Physical Examination Skills?

BACKGROUND

Few studies use objective structured clinical examinations (OSCEs) to measure physical examination skills of internal medicine residents. Little is known about performance by year of residency training.

PURPOSE

To determine differences between postgraduate year (PGY)-1 and PGY-3 residents on performance and comfort of physical examination skills.

METHODS

In a cross-sectional study, we tested 16 PGY-1 (weeks 0 and 4) and 8 PGY-3 internal medicine residents with a five-station OSCE.

RESULTS

PGY-3 residents performed better than PGY-1 week 0 residents (P = 0.03) but not PGY-1 week 4 residents (P = 0.42). PGY-1 resident performance improved after 1 month of inpatient wards experience (P < 0.001). PGY-3 residents had higher comfort compared to PGY-1 week 0 residents (P = 0.003) but not PGY-1 week 4 residents (P = 0.10).

CONCLUSIONS

Senior residents performed better and were more confident on physical examination skills, but the difference disappeared after 1 month of internship. This calls into question how much further learning occurs with physical examination throughout residency training.

Didactic lecture versus instructional standardized patient interaction in the surgical clerkship

BACKGROUND

Medical schools increasingly employ the standardized patient interaction (SPI) as a teaching and testing modality to prepare students for USMLE Step 2 Clinical Skills (Step 2 CS). However, little is known about the perceptions medical students have toward SPIs. We hypothesized that the SPI would increase the self-confidence of surgery students in their history and physical examination (H&P) skills as compared with the classic lecture format and that students would perceive the SPI as a valuable learning tool.

METHODS

Students (n = 126) on the junior surgery clerkship were randomized into two groups. Group A (n = 61) received a didactic lecture on acute appendicitis. Group B (n = 65) participated in an SPI in which the portrayed patient had acute appendicitis. Student perceptions were surveyed by written questionnaire pre- and postencounter using a 5-point Likert scale, with 5 being the most favorable rating. Data (perceptions, performances on the SPI, clerkship grades) were compared using a Student t test followed by the Mann-Whitney rank sums test.

RESULTS

Group A "enjoyed" the lecture format more than group B "enjoyed" the SPI format (3.4 +/- 0.1 versus 2.6 +/- 0.1, P <.001) and perceived the lecture as having more value to their overall education (3.5 +/- 0.1 versus 2.6 +/- 0.1, P <.001) than the SPI group. Surprisingly, student self-confidence in their H&P skills increased after the didactic lecture (from 3.6 +/- 0.1 to 4.1 +/- 0.1, P = .001) but not after the SPI (from 3.6 +/- 0.1 to 3.8 +/- 0.1, P = not significant).

CONCLUSIONS

The didactic lecture format was not only enjoyed and valued more than the SPI, but our surgery students also perceived it as superior to the SPI in building confidence in history and physical examination skills. These findings suggest that surgical educators should develop ways to improve students' perceptions and attitudes surrounding the surgical SPI.

Using standardized patients to assess the geriatrics medicine skills of medical students, internal medicine residents, and geriatrics medicine fellows

PURPOSE

Medical schools and residency programs are placing additional emphasis on including clinical geriatrics competencies within their curricula. An eight-station, Geriatric Medicine Standardized Patient Examination (GSPX) was studied as a method to assess bedside geriatrics clinical skills over the continuum of medical education from medical school through residency and fellowship training.

METHOD

The GSPX was administered to 39 medical students, 49 internal medicine residents, and 11 geriatrics medicine fellows in 2001-02. Reliability of standardized patient (SP) checklists and rating scales used to assess examinees' performance was measured by Cronbach's alpha. Validity was measured by surveying the examinees' assessment of fairness, individual case length, difficulty, and believability, and by faculty standard setting for each level of trainee.

RESULTS

Reliability was high (alpha =.89). All levels of examinees found the SPs to be believable, station lengths to be adequate, and rated the GSPX as a fair assessment. Students rated the cases as more difficult. Previous experience with similar real patients increased significantly with level of training (Pearson's r =.48, p <.0001). Faculty set passing scores that increased from students to residents to fellows. However, GSPX scores decreased with level of training (r = -.25, p =.01).

CONCLUSION

The GSPX is a reliable measure of geriatrics medicine skills with adequate face validity for examinees at all levels. However, GSPX scores did not increase with level of training, suggesting that a single form of the examination cannot be used across the continuum of training. Potential modifications to the GSPX that might provide more discrimination between levels of training are currently being explored.