What is the "ideal" grading system for the junior surgery clerkship?

Comparison of Two Veterinary Emergency and Critical Care Clerkship Grading Protocols

Universal guidelines for evaluating veterinary students' clinical clerkship performance are unavailable. At our institution, each service determines its own grading protocol. In this study, researchers compared clinician, staff, and student perceptions of a traditional and newly devised grading practice on the Emergency and Critical Care (ECC) clerkship. ECC clinicians and technicians were asked to assess the existing grading protocol for the clerkship. The system was then revised to better align with clerkship objectives. The revised protocol evaluated students on 12 items encompassing knowledge, clinical, and communication skills. Following the assignment of values for each category, letter grades were calculated automatically. Clinicians and staff were invited to rate the revised grading system. Throughout the study period, a corresponding survey was sent to students shortly after they had received their clerkship grades. Students' open-ended comments were analyzed qualitatively to identify common themes. Clinicians and technicians reported that the revised protocol was more inclusive and better able to provide fair and accurate assessments of students' performances. Students were generally satisfied with both grading protocols, however, in the open-ended comments students' frequently expressed desire for more directed and timely feedback on their performance. The results of this study indicate that the graders' believed that the revised evaluation protocol provides opportunities to provide fair and accurate assessments of student performance. Overall, students were satisfied with the new protocol and have a desire for tailored feedback provided in a timely fashion.

An Investigation of the Generalizability of Medical School Grades

Construct/Background: Medical school grades are currently unstandardized, and their level of reliability is unknown. This means their usefulness for reporting on student achievement is also not well documented. This study investigates grade reliability within 1 medical school.

APPROACH

Generalizability analyses are conducted on grades awarded. Grades from didactic and clerkship-based courses were treated as 2 levels of a fixed facet within a univariate mixed model. Grades from within the 2 levels (didactic and clerkship) were also entered in a multivariate generalizability study.

RESULTS

Grades from didactic courses were shown to produce a highly reliable mean score (G = .79) when averaged over as few as 5 courses. Although the universe score correlation between didactic and clerkship courses was high (r = .80), the clerkship courses required almost twice as many grades to reach a comparable level of reliability. When grades were converted to a Pass/Fail metric, almost all information contained in the grades was lost.

CONCLUSIONS

Although it has been suggested that the imprecision of medical school grades precludes their use as a reliable indicator of student achievement, these results suggest otherwise. While it is true that a Pass/Fail system of grading provides very little information about a student's level of performance, a multi-tiered grading system was shown to be a highly reliable indicator of student achievement within the medical school. Although grades awarded during the first 2 didactic years appear to be more reliable than clerkship grades, both yield useful information about student performance within the medical college.

Correlating surgical clerkship evaluations with performance on the National Board of Medical Examiners examination

BACKGROUND

Evaluation of medical students during the surgical clerkship is controversial. Performance is often based on subjective scoring, whereas objective knowledge is based on written examinations. Whether these measures correspond or are relevant to assess student performance is unknown. We hypothesized that student evaluations correlate with performance on the National Board Of Medical Examiners (NBME) examination.

METHODS

Data were collected from the 2011-2012 academic year. Medical students underwent a ward evaluation using a seven-point Likert scale assessing six educational competencies. Students also undertook the NBME examination, where performance was recorded as a percentile score adjusted to national standards.

RESULTS

A total of 129 medical students were studied. Scores on the NBME ranged from the 52nd to the 96th percentile with an average in the 75th percentile (±9). Clerkship scores ranged from 3.2-7.0 with a mean of 5.7 (±0.8). There was a strong positive association between higher NBME scores and higher clerkship evaluations shown by a Pearson correlation coefficient of 0.47 (P<0.001). Students clustered with below average ward evaluations (3.0-4.0) were in the 69.5th percentile of NBME scores, whereas students clustered with above average ward evaluations (6.0-7.0) were in the 79.2th percentile (P<0.001).

CONCLUSIONS

A strong positive relationship exists between subjective ward evaluations and NBME performance. These data may afford some confidence to surgical faculty and surgical resident ability to accurately evaluate medical students during clinical clerkships. Understanding factors in student performance may help in improving the surgical clerkship experience.

Using elements from an acute abdominal pain Objective Structured Clinical Examination (OSCE) leads to more standardized grading in the surgical clerkship for third-year medical students

BACKGROUND

There is poor reliability in the Likert-based assessments of patient interaction and general knowledge base for medical students in the surgical clerkship. The Objective Structured Clinical Examination (OSCE) can be used to assess these competencies.

OBJECTIVE

We hypothesize that using OSCE performance to replace the current Likert-based patient interaction and general knowledge base assessments will not affect the pass/fail rate for third-year medical students in the surgical clerkship.

METHODS

In this retrospective study, third-year medical student clerkship data from a three-station acute abdominal pain OSCE were collected from the 2009-2010 academic year. New patient interaction and general knowledge base assessments were derived from the performance data and substituted for original assessments to generate new clerkship scores and ordinal grades. Two-sided nonparametric statistics were used for comparative analyses, using an α = 0.05.

RESULTS

Seventy third-year medical students (50.0% female) were evaluated. A sign test showed a difference in the original (4.45/5) and the new (4.20/5) median patient interaction scores (p < 0.01). A sign test did not show a difference in the original (4.00/5) and the new (4.11/5) median general knowledge base scores (p = 0.28). Nine clerkship grades changed between these different grading schemes (p = 0.045), with an overall agreement of 87.1% and a kappa statistic of 0.81. There were no differences in the pass/fail rate (p > 0.99).

CONCLUSIONS

We conclude that there are no differences in pass/fail rate, but there is a more standardized distribution of patient interaction assessments and utilization of the full spectrum of possible passing grades. We recommend that the current patient interaction assessment for third-year medical students in the surgical clerkship be replaced with that found through trained standardized patients in this three-station acute abdominal pain OSCE.

Evaluation, grading, and use of the RIME vocabulary on internal medicine clerkships: results of a national survey and comparison to other clinical clerkships

BACKGROUND

Evaluation methods within and across clerkships are rapidly evolving, including greater emphasis or frameworks for descriptive evaluation and direct observation of competence.

PURPOSE

The purpose of this study is to describe current evaluation methods, use of the Reporter-Interpreter-Manager/Educator (RIME) framework, and grade assignment by internal medicine clerkship directors.

METHODS

In 2005, the Clerkship Directors in Internal Medicine surveyed its 109 institutional members. Topics included evaluation methods and grade contribution, use of evaluation sessions and/or RIME, and grade assignment (criterion referenced or normative).

RESULTS

Response rate was 81% (88/109). The evaluation methods were as follows: teachers' evaluations, 93% (64% of grade); National Board of Medical Examiners subject examination, 81% (25% of grade); faculty written exam, 34% (14% of grade); objective structured clinical examinations, 32% (12% of grade); direct observation, 22% (7% of grade). RIME is used by 42% of respondents. Many clerkship directors (43%) meet with teachers to discuss student performance. Criterion-referenced grading is used by 59%, and normative grading is used by 27%. Unsatisfactory grades are given for examination failures (72%), unprofessional behavior (49%), poor clinical performance (42%), and failure to meet requirements (18%).

CONCLUSIONS

Internal medicine clerkship directors emphasize description and observation of students. RIME and discussions with teachers are becoming commonplace.

Is the pass/fail system applicable to a medical school in Korea?

To determine whether a pass/fail system is more appropriate for medical education instead of a grade-based system, a survey of medical students and faculty members of Hallym University, Korea, was taken. A questionnaire was delivered to 54 junior students and 36 faculty members from a medical school in Korea and analyzed. Of these participants, 37.7% of students and 36.1% of faculty agreed to the pass/fail system, while 28.3% of students and 52.8% of faculty objected to it. The most frequent reason for objection was the potential decrease in learning achievement. A pass/fail system should be considered after persuasion of the students and faculty to think positively of this system.