A Model for Setting Performance Standards for Standardized Patient Examinations

Setting standards for complex performance assessments is difficult. This report describes and evaluates a model for setting performance standards for standardized patient examinations. A variation of a modified Angoff standard-setting procedure was used to set cutoffs for a standardized patient exercise. Analysis of the procedure was based on evaluation of the impact of the iterative process, the ability of the final standards to accurately classify trainees and the judges’confidence in the procedure and final standards. The practicality of the process also was assessed. The iterative steps in this procedure led to a reduction in the variance between the judges’ratings. All seven judges reported confidence in the final standards. The mean time commitment for each judge was 4.7 work hours. This model for setting performance standards successfully set useful standards for this exam, was practical, and can be utilized to set performance standards for other standardized patient examinations.

Making the case for mastery learning assessments: key issues in validation and justification

Theoretical and empirical support is increasing for mastery learning, in which learners must demonstrate a minimum level of proficiency before completing a given educational unit. Mastery learning approaches aim for uniform achievement of key objectives by allowing learning time to vary and as such are a course-level analogue to broader competency-based curricular strategies. Sound assessment is the cornerstone of mastery learning systems, yet the nature of assessment validity and justification for mastery learning differs in important ways from standard assessment models. Specific validity issues include (1) the need for careful definition of what is meant by "mastery" in terms of learners' achievement or readiness to proceed, the expected retention of mastery over time, and the completeness of content mastery required in a particular unit; (2) validity threats associated with increased retesting; (3) the need for reliability estimates that account for the specific measurement error at the mastery versus nonmastery cut score; and (4) changes in item- and test-level score variance over retesting, which complicate the analysis of evidence related to reliability, internal structure, and relationships to other variables. The positive and negative consequences for learners, educational systems, and patients resulting from the use of mastery learning assessments must be explored to determine whether a given mastery assessment and pass/fail cut score are valid and justified. In this article, the authors outline key considerations for the validation and justification of mastery learning assessments, with the goal of supporting insightful research and sound practice as the mastery model becomes more widespread.

How to set standards on performance-based examinations: AMEE Guide No. 85

This AMEE Guide offers an overview of methods used in determining passing scores for performance-based assessments. A consideration of various assessment purposes will provide context for discussion of standard setting methods, followed by a description of different types of standards that are typically set in health professions education. A step-by-step guide to the standard setting process will be presented. The Guide includes detailed explanations and examples of standard setting methods, and each section presents examples of research done using the method with performance-based assessments in health professions education. It is intended for use by those who are responsible for determining passing scores on tests and need a resource explaining methods for setting passing scores. The Guide contains a discussion of reasons for assessment, defines standards, and presents standard setting methods that have been researched with performance-based tests. The first section of the Guide addresses types of standards that are set. The next section provides guidance on preparing for a standard setting study. The following sections include conducting the meeting, selecting a method, implementing the passing score, and maintaining the standard. The Guide will support efforts to determine passing scores that are based on research, matched to the assessment purpose, and reproducible.

Using "standardized narratives" to explore new ways to represent faculty opinions of resident performance

PURPOSE

Most efforts to develop reliable evaluations of clinical competence have been oriented toward deconstructing the requisite competencies into separate scales. However, many are questioning the value of this approach on theoretical and empirical bases. This study uses "standardized narratives" to explore a different approach to assessing resident performance.

METHOD

In 2009, based on interviews with 19 experienced clinical faculty from two institutions, 16 narrative profiles were created to represent the range of resident competence that clinical faculty might encounter during supervision. Fourteen clinicians from three institutions independently grouped the profiles into as many categories as necessary to reflect various levels of performance, described their categories, then ranked the individual profiles within each category. Then, in groups of three or four, participants negotiated a final ranking and grouping of the 16 profiles.

RESULTS

Despite interesting idiosyncracies in the factors some participants identified as guiding their rankings, there was strong consistency across the 14 clinicians regarding the rankings (single-rater intraclass correlation [ICC] = 0.86) and groupings (single-rater ICC = 0.81) of the profiles. Similarly, across institutions, the four groups were highly consistent in their final negotiated rankings (single-group ICC = 0.91) and groupings (single-group ICC = 0.87) of the profiles.

CONCLUSIONS

Faculty showed more consistency in their decisions of what constitutes excellent, competent, and problematic performance in residents than implied by current assessment techniques that require deconstruction of resident competencies. This use of standardized narratives points to interesting opportunities for more authentically codifying faculty opinions of residents.

The use of standardized patient assessments for certification and licensure decisions

Although standardized patients have been employed for formative assessment for over 40 years, their use in high-stakes medical licensure examinations has been a relatively recent phenomenon. As part of the medical licensure process in the United States and Canada, the clinical skills of medical students, medical school graduates, and residents are evaluated in a simulated clinical environment. All of the evaluations attempt to provide the public with some assurance that the person who achieves a passing score has the knowledge and/or requisite skills to provide safe and effective medical services. Although the various standardized patient-based licensure examinations differ somewhat in terms of purpose, content, and scope, they share many commonalities. More important, given the extensive research that was conducted to support these testing initiatives, combined with their success in promoting educational activities and in identifying individuals with clinical skills deficiencies, they provide a framework for validating new simulation modalities and extending simulation-based assessment into other areas.

Developing performance-based medical school assessment programs in resource-limited environments

While the use of performance-based assessments in medical schools is widespread, there are unique issues associated with developing and establishing simulated clinical examinations when resources are limited. The faculty at the National University of Cuyo (NUC) in Mendoza, Argentina, has successfully implemented an OSCE as part of the medical school graduation requirements. This paper provides an overview of the issues faced by NUC faculty in setting up a standardized patient (SP)-based assessment with only limited resources, and contrasts their experiences with those of other groups, including certification and licensure bodies who have implemented similar types of assessments. Despite their novelty and a lack of technological, staffing, and budgetary resources, with adequate planning, an emphasis on faculty involvement, quality case development, local SP recruitment and training activities, educational programs focused on assessment, and flexibility in creating a physical space to conduct the exam, an effective program for assessing the clinical skills of medical students can be realized.

Comparison of a rational and an empirical standard setting procedure for an OSCE. Objective structured clinical examinations

PURPOSE

Earlier studies of absolute standard setting procedures for objective structured clinical examinations (OSCEs) show inconsistent results. This study compared a rational and an empirical standard setting procedure. Reliability and credibility were examined first. The impact of a reality check was then established.

METHODS

The OSCE included 16 stations and was taken by trainees in their final year of postgraduate training in general practice and experienced general practitioners. A modified Angoff (independent judgements, no group discussion) with and without a reality check was used as a rational procedure. A method related to the borderline group procedure, the borderline regression (BR) method, was used as an empirical procedure. Reliability was assessed using generalisability theory. Credibility was assessed by comparing pass rates and by relating the passing scores to test difficulty.

RESULTS

The passing scores were 73.4% for the Angoff procedure without reality check (Angoff I), 66.0% for the Angoff procedure with reality check (Angoff II) and 57.6% for the BR method. The reliabilities (expressed as root mean square errors) were 2.1% for Angoffs I and II, and 0.6% for the BR method. The pass rates of the trainees and GPs were 19% and 9% for Angoff I, 66% and 46% for Angoff II, and 95% and 80% for the BR method, respectively. The correlation between test difficulty and passing score was 0.69 for Angoff I, 0.88 for Angoff II and 0.86 for the BR method.

CONCLUSION

The BR method provides a more credible and reliable standard for an OSCE than a modified Angoff procedure. A reality check improves the credibility of the Angoff procedure but does not improve its reliability.

The consistency and uncertainty in examiners' definitions of pass/fail performance on OSCE (objective structured clinical examination) stations

The Medical Council of Canada has made use of examiners' pass/fail classifications of candidates' behaviors in objective structured clinical examination (OSCE) stations in defining cutting scores for these stations. This process assumes that there is consistency in the judgments of examiners employed in the same stations at different testing sites and in the cutting scores derived from these judgments. These assumptions were tested using the results of the fall 1993 administration of part 2 of the Medical Council of Canada's Evaluating Examination to 744 candidates. The results of this study provided evidence of the consistency of the pass/fail and cutting score definitions for the stations used across examiners.