Sequential testing in the objective structured clinical examination: selecting items for the screen

Introducing a model for optimal design of sequential objective structured clinical examinations

In a sequential OSCE which has been suggested to reduce testing costs, candidates take a short screening test and who fail the test, are asked to take the full OSCE. In order to introduce an effective and accurate sequential design, we developed a model for designing and evaluating screening OSCEs. Based on two datasets from a 10-station pre-internship OSCE and considering three factors, namely, the number of stations, the criteria for selecting the stations, and the cut-off score, several hypothetical tests were proposed. To investigate their accuracy, the positive predictive value (PPV), the pass rate, and the negative predictive value (NPV) were calculated. Also, a "desirable" composite outcome was defined as PPV = 100 %, pass rate ≥50 %, and NPV ≥25 %. Univariate and multiple logistic regression analyses were conducted to estimate the effects of independent factors on the occurrence of the desirable outcome. In half of the screening tests no false positive result was detected. Most of the screening OSCEs had acceptable levels of pass rate and NPV. Considering the desirable composite outcome 20 screening OSCEs could have successfully predicted the results of the corresponding full OSCE. The multiple regression analysis indicated significant contributions for the selection criteria (p values = 0.019) and the cut-off score (p values = 0.017). In order to have efficient screening OSCEs with the lowest probability of the error rate, careful selection of stations with high values of discrimination or item total correlation, and use of a relatively stringent cut-off score should be considered.

Sequential testing in a high stakes OSCE: Determining number of screening tests

OBJECTIVE

The sequential objective structured clinical exam (OSCE) is a stand-alone variation of the traditional OSCE whereby all students sit a screening test. Those who pass this initial assessment undergo no further testing while weakly performing students sit an additional (sequential) test to determine their overall pass/fail status. Our aim was to determine outcomes of adopting a sequential OSCE approach using different numbers of screening stations and pass marks.

METHOD

We carried out a retrospective, observational study of anonymised databases of two cohorts of student outcomes from the final OSCE examination at the University of Aberdeen Medical School. Data were accessed for students (n = 388) who sat the exam in the years 2013-2014. We used Stata simulate program to compare outcomes - in terms of sensitivity and specificity - across 5000 random selections of 6-14 OSCE stations using random selections of groups of 100 students (with different screening test pass marks) versus those obtained across 15 stations.

RESULTS

Across 6-14 stations, the sensitivity was ≥87% in 2013 and ≥84% in 2014 while the specificity ranged from 60% to 100% in both years. Specificity generally increased as the number of screening stations increased (with concomitant narrowing of the 95% confidence interval), while sensitivity varied between 84 and 98%. Similar sensitivities and specificities were found with screening pass marks of +1, +2 and +3 standard errors of measurement (SEM). Eight stations as a screening test appeared to be a reasonable compromise in terms of high sensitivity (88-89%) and specificity (83-86%).

CONCLUSION

This research extends current sequential OSCE literature using a novel and robust approach to identify the "ideal" in terms of number of screening stations and pass mark. We discuss the educational and resource implications of our findings and make recommendations for the use of the sequential OSCE in medical education.

ROC and loss function analysis in sequential testing

Sequential testing is applied to reduce costs in SP-based tests (OSCEs). Initially, all candidates take a screening test consisting of a part of the OSCE. Candidates who fail the screen sit the complete test, whereas those who pass the screen are qualified as a pass of the complete test. The procedure may result in a reduction of testing resources, but at the cost of false positives (candidates who pass the screen but would fail the complete test). In a diagnostic test an optimum cutpoint is obtained by minimizing the weighted sum of false negatives and false positives using Receiver Operator Characteristic (ROC) analysis. However, in the sequential procedure there are no false negatives, because the result of the complete test is considered the final outcome. Several authors ignored this difference and applied classic ROC analysis to a sequential test. Earlier, we doubted the validity of the procedure and proposed to set the cutpoint by minimizing the loss, defined as the weighted sum of the screen negatives and the false positives. Recently, Regehr and Colliver (2003) argued that under certain theoretically derived conditions the use of the loss formula in sequential testing is functionally identical to using classic ROC analysis. The current study shows (a) the conclusion of Regehr and Colliver is based on assumptions that can be challenged, and (b) ROC analysis indeed can be used in sequential testing, but only if the procedure is modified according to the results of a corresponding loss function analysis.

Using a standardized family to teach clinical skills to medical students

BACKGROUND

The use of standardized patients has been an accepted instructional methodology in medical education for many years. A logical evolution of this methodology is the creation of a standardized patient family.

DESCRIPTION

This article describes one such standardized family, the Jones family, and how the family is used to teach interpersonal skills, interviewing, communication, counseling, and history-taking skills to medical students.

EVALUATION

After several years of using the Jones family, we have found that more comprehensive scripts need to be developed, that recruitment and retention of standardized patients for a year long program does not seem to be a problem, and that the value added by a standardized family greatly enhances the educational experience for students. A standardized family seems a logical educational vehicle for teaching continuity of care, confidentiality, contextual placement of medical information within family dynamics, cultural beliefs, community orientation, and generalism.

CONCLUSION

A standardized family is a viable instructional methodology that deserves greater use in medical education.