A Bayesian Method for Evaluating Passing Scores: The PPoP Curve

A priori reliability of tests with cut score

The theoretical probability of misclassification in a mastery test is exactly computed using the raw score probability distribution (in the Rasch model) as a function of the examinee's latent ability. The resulting misclassification probability curve, together with the latent ability distribution in the group of examinees, completely determines the expected rate of classification errors. It is shown that several distinct ability thresholds, playing different roles in connection to classification reliability, can be associated to a test with a single cut score. In particular, it is possible to define (and compute) two relevant ability intervals, which encapsulate the functioning of a mastery test (about and far from the cut score, respectively); the dependence of these intervals on the item difficulty spectrum is investigated. Extension to the 2PL model is also discussed, with emphasis on the effects of weighted scoring.

Improving progress test score estimation using bayesian statistics

OBJECTIVES

Progress tests give a continuous measure of a student's growth in knowledge. However, the result at each test instance is subject to measurement error from a variety of sources. Previous tests contain useful information that might be used to reduce this error. A Bayesian statistical approach to using this prior information was investigated.

METHODS

We first developed a Bayesian model that used the result from only one preceding test to update both the current estimated test score and its standard error of measurement (SEM). This was then extended to include results from all previous tests.

RESULTS

The Bayesian model leads to an exponentially weighted combination of test scores. The results show smoothing of test scores when all previous tests are included in the model. The effective sample size is doubled, leading to a 30% reduction in measurement error.

CONCLUSIONS

A Bayesian approach can give improved score estimates and smaller SEMs. The method is simple to use with large cohorts of students and frequent tests. The smoothing of raw scores should give greater consistency in rank ordering of students and hence should better identify both high-performing students and those in need of remediation.