Improving the Assessment of Differential Item Functioning in Large-Scale Programs With Dual-Scale Purification of Rasch Models: The PISA Example

Combining Item Purification and Multiple Comparison Adjustment Methods in Detection of Differential Item Functioning

Many of the differential item functioning (DIF) detection methods rely on a principle of testing for DIF item by item, while considering the rest of the items or at least some of them being DIF-free. Computational algorithms of these DIF detection methods involve the selection of DIF-free items in an iterative procedure called item purification. Another aspect is the need to correct for multiple comparisons, which can be done with a number of existing multiple comparison adjustment methods. In this article, we demonstrate that implementation of these two controlling procedures together may have an impact on which items are detected as DIF items. We propose an iterative algorithm combining item purification and adjustment for multiple comparisons. Pleasant properties of the newly proposed algorithm are shown with a simulation study. The method is demonstrated on a real data example.

Examination of Gender-Related Differential Item Functioning Through Poly-BW Indices

The existing differential item functioning (DIF) detection approaches relying on item difficulty or item discrimination are limited for understanding the associates of DIF items, and consequently, DIF items were conventionally either deleted or ignored. Given the importance of minimizing DIF items in test construction, teachers or testing practitioners need more information regarding possible associates of DIF items. Using an example of a teacher-made mathematics achievement test, this study aimed to examine how the Poly-BW indices (power, defenselessness, disturbance, and hint) contributed to the properties of gender-related DIF items. Data from a 34-item mathematics achievement test that involved 1,439 seventh-grade students from Taiwan (51.01% boys and 48.99% girls) showed that the differences of the defenselessness (mp) and power (cp) indices between men and women served as salient predictors of the DIF measures estimated by the Poly Simultaneous Item Bias Test (Poly-SIBTEST) procedure and with satisfactory accuracy of hit rates. Items with relatively large defenselessness for men were likely to present male-favoring DIFs, whereas items with relatively large power for men were likely to present female-favoring DIFs. The Poly-BW indices yielded directions for modifying items for teachers in practice.

Detecting Rater Biases in Sparse Rater-Mediated Assessment Networks

Practical constraints in rater-mediated assessments limit the availability of complete data. Instead, most scoring procedures include one or two ratings for each performance, with overlapping performances across raters or linking sets of multiple-choice items to facilitate model estimation. These incomplete scoring designs present challenges for detecting rater biases, or differential rater functioning (DRF). The purpose of this study is to illustrate and explore the sensitivity of DRF indices in realistic sparse rating designs that have been documented in the literature that include different types and levels of connectivity among raters and students. The results indicated that it is possible to detect DRF in sparse rating designs, but the sensitivity of DRF indices varies across designs. We consider the implications of our findings for practice related to monitoring raters in performance assessments.