An Upgrading Procedure for Adaptive Assessment of Knowledge

PsycAssist: A Web-Based Artificial Intelligence System Designed for Adaptive Neuropsychological Assessment and Training

Assessing executive functions in individuals with disorders or clinical conditions can be challenging, as they may lack the abilities needed for conventional test formats. The use of more personalized test versions, such as adaptive assessments, might be helpful in evaluating individuals with specific needs. This paper introduces PsycAssist, a web-based artificial intelligence system designed for neuropsychological adaptive assessment and training. PsycAssist is a highly flexible and scalable system based on procedural knowledge space theory and may be used potentially with many types of tests. We present the architecture and adaptive assessment engine of PsycAssist and the two currently available tests: Adap-ToL, an adaptive version of the Tower of London-like test to assess planning skills, and MatriKS, a Raven-like test to evaluate fluid intelligence. Finally, we describe the results of an investigation of the usability of Adap-ToL and MatriKS: the evaluators perceived these tools as appropriate and well-suited for their intended purposes, and the test-takers perceived the assessment as a positive experience. To sum up, PsycAssist represents an innovative and promising tool to tailor evaluation and training to the specific characteristics of the individual, useful for clinical practice.

Enhancing Computerized Adaptive Testing with Batteries of Unidimensional Tests

The article presents a new computerized adaptive testing (CAT) procedure for use with batteries of unidimensional tests. At each step of testing, the estimate of a certain ability is updated on the basis of the response to the latest administered item and the current estimates of all other abilities measured by the battery. The information deriving from these abilities is incorporated into an empirical prior that is updated each time that new estimates of the abilities are computed. In two simulation studies, the performance of the proposed procedure is compared with that of a standard procedure for CAT with batteries of unidimensional tests. The proposed procedure yields more accurate ability estimates in fixed-length CATs, and a reduction of test length in variable-length CATs. These gains in accuracy and efficiency increase with the correlation between the abilities measured by the batteries.

Modeling Not-Reached Items in Cognitive Diagnostic Assessments

In cognitive diagnostic assessments with time limits, not-reached items (i.e., continuous nonresponses at the end of tests) frequently occur because examinees drop out of the test due to insufficient time. Oftentimes, the not-reached items are related to examinees’ specific cognitive attributes or knowledge structures. Thus, the underlying missing data mechanism of not-reached items is non-ignorable. In this study, a missing data model for not-reached items in cognitive diagnosis assessments was proposed. A sequential model with linear restrictions on item parameters for missing indicators was adopted; meanwhile, the deterministic inputs, noisy “and” gate model was used to model the responses. The higher-order structure was used to capture the correlation between higher-order ability parameters and dropping-out propensity parameters. A Bayesian Markov chain Monte Carlo method was used to estimate the model parameters. The simulation results showed that the proposed model improved diagnostic feedback results and produced accurate item parameters when the missing data mechanism was non-ignorable. The applicability of our model was demonstrated using a dataset from the Program for International Student Assessment 2018 computer-based mathematics cognitive test.

Cognitive Diagnosis Modeling Incorporating Item-Level Missing Data Mechanism

The aim of cognitive diagnosis is to classify respondents' mastery status of latent attributes from their responses on multiple items. Since respondents may answer some but not all items, item-level missing data often occur. Even if the primary interest is to provide diagnostic classification of respondents, misspecification of missing data mechanism may lead to biased conclusions. This paper proposes a joint cognitive diagnosis modeling of item responses and item-level missing data mechanism. A Bayesian Markov chain Monte Carlo (MCMC) method is developed for model parameter estimation. Our simulation studies examine the parameter recovery under different missing data mechanisms. The parameters could be recovered well with correct use of missing data mechanism for model fit, and missing that is not at random is less sensitive to incorrect use. The Program for International Student Assessment (PISA) 2015 computer-based mathematics data are applied to demonstrate the practical value of the proposed method.

Extending the Basic Local Independence Model to Polytomous Data

A probabilistic framework for the polytomous extension of knowledge space theory (KST) is proposed. It consists in a probabilistic model, called polytomous local independence model, that is developed as a generalization of the basic local independence model. The algorithms for computing "maximum likelihood" (ML) and "minimum discrepancy" (MD) estimates of the model parameters have been derived and tested in a simulation study. Results show that the algorithms differ in their capability of recovering the true parameter values. The ML algorithm correctly recovers the true values, regardless of the manipulated variables. This is not totally true for the MD algorithm. Finally, the model has been applied to a real polytomous data set collected in the area of psychological assessment. Results show that it can be successfully applied in practice, paving the way to a number of applications of KST outside the area of knowledge and learning assessment.

Testing the actual equivalence of automatically generated items

If the automatic item generation is used for generating test items, the question of how the equivalence among different instances may be tested is fundamental to assure an accurate assessment. In the present research, the question was dealt by using the knowledge space theory framework. Two different ways of considering the equivalence among instances are proposed: The former is at a deterministic level and it requires that all the instances of an item template must belong to exactly the same knowledge states; the latter adds a probabilistic level to the deterministic one. The former type of equivalence can be modeled by using the BLIM with a knowledge structure assuming equally informative instances; the latter can be modeled by a constrained BLIM. This model assumes equality constraints among the error parameters of the equivalent instances. An approach is proposed for testing the equivalence among instances, which is based on a series of model comparisons. A simulation study and an empirical application show the viability of the approach.