Construction of a computerized adaptive test (CAT-CCNB) for efficient neurocognitive and clinical psychopathology assessment

Validation of the structured interview section of the penn computerized adaptive test for neurocognitive and clinical psychopathology assessment (CAT GOASSESS)

Large-scale studies and burdened clinical settings require precise, efficient measures that assess multiple domains of psychopathology. Computerized adaptive tests (CATs) can reduce administration time without compromising data quality. We examined feasibility and validity of an adaptive psychopathology measure, GOASSESS, in a clinical community-based sample (N = 315; ages 18-35) comprising three groups: healthy controls, psychosis, mood/anxiety disorders. Assessment duration was compared between the Full and CAT GOASSESS. External validity was tested by comparing how the CAT and Full versions related to demographic variables, study group, and socioeconomic status. The relationships between scale scores and criteria were statistically compared within a mixed-model framework to account for dependency between relationships. Convergent validity was assessed by comparing scores of the CAT and the Full GOASSESS using Pearson correlations. The CAT GOASSESS reduced interview duration by more than 90 % across study groups and preserved relationships to external criteria and demographic variables as the Full GOASSESS. All CAT GOASSESS scales could replace those of the Full instrument. Overall, the CAT GOASSESS showed acceptable psychometric properties and demonstrated feasibility by markedly reducing assessment time compared to the Full GOASSESS. The adaptive version could be used in large-scale studies or clinical settings for intake screening.

Validation of the cognitive section of the Penn computerized adaptive test for neurocognitive and clinical psychopathology assessment (CAT-CCNB)

BACKGROUND

The Penn Computerized Neurocognitive Battery is an efficient tool for assessing brain-behavior domains, and its efficiency was augmented via computerized adaptive testing (CAT). This battery requires validation in a separate sample to establish psychometric properties.

METHODS

In a mixed community/clinical sample of N = 307 18-to-35-year-olds, we tested the relationships of the CAT tests with the full-form tests. We compared discriminability among recruitment groups (psychosis, mood, control) and examined how their scores relate to demographics. CAT-Full relationships were evaluated based on a minimum inter-test correlation of 0.70 or an inter-test correlation within at least 0.10 of the full-form correlation with a previous administration of the full battery. Differences in criterion relationships were tested via mixed models.

RESULTS

Most tests (15/17) met the minimum criteria for replacing the full-form with the updated CAT version (mean r = 0.67; range = 0.53-0.80) when compared to relationships of the full-forms with previous administrations of the full-forms (mean r = 0.68; range = 0.50-0.85). Most (16/17) CAT-based relationships with diagnostics and other validity criteria were indistinguishable (interaction p > 0.05) from their full-form counterparts.

CONCLUSIONS

The updated CNB shows psychometric properties acceptable for research. The full-forms of some tests should be retained due to insufficient time savings to justify the loss in precision.

Computerized adaptive test strategies for the matrix reasoning subtest of the Wechsler Adult Intelligence Scale, 4th Edition (WAIS-IV)

OBJECTIVE

Most neuropsychological tests were developed without the benefit of modern psychometric theory. We used item response theory (IRT) methods to determine whether a widely used test - the 26-item Matrix Reasoning subtest of the WAIS-IV - might be used more efficiently if it were administered using computerized adaptive testing (CAT).

METHOD

Data on the Matrix Reasoning subtest from 2197 participants enrolled in the National Neuropsychology Network (NNN) were analyzed using a two-parameter logistic (2PL) IRT model. Simulated CAT results were generated to examine optimal short forms using fixed-length CATs of 3, 6, and 12 items and scores were compared to the original full subtest score. CAT models further explored how many items were needed to achieve a selected precision of measurement (standard error ≤ .40).

RESULTS

The fixed-length CATs of 3, 6, and 12 items correlated well with full-length test results (with r = .90, .97 and .99, respectively). To achieve a standard error of .40 (approximate reliability = .84) only 3-7 items had to be administered for a large percentage of individuals.

CONCLUSIONS

This proof-of-concept investigation suggests that the widely used Matrix Reasoning subtest of the WAIS-IV might be shortened by more than 70% in most examinees while maintaining acceptable measurement precision. If similar savings could be realized in other tests, the accessibility of neuropsychological assessment might be markedly enhanced, and more efficient time use could lead to broader subdomain assessment.

The Misguided Veneration of Averageness in Clinical Neuroscience: A Call to Value Diversity over Typicality

Research and practice in clinical neurosciences often involve cognitive assessment. However, this has traditionally used a nomothetic approach, comparing the performance of patients to normative samples. This method of defining abnormality places the average test performance of neurologically healthy individuals at its center. However, evidence suggests that neurological 'abnormalities' are very common, as is the diversity of cognitive abilities. The veneration of central tendency in cognitive assessment, i.e., equating typicality with healthy or ideal, is, I argue, misguided on neurodiversity, bio-evolutionary, and cognitive neuroscientific grounds. Furthermore, the use of average performance as an anchor point for normal performance is unreliable in practice and frequently leads to the mischaracterization of cognitive impairments. Examples are explored of how individuals who are already vulnerable for socioeconomic reasons can easily be over-pathologized. At a practical level, by valuing diversity rather than typicality, cognitive assessments can become more idiographic and focused on change at the level of the individual. The use of existing methods that approach cognitive assessment ideographically is briefly discussed, including premorbid estimation methods and informant reports. Moving the focus away from averageness to valuing diversity for both clinical cognitive assessments and inclusion of diverse groups in research is, I argue, a more just and effective way forward for clinical neurosciences.