Review of Statistical and Methodological Issues in the Forensic Prediction of Malingering from Validity Tests: Part II-Methodological Issues

Forensic neuropsychological examinations to detect malingering in patients with neurocognitive, physical, and psychological dysfunction have tremendous social, legal, and economic importance. Thousands of studies have been published to develop and validate methods to forensically detect malingering based largely on approximately 50 validity tests, including embedded and stand-alone performance and symptom validity tests. This is Part II of a two-part review of statistical and methodological issues in the forensic prediction of malingering based on validity tests. The Part I companion paper explored key statistical issues. Part II examines related methodological issues through conceptual analysis, statistical simulations, and reanalysis of findings from prior validity test validation studies. Methodological issues examined include the distinction between analog simulation and forensic studies, the effect of excluding too-close-to-call (TCTC) cases from analyses, the distinction between criterion-related and construct validation studies, and the application of the Revised Quality Assessment of Diagnostic Accuracy Studies tool (QUADAS-2) in all Test of Memory Malingering (TOMM) validation studies published within approximately the first 20 years following its initial publication to assess risk of bias. Findings include that analog studies are commonly confused for forensic validation studies, and that construct validation studies are routinely presented as if they were criterion-reference validation studies. After accounting for the exclusion of TCTC cases, actual classification accuracy was found to be well below claimed levels. QUADAS-2 results revealed that extant TOMM validation studies all had a high risk of bias, with not a single TOMM validation study with low risk of bias. Recommendations include adoption of well-established guidelines from the biomedical diagnostics literature for good quality criterion-referenced validation studies and examination of implications for malingering determination practices. Design of future studies may hinge on the availability of an incontrovertible reference standard of the malingering status of examinees.

Review of Statistical and Methodological Issues in the Forensic Prediction of Malingering from Validity Tests: Part I: Statistical Issues

Forensic neuropsychological examinations with determination of malingering have tremendous social, legal, and economic consequences. Thousands of studies have been published aimed at developing and validating methods to diagnose malingering in forensic settings, based largely on approximately 50 validity tests, including embedded and stand-alone performance validity tests. This is the first part of a two-part review. Part I explores three statistical issues related to the validation of validity tests as predictors of malingering, including (a) the need to report a complete set of classification accuracy statistics, (b) how to detect and handle collinearity among validity tests, and (c) how to assess the classification accuracy of algorithms for aggregating information from multiple validity tests. In the Part II companion paper, three closely related research methodological issues will be examined. Statistical issues are explored through conceptual analysis, statistical simulations, and through reanalysis of findings from prior validation studies. Findings suggest extant neuropsychological validity tests are collinear and contribute redundant information to the prediction of malingering among forensic examinees. Findings further suggest that existing diagnostic algorithms may miss diagnostic accuracy targets under most realistic conditions. The review makes several recommendations to address these concerns, including (a) reporting of full confusion table statistics with 95% confidence intervals in diagnostic trials, (b) the use of logistic regression, and (c) adoption of the consensus model on the "transparent reporting of multivariate prediction models for individual prognosis or diagnosis" (TRIPOD) in the malingering literature.

Performance Validity Testing in Justice-Involved Adults with Fetal Alcohol Spectrum Disorder

OBJECTIVES

A number of commonly used performance validity tests (PVTs) may be prone to high failure rates when used for individuals with severe neurocognitive deficits. This study investigated the validity of 10 PVT scores in justice-involved adults with fetal alcohol spectrum disorder (FASD), a neurodevelopmental disability stemming from prenatal alcohol exposure and linked with severe neurocognitive deficits.

METHOD

The sample comprised 80 justice-involved adults (ages 19-40) including 25 with confirmed or possible FASD and 55 where FASD was ruled out. Ten PVT scores were calculated, derived from Word Memory Test, Genuine Memory Impairment Profile, Advanced Clinical Solutions (Word Choice), the Wechsler Adult Intelligence Scale - Fourth Edition (Reliable Digit Span and age-corrected scaled scores (ACSS) from Digit Span, Coding, Symbol Search, Coding - Symbol Search, Vocabulary - Digit Span), and the Wechsler Memory Scale - Fourth Edition (Logical Memory II Recognition).

RESULTS

Participants with diagnosed/possible FASD were more likely to fail any single PVT, and failed a greater number of PVTs overall, compared to those without FASD. They were also more likely to fail based on Word Memory Test, Digit Span ACSS, Coding ACSS, Symbol Search ACSS, and Logical Memory II Recognition, compared to controls (35-76%). Across both groups, substantially more participants with IQ <70 failed two or more PVTs (90%), compared to those with an IQ ≥70 (44%).

CONCLUSIONS

Results highlight the need for additional research examining the use of PVTs in justice-involved populations with FASD.

Malingering Detection of Cognitive Impairment With the b Test Is Boosted Using Machine Learning

Objective: Here we report an investigation on the accuracy of the b Test, a measure to identify malingering of cognitive symptoms, in detecting malingerers of mild cognitive impairment.

Method: Three groups of participants, patients with Mild Neurocognitive Disorder (n = 21), healthy elders (controls, n = 21), and healthy elders instructed to simulate mild cognitive disorder (malingerers, n = 21) were administered two background neuropsychological tests (MMSE, FAB) as well as the b Test.

Results: Malingerers performed significantly worse on all error scores as compared to patients and controls, and performed poorly than controls, but comparably to patients, on the time score. Patients performed significantly worse than controls on all scores, but both groups showed the same pattern of more omission than commission errors. By contrast, malingerers exhibited the opposite pattern with more commission errors than omission errors. Machine learning models achieve an overall accuracy higher than 90% in distinguishing patients from malingerers on the basis of b Test results alone.

Conclusions: Our findings suggest that b Test error scores accurately distinguish patients with Mild Neurocognitive Disorder from malingerers and may complement other validated procedures such as the Medical Symptom Validity Test.

Assessment of feigned cognitive impairment in severe traumatic brain injury patients with the Forced-choice Graphics Memory Test

INTRODUCTION

The Forced-choice Graphics Memory Test (FGMT) is a newly developed measure to assess feigned cognitive impairment. This study investigated the ability and reliability of FGMT for identification of malingering in patients with traumatic brain injury (TBI).

METHODS

The FGMT was administered to 40 healthy volunteers instructed to respond validly (Healthy Control, H-C), 40 healthy volunteers instructed to feign cognitive impairment (Healthy Malingering, H-M), 40 severe TBI patients who responded validly (TBI control, TBI-C), and 30 severe TBI patients who evidenced invalid performance (TBI malingering, TBI-M).

RESULTS

Both malingering groups (H-M and TBI-M) performed much more poorly than the nonmalingering groups (H-C and TBI-C). The FGMT overall total score, score on easy items, and score on hard items differed significantly across the four groups. The total score showed the highest classification accuracy in differentiating malingering from nonmalingering. A cutoff of less than 18 (total items) successfully identified 95% of TBI-C and 93.3% of TBI-M participants. The FGMT also demonstrated high test-retest reliability and internal consistency. FGMT scores were not affected by TBI patients' education, gender, age, or intelligence.

CONCLUSION

Our results suggest that the FGMT can be used as a fast and reliable tool for identification of feigned cognitive impairment in patients with TBI.

Psychological characteristics of individuals who put forth inadequate cognitive effort in a secondary gain context

The current study sought to characterize the psychological architecture of individuals who put forth inadequate effort. The Minnesota Multiphasic Personality Inventory, 2nd Edition-Restructured Form was used to identify dimensions of psychological functioning in a mixed outpatient sample of U.S. Veterans referred for neuropsychological evaluation as part of their clinical care. After accounting for external financial incentive and symptom overreporting, results showed that the inadequate effort group (n = 23, mean age = 42.48) scored higher than the adequate effort group (n = 29, mean age = 44.31) on neurologic complaints (NUC) and lower on behavioral/externalizing dysfunction (BXD), antisocial behaviors (RC4), and disconstraint (DISC-r). Lower scores on BXD, RC4, and DISC-r could indicate higher behavioral constraint-a psychological characteristic that has been linked to the pursuit of high-value future rewards. Alternatively, lower scores on these scales could have reflected a self-presentation strategy aimed at minimizing externalizing and RC4 in order to appear more psychological healthy. Implications of each of these interpretations are discussed.

Unreliable block span reveals simulated intellectual disability on the stanford-binet intelligence scales-fifth edition

Previously developed validity measures are largely lacking in their ability to accurately distinguish malingerers from individuals with genuine intellectual disabilities (ID). The purpose of the present study was to develop a validity index for the Stanford-Binet Intelligence Scales-Fifth Edition (SB-5) aimed at detecting malingered ID. Fifty-four college students were instructed to feign impairment consistent with mental retardation while completing the SB-5. Simulated malingerers performed significantly poorer on the Block Span task of the SB-5 than did individuals with genuine impairment. A Block Span Validity Index (BSVI) cutoff score of less than 10 showed perfect specificity by correctly identifying all persons with true ID and demonstrated 52% sensitivity in identifying persons feigning impairment. For the detection of malingered ID, the BSVI yielded favorable results, although more research is needed to validate the BSVI using a known-groups study design.

Specificity and false positive rates of the Test of Memory Malingering, Rey 15-item Test, and Rey Word Recognition Test among forensic inpatients with intellectual disabilities

This study evaluated the specificity and false positive (FP) rates of the Rey 15-Item Test (FIT), Word Recognition Test (WRT), and Test of Memory Malingering (TOMM) in a sample of 21 forensic inpatients with mild intellectual disability (ID). The FIT demonstrated an FP rate of 23.8% with the standard quantitative cutoff score. Certain qualitative error types on the FIT showed promise and had low FP rates. The WRT obtained an FP rate of 0.0% with previously reported cutoff scores. Finally, the TOMM demonstrated low FP rates of 4.8% and 0.0% on Trial 2 and the Retention Trial, respectively, when applying the standard cutoff score. FP rates are reported for a range of cutoff scores and compared with published research on individuals diagnosed with ID. Results indicated that although the quantitative variables on the FIT had unacceptably high FP rates, the TOMM and WRT had low FP rates, increasing the confidence clinicians can place in scores reflecting poor effort on these measures during ID evaluations.

Validity issues in Atkins death cases

Justice Scalia warned in his dissent in ATKINS v. VIRGINIA, 536 U.S. 304, 122S.Ct. 2242 (2002) that the problem of feigned mental retardation would complicate the findings in cases involving the death penalty in low IQ individuals. Validity measurement in low IQ individuals has been criticized, largely with questions concerning specificity of performance validity tests (PVTs; Salekin & Doane, 2009, Applied Neuropsychology, 16, 105). In this article, our purpose is to examine the false positive rates of specific PVTs in low IQ individuals, particularly with reference to a Symptom Validity Scale previously developed for low functioning individuals (Chafetz, Abrahams, & Kohlmaier, 2007, Archives of Clinical Neuropsychology, 22, 1). The findings show that the PVTs analyzed have few false positives in a low IQ range of 60-75 when these individuals are well motivated to perform highly on testing, which allows these PVTs to be used in high stakes cases to provide evidence concerning malingering. Principles of dealing with performance validity in low functioning individuals are discussed with reference to the issues in capital cases. A practical summary guide is supplied.

Development and validation of the stanford binet-5 rarely missed items-nonverbal index for the detection of malingered mental retardation

Current research suggests that effort indices designed for the detection of malingered neurocognitive functioning do not have adequate sensitivity or specificity for use in cases where malingered mental retardation (MR) is the issue. Therefore, development and validation of reliable, objective measures for the detection of malingered MR have become imperative for both forensic and disability cases in recent years. The purpose of this study was to develop and validate an embedded malingering index for the Stanford Binet Intelligence Scales, Fifth Edition. Data from individuals in the SB5 standardization sample, who had intellectual deficits in the MR range, were used. Items that were rarely missed by the MR sample were pooled and validated using a sample of 54 college students asked to feign MR. Nonverbal items that were missed significantly more frequently by the malingering "analog MR sample" were retained and composed the Stanford Binet Rarely Missed Items-Nonverbal (SBRMI-NV) index. When only individuals who successfully malingered MR (FSIQ < 71) were included, sensitivity of 0.88 and specificity of 1.00 were obtained. Results indicate that although the SBRMI-NV needs further validation, it shows great promise in the detection of malingered MR.

The Utility of the Structured Interview of Reported Symptoms in a Sample of Individuals With Intellectual Disabilities

The challenges of accurate forensic assessment are aggravated when evaluatees have intellectual disabilities. Few studies have addressed the efficacy of forensic assessment in samples diagnosed with an intellectual disability, and those that have typically focus on measures of cognitive effort rather than on feigned psychiatric symptoms. This study focuses on the applicability of the original and revised versions of the Structured Interview of Reported Symptoms (SIRS) in a sample of participants with genuine intellectual disabilities. The SIRS was administered to 43 individuals diagnosed with intellectual disabilities, all of whom were asked to respond honestly. A considerable proportion of these respondents were misclassified as feigning psychiatric symptoms. These misclassifications were most frequent when the respondents had comorbid psychiatric diagnoses. The updated scoring algorithm of the SIRS-2 generated a rate of misclassifications that was substantially lower but that still exceeded published normative data. The implications for forensic assessment are discussed.