Computerized adaptive test strategies for the matrix reasoning subtest of the Wechsler Adult Intelligence Scale, 4th Edition (WAIS-IV)

OBJECTIVE

Most neuropsychological tests were developed without the benefit of modern psychometric theory. We used item response theory (IRT) methods to determine whether a widely used test - the 26-item Matrix Reasoning subtest of the WAIS-IV - might be used more efficiently if it were administered using computerized adaptive testing (CAT).

METHOD

Data on the Matrix Reasoning subtest from 2197 participants enrolled in the National Neuropsychology Network (NNN) were analyzed using a two-parameter logistic (2PL) IRT model. Simulated CAT results were generated to examine optimal short forms using fixed-length CATs of 3, 6, and 12 items and scores were compared to the original full subtest score. CAT models further explored how many items were needed to achieve a selected precision of measurement (standard error ≤ .40).

RESULTS

The fixed-length CATs of 3, 6, and 12 items correlated well with full-length test results (with r = .90, .97 and .99, respectively). To achieve a standard error of .40 (approximate reliability = .84) only 3-7 items had to be administered for a large percentage of individuals.

CONCLUSIONS

This proof-of-concept investigation suggests that the widely used Matrix Reasoning subtest of the WAIS-IV might be shortened by more than 70% in most examinees while maintaining acceptable measurement precision. If similar savings could be realized in other tests, the accessibility of neuropsychological assessment might be markedly enhanced, and more efficient time use could lead to broader subdomain assessment.

Estimating individual health-related quality of life changes in low back pain patients

BACKGROUND

There is a need to evaluate different options for estimating individual change in health-related quality of life for patients with low back pain.

METHODS

Secondary analysis of data collected at baseline and 6 weeks later in a randomized trial of 749 adults with low back pain receiving usual medical care (UMC) or UMC plus chiropractic care at a small hospital at a military training site or two large military medical centers. The mean age was 31; 76% were male and 67% were White. The study participants completed the Patient-Reported Outcomes Measurement Information System (PROMIS®)-29 v 1.0 physical function, pain interference, pain intensity, fatigue, sleep disturbance, depression, anxiety, satisfaction with participation in social roles, physical summary, and mental health summary scores (T-scored with mean = 50 and standard deviation (SD) = 10 in the U.S. general population).

RESULTS

Reliability estimates at the baseline ranged from 0.700 to 0.969. Six-week test-retest intraclass correlation estimates were substantially lower than these estimates: the median test-retest intraclass correlation for the two-way mixed-effects model was 0. 532. Restricting the test-retest reliability estimates to the subset who reported they were about the same as at baseline on a retrospective rating of change item increased the median test-retest reliability to 0.686. The amount of individual change that was statistically significant varied by how reliability was estimated, and which SD was used. The smallest change needed was found when internal consistency reliability and the SD at baseline were used. When these values were used, the amount of change needed to be statistically significant (p < .05) at the individual level ranged from 3.33 (mental health summary scale) to 12.30 (pain intensity item) T-score points.

CONCLUSIONS

We recommend that in research studies estimates of the magnitude of individual change needed for statistical significance be provided for multiple reliability and standard deviation estimates. Whenever possible, patients should be classified based on whether they 1) improved significantly and perceived they got better, 2) improved significantly but did not perceive they were better, 3) did not improve significantly but felt they got better, or 4) did not improve significantly or report getting better.

Clinical Validity of the PROMIS Healthcare Engagement 8-Item Short Form

BACKGROUND

Healthcare engagement is a key measurement target for value-based healthcare, but a reliable and valid patient-reported measure has not yet been widely adopted.

OBJECTIVE

To assess the validity of a newly developed patient-reported measure of healthcare engagement, the 8-item PROMIS Healthcare Engagement (PHE-8a).

DESIGN

Prospective cohort study of the association between healthcare engagement and quality of care over 1 year. We fit mixed effects models of quality indicators as a function of engagement scores, adjusting for age, race/ethnicity, rural residence, and risk scores.

PARTICIPANTS

National stratified random sample of 9552 Veterans receiving Veterans Health Administration care for chronic conditions (hypertension, diabetes) or mental health conditions (depression, post-traumatic stress disorder).

MAIN MEASURES

Patient experience: Consumer Assessment of Health Plans and Systems communication and self-management support composites; no-show rates for primary care and mental health appointments; use of patient portal My HealtheVet; and Healthcare Effectiveness Data and Information Set electronic quality measures: HbA1c poor control, controlling high blood pressure, and hyperlipidemia therapy adherence.

KEY RESULTS

Higher engagement scores were associated with better healthcare quality across all outcomes, with each 5-point increase (1/2 standard deviation) in engagement scores associated with statistically significant and clinically meaningful gains in quality. Across the continuum of low to high engagement scores, we observed a concomitant reduction in primary care no-show rates of 37% and 24% for mental health clinics; an increased likelihood of My HealtheVet use of 15.4%; and a decreased likelihood of poor diabetes control of 44%.

CONCLUSIONS

The PHE-8a is a brief, reliable, and valid patient-reported measure of healthcare engagement. These results confirm previously untested hypotheses that patient engagement can promote healthcare quality.

Response Category Functioning on the Health Care Engagement Measure Using the Nominal Response Model

As part of a scale development project, we fit a nominal response item response theory model to responses to the Health Care Engagement Measure (HEM). When using the original 5-point response format, categories were not ordered as intended for six of the 23 items. For the remaining, the category boundary discrimination between Categories 0 (not at all true) and 1 (a little bit true) was only weakly discriminating, suggesting uninformative categories. When the lowest two categories were collapsed, psychometric properties improved greatly. Category boundary discriminations within items, however, varied significantly. Specifically, higher response category distinctions, such as responding 3 (very true) versus 2 (mostly true) were considerably more discriminating than lower response category distinctions. Implications for HEM scoring and for improving measurement precision at lower levels of the construct are presented as is the unique role of the nominal response model in category analysis.

Construct identification in the neuropsychological battery: What are we measuring?

OBJECTIVE

Major obstacles to data harmonization in neuropsychology include lack of consensus about what constructs and tests are most important and invariant across healthy and clinical populations. This study addressed these challenges using data from the National Neuropsychology Network (NNN).

METHOD

Data were obtained from 5,000 NNN participants and Pearson standardization samples. Analyses included variables from four instruments: Wechsler Adult Intelligence Scale, 4th Edition (WAIS-IV); Wechsler Memory Scale, 4th Edition (WMS-IV); California Verbal Learning Test, 3rd Edition (CVLT3); and Delis-Kaplan Executive Function System (D-KEFS). We used confirmatory factor analysis to evaluate models suggested by prior work and examined fit statistics and measurement invariance across samples. We examined relations of factor scores to demographic and clinical characteristics.

RESULTS

For each instrument, we identified four first-order and one second-order factor. Optimal models in patients generally paralleled the best-fitting models in the standardization samples, including task-specific factors. Analysis of the NNN data prompted specification of a Recognition-Familiarity factor on the WMS-IV and an Inhibition-Switching factor on the D-KEFS. Analyses showed strong to strict factorial invariance across samples with expected differences in factor means and variances. The Recognition-Familiarity factor correlated with age more strongly in NNN than in the standardization sample.

CONCLUSIONS

Factor models derived from healthy groups generally fit well in patients. NNN data helped identify novel Recognition-Familiarity and Inhibition-Switching factors that were also invariant across samples and may be clinically useful. The findings support efforts to identify evidence-based and optimally efficient measurements of neuropsychological constructs that are valid across groups. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Friendship Network Satisfaction: A multifaceted construct scored as a unidimensional scale

Although satisfying friendships are crucial for well-being throughout adulthood, measures of friendship satisfaction have been limited by: (1) item content relevant to children only, (2) a focus on single relationships rather than the friendship network, and (3) disagreement about the number of dimensions necessary to capture the construct. To overcome these limitations, we assembled an item pool from a number of existing measures, created additional items drawn from research on friendships, and then examined the structure and psychometric properties of those items in two online surveys of over 2000 respondents each. Factor analyses consistently identified two correlated factors-closeness and socializing-but bi-factor modeling revealed that scores on both subscales load strongly on a general factor, suggesting that the multifaceted content can be scored efficiently as a unidimensional composite. Analyses using item response theory (IRT) supported the creation of a reliable 14-item instrument that demonstrated adequate convergent and predictive validity. Thus, the Friendship Network Satisfaction (FNS) Scale is a psychometrically sound tool to advance research on friendships across the lifespan.

Impact of stress resilience and susceptibility on fear learning, anxiety, and alcohol intake

Post-traumatic stress disorder (PTSD) can develop after exposure to traumatic events and severely impacts the quality of life. PTSD is frequently comorbid with substance use disorders, with alcoholism being particularly common. However, not everyone who experiences trauma develops PTSD and the factors that render individuals susceptible or resilient to the effects of stress are unknown although gender appears to play an important role. Rodent models of stress exposure such as stress-enhanced fear learning (SEFL) recapitulate some aspects of PTSD symptomology, making them an invaluable tool for studying this disorder. This study examined whether exposure to a modified version of the SEFL procedure (4 footshocks instead of the standard 15 over 90 min) would reveal both susceptible and resilient subjects. Following stress exposure, distinct susceptible and resilient groups emerged that differed in fear learning and anxiety-related behavior as well as voluntary alcohol intake. Some aspects of stress susceptibility manifested differently in males compared to females, with susceptibility associated with increased alcohol intake in males and increased baseline anxiety in females.

Matching IRT Models to Patient-Reported Outcomes Constructs: The Graded Response and Log-Logistic Models for Scaling Depression

Item response theory (IRT) model applications extend well beyond cognitive ability testing, and various patient-reported outcomes (PRO) measures are among the more prominent examples. PRO (and like) constructs differ from cognitive ability constructs in many ways, and these differences have model fitting implications. With a few notable exceptions, however, most IRT applications to PRO constructs rely on traditional IRT models, such as the graded response model. We review some notable differences between cognitive and PRO constructs and how these differences can present challenges for traditional IRT model applications. We then apply two models (the traditional graded response model and an alternative log-logistic model) to depression measure data drawn from the Patient-Reported Outcomes Measurement Information System project. We do not claim that one model is "a better fit" or more "valid" than the other; rather, we show that the log-logistic model may be more consistent with the construct of depression as a unipolar phenomenon. Clearly, the graded response and log-logistic models can lead to different conclusions about the psychometrics of an instrument and the scaling of individual differences. We underscore, too, that, in general, explorations of which model may be more appropriate cannot be decided only by fit index comparisons; these decisions may require the integration of psychometrics with theory and research findings on the construct of interest.

Using Item Response Theory to Identify Responders to Treatment: Examples with the Patient-Reported Outcomes Measurement Information System (PROMIS®) Physical Function Scale and Emotional Distress Composite

The reliable change index has been used to evaluate the significance of individual change in health-related quality of life. We estimate reliable change for two measures (physical function and emotional distress) in the Patient-Reported Outcomes Measurement Information System (PROMIS®) 29-item health-related quality of life measure (PROMIS-29 v2.1). Using two waves of data collected 3 months apart in a longitudinal observational study of chronic low back pain and chronic neck pain patients receiving chiropractic care, and simulations, we compare estimates of reliable change from classical test theory fixed standard errors with item response theory standard errors from the graded response model. We find that unless true change in the PROMIS physical function and emotional distress scales is substantial, classical test theory estimates of significant individual change are much more optimistic than estimates of change based on item response theory.

Psychometric evaluation of a patient-reported item bank for healthcare engagement

PURPOSE

Healthcare engagement is a core measurement target for efforts to improve healthcare systems. This construct is broadly defined as the extent to which healthcare services represent collaborative partnerships with patients. Previous qualitative work operationalized healthcare engagement as generalized self-efficacy in four related subdomains: self-management, collaborative communication, health information use, and healthcare navigation. Building on this work, our objective was to establish a healthcare engagement instrument that is sufficiently unidimensional to yield a single score.

METHOD

We conducted cognitive interviews followed by a nation-wide mail survey of US Veteran Administration (VA) healthcare users. Data were collected on 49 candidate healthcare engagement items, as well as measures of self-efficacy for managing symptoms, provider communication, and perceived access. Items were subjected to exploratory bifactor, statistical learning, and IRT analyses.

RESULTS

Cognitive interviews were completed by 56 patients and 9552 VA healthcare users with chronic conditions completed the mail survey. Participants were mostly white and male but with sizable minority participation. Psychometric analyses and content considerations reduced the item pool to 23 items, which demonstrated a strong general factor (OmegaH of .89). IRT analyses revealed a high level of reliability across the trait range and little DIF across groups. Most health information use items were removed during analyses, suggesting a more independent role for this domain.

CONCLUSION

We provide quantitative evidence for a relatively unidimensional measure of healthcare engagement. Despite developed with VA healthcare users, the measure is intended for general use. Future work includes short-form development and validation with other patient groups.

Extensions of Multiple-Group Item Response Theory Alignment: Application to Psychiatric Phenotypes in an International Genomics Consortium

Large-scale studies spanning diverse project sites, populations, languages, and measurements are increasingly important to relate psychological to biological variables. National and international consortia already are collecting and executing mega-analyses on aggregated data from individuals, with different measures on each person. In this research, we show that Asparouhov and Muthén's alignment method can be adapted to align data from disparate item sets and response formats. We argue that with these adaptations, the alignment method is well suited for combining data across multiple sites even when they use different measurement instruments. The approach is illustrated using data from the Whole Genome Sequencing in Psychiatric Disorders consortium and a real-data-based simulation is used to verify accurate parameter recovery. Factor alignment appears to increase precision of measurement and validity of scores with respect to external criteria. The resulting parameter estimates may further inform development of more effective and efficient methods to assess the same constructs in prospectively designed studies.

Don’t Forget the Model in Your Model-based Reliability Coefficients: A Reply to McNeish (2018)

McNeish (2018) advocates that researchers abandon coefficient alpha in favor of alternative reliability measures, such as the 1-factor reliability (coefficient omega), a total reliability coefficient based on an exploratory bifactor solution (“Revelle’s omega total”), and the glb (“greatest lower bound”). McNeish supports this argument by demonstrating that these coefficients produce higher sample values in several examples. We express three main disagreements with this article. First, we show that McNeish exaggerates the extent to which alpha is different from omega when unidimensionality holds. Second, we argue that, when unidimensionality is violated, most alternative reliability coefficients are model-based, and it is critical to carefully select the underlying latent variable model rather than relying on software defaults. Third, we point out that higher sample reliability values do not necessarily capture population reliability better: many alternative reliability coefficients are upwardly biased except in very large samples. We conclude with a set of alternative recommendations for researchers.

Neuropsychological tests of the future: How do we get there from here?

OBJECTIVE

This article reviews current approaches to neuropsychological assessment, identifies opportunities for development of new methods using modern psychometric theory and advances in technology, and suggests a transition path that promotes application of novel methods without sacrificing validity.

METHODS

Theoretical/state-of-the-art review.

CONCLUSIONS

Clinical neuropsychological assessment today does not reflect advances in neuroscience, modern psychometrics, or technology. Major opportunities for improving practice include both psychometric and technological strategies. Modern psychometric approaches including item response theory (IRT) enable linking procedures that can place different measures on common scales; adaptive testing algorithms that can dramatically increase efficiency of assessment; examination of differential item functioning (DIF) to detect measures that behave differently in different groups; and person fit statistics to detect aberrant patterns of responding of high value for performance validity testing. Opportunities to introduce novel technologies include computerized adaptive testing, Web-based assessment, healthcare- and bio-informatics strategies, mobile platforms, wearables, and the 'internet-of-things'. To overcome inertia in current practices, new methods must satisfy requirements for back-compatibility with legacy instrumentation, enabling us to leverage the wealth of validity data already accrued for classic procedures. A path to achieve these goals involves creation of a global network to aggregate item-level data into a shared repository that will enable modern psychometric analyses to refine existing methods, and serve as a platform to evolve novel assessment strategies, which over time can revolutionize neuropsychological assessment practices world-wide.

Ecological validity of a quantitative classification system for mental illness in treatment-seeking adults

Quantitative models of mental illness, such as the Hierarchical Taxonomy of Psychopathology (HiTOP), aim to optimize clinical assessment, which conventionally follows categorical diagnostic rubrics. The evidence base for these models is robust, but also uniform; available data come mostly from structured diagnostic interviews in nationally representative samples. It remains to be seen whether HiTOP adequately reflects mental illness as evaluated in routine clinical care, where diagnosis is often unsystematic and incomplete, relative to controlled research conditions. To test the generalizability of a quantitative nosology to real-world assessment contexts, we fit the HiTOP model to diagnoses in a large sample (N = 25,002) of treatment-seeking university students who were seen by health professionals in everyday practice. We then examined the criterion validity of model components in relation to clinically relevant outcomes (i.e., suicide attempts, self-injury, and binge drinking). Three related structures fit the data well: a correlated-factor model with internalizing, externalizing, and eating pathology dimensions; a higher-order model that added a general factor of psychopathology that spanned these 3 first-order factors; and a bifactor model that partitioned diagnostic (co)variance across a general factor and 3 orthogonal group factors. The first-order factors had expected patterns of criterion validity, and the general factor was a strong predictor of suicidality and self-injury, paralleling past research. Bifactor model group factors had interpretative problems, however. Across models, categorical diagnoses consistently offered minimal incremental validity relative to the transdiagnostic factors. We conclude that HiTOP is ecologically valid-explaining comorbidity patterns among diagnoses assigned "in the field"-and is poised to enhance clinical assessment and decision-making in routine care. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Bifactor Modeling of the Positive and Negative Syndrome Scale: Generalized Psychosis Spans Schizoaffective, Bipolar, and Schizophrenia Diagnoses

OBJECTIVE

Common genetic variation spans schizophrenia, schizoaffective and bipolar disorders, but historically, these syndromes have been distinguished categorically. A symptom dimension shared across these syndromes, if such a general factor exists, might provide a clearer target for understanding and treating mental illnesses that share core biological bases.

METHOD

We tested the hypothesis that a bifactor model of the Positive and Negative Syndrome Scale (PANSS), containing 1 general factor and 5 specific factors (positive, negative, disorganized, excited, anxiety), explains the cross-diagnostic structure of symptoms better than the traditional 5-factor model, and examined the extent to which a general factor reflects the overall severity of symptoms spanning diagnoses in 5094 total patients with a diagnosis of schizophrenia, schizoaffective, and bipolar disorder.

RESULTS

The bifactor model provided superior fit across diagnoses, and was closer to the "true" model, compared to the traditional 5-factor model (Vuong test; P < .001). The general factor included high loadings on 28 of the 30 PANSS items, omitting symptoms associated with the excitement and anxiety/depression domains. The general factor had highest total loadings on symptoms that are often associated with the positive and disorganization syndromes, but there were also substantial loadings on the negative syndrome thus leading to the interpretation of this factor as reflecting generalized psychosis.

CONCLUSIONS

A bifactor model derived from the PANSS can provide a stronger framework for measuring cross-diagnostic psychopathology than a 5-factor model, and includes a generalized psychosis dimension shared at least across schizophrenia, schizoaffective, and bipolar disorder.

A 10-minute measure of global cognition: Validation of the Brief Cognitive Assessment Tool for Schizophrenia (B-CATS)

INTRODUCTION

Schizophrenia is marked by a global cognitive impairment that contributes significantly to chronic disability and unemployment. As new treatments are developed for cognition in schizophrenia, clinicians require easily administered instruments to assess cognition. We previously developed a very brief cognitive battery (Bell et al., 2005). The Brief Cognitive Assessment Tool for Schizophrenia (B-CATS) was developed specifically to provide clinicians with a way to assess cognition in their patients with schizophrenia. Here, we report the results of a validity study comparing B-CATS to a larger neurocognitive battery, the Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS) battery.

METHODS

Outpatients with schizophrenia (N=91) were administered the B-CATS and the non-overlapping tests of the MATRICS battery at two time points separated by 1month. They were also administered the UCSD Performance-based Skills Assessment-Brief (UPSA-B), a measure of functional capacity.

RESULT

The B-CATS has an administration time of approximately 10min. It demonstrates good test-retest reliability and internal consistency. It correlates 0.76 (p<0.01) with the MATRICS battery. The shorter B-CATS and the MATRICS battery correlate with the UPSA-B at 0.50 and 0.58 respectively.

CONCLUSION

A 10-minute version of the B-CATS correlates highly with the "gold standard" neurocognitive battery that has an administration time of over 60min. Both measures correlate moderately with a measure of functional capacity. This brief battery was designed to allow clinicians to monitor cognitive change and better inform treatment decisions.

Development and public release of a computerized adaptive (CAT) version of the Schizotypal Personality Questionnaire

One of the most widely used measures of psychosis-related symptoms and characteristics is the 74-item Schizotypal Personality Questionnaire (SPQ). Using multidimensional Item Response Theory (bifactor 2-parameter model), we calibrated SPQ items in a sample of 375 youths aged 9-24 years and constructed a fully functional computerized adaptive form of the SPQ on an open-source platform for public use. To assess validity, we used the above parameters to simulate CAT sessions in a separate validation sample (N = 100) using three test-length-based stopping rules: 8 items, 16 items, and 32 items. Those scores were then compared to full-form and SPQ-Brief scores on their abilities to predict psychosis or clinical risk status. Areas under the receiver operating characteristic curves indicated mediocre predictive ability, but did not differ among any of the forms, even when only eight adaptive items were administered. The Youden index for the 16-item adaptive version was higher than that for the 22-item SPQ-Brief. Classification accuracy for the full SPQ was 73% compared to 66% for the both the SPQ-Brief and adaptive versions (average of three stopping rules). The SPQ-CAT shows promise as a much shorter but valid assessment of schizotypy which can save time with minimal loss of information.

The global therapist competence scale for youth psychosocial treatment: Development and initial validation

OBJECTIVE

We describe the development and initial psychometric properties of the observer-rated Global Therapist Competence Scale for Youth Psychosocial Treatment (G-COMP) in the context of cognitive-behavioral treatment (CBT) for youth anxiety disorders.

METHOD

Independent coders rated 744 sessions from a sample of 68 youth (mean age = 10.56 years) using the G-COMP and the instruments of alliance, involvement, CBT adherence, CBT competence.

RESULTS

Inter-rater reliability coefficients, ICC(2,2), were greater than .60 for the 5 G-COMP domain scores. G-COMP scores yielded small to medium correlations with instruments of alliance (rs = .17-.44) and youth involvement in treatment (rs = .08-.53), and medium to large correlations with instruments of CBT competence and adherence (rs = .26-.63). Therapists in the research setting were rated higher compared to newly trained therapists in community clinics.

CONCLUSION

Preliminary reliability and validity of the G-COMP are promising, but future research is needed with non-CBT samples.

Disparity between General Symptom Relief and Remission Criteria in the Positive and Negative Syndrome Scale (PANSS): A Post-treatment Bifactor Item Response Theory Model

Objective: Total scale scores derived by summing ratings from the 30-item PANSS are commonly used in clinical trial research to measure overall symptom severity, and percentage reductions in the total scores are sometimes used to document the efficacy of treatment. Acknowledging that some patients may have substantial changes in PANSS total scores but still be sufficiently symptomatic to warrant diagnosis, ratings on a subset of 8 items, referred to here as the "Remission set," are sometimes used to determine if patients' symptoms no longer satisfy diagnostic criteria. An unanswered question remains: is the goal of treatment better conceptualized as reduction in overall symptom severity, or reduction in symptoms below the threshold for diagnosis? We evaluated the psychometric properties of PANSS total scores, to assess whether having low symptom severity post-treatment is equivalent to attaining Remission. Design: We applied a bifactor item response theory (IRT) model to post-treatment PANSS ratings of 3,647 subjects diagnosed with schizophrenia assessed at the termination of 11 clinical trials. The bifactor model specified one general dimension to reflect overall symptom severity, and five domain-specific dimensions. We assessed how PANSS item discrimination and information parameters varied across the range of overall symptom severity (θ), with a special focus on low levels of symptoms (i.e., θ<-1), which we refer to as "Relief" from symptoms. A score of θ=-1 corresponds to an expected PANSS item score of 1.83, a rating between "Absent" and "Minimal" for a PANSS symptom. Results: The application of the bifactor IRT model revealed: (1) 88% of total score variation was attributable to variation in general symptom severity, and only 8% reflected secondary domain factors. This implies that a general factor may provide a good indicator of symptom severity, and that interpretation is not overly complicated by multidimensionality; (2) Post-treatment, 534 individuals (about 15% of the whole sample) scored in the "Relief" range of general symptom severity, but more than twice that number (n = 1351) satisfied Remission criteria (37%). 2 in 3 Remitted patients had scores that were not in a low symptom range (corresponding to Absent or Minimal item scores); (3) PANSS items vary greatly in their ability to measure the general symptom severity dimension; while many items are highly discriminating and relatively "pure" indicators of general symptom severity (delusions, conceptual disorganization), others are better indicators of specific dimensions (blunted affect, depression). The utility of a given PANSS item for assessing a patient depended on the illness level of the patient. Conclusion: Satisfying conventional Remission criteria was not strongly associated with low levels of symptoms. The items providing the most information for patients in the symptom Relief range were Delusions, Preoccupation, Suspiciousness Persecution, Unusual Thought Content, Conceptual Disorganization, Stereotyped Thinking, Active Social Avoidance, and Lack of Judgment and Insight. Lower scores on these items (item scores ≤2) were strongly associated with having a low latent trait θ or experiencing overall symptom relief. The inter-rater agreement between Remission and Relief subjects suggested that these criteria identified different subsets of patients. Alternative subsets of items may offer better indicators of general symptom severity and provide better discrimination (and lower standard errors) for scaling individuals and judging symptom relief, where the "best" subset of items ultimately depends on the illness range and treatment phase being evaluated.

Measuring pathology using the PANSS across diagnoses: Inconsistency of the positive symptom domain across schizophrenia, schizoaffective, and bipolar disorder

Although the Positive and Negative Syndrome Scale (PANSS) was developed for use in schizophrenia (SZ), antipsychotic drug trials use the PANSS to measure symptom change also for bipolar (BP) and schizoaffective (SA) disorder, extending beyond its original indications. If the dimensions measured by the PANSS are different across diagnoses, then the same score change for the same drug condition may have different meanings depending on which group is being studied. Here, we evaluated whether the factor structure in the PANSS was consistent across schizophrenia (n = 3647), bipolar disorder (n = 858), and schizoaffective disorder (n = 592). Along with congruency coefficients, Hancock's H, and Jaccard indices, we used target rotations and statistical tests of invariance based on confirmatory factor models. We found the five symptom dimensions measured by the 30-item PANSS did not generalize well to schizoaffective and bipolar disorders. A model based on an 18-item version of the PANSS generalized better across SZ and BP groups, but significant problems remained in generalizing some of the factors to the SA sample. Schizophrenia and bipolar disorder showed greater similarity in factor structure than did schizophrenia and schizoaffective disorder. The Anxiety/Depression factor was the most consistent across disorders, while the Positive factor was the least consistent.

Alternative Approaches to Addressing Non-Normal Distributions in the Application of IRT Models to Personality Measures

It is generally assumed that the latent trait is normally distributed in the population when estimating logistic item response theory (IRT) model parameters. This assumption requires that the latent trait be fully continuous and the population homogenous (i.e., not a mixture). When this normality assumption is violated, models are misspecified, and item and person parameter estimates are inaccurate. When normality cannot be assumed, it might be appropriate to consider alternative modeling approaches: (a) a zero-inflated mixture, (b) a log-logistic, (c) a Ramsay curve, or (d) a heteroskedastic-skew model. The first 2 models were developed to address modeling problems associated with so-called quasi-continuous or unipolar constructs, which apply only to a subset of the population, or are meaningful at one end of the continuum only. The second 2 models were developed to address non-normal latent trait distributions and violations of homogeneity of error variance, respectively. To introduce these alternative IRT models and illustrate their strengths and weaknesses, we performed real data application comparing results to those from a graded response model. We review both statistical and theoretical challenges in applying these models and choosing among them. Future applications of these and other alternative models (e.g., unfolding, diffusion) are needed to advance understanding about model choice in particular situations.

Responsiveness to Change and Minimally Important Differences of the Patient-Reported Outcomes Measurement Information System Gastrointestinal Symptoms Scales

BACKGROUND

The NIH-sponsored Patient-Reported Outcomes Measurement Information System (PROMIS) Gastrointestinal (GI) Symptoms scales were developed to assess patients' GI symptoms in clinical settings.

AIMS

To assess responsiveness to change and provide minimally important difference (MID) estimates for the PROMIS GI Symptoms scales.

METHODS

A sample of 256 GI outpatients self-administered the eight PROMIS GI Symptoms scales (gastroesophageal reflux, disrupted swallowing, diarrhea, bowel incontinence/soilage, nausea and vomiting, constipation, belly pain, and gas/bloating/flatulence) at two visits. Patient self-reported and physician-reported assessments of the subjects' overall GI condition were employed as change anchors. In addition, we prospectively assessed change at both visits using a GI-symptom anchor, the Gastrointestinal Symptom Rating Scale (GSRS). Responsiveness to change was assessed using F-statistics. The minimally changed group was those somewhat better or somewhat worse on the retrospective anchors and changing by one category on the modified GSRS (e.g., from slight to mild discomfort to moderate to moderately severe discomfort).

RESULTS

Responsiveness to change was statistically significant for 6 of 8 PROMIS scales using the self-report GI anchor, 3 of 8 scales using the physician-reported anchor, and 5 of 5 scales using the corresponding GSRS scales as anchors. The MID estimates for scales for improvement and worsening were about 0.5-0.6 SD using the GSRS anchor and generally larger in magnitude than the change for the "about the same" group.

CONCLUSIONS

The responsiveness and MID estimates provided here for the PROMIS GI Symptoms scales can aid in scale score interpretation in clinical trials and observational studies.

Differential item functioning of the patient-reported outcomes information system (PROMIS®) pain interference item bank by language (Spanish versus English)

BACKGROUND

About 70% of Latinos, 5 years old or older, in the United States speak Spanish at home. Measurement equivalence of the PROMIS^® pain interference (PI) item bank by language of administration (English versus Spanish) has not been evaluated.

METHODS

A sample of 527 adult Spanish-speaking Latinos completed the Spanish version of the 41-item PROMIS^® pain interference item bank. We evaluate dimensionality, monotonicity and local independence of the Spanish-language items. Then we evaluate differential item functioning (DIF) using ordinal logistic regression with item response theory scores estimated from DIF-free "anchor" items.

RESULTS

One of the 41 items in the Spanish version of the PROMIS^® PI item bank was identified as having significant uniform DIF.

CONCLUSIONS

English- and Spanish-speaking subjects with the same level of pain interference responded differently to 1 of the 41 items in the PROMIS^® PI item bank. This item was not retained due to proprietary issues. The original English language item parameters can be used when estimating PROMIS^® PI scores.

Finding Pure Sub-Models for Improved Differentiation of Bi-Factor and Second-Order Models

Several studies have indicated that bi-factor models fit a broad range of psychometric data better than alternative multidimensional models such as second-order models, e.g Rodriguez, Reise and Haviland (2016), Gignac (2016), and Carnivez (2016). Murray and Johnson (2013) and Gignac (2016) argue that this phenomenon is partially due to un-modeled complexities (e.g. un-modeled cross-factor loadings) that induce a bias in standard statistical measures that favors bi-factor models over second-order models. We extend the Murray and Johnson simulation studies to show how the ability to distinguish second-order and bi-factor models diminishes as the amount of un-modeled complexity increases. By using theorems about rank constraints on the covariance matrix to find sub-models of measurement models that have less un-modeled complexity, we are able to reduce the statistical bias in favor of bi-factor models; this allows researchers to reliably distinguish between bi-factor and second-order models.

Assessment of attachment in psychosis: A psychometric cause for concern

Attachment has recently been proposed as a key developmental construct in psychosis, in particular with respect to interpersonal functioning and social cognition. The current study examined the latent structure of the self-report Psychosis Attachment Measure (PAM) and its relationship to lower-level perceptual and higher-order inferential social cognitive processes. The PAM was administered to 138 psychiatrically stable outpatients with schizophrenia alongside a battery of symptom, social cognitive, and functional measures. PAM responses were analyzed using latent variable measurement models, which did not yield evidence of the coherent two-dimensional structure predicted by previous literature. A unidimensional subscale comprising 6 of the 16 original PAM items possessed the strongest psychometric properties. This subscale was generally uncorrelated with social cognitive measures and showed weak correlations with some symptoms measures and with community functioning. These results suggest that either the PAM may not measure attachment in psychosis or it may measure only attachment anxiety but demonstrate little construct validity in this population. These results tell a cautionary tale regarding making theoretical inferences on the basis of measures without coherent latent structure. Attachment measures with stronger psychometric properties will help clarify putative relationships between attachment and social cognitive processes in psychosis.

Is the Bifactor Model a Better Model or Is It Just Better at Modeling Implausible Responses? Application of Iteratively Reweighted Least Squares to the Rosenberg Self-Esteem Scale

Although the structure of the Rosenberg Self-Esteem Scale (RSES) has been exhaustively evaluated, questions regarding dimensionality and direction of wording effects continue to be debated. To shed new light on these issues, we ask (a) for what percentage of individuals is a unidimensional model adequate, (b) what additional percentage of individuals can be modeled with multidimensional specifications, and (c) what percentage of individuals respond so inconsistently that they cannot be well modeled? To estimate these percentages, we applied iteratively reweighted least squares (IRLS) to examine the structure of the RSES in a large, publicly available data set. A distance measure, d_s, reflecting a distance between a response pattern and an estimated model, was used for case weighting. We found that a bifactor model provided the best overall model fit, with one general factor and two wording-related group factors. However, on the basis of d_r values, a distance measure based on individual residuals, we concluded that approximately 86% of cases were adequately modeled through a unidimensional structure, and only an additional 3% required a bifactor model. Roughly 11% of cases were judged as "unmodelable" due to their significant residuals in all models considered. Finally, analysis of d_s revealed that some, but not all, of the superior fit of the bifactor model is owed to that model's ability to better accommodate implausible and possibly invalid response patterns, and not necessarily because it better accounts for the effects of direction of wording.

Exploratory Bifactor Analysis: The Schmid-Leiman Orthogonalization and Jennrich-Bentler Analytic Rotations

Analytic bifactor rotations have been recently developed and made generally available, but they are not well understood. The Jennrich-Bentler analytic bifactor rotations (bi-quartimin and bi-geomin) are an alternative to, and arguably an improvement upon, the less technically sophisticated Schmid-Leiman orthogonalization. We review the technical details that underlie the Schmid-Leiman and Jennrich-Bentler bifactor rotations, using simulated data structures to illustrate important features and limitations. For the Schmid-Leiman, we review the problem of inaccurate parameter estimates caused by the linear dependencies, sometimes called "proportionality constraints," that are required to expand a p correlated factors solution into a (p + 1) (bi)factor space. We also review the complexities involved when the data depart from perfect cluster structure (e.g., item cross-loading on group factors). For the Jennrich-Bentler rotations, we describe problems in parameter estimation caused by departures from perfect cluster structure. In addition, we illustrate the related problems of (a) solutions that are not invariant under different starting values (i.e., local minima problems) and (b) group factors collapsing onto the general factor. Recommendations are made for substantive researchers including examining all local minima and applying multiple exploratory techniques in an effort to identify an accurate model.

An Empirical Comparison of Acculturation Models

The unidirectional and bidirectional models of acculturation were compared in a sample of 291 Asian Americans. Both models produced good predictions of Asian preferences, cultural knowledge, ethnic identification, and generational status. The bidirectional model, however, failed to demonstrate its reputed independence across home culture and host culture orientations. The unidirectional model is recommended as an economical proxy measure of acculturation, the bidirectional model is recommended for full theoretical investigations of acculturation, and a speculative tridirectional model is proposed to clarify the distinction between acculturation and ethnogenesis (the creation of a new ethnicity).

Item response theory and the measurement of psychiatric constructs: some empirical and conceptual issues and challenges

Item response theory (IRT) measurement models are now commonly used in educational, psychological, and health-outcomes measurement, but their impact in the evaluation of measures of psychiatric constructs remains limited. Herein we present two, somewhat contradictory, theses. The first is that, when skillfully applied, IRT has much to offer psychiatric measurement in terms of scale development, psychometric analysis, and scoring. The second argument, however, is that psychiatric measurement presents some unique challenges to the application of IRT - challenges that may not be easily addressed by application of conventional IRT models and methods. These challenges include, but are not limited to, the modeling of conceptually narrow constructs and their associated limited item pools, and unipolar constructs where the expected latent trait distribution is highly skewed.

Multilevel Modeling and its Application in Counseling Psychology Research

Multilevel modeling (MLM) should be used when a researcher has collected hierarchical data. For example, when a researcher investigates an outcome variable (e.g., depression) with several clients drawn from different clinicians, the data set has a hierarchical structure. Herein, we describe the use of MLM in counseling research. The goals include the following: (a) to specify research contexts where MLM may be applied, (b) to describe how to conduct data analyses using MLM, and (c) to highlight key statistical and design issues encountered when analyzing hierarchical data. We also highlight how MLM can be used (a) to provide valid statistical inference in the presence of hierarchical data structure, (b) to separate the within-group effects from between-group effects for predictor variables, and (c) to study the interactions among predictor variables drawn from different levels (e.g., variables drawn from both clients and their clinicians).

Item Response Theory

Item response theory (IRT) is an increasingly popular approach to the development, evaluation, and administration of psychological measures. We introduce, first, three IRT fundamentals: (a) item response functions, (b) information functions, and (c) invariance. We next illustrate how IRT modeling can improve the quality of psychological measurement. Available evidence suggests that the differences between IRT and traditional psychometric methods are not trivial; IRT applications can improve the precision and validity of psychological research across a wide range of subjects.

The Emergence of Item Response Theory Models and the Patient Reported Outcomes Measurement Information Systems

Item response theory (IRT) models emerged to solve practical testing problems in large-scale cognitive achievement and aptitude assessment. Within the last decade, an explosion of IRT applications have occurred in the non-cognitive domain. In this report, I highlight the development, implementation, and results of a single project: Patient Reported Outcomes Measurement Information Systems (PROMIS). The PROMIS projectreflects the state-of-the-art application of IRT in the non-cognitive domain, and has produced important advancements in patient reported outcomes measurement.However, the project also illustrates challenges that confront researchers wishing to apply IRT to non-cognitive constructs. These challenges are: a) selecting a population to set the metric for interpretation of item parameters, b) working with non-normal quasi-continuous latent traits, and c) working with narrow-bandwidth constructs that potentially have a limitedpool of potential indicators. Differences between cognitive and non-cognitive measurement contexts are discussed and directions for future research suggested.

Development of an itemwise efficiency scoring method: Concurrent, convergent, discriminant, and neuroimaging-based predictive validity assessed in a large community sample

Traditional "paper-and-pencil" testing is imprecise in measuring speed and hence limited in assessing performance efficiency, but computerized testing permits precision in measuring itemwise response time. We present a method of scoring performance efficiency (combining information from accuracy and speed) at the item level. Using a community sample of 9,498 youths age 8-21, we calculated item-level efficiency scores on 4 neurocognitive tests, and compared the concurrent, convergent, discriminant, and predictive validity of these scores with simple averaging of standardized speed and accuracy-summed scores. Concurrent validity was measured by the scores' abilities to distinguish men from women and their correlations with age; convergent and discriminant validity were measured by correlations with other scores inside and outside of their neurocognitive domains; predictive validity was measured by correlations with brain volume in regions associated with the specific neurocognitive abilities. Results provide support for the ability of itemwise efficiency scoring to detect signals as strong as those detected by standard efficiency scoring methods. We find no evidence of superior validity of the itemwise scores over traditional scores, but point out several advantages of the former. The itemwise efficiency scoring method shows promise as an alternative to standard efficiency scoring methods, with overall moderate support from tests of 4 different types of validity. This method allows the use of existing item analysis methods and provides the convenient ability to adjust the overall emphasis of accuracy versus speed in the efficiency score, thus adjusting the scoring to the real-world demands the test is aiming to fulfill. (PsycINFO Database Record

Evaluating bifactor models: Calculating and interpreting statistical indices

Bifactor measurement models are increasingly being applied to personality and psychopathology measures (Reise, 2012). In this work, authors generally have emphasized model fit, and their typical conclusion is that a bifactor model provides a superior fit relative to alternative subordinate models. Often unexplored, however, are important statistical indices that can substantially improve the psychometric analysis of a measure. We provide a review of the particularly valuable statistical indices one can derive from bifactor models. They include omega reliability coefficients, factor determinacy, construct reliability, explained common variance, and percentage of uncontaminated correlations. We describe how these indices can be calculated and used to inform: (a) the quality of unit-weighted total and subscale score composites, as well as factor score estimates, and (b) the specification and quality of a measurement model in structural equation modeling. (PsycINFO Database Record

Applying Bifactor Statistical Indices in the Evaluation of Psychological Measures

The purpose of this study was to apply a set of rarely reported psychometric indices that, nevertheless, are important to consider when evaluating psychological measures. All can be derived from a standardized loading matrix in a confirmatory bifactor model: omega reliability coefficients, factor determinacy, construct replicability, explained common variance, and percentage of uncontaminated correlations. We calculated these indices and extended the findings of 50 recent bifactor model estimation studies published in psychopathology, personality, and assessment journals. These bifactor derived indices (most not presented in the articles) provided a clearer and more complete picture of the psychometric properties of the assessment instruments. We reached 2 firm conclusions. First, although all measures had been tagged "multidimensional," unit-weighted total scores overwhelmingly reflected variance due to a single latent variable. Second, unit-weighted subscale scores often have ambiguous interpretations because their variance mostly reflects the general, not the specific, trait. Finally, we review the implications of our evaluations and consider the limits of inferences drawn from a bifactor modeling approach.

Item Response Theory Analysis of ADHD Symptoms in Children With and Without ADHD

Item response theory (IRT) was separately applied to parent- and teacher-rated symptoms of attention-deficit/hyperactivity disorder (ADHD) from a pooled sample of 526 six- to twelve-year-old children with and without ADHD. The dimensional structure ADHD was first examined using confirmatory factor analyses, including the bifactor model. A general ADHD factor and two group factors, representing inattentive and hyperactive/impulsive dimensions, optimally fit the data. Using the graded response model, we estimated discrimination and location parameters and information functions for all 18 symptoms of ADHD. Parent- and teacher-rated symptoms demonstrated adequate discrimination and location values, although these estimates varied substantially. For parent ratings, the test information curve peaked between -2 and +2 SD, suggesting that ADHD symptoms exhibited excellent overall reliability at measuring children in the low to moderate range of the general ADHD factor, but not in the extreme ranges. Similar results emerged for teacher ratings, in which the peak range of measurement precision was from -1.40 to 1.90 SD Several symptoms were comparatively more informative than others; for example, is often easily distracted ("Distracted") was the most informative parent- and teacher-rated symptom across the latent trait continuum. Clinical implications for the assessment of ADHD as well as relevant considerations for future revisions to diagnostic criteria are discussed.

Initial development of a treatment adherence measure for cognitive-behavioral therapy for child anxiety

The measurement of treatment adherence (a component of treatment integrity defined as the extent to which a treatment is delivered as intended) is a critical element in treatment evaluation research. This article presents initial psychometric data for scores on the Cognitive-Behavioral Therapy Adherence Scale for Youth Anxiety (CBAY-A), an observational measure designed to be sensitive to common practice elements found in individual cognitive-behavioral therapy (ICBT) for youth anxiety. Therapy sessions (N = 954) from 1 efficacy and 1 effectiveness study of ICBT for youth anxiety were independently rated by 2 coders. Interrater reliability (as gauged by intraclass correlation coefficients) for the item scores averaged 0.77 (SD = 0.15; range .48 to .80). The CBAY-A item and scale (skills, model, total) scores demonstrated evidence of convergent and discriminant validity with an observational measure of therapeutic interventions and an observational measure of the alliance. The CBAY-A item and scale scores also discriminated between therapists delivering ICBT in research and practice settings and therapists delivering nonmanualized usual clinical care. We discuss the importance of replicating these psychometric findings in different samples and highlight possible application of an adherence measure in testing integrity-outcome relations.

Development of an abbreviated form of the Penn Line Orientation Test using large samples and computerized adaptive test simulation

Visuospatial processing is a commonly assessed neurocognitive domain with deficits linked to dysfunction in right posterior regions of the brain. With the growth of large-scale clinical research studies, there is an increased need for efficient and scalable assessments of neurocognition, including visuospatial processing. The purpose of the current study was to use a novel method that combines item response theory (IRT) and computerized adaptive testing (CAT) approaches to create an abbreviated form of the computerized Penn Line Orientation Test (PLOT). The 24-item PLOT was administered to 8,498 youths (aged 8-21 years) as part of the Philadelphia Neurodevelopmental Cohort study and, by Web-based data collection, in an independent sample of 4,593 adults from Great Britain as part of a TV documentary. IRT-based CAT simulations were used to select the best PLOT items for an abbreviated form by performing separate simulations in each group and choosing only items that were selected as useful (i.e., high item discrimination and in the appropriate difficulty range) in at least 1 of the simulations. Fifteen items were chosen for the final, short form of the PLOT, indicating substantial agreement among the models in how they evaluated each item's usefulness. Moreover, this abbreviated version performed comparably to the full version in tests of sensitivity to age and sex effects. This abbreviated version of the PLOT cuts administration time by 50% without detectable loss of information, which points to its feasibility for large-scale clinical and genomic studies.

Iteration of Partially Specified Target Matrices: Applications in Exploratory and Bayesian Confirmatory Factor Analysis

We describe and evaluate a factor rotation algorithm, iterated target rotation (ITR). Whereas target rotation (Browne, 2001) requires a user to specify a target matrix a priori based on theory or prior research, ITR begins with a standard analytic factor rotation (i.e., an empirically informed target) followed by an iterative search procedure to update the target matrix. In Study 1, Monte Carlo simulations were conducted to evaluate the performance of ITR relative to analytic rotations from the Crawford-Ferguson family with population factor structures varying in complexity. Simulation results: (a) suggested that ITR analyses will be particularly useful when evaluating data with complex structures (i.e., multiple cross-loadings) and (b) showed that the rotation method used to define an initial target matrix did not materially affect the accuracy of the various ITRs. In Study 2, we: (a) demonstrated the application of ITR as a way to determine empirically informed priors in a Bayesian confirmatory factor analysis (BCFA; Muthén & Asparouhov, 2012) of a rater-report alexithymia measure (Haviland, Warren, & Riggs, 2000) and (b) highlighted some of the challenges when specifying empirically based priors and assessing item and overall model fit.

Structure and correlates of self-reported empathy in schizophrenia

Research on empathy in schizophrenia has relied on dated self-report scales that do not conform to contemporary social neuroscience models of empathy. The current study evaluated the structure and correlates of the recently-developed Questionnaire of Cognitive and Affective Empathy (QCAE) in schizophrenia. This measure, whose structure and validity was established in healthy individuals, includes separate scales to assess the two main components of empathy: Cognitive Empathy (assessed by two subscales) and Affective Empathy (assessed by three subscales). Stable outpatients with schizophrenia (n = 145) and healthy individuals (n = 45) completed the QCAE, alternative measures of empathy, and assessments of clinical symptoms, neurocognition, and functional outcome. Exploratory and confirmatory factor analyses provided consistent support for a two-factor solution in the schizophrenia group, justifying the use of separate cognitive and affective empathy scales in this population. However, one of the three Affective Empathy subscales was not psychometrically sound and was excluded from further analyses. Patients reported significantly lower Cognitive Empathy but higher Affective Empathy than controls. Among patients, the QCAE scales showed significant correlations with an alternative self-report empathy scale, but not with performance on an empathic accuracy task. The QCAE Cognitive Empathy subscales also showed significant, though modest, correlations with negative symptoms and functional outcome. These findings indicate that structure of self-reported empathy is similar in people with schizophrenia and healthy subjects, and can be meaningfully compared between groups. They also contribute to emerging evidence that some aspects of empathy may be intact or hyper-responsive in schizophrenia.

Psychometric properties of the Penn Computerized Neurocognitive Battery

OBJECTIVE

The Penn Computerized Neurocognitive Battery (CNB) was designed to measure performance accuracy and speed on specific neurobehavioral domains using tests that were previously validated with functional neuroimaging. The goal of the present study was to evaluate the neuropsychological theory used to construct the CNB by confirming the factor structure of the tests composing it.

METHOD

In a large community sample (N = 9,138; age range 8-21), we performed a correlated-traits confirmatory factor analysis (CFA) and multiple exploratory factor analyses (EFAs) on the 12 CNB measures of Efficiency (which combine Accuracy and Speed). We then performed EFAs of the Accuracy and Speed measures separately. Finally, we performed a confirmatory bifactor analysis of the Efficiency scores. All analyses were performed with Mplus using maximum likelihood estimation.

RESULTS

RESULTS strongly support the a priori theory used to construct the CNB, showing that tests designed to measure executive, episodic memory, complex cognition, and social cognition aggregate their loadings within these domains. When Accuracy and Speed were analyzed separately, Accuracy produced 3 reliable factors: executive and complex cognition, episodic memory, and social cognition, while speed produced 2 factors: tests that require fast responses and those where each item requires deliberation. The statistical "Fit" of almost all models described above was acceptable (usually excellent).

CONCLUSIONS

Based on the analysis from these large-scale data, the CNB offers an effective means for measuring the integrity of intended neurocognitive domains in about 1 hour of testing and is thus suitable for large-scale clinical and genomic studies.

A Bifactor Model of Disgust Proneness

The current research evaluated a bifactor model for the Disgust Emotion Scale (DES) in three samples: N = 1,318 nonclinical participants, N = 152 clinic-referred patients, and N = 352 nonclinical participants. The primary goals were to (a) use bifactor modeling to examine the latent structure of the DES and in turn (b) evaluate whether the DES should be scored as a unidimensional scale or whether subscales should also be interpreted. Results suggested that a bifactor model fit the DES data well and that all DES items were strongly influenced by a general disgust proneness dimension and by five content dimensions. Moreover, model-based reliability analyses suggested that scoring a general disgust dimension is justified despite the confirmed multidimensional structure. However, subscales were found to be unreliable after controlling for the general disgust factor with the potential exception of the Mutilation/Death and Animals subscale. Subsequent analysis also showed that only the general disgust factor robustly predicted an obsessive-compulsive disorder symptom latent factor—a clinical condition closely related to disgust proneness; latent variables representing DES domains displayed weak relations with an obsessive-compulsive disorder factor above and beyond the general disgust factor. Implications for better understanding the structure of DES responses and its use in clinical research are discussed.

Adapting social neuroscience measures for schizophrenia clinical trials, part 3: fathoming external validity

It is unknown whether measures adapted from social neuroscience linked to specific neural systems will demonstrate relationships to external variables. Four paradigms adapted from social neuroscience were administered to 173 clinically stable outpatients with schizophrenia to determine their relationships to functionally meaningful variables and to investigate their incremental validity beyond standard measures of social and nonsocial cognition. The 4 paradigms included 2 that assess perception of nonverbal social and action cues (basic biological motion and emotion in biological motion) and 2 that involve higher level inferences about self and others' mental states (self-referential memory and empathic accuracy). Overall, social neuroscience paradigms showed significant relationships to functional capacity but weak relationships to community functioning; the paradigms also showed weak correlations to clinical symptoms. Evidence for incremental validity beyond standard measures of social and nonsocial cognition was mixed with additional predictive power shown for functional capacity but not community functioning. Of the newly adapted paradigms, the empathic accuracy task had the broadest external validity. These results underscore the difficulty of translating developments from neuroscience into clinically useful tasks with functional significance.

Adapting social neuroscience measures for schizophrenia clinical trials, Part 2: trolling the depths of psychometric properties

The psychometric properties of 4 paradigms adapted from the social neuroscience literature were evaluated to determine their suitability for use in clinical trials of schizophrenia. This 2-site study (University of California, Los Angeles and University of North Carolina) included 173 clinically stable schizophrenia outpatients and 88 healthy controls. The social cognition battery was administered twice to the schizophrenia group (baseline, 4-week retest) and once to the control group. The 4 paradigms included 2 that assess perception of nonverbal social and action cues (basic biological motion and emotion in biological motion) and 2 that involve higher level inferences about self and others' mental states (self-referential memory and empathic accuracy). Each paradigm was evaluated on (1) patient vs healthy control group differences, (2) test-retest reliability, (3) utility as a repeated measure, and (4) tolerability. Of the 4 paradigms, empathic accuracy demonstrated the strongest characteristics, including large between-group differences, adequate test-retest reliability (.72), negligible practice effects, and good tolerability ratings. The other paradigms showed weaker psychometric characteristics in their current forms. These findings highlight challenges in adapting social neuroscience paradigms for use in clinical trials.

The Cognitive Assessment Interview (CAI): reliability and validity of a brief interview-based measure of cognition

OBJECTIVE

To obtain Food and Drug Administration approval for the treatment of cognitive impairments associated with schizophrenia, a drug will need to demonstrate benefits beyond those that may be documented on objective cognitive tests. Interview-based measures of cognition such as the Cognitive Assessment Interview (CAI) are candidate coprimary outcome measures.

METHODS

Psychiatrically stable schizophrenia outpatients (n=150) were studied using the CAI to obtain information about cognitive functioning from both the patient and an informant. Patients also received objective assessments of neurocognition, functional capacity, functional outcome, and symptoms, at baseline and 1 month later.

RESULTS

The CAI had good internal consistency (Cronbach's alpha=.92) and good test-retest reliability (r=.83). The CAI was moderately correlated with objective neurocognitive test scores (r's=-.39 to -.41) and moderately correlated with social functioning (r=-.38), work functioning (r=-.48), and overall functional outcome (r=-.49). The correlations of CAI scores with external validity indicators did not differ significantly by source of information (patient alone ratings were valid). Overall functional outcome correlated more strongly with patient CAI scores (r=-.50) than with objective neurocognitive test scores (r=.29) or functional capacity (r=.29).

CONCLUSIONS

Field testing of the CAI produced reliable ratings of cognitive functioning that were correlated with functional outcome. Patient ratings alone yielded scores with reliability and validity values appropriate for use in clinical trials. The CAI appears to provide useful complementary information and possesses practical advantages for rating cognitive functioning including an interview-based method of administration, brief assessment time (15 min for the patient assessment), little or no practice effects, and ease of scoring.

The Barratt Impulsiveness Scale-11: reassessment of its structure in a community sample

The Barratt Impulsiveness Scale (Version 11; BIS-11; Patton, Stanford, & Barratt, 1995) is a gold-standard measure that has been influential in shaping current theories of impulse control, and has played a key role in studies of impulsivity and its biological, psychological, and behavioral correlates. Psychometric research on the structure of the BIS-11, however, has been scant. We therefore applied exploratory and confirmatory factor analyses to data collected using the BIS-11 in a community sample (N = 691). Our goal was to test 4 theories of the BIS-11 structure: (a) a unidimensional model, (b) a 6 correlated first-order factor model, (c) a 3 second-order factor model, and (d) a bifactor model. Among the problems identified were (a) low or near-zero correlations of some items with others; (b) highly redundant content of numerous item pairs; (c) items with salient cross-loadings in multidimensional solutions; and, ultimately, (d) poor fit to confirmatory models. We conclude that use of the BIS-11 total score as reflecting individual differences on a common dimension of impulsivity presents challenges in interpretation. Also, the theory that the BIS-11 measures 3 subdomains of impulsivity (attention, motor, and nonplanning) was not empirically supported. A 2-factor model is offered as an alternative multidimensional structural representation.

The Clinical Assessment Interview for Negative Symptoms (CAINS): final development and validation

OBJECTIVE

A major barrier to developing treatments for negative symptoms has been measurement concerns with existing assessment tools. Fulfilling the top recommendation of the National Institute of Mental Health's Consensus Development Conference on Negative Symptoms, the Clinical Assessment Interview for Negative Symptoms (CAINS) was developed using an iterative, empirical approach, and includes items assessing motivation, pleasure, and emotion expression. The authors employed multiple analytic techniques to develop the CAINS and here provide final development and validation results.

METHOD

The CAINS structure, interrater agreement, test-retest reliability, and convergent and discriminant validity were assessed in a large and diverse sample of 162 outpatients with schizophrenia or schizoaffective disorder recruited from four sites.

RESULTS

Three items with poor psychometric properties were removed, resulting in a 13-item CAINS. The CAINS factor structure was replicated, demonstrating two modestly correlated scales: expression (four items) and motivation/pleasure (nine items). The scales demonstrated good internal consistency, test-retest stability, and interrater agreement. Strong convergent validity was demonstrated by linkages with other negative symptom measures, self-report scales of sociality, pleasure, and motivation, and coded facial expressions. Discriminant validity was shown by independence from depression, medication side effects, and cognition. Notably, the CAINS scales were related to real-world vocational, independent living, and social/familial functioning.

CONCLUSIONS

The CAINS is an empirically developed and evaluated measure of negative symptoms. Findings indicate that the CAINS is brief yet comprehensive and employable across a wide range of research and clinical contexts.

The importance of modeling method effects: resolving the (uni)dimensionality of the loneliness questionnaire

This study sought to resolve the dimensionality of the Loneliness Questionnaire (LQ; Asher, Hymel, & Renshaw, 1984) by applying recommended confirmatory factor analytic procedures that control for method effects (Brown, 2003). This study was needed given that inconsistent findings have been reported recently regarding the structure of this instrument (Bagner, Storch, & Roberti, 2004) and all models to date have not accounted for method effects due to the non-reversed-worded and reversed-worded items of this instrument. Using a large sample of youth in Grades 2 through 12 (N = 11,725), we compared the previously reported 1- and 2-factor models with a newly posited 1-factor model that incorporated correlated error terms to account for method effects. We found that the 1-factor model that included correlated error terms fit the data best, and that this factor structure evidenced measurement invariance across boys and girls in childhood, but not in adolescence. The meaning of the LQ indicators was also consistent for boys across development, but evidenced differences for girls in childhood versus adolescence. More generally, it was demonstrated that modeling method effects is vital to accurately understanding the dimensionality of loneliness when reversed-worded and non-reversed-worded items are used as indicators. The measurement and clinical implications of these findings are discussed.