Recovering Predictor-Criterion Relations Using Covariate-Informed Factor Score Estimates

Early childhood measurement invariance of the Strengths and Difficulties Questionnaire across age, race, sex, and socioeconomic status

Research suggests there are differences in children's internalizing and externalizing symptoms as a function of age, race, sex, and socioeconomic status (SES). Males, Black children, and children experiencing lower SES have been rated as having more externalizing problems. Female and older children have been rated as having higher internalizing symptoms. The validity of these findings rests on the assumption that the measures mean the same thing across groups and developmental time (i.e., measurement invariance [MI]). Without assuring MI, results may represent differences in measurement and not true differences in the underlying construct. The Strengths and Difficulties Questionnaire (SDQ) is a widely used tool to measure internalizing and externalizing symptoms. Papers have evaluated MI of the SDQ in school-aged children. However, to our knowledge, no studies of young children have examined MI across Black and White families from diverse SES backgrounds. Data from the Family Life Project were used to evaluate MI of the SDQ across child age (35 to 90 months), race, sex, and SES. Using moderated nonlinear factor analysis (MNLFA), multiple SDQ items demonstrated measurement noninvariance as a function of child demographic variables. Results suggest that it is important to test and adjust for noninvariance with the SDQ when applied to early childhood populations comprising Black and White children from diverse SES backgrounds. An MNLFA approach improves our ability to validly measure and compare symptoms of psychopathology in diverse early childhood populations. This could have implications for our understanding of rates of mental health challenges and treatment in early childhood. (PsycInfo Database Record (c) 2025 APA, all rights reserved).

Practical Implications of Sum Scores Being Psychometrics' Greatest Accomplishment

This paper reflects on some practical implications of the excellent treatment of sum scoring and classical test theory (CTT) by Sijtsma et al. (Psychometrika 89(1):84-117, 2024). I have no major disagreements about the content they present and found it to be an informative clarification of the properties and possible extensions of CTT. In this paper, I focus on whether sum scores-despite their mathematical justification-are positioned to improve psychometric practice in empirical studies in psychology, education, and adjacent areas. First, I summarize recent reviews of psychometric practice in empirical studies, subsequent calls for greater psychometric transparency and validity, and how sum scores may or may not be positioned to adhere to such calls. Second, I consider limitations of sum scores for prediction, especially in the presence of common features like ordinal or Likert response scales, multidimensional constructs, and moderated or heterogeneous associations. Third, I review previous research outlining potential limitations of using sum scores as outcomes in subsequent analyses where rank ordering is not always sufficient to successfully characterize group differences or change over time. Fourth, I cover potential challenges for providing validity evidence for whether sum scores represent a single construct, particularly if one wishes to maintain minimal CTT assumptions. I conclude with thoughts about whether sum scores-even if mathematically justified-are positioned to improve psychometric practice in empirical studies.

Systematic Integration of Multi-Informant Externalizing Ratings in Clinical Settings

Best practice clinical assessment of externalizing problems often necessitates collection of information from parents, youth themselves, and teachers. The present study tested the predictive validity of a psychometrically-driven scoring procedure to integrate multi-informant, dimensional ratings of externalizing problems. Participants were 2264 clinic-referred youth ages 6-18. Parents, teachers, and youth completed questionnaire ratings of externalizing problems (hyperactivity-inattention, conduct problems, and oppositionality-defiance) prior to an initial clinical appointment. The predictive validity of simple (highest informant rating; and all informant ratings separately) and more complex (latent S-1 bifactor model with specific informant factors; and moderated nonlinear factor analysis accounting for child age and sex) methods of informant integration was tested in predicting impairment, comorbidity, and number of clinical encounters. A simple model, in which all informant ratings were included, showed the best predictive validity across outcomes, performing as well or better than the use of the highest informant ratings or more complex latent variable models. The addition of child age and sex as moderators in the factor model did not improve predictive validity. Each informant (parent, teacher, and youth) contributes important information to the prediction of clinically-relevant outcomes. There is insufficient evidence at present to suggest that complex latent variable models should be favored over simpler models that preserve each informant's ratings.

Unreliable Continuous Treatment Indicators in Propensity Score Analysis

Propensity score analyses (PSA) of continuous treatments often operationalize the treatment as a multi-indicator composite, and its composite reliability is unreported. Latent variables or factor scores accounting for this unreliability are seldom used as alternatives to composites. This study examines the effects of the unreliability of indicators of a latent treatment in PSA using the generalized propensity score (GPS). A Monte Carlo simulation study was conducted varying composite reliability, continuous treatment representation, variability of factor loadings, sample size, and number of treatment indicators to assess whether Average Treatment Effect (ATE) estimates differed in their relative bias, Root Mean Squared Error, and coverage rates. Results indicate that low composite reliability leads to underestimation of the ATE of latent continuous treatments, while the number of treatment indicators and variability of factor loadings show little effect on ATE estimates, after controlling for overall composite reliability. The results also show that, in correctly specified GPS models, the effects of low composite reliability can be somewhat ameliorated by using factor scores that were estimated including covariates. An illustrative example is provided using survey data to estimate the effect of teacher adoption of a workbook related to a virtual learning environment in the classroom.

Psychometric properties of sum scores and factor scores differ even when their correlation is 0.98: A response to Widaman and Revelle

Commentary in Widaman and Revelle (2022) argued that sum scoring is justified as long as unidimensionality holds because sum score reliability is defined. My response begins with a review of the literature supporting the perspective we adopted in the original article. I then conduct simulation studies to assess the psychometric properties of sum scores created using Widaman and Revelle's justification relative to scores created by the weighted factor score approach in the original article. In my simulations, I generate data where sum and factor scores are correlated at 0.96 or 0.98 because high factor-sum score correlations are often used to support the contention that sum and factor scores have interchangeable psychometric properties. I explore (a) correlations between estimated scores and true scores, (b) classification accuracy of sum and factor scores, and (c) reliability of sum and factor scores. Results show that factor scores have (a) higher correlations with true scores (Δ = 0.02-0.04), (b) higher sensitivity (Δ = 4-8 percentage points), and (c) higher reliability (Δ = 0.04-0.07). Factor score performance metrics also have less sampling variability in most conditions. Psychometric properties of sum scores-even when highly correlated with factor scores-remain less desirable than those of factor scores. Additional considerations like models with multiple factors and measurement invariance are also discussed. Essentially, even if accepting Widaman and Revelle's justification for sum scoring, it is uncertain whether researchers generally would want to sum score after fitting a factor analysis unless sum and factor scores correlate at (and not merely close to) 1.00.

Informing Harmonization Decisions in Integrative Data Analysis: Exploring the Measurement Multiverse

Combining datasets in an integrative data analysis (IDA) requires researchers to make a number of decisions about how best to harmonize item responses across datasets. This entails two sets of steps: logical harmonization, which involves combining items which appear similar across datasets, and analytic harmonization, which involves using psychometric models to find and account for cross-study differences in measurement. Embedded in logical and analytic harmonization are many decisions, from deciding whether items can be combined prima facie to how best to find covariate effects on specific items. Researchers may not have specific hypotheses about these decisions, and each individual choice may seem arbitrary, but the cumulative effects of these decisions are unknown. In the current study, we conducted an IDA of the relationship between alcohol use and delinquency using three datasets (total N = 2245). For analytic harmonization, we used moderated nonlinear factor analysis (MNLFA) to generate factor scores for delinquency. We conducted both logical and analytic harmonization 72 times, each time making a different set of decisions. We assessed the cumulative influence of these decisions on MNLFA parameter estimates, factor scores, and estimates of the relationship between delinquency and alcohol use. There were differences across paths in MNLFA parameter estimates, but fewer differences in estimates of factor scores and regression parameters linking delinquency to alcohol use. These results suggest that factor scores may be relatively robust to subtly different decisions in data harmonization, and measurement model parameters are less so.

Enhancing measurement validity in diverse populations: Modern approaches to evaluating differential item functioning

When developing and evaluating psychometric measures, a key concern is to ensure that they accurately capture individual differences on the intended construct across the entire population of interest. Inaccurate assessments of individual differences can occur when responses to some items reflect not only the intended construct but also construct-irrelevant characteristics, like a person's race or sex. Unaccounted for, this item bias can lead to apparent differences on the scores that do not reflect true differences, invalidating comparisons between people with different backgrounds. Accordingly, empirically identifying which items manifest bias through the evaluation of differential item functioning (DIF) has been a longstanding focus of much psychometric research. The majority of this work has focused on evaluating DIF across two (or a few) groups. Modern conceptualizations of identity, however, emphasize its multi-determined and intersectional nature, with some aspects better represented as dimensional than categorical. Fortunately, many model-based approaches to modelling DIF now exist that allow for simultaneous evaluation of multiple background variables, including both continuous and categorical variables, and potential interactions among background variables. This paper provides a comparative, integrative review of these new approaches to modelling DIF and clarifies both the opportunities and challenges associated with their application in psychometric research.

Can severity of substance use be measured across drug classes? Estimating differential item functioning by drug class in two general measures of substance use severity

BACKGROUND

Substance use severity is frequently measured using generic (i.e., non-drug specific) items. Yet, the measurement properties of these items must be evaluated for measurement invariance across inidividuals who use differing substances to ensure total scores can be compared across groups.

METHOD

This study used data from two independent samples (n1 = 474; n2 = 5183) and two measures of general substance use severity with generic items, the Patient Reported Outcomes Measurement Information System (PROMIS) Severity of Substance Use and DAST-10, to examine for differential item functioning (DIF) across substances (i.e., sedatives, opioids, amphetamines, cocaine, and cannabis). We utilized moderated nonlinear factor analysis to estimate DIF. Finally, we compared factor scores across estimation methods with and without accounting for DIF to examine the impact of DIF.

RESULTS

A minority of items showed statistically significant DIF in each scale (Items with DIF: PROMIS Sample 1: 5/37; PROMIS Sample 2: 7/20; DAST-10 Sample 2: 3/10). Factor scores across scoring methods showed extremely high correlations (0.994 - 0.999), estimates of mean differences across substance groups did not vary considerably across scoring methods, but measurement differences were correlated with factor scores.

DISCUSSION

These findings suggest that these two measures of substance use severity can be used across individuals using different substances. Factor scores appear similar across scoring methods and mean differences do not appear to be substantially biases. Measures with generic items may offer a parsimonious alternative to measures with drug specific items but more research is needed to evaluate the robustness of these findings.

Measurement invariance of maternal depressive symptoms across the first 2 years since birth and across racial group, education, income, primiparity, and age

Up to 19% of postpartum mothers experience depressive symptoms, which are associated with infant development. Thus, research examining postpartum depression has implications for mothers' and infants' well-being. However, this research relies on the often-untested assumption of measurement invariance-that measures capture the same construct across time and sociodemographic characteristics. In the absence of invariance, measurement bias may confound differences across time and group, contributing to invalid inferences. In a sociodemographically diverse (40.7% African American, 58.9% White; 67.9% below two times the federal poverty line; 19.4% with less than high school education), rural, longitudinal sample (N = 1,275) of mothers, we used moderated nonlinear factor analysis (MNLFA) to examine measurement invariance of the Brief Symptom Inventory-18 (BSI-18) Depressive Symptoms subscale across time since birth, racial group, education, income, primiparity, and maternal age at childbirth. We identified evidence of differential item functioning (DIF; i.e., measurement noninvariance) as a function of racial group and education. Subsequent analyses indicated, however, that the DIF-induced bias had minimal impacts on substantive comparisons examining change over time since birth and group differences. Thus, the presence of measurement noninvariance does not appear to bias substantive comparisons using the BSI-18 Depressive Symptoms subscale across the first 2 years since birth in a sample comprising primarily African American and White mothers living in predominately rural, low-income communities. This study demonstrates the importance of assessing measurement invariance and highlights MNLFA for evaluating the impact of noninvariance as a preliminary step that increases confidence in the validity of substantive inferences. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Leveraging item-level accuracy and reaction time to address ceiling effects in the measurement of inhibitory control in preschool-aged children

Preschool-aged children's performance on inhibitory control tasks is typically represented by the overall accuracy of their item responses (e.g., mean proportion correct). However, in settings where children vary widely in age or ability level, inhibitory control tasks are susceptible to ceiling effects, which undermine measurement precision. We have previously demonstrated a general approach for scoring inhibitory control tasks that combines item-level accuracy and reaction-time information to minimize ceiling effects. Here, we extend that approach by incorporating additional item-level reaction time data from an adjunct (simple reaction time) task. We contrast three approaches for scoring inhibitory control tasks, two of which rely exclusively on item accuracy information and a third which also considers item reaction time information. We demonstrate the impacts of these different approaches to scoring with two inhibitory control tasks that were included in a recent evaluation of the Red Light, Purple Light intervention in preprimary classrooms in Nairobi County, Kenya. We limited our study to children who met inclusion criteria at pre-test (N = 418; 51% male; mean age = 4.8 years) or post-test (N = 386; 51% male; mean age = 4.8 years). Children's performance on individual inhibitory control tasks was strongly correlated regardless of the scoring approach (rs = 0.73-0.97 across two tasks). However, the combined accuracy and reaction time scores eliminated ceiling effects that were common when only accuracy information was used. The combined accuracy and reaction time models also distinguished item-level RT into inhibitory control and processing speed components, which are distinct constructs. Results are discussed with respect to the challenges and nuances of the estimation and interpretation of inhibitory control task scores with children of varied ages and ability levels.

Utilizing Moderated Non-linear Factor Analysis Models for Integrative Data Analysis: A Tutorial

Integrative data analysis (IDA) is an analytic tool that allows researchers to combine raw data across multiple, independent studies, providing improved measurement of latent constructs as compared to single study analysis or meta-analyses. This is often achieved through implementation of moderated nonlinear factor analysis (MNLFA), an advanced modeling approach that allows for covariate moderation of item and factor parameters. The current paper provides an overview of this modeling technique, highlighting distinct advantages most apt for IDA. We further illustrate the complex modeling building process involved in MNLFA by providing a tutorial using empirical data from five separate prevention trials. The code and data used for analyses are also provided.

Causally Interpretable Meta-analysis: Application in Adolescent HIV Prevention

Endowing meta-analytic results with a causal interpretation is challenging when there are differences in the distribution of effect modifiers among the populations underlying the included trials and the target population where the results of the meta-analysis will be applied. Recent work on transportability methods has described identifiability conditions under which the collection of randomized trials in a meta-analysis can be used to draw causal inferences about the target population. When the conditions hold, the methods enable estimation of causal quantities such as the average treatment effect and conditional average treatment effect in target populations that differ from the populations underlying the trial samples. The methods also facilitate comparison of treatments not directly compared in a head-to-head trial and assessment of comparative effectiveness within subgroups of the target population. We briefly describe these methods and present a worked example using individual participant data from three HIV prevention trials among adolescents in mental health care. We describe practical challenges in defining the target population, obtaining individual participant data from included trials and a sample of the target population, and addressing systematic missing data across datasets. When fully realized, methods for causally interpretable meta-analysis can provide decision-makers valid estimates of how treatments will work in target populations of substantive interest as well as in subgroups of these populations.

Integrative data analysis of self-efficacy in 4 clinical trials for alcohol use disorder

BACKGROUND

Self-efficacy has been proposed as a key predictor of alcohol treatment outcomes and a potential mechanism of success in achieving abstinence or drinking reductions following alcohol treatment. Integrative data analysis, where data from multiple studies are combined for analyses, can be used to synthesize analyses across multiple alcohol treatment trials by creating a commensurate measure and controlling for differential item functioning (DIF) to determine whether alcohol treatments improve self-efficacy.

METHOD

The current study used moderated nonlinear factor analysis (MNLFA) to examine the effect of treatment on self-efficacy across four different treatment studies (N = 3720; 72.5% male, 68.4% non-Hispanic white). Self-efficacy was measured using the Alcohol Abstinence Self-Efficacy Scale (AASE) in the COMBINE Study (n = 1383) and Project MATCH (n = 1726), and the Drug Taking Confidence Questionnaire (DTCQ) in two studies of Telephone Continuing Care (TEL Study 1: n = 303; TEL Study 2: n = 212). DIF was examined across time, study, treatment condition, marital status, age, and sex.

RESULTS

We identified 12 items from the AASE and DTCQ to create a commensurate measure of self-efficacy using MNLFA. All active treatments, including cognitive-behavioral treatment, a combined behavioral intervention, medication management, motivation enhancement treatment, telephone continuing care, twelve-step facilitation, and relapse prevention, were associated with significant increases in self-efficacy from baseline to posttreatment that were maintained for up to a year. Importantly, treatment as usual in community settings, which consisted of weekly group therapy that included addiction counseling and twelve-step recovery support, was not associated with significant increases in self-efficacy.

CONCLUSIONS

Alcohol self-efficacy increases following treatment and numerous evidence-based treatments are associated with significant increases in self-efficacy, which are maintained over time. Community treatment that focuses solely on addiction counseling and twelve-step support may not promote increases in self-efficacy.

Individual Differences in the Stability and Change of Childhood Depression: A Growth Mixture Model With Structured Residuals

Studies of developmental trajectories of depression are important for understanding depression etiology. Existing studies have been limited by short time frames and no studies have explored a key factor: differential patterns of responding to life events. This article introduces a novel analytic technique, growth mixture modeling with structured residuals, to examine the course of youth depression in a large, prospective cohort (N = 11,641, ages 4-16.5, 96% White). Age-specific critical points were identified at ages 8 and 13 where depression symptoms spiked for a minority of children. Most depression risk was due to dynamic responses to environmental events, drawn not from a small pool of persistently depressed children, but a larger pool of children who varied across higher and lower symptom levels.

Capturing Environmental Dimensions of Adversity and Resources in the Context of Poverty Across Infancy Through Early Adolescence: A Moderated Nonlinear Factor Model

Income, education, and cumulative-risk indices likely obscure meaningful heterogeneity in the mechanisms through which poverty impacts child outcomes. This study draws from contemporary theory to specify multiple dimensions of poverty-related adversity and resources, with the aim of better capturing these nuances. Using data from the Family Life Project (N = 1,292), we leveraged moderated nonlinear factor analysis (Bauer, 2017) to establish group- and longitudinally invariant environmental measures from infancy to early adolescence. Results indicated three latent factors-material deprivation, psychosocial threat, and sociocognitive resources-were distinct from each other and from family income. Each was largely invariant across site, racial group, and development and showed convergent and discriminant relations with age-twelve criterion measures. Implications for ensuring socioculturally valid measurements of poverty are discussed.

Psychometric Models for Scoring Multiple Reporter Assessments: Applications to Integrative Data Analysis in Prevention Science and Beyond

Conducting valid and reliable empirical research in the prevention sciences is an inherently difficult and challenging task. Chief among these is the need to obtain numerical scores of underlying theoretical constructs for use in subsequent analysis. This challenge is further exacerbated by the increasingly common need to consider multiple reporter assessments, particularly when using integrative data analysis to fit models to data that have been pooled across two or more independent samples. The current paper uses both simulated and real data to examine the utility of a recently proposed psychometric model for multiple reporter data called the trifactor model (TFM) in settings that might be commonly found in prevention research. Results suggest that numerical scores obtained using the TFM are superior to more traditional methods, particularly when pooling samples that contribute different reporter perspectives.

Secondary Prevention of Depressive Prodrome in Adolescents: Before and After Attending a Jogging Program on Campus

The adolescent depressive prodrome has been conceptualized as an early integrated sign of depressive symptoms, which may develop to a first episode of depression or return to normal for the adolescents. In this study, depressive prodrome presented the early self-rated depressive symptoms for the sample participants. By referring to the Kutcher Adolescent Depression Scale and the psychometric characteristics of the Adolescent Depression Scale (ADR), we proposed a self-rated questionnaire to assess the severity of the depressive symptoms in adolescents before and after attending the jogging program on a high school campus in Taiwan. With the parental co-signature and self-signed informed consent form, 284 high school students under the average age of 15 years, participated in this study in March 2019. Through the software of IBMSPSS 25, we used a binary logistic model, principal component analysis (PCA), multiple-dimensional analysis, and receiver operating characteristic curve (ROC) to analyze the severity of the depressive prodrome via the threshold severity score (SC) and false positive rate (FPR). Findings revealed that attending the 15-week jogging program (3 times a week, 45 min each) on campus can change the severity status and reduce the prevalence of moderate-severe depressive prodrome by 26%. The two-dimensional approach identified three symptoms, which were the crying spell, loss of pleasure doing daily activities, and feeling the decline in memory. They kept being invariant symptoms during the course of depressive prodrome assessment for sample participants. In this study, the campus jogging program appeared to be able to affect the FPR of the measure of depressive prodrome. Compared with the subthreshold depression, the depressive prodrome emphasized the assessment from the view of the secondary prevention by representing the change from a person's premorbid functioning up until the first onset of depression or returning to normal. However, the subthreshold depression is a form of minor depression according to DSM-5 criteria varying on the number of symptoms and duration required, highly prevalent in the concern of primary care.

Harmonizing altered measures in integrative data analysis: A methods analogue study

In the current study, we used an analogue integrative data analysis (IDA) design to test optimal scoring strategies for harmonizing alcohol- and drug-use consequence measures with varying degrees of alteration across four study conditions. We evaluated performance of mean, confirmatory factor analysis (CFA), and moderated nonlinear factor analysis (MNLFA) scores based on traditional indices of reliability (test-retest, internal, and score recovery or parallel forms) and validity. Participants in the analogue study included 854 college students (46% male; 21% African American, 5% Hispanic/Latino, 56% European American) who completed two versions of the altered measures at two sessions, separated by 2 weeks. As expected, mean, CFA, and MNLFA scores all resulted in scales with lower reliability given increasing scale alteration (with less fidelity to formerly developed scales) and shorter scale length. MNLFA and CFA scores, however, showed greater validity than mean scores, demonstrating stronger relationships with external correlates. Implications for measurement harmonization in the context of IDA are discussed.

Estimation of Response Styles Using the Multidimensional Nominal Response Model: A Tutorial and Comparison With Sum Scores

Recent years have seen a dramatic increase in item response models for measuring response styles on Likert-type items. These model-based approaches stand in contrast to traditional sum-score-based methods where researchers count the number of times that participants selected certain response options. The multidimensional nominal response model (MNRM) offers a flexible model-based approach that may be intuitive to those familiar with sum score approaches. This paper presents a tutorial on the model along with code for estimating it using three different software packages: flexMIRT®, mirt, and Mplus. We focus on specification and interpretation of response functions. In addition, we provide analytical details on how sum score to scale score conversion can be done with the MNRM. In the context of a real data example, three different scoring approaches are then compared. This example illustrates how sum-score-based approaches can sometimes yield scores that are confounded with substantive content. We expect that the current paper will facilitate further investigations as to whether different substantive conclusions are reached under alternative approaches to measuring response styles.

Simplifying the Assessment of Measurement Invariance over Multiple Background Variables: Using Regularized Moderated Nonlinear Factor Analysis to Detect Differential Item Functioning

Determining whether measures are equally valid for all individuals is a core component of psychometric analysis. Traditionally, the evaluation of measurement invariance (MI) involves comparing independent groups defined by a single categorical covariate (e.g., men and women) to determine if there are any items that display differential item functioning (DIF). More recently, Moderated Nonlinear Factor Analysis (MNLFA) has been advanced as an approach for evaluating MI/DIF simultaneously over multiple background variables, categorical and continuous. Unfortunately, conventional procedures for detecting DIF do not scale well to the more complex MNLFA. The current manuscript therefore proposes a regularization approach to MNLFA estimation that penalizes the likelihood for DIF parameters (i.e., rewarding sparse DIF). This procedure avoids the pitfalls of sequential inference tests, is automated for end users, and is shown to perform well in both a small-scale simulation and an empirical validation study.

An External Validity Approach for Assessing Essential Unidimensionality in Correlated-Factor Models

Many psychometric measures yield data that are compatible with (a) an essentially unidimensional factor analysis solution and (b) a correlated-factor solution. Deciding which of these structures is the most appropriate and useful is of considerable importance, and various procedures have been proposed to help in this decision. The only fully developed procedures available to date, however, are internal, and they use only the information contained in the item scores. In contrast, this article proposes an external auxiliary procedure in which primary factor scores and general factor scores are related to relevant external variables. Our proposal consists of two groups of procedures. The procedures in the first group (differential validity procedures) assess the extent to which the primary factor scores relate differentially to the external variables. Procedures in the second group (incremental validity procedures) assess the extent to which the primary factor scores yield predictive validity increments with respect to the single general factor scores. Both groups of procedures are based on a second-order structural model with latent variables from which new methodological results are obtained. The functioning of the proposal is assessed by means of a simulation study, and its usefulness is illustrated with a real-data example in the personality domain.

Approaches for creating comparable measures of alcohol use symptoms: Harmonization with eight studies of criminal justice populations

BACKGROUND

With increasing data archives comprised of studies with similar measurement, optimal methods for data harmonization and measurement scoring are a pressing need. We compare three methods for harmonizing and scoring the AUDIT as administered with minimal variation across 11 samples from eight study sites within the STTR (Seek-Test-Treat-Retain) Research Harmonization Initiative. Descriptive statistics and predictive validity results for cut-scores, sum scores, and Moderated Nonlinear Factor Analysis scores (MNLFA; a psychometric harmonization method) are presented.

METHODS

Across the eight study sites, sample sizes ranged from 50 to 2405 and target populations varied based on sampling frame, location, and inclusion/exclusion criteria. The pooled sample included 4667 participants (82% male, 52% Black, 24% White, 13% Hispanic, and 8% Asian/ Pacific Islander; mean age of 38.9 years). Participants completed the AUDIT at baseline in all studies.

RESULTS

After logical harmonization of items, we scored the AUDIT using three methods: published cut-scores, sum scores, and MNLFA. We found greater variation, fewer floor effects, and the ability to directly address missing data in MNLFA scores as compared to cut-scores and sum scores. MNLFA scores showed stronger associations with binge drinking and clearer study differences than did other scores.

CONCLUSIONS

MNLFA scores are a promising tool for data harmonization and scoring in pooled data analysis. Model complexity with large multi-study applications, however, may require new statistical advances to fully realize the benefits of this approach.