Sex differences in sum scores may be hard to interpret: the importance of measurement invariance

Psychometric properties of sum scores and factor scores differ even when their correlation is 0.98: A response to Widaman and Revelle

Commentary in Widaman and Revelle (2022) argued that sum scoring is justified as long as unidimensionality holds because sum score reliability is defined. My response begins with a review of the literature supporting the perspective we adopted in the original article. I then conduct simulation studies to assess the psychometric properties of sum scores created using Widaman and Revelle's justification relative to scores created by the weighted factor score approach in the original article. In my simulations, I generate data where sum and factor scores are correlated at 0.96 or 0.98 because high factor-sum score correlations are often used to support the contention that sum and factor scores have interchangeable psychometric properties. I explore (a) correlations between estimated scores and true scores, (b) classification accuracy of sum and factor scores, and (c) reliability of sum and factor scores. Results show that factor scores have (a) higher correlations with true scores (Δ = 0.02-0.04), (b) higher sensitivity (Δ = 4-8 percentage points), and (c) higher reliability (Δ = 0.04-0.07). Factor score performance metrics also have less sampling variability in most conditions. Psychometric properties of sum scores-even when highly correlated with factor scores-remain less desirable than those of factor scores. Additional considerations like models with multiple factors and measurement invariance are also discussed. Essentially, even if accepting Widaman and Revelle's justification for sum scoring, it is uncertain whether researchers generally would want to sum score after fitting a factor analysis unless sum and factor scores correlate at (and not merely close to) 1.00.

Measurement Invariance of the Satisfaction with Life Scale by Sexual Orientation

The Satisfaction with Life Scale (SWLS) has shown strong evidence of reliability, validity, and gender invariance, and there is some evidence of invariance across age, culture, and educational levels. So far, invariance across sexual orientation has not been studied, despite the number of works that relate well-being to sexual orientation. The SWLS should be invariant across sexual orientation to be able to compare group means. This study aimed to explore the invariance of the SWLS across sexual orientation. A non-probabilistic sample with 553 Spanish adults (208 males, 345 females; 212 heterosexuals, 182 gays, and 138 bisexuals among other sexual orientations) participated in a survey. We tested a one-factor model using confirmatory factor analysis. We tested the configural, metric, and scalar invariance of the factorial structure of the SWLS across sexual orientation with heterosexual, lesbian/gay, and bisexual groups. According to our results, the Spanish version of the SWLS shows scalar invariance across sexual orientations, allowing a valid comparison between sexual minority and heterosexual people. Moreover, in our sample, lesbian/gay and bisexual participants obtained lower scores in life satisfaction than heterosexual participants. Bisexual people obtained the lowest score in well-being compared with the other groups. Implications related to the importance of checking instrument invariance before comparing mean differences between groups are discussed.

Dynamic fit index cutoffs for one-factor models

Assessing whether a multiple-item scale can be represented with a one-factor model is a frequent interest in behavioral research. Often, this is done in a factor analysis framework with approximate fit indices like RMSEA, CFI, or SRMR. These fit indices are continuous measures, so values indicating acceptable fit are up to interpretation. Cutoffs suggested by Hu and Bentler (1999) are a common guideline used in empirical research. However, these cutoffs were derived with intent to detect omitted cross-loadings or omitted factor covariances in multifactor models. These types of misspecifications cannot exist in one-factor models, so the appropriateness of using these guidelines in one-factor models is uncertain. This paper uses a simulation study to address whether traditional fit index cutoffs are sensitive to the types of misspecifications common in one-factor models. The results showed that traditional cutoffs have very poor sensitivity to misspecification in one-factor models and that the traditional cutoffs generalize poorly to one-factor contexts. As an alternative, we investigate the accuracy and stability of the recently introduced dynamic fit cutoff approach for creating fit index cutoffs for one-factor models. Simulation results indicated excellent performance of dynamic fit index cutoffs to classify correct or misspecified one-factor models and that dynamic fit index cutoffs are a promising approach for more accurate assessment of model fit in one-factor contexts.

Evaluating the psychometric properties of the e-cigarette barriers to cessation scale

Evidence suggests that electronic cigarette (e-cigarette) users have difficulty quitting e-cigarettes. The Electronic Cigarette Barriers to Cessation Scale (E-BCS) was developed to assess barriers to cessation among e-cigarette users. Yet, no work has evaluated the psychometric properties of the E-BCS. The current study utilized data from two independent samples of adult e-cigarette users to evaluate the psychometric properties and structure of the E-BCS. In Study 1, competing models of the E-BCS were evaluated, and initial construct validity and reliability were explored using data from 599 adult e-cigarettes users (52.3% female, M_age = 35.16 years, SD = 10.30). Results supported a unidimensional structure for the E-BCS comprised of 18 items. In Study 2, the unidimensional structure of the 18-item E-BCS and construct validity were evaluated among a sample of 516 current e-cigarette using adults (52.3% female; M_age = 34.74, SD = 11.37). Results confirmed the unidimensional structure, measurement invariance across sex, and construct validity across related e-cigarette behavior constructs and psychopathology for the E-BCS. These findings suggest that the E-BCS is a potentially useful clinical instrument for measuring barriers that may interfere with e-cigarette cessation.

Measurement in Intensive Longitudinal Data

Technological advances have increased the prevalence of intensive longitudinal data as well as statistical techniques appropriate for these data, such as dynamic structural equation modeling (DSEM). Intensive longitudinal designs often investigate constructs related to affect or mood and do so with multiple item scales. However, applications of intensive longitudinal methods often rely on simple sums or averages of the administered items rather than considering a proper measurement model. This paper demonstrates how to incorporate measurement models into DSEM to (1) provide more rigorous measurement of constructs used in intensive longitudinal studies and (2) assess whether scales are invariant across time and across people, which is not possible when item responses are summed or averaged. We provide an example from an ecological momentary assessment study on self-regulation in adults with binge eating disorder and walkthrough how to fit the model in Mplus and how to interpret the results.

Factor Structure and Measurement Invariance of Youth Psychopathic Traits Inventory-Child Version in Chinese Children

The primary goal of the present study was to examine the latent factor structure and measurement invariance (MI) of the Youth Psychopathic Traits Inventory-Child Version (YPI-CV) in a sample of Chinese children. 299 school children (aged 9-12, 47.3% female) completed the Chinese version of the YPI-CV, and their parents completed a different measure of psychopathic traits, as well as ones for other measures: the Child Problematic Traits Inventory (CPTI), the Strength and Difficulty Questionnaire (SDQ), and the Social Competence - Parent Version (SCPV). Results showed that a bifactor model at item level fit the data best and was invariant across gender. Specifically, the general psychopathy factor influenced the 18 items strongly, suggesting that the YPI-CV is unidimensional rather than multidimensional. Overall, findings suggest that the bifactor structure of the YPI-CV should be used when examining relationships with outcome variables in Chinese children, with a focus on the total score of the YPI-CV, while factor scores should explain with caution.

Differential Item Functioning in Brief Instruments of Disordered Eating

Abstract. Most instruments for the assessment of disordered eating were developed and validated in young female samples. However, they are often used in heterogeneous general population samples. Therefore, brief instruments of disordered eating should assess the severity of disordered eating equally well between individuals with different gender, age, body mass index (BMI), and socioeconomic status (SES). Differential item functioning (DIF) of two brief instruments of disordered eating (SCOFF, Eating Attitudes Test [EAT-8]) was modeled in a representative sample of the German population ( N = 2,527) using a multigroup item response theory (IRT) and a multiple-indicator multiple-cause (MIMIC) structural equation model (SEM) approach. No DIF by age was found in both questionnaires. Three items of the EAT-8 showed DIF across gender, indicating that females are more likely to agree than males, given the same severity of disordered eating. One item of the EAT-8 revealed slight DIF by BMI. DIF with respect to the SCOFF seemed to be negligible. Both questionnaires are equally fair across people with different age and SES. The DIF by gender that we found with respect to the EAT-8 as screening instrument may be also reflected in the use of different cutoff values for men and women. In general, both brief instruments assessing disordered eating revealed their strengths and limitations concerning test fairness for different groups.

The Body Shape Questionnaire Is Not Invariant Across Sex: Evidence From Portuguese-Speaking University Students

The Body Shape Questionnaire (BSQ) is a widely used measure of body shape concerns that was originally designed for use with women but has more recently been used with boys and men. The latter use may be problematic, given that no previous study has demonstrated sex invariance for BSQ scores. To determine the extent to which BSQ scores are sex invariant, we asked Portuguese-speaking women ( n = 1,613) and men ( n = 871) to complete the full BSQ (34 items). Confirmatory factor analysis indicated that a hypothesized 32-item model of BSQ scores and shorter versions had acceptable fit indices in women and men, separately. However, multigroup confirmatory factor analysis showed that these BSQ model scores had configural but not metric, scalar, or strict sex invariance. Differential item analysis indicated significant item-functioning differences on 19 of the 32 retained BSQ items. Thus, BSQ scores are not sex invariant, making problematic the results of previous studies that have compared latent BSQ scores across sex.

Measurement Invariance Between Genders on Two Measures of Borderline Personality Disorder

Gender bias in the diagnosis and assessment of borderline personality disorder (BPD) has been the subject of much controversy in the psychological research literature. Evidence regarding differential prevalence rates and diagnostic rates in clinical settings between genders has been mixed, and measurement bias is one potential explanation for these mixed findings. A total of 340 participants were recruited from a Midwestern university and four outpatient community mental health clinics. They were administered two structured clinical interviews for BPD that were then evaluated for measurement invariance. Both the SCID-II BPD scale and the PDI-IV BPD scale were found to be measurement invariant between genders. No significant differences were found on the SCID-II BPD scale in diagnosis or overall number of criteria endorsed. However, in the rates of endorsement of individual criteria, differences were found in the unstable relationships item.

Sum Scores in Twin Growth Curve Models: Practicality Versus Bias

To study behavioral or psychiatric phenotypes, multiple indices of the behavior or disorder are often collected that are thought to best reflect the phenotype. Combining these items into a single score (e.g. a sum score) is a simple and practical approach for modeling such data, but this simplicity can come at a cost in longitudinal studies, where the relevance of individual items often changes as a function of age. Such changes violate the assumptions of longitudinal measurement invariance (MI), and this violation has the potential to obfuscate the interpretation of the results of latent growth models fit to sum scores. The objectives of this study are (1) to investigate the extent to which violations of longitudinal MI lead to bias in parameter estimates of the average growth curve trajectory, and (2) whether absence of MI affects estimates of the heritability of these growth curve parameters. To this end, we analytically derive the bias in the estimated means and variances of the latent growth factors fit to sum scores when the assumption of longitudinal MI is violated. This bias is further quantified via Monte Carlo simulation, and is illustrated in an empirical analysis of aggression in children aged 3-12 years. These analyses show that measurement non-invariance across age can indeed bias growth curve mean and variance estimates, and our quantification of this bias permits researchers to weigh the costs of using a simple sum score in longitudinal studies. Simulation results indicate that the genetic variance decomposition of growth factors is, however, not biased due to measurement non-invariance across age, provided the phenotype is measurement invariant across birth-order and zygosity in twins.

Substance Use Disorders, Violence, Mental Health, and HIV: Differentiating a Syndemic Factor by Gender and Sexuality

This paper measures syndemic substance use disorder, violence, and mental health and compares the syndemic among HIV-infected heterosexual men, heterosexual women, and men who have sex with men (MSM). Data were from a sample of high needs substance-using, HIV-infected people in South Florida between 2010 and 2012 (n = 481). We used confirmatory factor analysis to measure a syndemic latent variable and applied measurement invariance models to identify group differences in the data structure of syndemic co-morbidities among heterosexual men, heterosexual women, and MSM. We found that variables used to measure the syndemic fit each sub-group, supporting that substance use disorder, violence, and mental health coincide in HIV-infected individuals. Heterosexual men and MSM demonstrated similar syndemic latent variable factor loadings, but significantly different item intercepts, indicating that heterosexual men had larger mean values on substance use disorder, anxiety, and depression than MSM. Heterosexual men and heterosexual women demonstrated significantly different syndemic variable factor loadings, indicating that anxiety and depression contribute more (and substance use contributes less) to the syndemic in heterosexual men compared to heterosexual women. MSM and heterosexual women demonstrated similar syndemic latent variable factor loadings and intercepts, but had significantly different factor residual variances indicating more variance in violent victimization and depression for MSM and more variance in stress for heterosexual women than what is captured by the observed syndemic indicators. Furthermore, heterosexual women had a larger syndemic factor mean than MSM, indicating that the syndemic burden is greater among heterosexual women than MSM. Our findings support that measurement invariance can elucidate differences in the syndemic to tailor interventions to sub-group needs.

Chinese version of the Schizotypal Personality Questionnaire: Factor structure replication and invariance across sex

INTRODUCTION

The Schizotypal Personality Questionnaire (SPQ) is a self-report measure assessing symptoms of schizotypy. The SPQ has been used in both normative and clinical samples and has much theoretical and empirical support. A three-factor structure of the SPQ, derived on the basis of work in schizophrenia, consisting of Cognitive-Perceptual, Interpersonal, and Disorganized factors, has been well replicated. The present study aimed to (i) validate this three-factor structure in the Chinese version of the SPQ in a sample of individuals of Chinese ethnicity, and (ii) test for invariance across sex.

METHODS

A total of 209 (99 males) undergraduate university students (Mage  = 19.5, SD = 1.6) were administered the SPQ.

RESULTS

Confirmatory factor analysis indicated a better fit between the data and the three-factor model compared with a one-factor model. Multigroup confirmatory factor analysis also found strong measurement invariance across sex.

DISCUSSION

The current results add to a growing body of literature evidencing cross-cultural validity of the SPQ and its invariance across sex. Research and clinical implications of the current results are discussed.

Measuring University Students’ Approaches to Learning Statistics

The aim of the current study was to provide evidence that an abbreviated version of the Approaches and Study Skills Inventory for Students (ASSIST) was invariant across different languages and educational contexts in measuring university students’ learning approaches to statistics. Data were collected on samples of university students attending undergraduate introductory statistics courses in five countries (Argentina, Italy, Australia, Turkey, and Vietnam). Using factor analysis, we confirmed the three-factor (Deep, Surface, and Strategic approach) model holds across the five samples, and we provided evidence of configural and measurement invariance. The current version of the ASSIST for statistics learners is a suitable scale for researchers and teachers working in the field of statistics education and represents promising tool for multinational studies on approaches to learning statistics.

Childhood ODD and ADHD Behavior: The Effect of Classroom Sharing, Gender, Teacher Gender and Their Interactions

One criterion for a diagnostic and statistical manual of mental disorders (DSM-IV) diagnosis of attention deficit hyperactivity disorder (ADHD) and oppositional defiant disorder (ODD) is that symptoms are present in at least two settings, and often teacher ratings are taken into account. The short Conners' Teacher Rating Scales-Revised (CTRS-R) is a widely used standardized instrument measuring ODD and ADHD behavior in a school setting. In the current study CTRS-R data were available for 7, 9 and 12-year-old twins from the Netherlands Twin Register. Measurement invariance (MI) across student gender and teacher gender was established for three of the four scales (Oppositional Behavior, Hyperactivity and ADHD Index) of the CTRS-R. The fourth scale (ATT) showed an unacceptable model fit even without constraints on the data and revision of this scale is recommended. Gene-environment (GxE) interaction models revealed that heritability was larger for children sharing a classroom. There were some gender differences in the heritability of ODD and ADHD behavior and there was a moderating effect of teacher's gender at some of the ages. Taken together, this indicates that there was evidence for GxE interaction for classroom sharing, gender of the student and gender of the teacher.

Italian version of the Dutch Eating Behavior Questionnaire. Psychometric proprieties and measurement invariance across sex, BMI-status and age

The purpose of this study was to examine the basic psychometric proprieties of the Dutch Eating Behavior Questionnaire (DEBQ) and its measurement invariance across sex, BMI-status (normal weight/overweight), and age in a community sample of 990 Italian adults. The analysis of the dimensionality of the DEBQ using exploratory factor analysis revealed the existence of three major factors - emotional, restrained and external eating. Single and multi-group confirmatory factor analyses replicated the three-factor structure, and this dimensional structure proved to be invariant across sex, BMI-status, and age. Findings upheld the criterion-related validity (e.g., via its associations with Eating Attitudes Test-26). The DEBQ's subscales displayed high internal consistency and test-retest reliability over a 4-week period. Statistically significant differences were found when sex, BMI and age groups are compared in the latent means of emotional, external and restrained eating and they are discussed with reference to theory, past and recent empirical findings. Overall, results support the measurement invariance of the DEBQ and suggest that the Italian version is a psychometrically reliable, valid and useful measurement instrument for assessing adult eating behaviors.

Analysis of the factor structure of the Sociocultural Attitudes Towards Appearance Questionnaire (SATAQ-3) in Spanish secondary-school students through exploratory structural equation modeling

The aims of the present study were: (1) to assess the factor structure of the SATAQ-3 in Spanish secondary-school students by means of exploratory factor analysis (EFA), confirmatory factor analysis (CFA) and exploratory structural equation modeling (ESEM) models; and (2) to study its invariance by sex and school grade. ESEM is a technique that has been proposed for the analysis of internal structure that overcomes some of the limitations of EFA and CFA. Participants were 1559 boys and girls in grades seventh to tenth. The results support the four-factor solution of the original version, and reveal that the best fit was obtained with ESEM, excluding Item 20 and with correlated uniqueness between reverse-keyed items. Our version shows invariance by sex and grade. The differences between scores of different groups are in the expected direction, and support the validity of the questionnaire. We recommend a version excluding Item 20 and without reverse-keyed items.

Stable genetic effects on symptoms of alcohol abuse and dependence from adolescence into early adulthood

Relatively little is known about how genetic influences on alcohol abuse and dependence (AAD) change with age. We examined the change in influence of genetic and environmental factors which explain symptoms of AAD from adolescence into early adulthood. Symptoms of AAD were assessed using the four AAD screening questions of the CAGE inventory. Data were obtained up to six times by self-report questionnaires for 8,398 twins from the Netherlands Twin Register aged between 15 and 32 years. Longitudinal genetic simplex modeling was performed with Mx. Results showed that shared environmental influences were present for age 15-17 (57%) and age 18-20 (18%). Unique environmental influences gained importance over time, contributing 15% of the variance at age 15-17 and 48% at age 30-32. At younger ages, unique environmental influences were largely age-specific, while at later ages, age-specific influences became less important. Genetic influences on AAD symptoms over age could be accounted for by one factor, with the relative influence of this factor differing across ages. Genetic influences increased from 28% at age 15-17 to 58% at age 21-23 and remained high in magnitude thereafter. These results are in line with a developmentally stable hypothesis that predicts that a single set of genetic risk factors acts on symptoms of AAD from adolescence into young adulthood.

Measurement invariance of the kidney disease and quality of life instrument (KDQOL-SF) across veterans and non-veterans

BACKGROUND

Studies have demonstrated that perceived health-related quality of life (HRQOL) of patients receiving hemodialysis is significantly impaired. Since HRQOL outcome data are often used to compare groups to determine health care effectiveness it is imperative that measures of HRQOL are valid. However, valid HRQOL comparisons between groups can only be made if instrument invariance is demonstrated. The Kidney Disease Quality of Life-Short Form (KDQOL-SF) is a widely used HRQOL measure for patients with chronic kidney disease (CKD) however, it has not been validated in the Veteran population. Therefore, the purpose of this study was to examine the measurement invariance of the KDQOL-SF across Veterans and non-Veterans with CKD.

METHODS

Data for this study were from two large prospective observational studies of patients receiving hemodialysis: 1) Veteran End-Stage Renal Disease Study (VETERAN) (N = 314) and 2) Dialysis Outcomes and Practice Patterns Study (DOPPS) (N = 3,300). Health-related quality of life was measured with the KDQOL-SF, which consists of the SF-36 and the Kidney Disease Component Summary (KDCS). Single-group confirmatory factor analysis was used to evaluate the goodness-of-fit of the hypothesized measurement model for responses to the subscales of the KDCS and SF-36 instruments when analyzed together; and given acceptable goodness-of-fit in each group, multigroup CFA was used to compare the structure of this factor model in the two samples. Pattern of factor loadings (configural invariance), the magnitude of factor loadings (metric invariance), and the magnitude of item intercepts (scalar invariance) were assessed as well as the degree to which factors have the same variances, covariances, and means across groups (structural invariance).

RESULTS

CFA demonstrated that the hypothesized two-factor model (KDCS and SF-36) fit the data of both the Veteran and DOPPS samples well, supporting configural invariance. Multigroup CFA results concerning metric and scalar invariance suggested partial strict invariance for the SF-36, but only weak invariance for the KDCS. Structural invariance was not supported.

CONCLUSIONS

Results suggest that Veterans may interpret the KDQOL-SF differently than non-Veterans. Further evaluation of measurement invariance of the KDQOL-SF between Veterans and non-Veterans is needed using large, randomly selected samples before comparisons between these two groups using the KDQOL-SF can be done reliably.