Variability in intensively assessed mood: Systematic sources and factor structure in outpatients with opioid use disorder

In intensive longitudinal studies using ecological momentary assessment, mood is typically assessed by repeatedly obtaining ratings for a large set of adjectives. Summarizing and analyzing these mood data can be problematic because the reliability and factor structure of such measures have rarely been evaluated in this context, which-unlike cross-sectional studies-captures between- and within-person processes. Our study examined how mood ratings (obtained thrice daily for 8 weeks; n = 306, person moments = 39,321) systematically vary and covary in outpatients receiving medication for opioid use disorder (MOUD). We used generalizability theory to quantify several aspects of reliability, and multilevel confirmatory factor analysis (MCFA) to detect factor structures within and across people. Generalizability analyses showed that the largest proportion of systematic variance across mood items was at the person level, followed by the person-by-day interaction and the (comparatively small) person-by-moment interaction for items reflecting low arousal. The best-fitting MCFA model had a three-factor structure both at the between- and within-person levels: positive mood, negative mood, and low-arousal states (with low arousal considered as either a separate factor or a subfactor of negative mood). We conclude that (a) mood varied more between days than between moments and (b) low arousal may be worth scoring and reporting separately from positive and negative mood states, at least in a MOUD population. Our three-factor structure differs from prior analyses of mood; more work is needed to understand the extent to which it generalizes to other populations. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Rasch Modeling and Multilevel Confirmatory Factor Analysis for the Usability of the Impact of Event Scale-Revised (IES-R) during the COVID-19 Pandemic

BACKGROUND

Several instruments are currently used to assess Coronavirus Disease 2019 (COVID-19) -induced psychological distress, including the 22-item Impact of Event Scale-Revised (IES-R). The IES-R is a self-administered scale used to assess post-traumatic stress disorder (PTSD). The current study aimed to examine the construct validity of the IES-R, based on the Rasch model, with COVID-19-related data, as well as to test the multilevel construct validity of the IES-R within and among countries during the pandemic crisis.

METHODS

A multi-country web-based cross-sectional survey was conducted utilizing the 22-item IES-R. A total of 1020 participants enrolled in our survey, of whom 999 were included in the analyses. Data were analyzed using Rasch modeling and multilevel confirmatory factor analysis (MCFA).

RESULTS

The Rasch modeling results of the IES-R demonstrated that the IES-R is a satisfactory instrument with the five-point Likert scale, asserting that its 22 items are significant contributors to assessing PTSD as a unidimensional construct covered by the items of the IES-R. The MCFA confirmed that the 22-item IES-R, with its three factors, including intrusion, avoidance, and hyperarousal, demonstrates adequate construct validity at the within- and among-country levels. However, the results of the Akaike information criterion (AIC) model determined that the 16-item IES-R is better than the 22-item IES-R.

CONCLUSION

The results suggested that the 22-item IES-R is a reliable screening instrument for measuring PTSD related to the COVID-19 pandemic, and can be utilized to provide timely psychological health support, when needed, based on the screening results.

Assessing Construct Validity in Math Achievement: An Application of Multilevel Structural Equation Modeling (MSEM)

The purpose of the present study was to model math achievement at both the person and university levels of the analyses in order to understand the optimal factor structure of math competency. Data involved 2,881 students who took a national mathematics examination as part of their entry at the university public system in Saudi Arabia. Four factors from the National math examination comprised the math achievement measure, namely, numbers and operations, algebra and analysis, geometry and measurement, and, statistics and probabilities. Data were analyzed using the aggregate method and by use of Multilevel Structural Equation Modeling (MSEM). Results indicated that both a unidimensional and a 4-factor correlated model fitted the data equally well using aggregate data, where for reasons of parsimony the unidimensional model was the preferred choice with these data. When modeling data including clustering, results pointed to alternative factor structures at the person and university levels. Thus, a unidimensional model provided the best fit at the University level, whereas a four-factor correlated model was most descriptive for person level data. The optimal simple structure was evaluated using the Ryu and West (2009) methodology for partially saturating the MSEM model and also met criteria for discriminant validation as described in Gorsuch (1983). Furthermore, a university level variable, namely the year of establishment, pointed to the superiority of older institutions with regard to math achievement. It is concluded that ignoring a multilevel structure in the data may result in erroneous conclusions with regard to the optimal factor structure and the tests of structural models following that.

Reconsidering Cluster Bias in Multilevel Data: A Monte Carlo Comparison of Free and Constrained Baseline Approaches

The test for item level cluster bias examines the improvement in model fit that results from freeing an item's between level residual variance from a baseline model with equal within and between level factor loadings and between level residual variances fixed at zero. A potential problem is that this approach may include a misspecified unrestricted model if any non-invariance is present, but the log-likelihood difference test requires that the unrestricted model is correctly specified. A free baseline approach where the unrestricted model includes only the restrictions needed for model identification should lead to better decision accuracy, but no studies have examined this yet. We ran a Monte Carlo study to investigate this issue. When the referent item is unbiased, compared to the free baseline approach, the constrained baseline approach led to similar true positive (power) rates but much higher false positive (Type I error) rates. The free baseline approach should be preferred when the referent indicator is unbiased. When the referent assumption is violated, the false positive rate was unacceptably high for both free and constrained baseline approaches, and the true positive rate was poor regardless of whether the free or constrained baseline approach was used. Neither the free or constrained baseline approach can be recommended when the referent indicator is biased. We recommend paying close attention to ensuring the referent indicator is unbiased in tests of cluster bias. All Mplus input and output files, R, and short Python scripts used to execute this simulation study are uploaded to an open access repository.

Relating Measurement Invariance, Cross-Level Invariance, and Multilevel Reliability

Data often have a nested, multilevel structure, for example when data are collected from children in classrooms. This kind of data complicate the evaluation of reliability and measurement invariance, because several properties can be evaluated at both the individual level and the cluster level, as well as across levels. For example, cross-level invariance implies equal factor loadings across levels, which is needed to give latent variables at the two levels a similar interpretation. Reliability at a specific level refers to the ratio of true score variance over total variance at that level. This paper aims to shine light on the relation between reliability, cross-level invariance, and strong factorial invariance across clusters in multilevel data. Specifically, we will illustrate how strong factorial invariance across clusters implies cross-level invariance and perfect reliability at the between level in multilevel factor models.

A Solution to Modeling Multilevel Confirmatory Factor Analysis with Data Obtained from Complex Survey Sampling to Avoid Conflated Parameter Estimates

The issue of equality in the between-and within-level structures in Multilevel Confirmatory Factor Analysis (MCFA) models has been influential for obtaining unbiased parameter estimates and statistical inferences. A commonly seen condition is the inequality of factor loadings under equal level-varying structures. With mathematical investigation and Monte Carlo simulation, this study compared the robustness of five statistical models including two model-based (a true and a mis-specified models), one design-based, and two maximum models (two models where the full rank of variance-covariance matrix is estimated in between level and within level, respectively) in analyzing complex survey measurement data with level-varying factor loadings. The empirical data of 120 3rd graders' (from 40 classrooms) perceived Harter competence scale were modeled using MCFA and the parameter estimates were used as true parameters to perform the Monte Carlo simulation study. Results showed maximum models was robust to unequal factor loadings while the design-based and the miss-specified model-based approaches produced conflated results and spurious statistical inferences. We recommend the use of maximum models if researchers have limited information about the pattern of factor loadings and measurement structures. Measurement models are key components of Structural Equation Modeling (SEM); therefore, the findings can be generalized to multilevel SEM and CFA models. Mplus codes are provided for maximum models and other analytical models.

Multilevel Factor Analysis: Reporting Guidelines and a Review of Reporting Practices

We provide reporting guidelines for multilevel factor analysis (MFA) and use these guidelines to systematically review 72 MFA applications in journals across a range of disciplines (e.g., education, health/nursing, management, and psychology) published between 1994 and 2014. Results are organized in terms of the (a) characteristics of the MFA application (e.g., construct measured), (b) purpose (e.g., measurement validation), (c) data source (e.g., number of cases at Level 1 and Level 2), (d) statistical approach (e.g., maximum likelihood), and (e) results reported (e.g., intraclass correlations for indicators and latent variables, standardized factor loadings, fit indices). Results from this review have implications for applied researchers interested in expanding their approaches to psychometric analyses and construct validation within a multilevel framework and for methodologists using Monte Carlo methods to explore technical and methodological issues grounded in realistic research design conditions.

Collinear Latent Variables in Multilevel Confirmatory Factor Analysis: A Comparison of Maximum Likelihood and Bayesian Estimations

Because variables may be correlated in the social and behavioral sciences, multicollinearity might be problematic. This study investigates the effect of collinearity manipulated in within and between levels of a two-level confirmatory factor analysis by Monte Carlo simulation. Furthermore, the influence of the size of the intraclass correlation coefficient (ICC) and estimation method; maximum likelihood estimation with robust chi-squares and standard errors and Bayesian estimation, on the convergence rate are investigated. The other variables of interest were rate of inadmissible solutions and the relative parameter and standard error bias on the between level. The results showed that inadmissible solutions were obtained when there was between level collinearity and the estimation method was maximum likelihood. In the within level multicollinearity condition, all of the solutions were admissible but the bias values were higher compared with the between level collinearity condition. Bayesian estimation appeared to be robust in obtaining admissible parameters but the relative bias was higher than for maximum likelihood estimation. Finally, as expected, high ICC produced less biased results compared to medium ICC conditions.