Constructed Measures and Causal Inference: Towards a New Model of Measurement for Psychosocial Constructs

Early life stress and functional network topology in children

Brain networks are continuously modified throughout development, yet this plasticity can also make functional networks vulnerable to early life stress. Little is currently known about the effect of early life stress on the functional organization of the brain. The current study investigated the association between environmental stressors and network topology using data from the Adolescent Brain Cognitive DevelopmentSM (ABCD®) Study. Hierarchical modeling identified a general factor of environmental stress, representing the common variance across multiple stressors, as well as four subfactors including familial dynamics, interpersonal support, neighborhood SES deprivation, and urbanicity. Functional network topology metrics were obtained using graph theory at rest and during tasks of reward processing, inhibition, and affective working memory. The general factor of environmental stress was associated with less specialization of networks, represented by lower modularity at rest. Local metrics indicated that general environmental stress was also associated with less efficiency in the subcortical-cerebellar and visual networks while showing greater efficiency in the default mode network at rest. Subfactors of environmental stress were associated with differences in specialization and efficiency in select networks. The current study illustrates that a wide range of stressors in a child's environment are associated with differences in brain network topology.

Causal complexity in human research: On the shared challenges of behavior genetics, medical genetics, and environmentally oriented social science

We received 23 spirited commentaries on our target article from across the disciplines of philosophy, economics, evolutionary genetics, molecular biology, criminology, epidemiology, and law. We organize our reply around three overarching questions: (1) What is a cause? (2) How are randomized controlled trials (RCTs) and within-family genome-wide association studies (GWASs) alike and unalike? (3) Is behavior genetics a qualitatively different enterprise? Throughout our discussion of these questions, we advocate for the idea that behavior genetics shares many of the same pitfalls and promises as environmentally oriented research, medical genetics, and other arenas of the social and behavioral sciences.

Niche diversity effects on personality measurement - evidence from large national samples during the COVID-19 pandemic

We report systematic variability in the psychometric properties of a brief personality inventory during the early stages of the COVID-19 pandemic. Drawing upon recent discussions about the universality vs cultural relativism of personality measures, we review and comparatively test theories predicting systematic variability in personality measurement across cultures using an established brief personality measure applied to population samples in 16 nations during the first wave of the COVID-19 pandemic (N = 35,052). We found systematic variation in factor replicability and effective dimensionality. In line with previous theorizing, factors replicated better in contexts with greater niche diversity. Examining possible drivers underlying this association, the investigation of the individual components in the niche construction index suggested that life expectancy and to a lesser degree economic complexity are associated with greater personality structure differentiation. Population-level indicators of acute threat due to COVID-19 did not show credible effects. These patterns suggest that a) investigation of personality structure in population samples can provide useful insights into personality dynamics, b) socioecological factors have a systematic impact on survey responses, but c) we also need better theorizing and research about both personality and culture to understand how niche construction dynamics operate.

Identifying causal subsequent memory effects

Over 40 y of accumulated research has detailed associations between neuroimaging signals measured during a memory encoding task and later memory performance, across a variety of brain regions, measurement tools, statistical approaches, and behavioral tasks. But the interpretation of these subsequent memory effects (SMEs) remains unclear: if the identified signals reflect cognitive and neural mechanisms of memory encoding, then the underlying neural activity must be causally related to future memory. However, almost all previous SME analyses do not control for potential confounders of this causal interpretation, such as serial position and item effects. We collect a large fMRI dataset and use an experimental design and analysis approach that allows us to statistically adjust for nearly all known exogenous confounding variables. We find that, using standard approaches without adjustment, we replicate several univariate and multivariate subsequent memory effects and are able to predict memory performance across people. However, we are unable to identify any signal that reliably predicts subsequent memory after adjusting for confounding variables, bringing into doubt the causal status of these effects. We apply the same approach to subjects' judgments of learning collected following an encoding period and show that these behavioral measures of mnemonic status do predict memory after adjustments, suggesting that it is possible to measure signals near the time of encoding that reflect causal mechanisms but that existing neuroimaging measures, at least in our data, may not have the precision and specificity to do so.

A statistical test to reject the structural interpretation of a latent factor model

Factor analysis is often used to assess whether a single univariate latent variable is sufficient to explain most of the covariance among a set of indicators for some underlying construct. When evidence suggests that a single factor is adequate, research often proceeds by using a univariate summary of the indicators in subsequent research. Implicit in such practices is the assumption that it is the underlying latent, rather than the indicators, that is causally efficacious. The assumption that the indicators do not have effects on anything subsequent, and that they are themselves only affected by antecedents through the underlying latent is a strong assumption, effectively imposing a structural interpretation on the latent factor model. In this paper, we show that this structural assumption has empirically testable implications, even though the latent variable itself is unobserved. We develop a statistical test to potentially reject the structural interpretation of a latent factor model. We apply this test to data concerning associations between the Satisfaction-with-Life-Scale and subsequent all-cause mortality, which provides strong evidence against a structural interpretation for a univariate latent underlying the scale. Discussion is given to the implications of this result for the development, evaluation, and use of measures, and for the use of factor analysis itself.

Comparing the Robustness of the Structural after Measurement (SAM) Approach to Structural Equation Modeling (SEM) against Local Model Misspecifications with Alternative Estimation Approaches

Structural equation models (SEM), or confirmatory factor analysis as a special case, contain model parameters at the measurement part and the structural part. In most social-science SEM applications, all parameters are simultaneously estimated in a one-step approach (e.g., with maximum likelihood estimation). In a recent article, Rosseel and Loh (2022, Psychol. Methods) proposed a two-step structural after measurement (SAM) approach to SEM that estimates the parameters of the measurement model in the first step and the parameters of the structural model in the second step. Rosseel and Loh claimed that SAM is more robust to local model misspecifications (i.e., cross loadings and residual correlations) than one-step maximum likelihood estimation. In this article, it is demonstrated with analytical derivations and simulation studies that SAM is generally not more robust to misspecifications than one-step estimation approaches. Alternative estimation methods are proposed that provide more robustness to misspecifications. SAM suffers from finite-sample bias that depends on the size of factor reliability and factor correlations. A bootstrap-bias-corrected LSAM estimate provides less biased estimates in finite samples. Nevertheless, we argue in the discussion section that applied researchers should nevertheless adopt SAM because robustness to local misspecifications is an irrelevant property when applying SAM. Parameter estimates in a structural model are of interest because intentionally misspecified SEMs frequently offer clearly interpretable factors. In contrast, SEMs with some empirically driven model modifications will result in biased estimates of the structural parameters because the meaning of factors is unintentionally changed.

Exploring the Multiverse of Analytical Decisions in Scaling Educational Large-Scale Assessment Data: A Specification Curve Analysis for PISA 2018 Mathematics Data

In educational large-scale assessment (LSA) studies such as PISA, item response theory (IRT) scaling models summarize students’ performance on cognitive test items across countries. This article investigates the impact of different factors in model specifications for the PISA 2018 mathematics study. The diverse options of the model specification also firm under the labels multiverse analysis or specification curve analysis in the social sciences. In this article, we investigate the following five factors of model specification in the PISA scaling model for obtaining the two country distribution parameters; country means and country standard deviations: (1) the choice of the functional form of the IRT model, (2) the treatment of differential item functioning at the country level, (3) the treatment of missing item responses, (4) the impact of item selection in the PISA test, and (5) the impact of test position effects. In our multiverse analysis, it turned out that model uncertainty had almost the same impact on variability in the country means as sampling errors due to the sampling of students. Model uncertainty had an even larger impact than standard errors for country standard deviations. Overall, each of the five specification factors in the multiverse analysis had at least a moderate effect on either country means or standard deviations. In the discussion section, we critically evaluate the current practice of model specification decisions in LSA studies. It is argued that we would either prefer reporting the variability in model uncertainty or choosing a particular model specification that might provide the strategy that is most valid. It is emphasized that model fit should not play a role in selecting a scaling strategy for LSA applications.