The Multiverse of Methods: Extending the Multiverse Analysis to Address Data-Collection Decisions

Optimal processing of surface facial EMG to identify emotional expressions: A data-driven approach

Surface facial electromyography (EMG) is commonly used to detect emotions from subtle facial expressions. Although there are established procedures for collecting EMG data and some aspects of their processing, there is little agreement among researchers about the optimal way to process the EMG signal, so that the study-unrelated variability (noise) is removed, and the emotion-related variability is best detected. The aim of the current paper was to establish an optimal processing pipeline for EMG data for identifying emotional expressions in facial muscles. We identified the most common processing steps from existing literature and created 72 processing pipelines that represented all the different processing choices. We applied these pipelines to a previously published dataset from a facial mimicry experiment, where 100 adult participants observed happy and sad facial expressions, whilst the activity of their facial muscles, zygomaticus major and corrugator supercilii, was recorded with EMG. We used a resampling approach and subsets of the original data to investigate the effect and robustness of different processing choices on the performance of a logistic regression model that predicted the mimicked emotion (happy/sad) from the EMG signal. In addition, we used a random forest model to identify the most important processing steps for the sensitivity of the logistic regression model. Three processing steps were found to be most impactful: baseline correction, standardisation within muscles, and standardisation within subjects. The chosen feature of interest and the signal averaging had little influence on the sensitivity to the effect. We recommend an optimal processing pipeline, share our code and data, and provide a step-by-step walkthrough for researchers.

First steps into the pupillometry multiverse of developmental science

Pupillometry has been widely implemented to investigate cognitive functioning since infancy. Like most psychophysiological and behavioral measures, it implies hierarchical levels of arbitrariness in preprocessing before statistical data analysis. By means of an illustrative example, we checked the robustness of the results of a familiarization procedure that compared the impact of audiovisual and visual stimuli in 12-month-olds. We adopted a multiverse approach to pupillometry data analysis to explore the role of (1) the preprocessing phase, that is, handling of extreme values, selection of the areas of interest, management of blinks, baseline correction, participant inclusion/exclusion and (2) the modeling structure, that is, the incorporation of smoothers, fixed and random effects structure, in guiding the parameter estimation. The multiverse of analyses shows how the preprocessing steps influenced the regression results, and when visual stimuli plausibly predicted an increase of resource allocation compared with audiovisual stimuli. Importantly, smoothing time in statistical models increased the plausibility of the results compared to those nested models that do not weigh the impact of time. Finally, we share theoretical and methodological tools to move the first steps into (rather than being afraid of) the inherent uncertainty of infant pupillometry.

Beyond single paradigms, pipelines, and outcomes: Embracing multiverse analyses in psychophysiology

Psychophysiological research is an inherently complex undertaking due to the nature of the data, and its analysis is characterized by many decision points that shape the final dataset and a study's findings. These decisions create a "multiverse" of possible outcomes, and each decision from study conceptualization to statistical analysis can lead to different results and interpretations. This review describes the concept of multiverse analyses, a methodological approach designed to understand the impact of different decisions on the robustness of a study's findings and interpretation. The emphasis is on transparently showcasing different reasonable approaches for constructing a final dataset and on highlighting the influence of various decision points, from experimental design to data processing and outcome selection. For example, the choice of an experimental task can significantly impact event-related brain potential (ERP) scores or skin conductance responses (SCRs), and different tasks might elicit unique variances in each measure. This review underscores the importance of transparently embracing the flexibility inherent in psychophysiological research and the potential consequences of not understanding the fragility or robustness of experimental findings. By navigating the intricate terrain of the psychophysiological multiverse, this review serves as an introduction, helping researchers to make informed decisions, improve the collective understanding of psychophysiological findings, and push the boundaries of the field.

Exploring the multiverse: the impact of researchers' analytic decisions on relationships between depression and inflammatory markers

In recent years, a replication crisis in psychiatry has led to a growing focus on the impact of researchers' analytic decisions on the results from studies. Multiverse analyses involve examining results across a wide array of possible analytic decisions (e.g., log-transforming variables, number of covariates, or treatment of outliers) and identifying if study results are robust to researchers' analytic decisions. Studies have begun to use multiverse analysis for well-studied relationships that have some heterogeneity in results/conclusions across studies.We examine the well-studied relationship between peripheral inflammatory markers (PIMs; e.g., white blood cell count (WBC) and C-reactive protein (CRP)) and depression severity in the large NHANES dataset (n = 25,962). Specification curve analyses tested the impact of 9 common analytic decisions (comprising of 58,000+ possible combinations) on the association of PIMs and depression severity. Relationships of PIMs and total depression severity are robust to analytic decisions (based on tests of inference jointly examining effect sizes and p-values). However, moderate/large differences are noted in effect sizes based on analytic decisions and the majority of analyses do not result in significant findings, with the percentage of analyses with statistically significant results being 46.1% for WBC and 43.8% for CRP. For associations of PIMs with specific symptoms of depression, some associations (e.g., sleep, appetite) in males (but not females) were robust to analytic decisions. We discuss how multiverse analyses can be used to guide research and also the need for authors, reviewers, and editors to incorporate multiverse analyses to enhance replicability of research findings.

Systematically investigating the role of context on effect replicability in reinstatement of fear in humans

Context is crucial in guiding behavior in an ever-changing world and contextual information plays a crucial role in associative learning processes. For instance, the return of fear (RoF) after successful extinction, which is used to study the mechanisms underlying relapse phenomena in fear- and stress-related disorders in an experimental model, is known to be context dependent as evident from phenomena such as renewal (contextual change) and reinstatement (re-exposure to an aversive event). Human adaptions of reinstatement paradigms have resulted in mixed findings: CS specific as well as unspecific RoF or unexpected "reinstated" conditioned responding in no reinstatement US control groups. Here, we systematically investigate the role of context (i.e., cue-context compound) on reinstatement-induced RoF in a human differential fear conditioning paradigm using subjective and psychophysiological measures in a large sample (N = 212) including reinstatement and control groups. Overall, response patterns in reinstatement-groups mirrored results from single-cue rodent work. Yet, only generalized, not differential RoF was observed. Remarkably, depending on outcome measure RoF was also observed under identical experimental context conditions without US-re-exposure, underlining effects of contextual change beyond the reinstatement-US and challenging reinstatement research in human subjects and highlight that future reinstatement work should focus on the operationalization of context.

How robust is the relationship between neural processing speed and cognitive abilities?

Individual differences in processing speed are consistently related to individual differences in cognitive abilities, but the mechanisms through which a higher processing speed facilitates reasoning remain largely unknown. To identify these mechanisms, researchers have been using latencies of the event-related potential (ERP) to study how the speed of cognitive processes associated with specific ERP components is related to cognitive abilities. Although there is some evidence that latencies of ERP components associated with higher-order cognitive processes are related to intelligence, results are overall quite inconsistent. These inconsistencies likely result from variations in analytic procedures and little consideration of the psychometric properties of ERP latencies in relatively small sample studies. Here we used a multiverse approach to evaluate how different analytical choices regarding references, low-pass filter cutoffs, and latency measures affect the psychometric properties of P2, N2, and P3 latencies and their relations with cognitive abilities in a sample of 148 participants. Latent correlations between neural processing speed and cognitive abilities ranged from -.49 to -.78. ERP latency measures contained about equal parts of measurement error variance and systematic variance, and only about half of the systematic variance was related to cognitive abilities, whereas the other half reflected nuisance factors. We recommend addressing these problematic psychometric properties by recording EEG data from multiple tasks and modeling relations between ERP latencies and covariates in latent variable models. All in all, our results indicate that there is a substantial and robust relationship between neural processing speed and cognitive abilities when those issues are addressed.

Context-dependent amygdala-prefrontal connectivity during the dot-probe task varies by irritability and attention bias to angry faces

Irritability, defined as proneness to anger, is among the most common reasons youth are seen for psychiatric care. Youth with irritability demonstrate aberrant processing of anger-related stimuli; however, the neural mechanisms remain unknown. We applied a drift-diffusion model (DDM), a computational tool, to derive a latent behavioral metric of attentional bias to angry faces in youth with varying levels of irritability during functional magnetic resonance imaging (fMRI). We examined associations among irritability, task behavior using a DDM-based index for preferential allocation of attention to angry faces (i.e., extra-decisional time bias; Δt₀), and amygdala context-dependent connectivity during the dot-probe task. Our transdiagnostic sample, enriched for irritability, included 351 youth (ages 8-18; M = 12.92 years, 51% male, with primary diagnoses of either attention deficit/hyperactivity disorder [ADHD], disruptive mood dysregulation disorder [DMDD], an anxiety disorder, or healthy controls). Models accounted for age, sex, in-scanner motion, and co-occurring symptoms of anxiety. Youth and parents rated youth's irritability using the Affective Reactivity Index. An fMRI dot-probe task was used to assess attention orienting to angry faces. In the angry-incongruent vs. angry-congruent contrast, amygdala connectivity with the bilateral inferior frontal gyrus (IFG), insula, caudate, and thalamus/pulvinar was modulated by irritability level and attention bias to angry faces, Δt₀, all ts₃₅₀ > 4.46, ps < 0.001. In youth with high irritability, elevated Δt₀ was associated with a weaker amygdala connectivity. In contrast, in youth with low irritability, elevated Δt₀ was associated with stronger connectivity in those regions. No main effect emerged for irritability. As irritability is associated with reactive aggression, these results suggest a potential neural regulatory deficit in irritable youth who have elevated attention bias to angry cues.

The thresholding problem and variability in the EEG graph network parameters

Graph thresholding is a frequently used practice of eliminating the weak connections in brain functional connectivity graphs. The main aim of the procedure is to delete the spurious connections in the data. However, the choice of the threshold is arbitrary, and the effect of the threshold choice is not fully understood. Here we present the description of the changes in the global measures of a functional connectivity graph depending on the different proportional thresholds based on the 146 resting-state EEG recordings. The dynamics is presented in five different synchronization measures (wPLI, ImCoh, Coherence, ciPLV, PPC) in sensors and source spaces. The analysis shows significant changes in the graph's global connectivity measures as a function of the chosen threshold which may influence the outcome of the study. The choice of the threshold could lead to different study conclusions; thus it is necessary to improve the reasoning behind the choice of the different analytic options and consider the adoption of different analytic approaches. We also proposed some ways of improving the procedure of thresholding in functional connectivity research.

Exploring the Multiverse of Analytical Decisions in Scaling Educational Large-Scale Assessment Data: A Specification Curve Analysis for PISA 2018 Mathematics Data

In educational large-scale assessment (LSA) studies such as PISA, item response theory (IRT) scaling models summarize students’ performance on cognitive test items across countries. This article investigates the impact of different factors in model specifications for the PISA 2018 mathematics study. The diverse options of the model specification also firm under the labels multiverse analysis or specification curve analysis in the social sciences. In this article, we investigate the following five factors of model specification in the PISA scaling model for obtaining the two country distribution parameters; country means and country standard deviations: (1) the choice of the functional form of the IRT model, (2) the treatment of differential item functioning at the country level, (3) the treatment of missing item responses, (4) the impact of item selection in the PISA test, and (5) the impact of test position effects. In our multiverse analysis, it turned out that model uncertainty had almost the same impact on variability in the country means as sampling errors due to the sampling of students. Model uncertainty had an even larger impact than standard errors for country standard deviations. Overall, each of the five specification factors in the multiverse analysis had at least a moderate effect on either country means or standard deviations. In the discussion section, we critically evaluate the current practice of model specification decisions in LSA studies. It is argued that we would either prefer reporting the variability in model uncertainty or choosing a particular model specification that might provide the strategy that is most valid. It is emphasized that model fit should not play a role in selecting a scaling strategy for LSA applications.

Interpolated testing and content pretesting as interventions to reduce task-unrelated thoughts during a video lecture

Considerable research has examined the prevalence and apparent consequences of task-unrelated thoughts (TUTs) in both laboratory and authentic educational settings. Few studies, however, have explored methods to reduce TUTs during learning; those few studies tested small samples or used unvalidated TUT assessments. The present experimental study attempted to conceptually replicate or extend previous findings of interpolated testing and pretesting effects on TUT and learning. In a study of 195 U.S. undergraduates, we investigated whether interpolated testing (compared to interpolated restudy) and pretesting on lecture-relevant materials (compared to pretesting on conceptually related but lecture-irrelevant materials) would reduce TUTs during a video lecture on introductory statistics. Subjects completed either a content-matched or content-mismatched pretest on statistics concepts and then watched a narrated lecture slideshow. During the lecture, half of the sample completed interpolated tests on the lecture material and half completed interpolated restudy of that material. All subjects responded to unpredictably presented thought probes during the video to assess their immediately preceding thoughts, including TUTs. Following the lecture, students reported on their situational interest elicited by the lecture and then completed a posttest. Interpolated testing significantly reduced TUT rates during the lecture compared to restudying, conceptually replicating previous findings—but with a small effect size and no supporting Bayes-factor evidence. We found statistical evidence for neither an interpolated testing effect on learning, nor an effect of matched-content pretesting on TUT rates or learning. Interpolated testing might have limited utility to support students’ attention, but varying effect sizes across studies warrants further work.

Systematically assessing microbiome–disease associations identifies drivers of inconsistency in metagenomic research

Evaluating the relationship between the human gut microbiome and disease requires computing reliable statistical associations. Here, using millions of different association modeling strategies, we evaluated the consistency—or robustness—of microbiome-based disease indicators for 6 prevalent and well-studied phenotypes (across 15 public cohorts and 2,343 individuals). We were able to discriminate between analytically robust versus nonrobust results. In many cases, different models yielded contradictory associations for the same taxon–disease pairing, some showing positive correlations and others negative. When querying a subset of 581 microbe–disease associations that have been previously reported in the literature, 1 out of 3 taxa demonstrated substantial inconsistency in association sign. Notably, >90% of published findings for type 1 diabetes (T1D) and type 2 diabetes (T2D) were particularly nonrobust in this regard. We additionally quantified how potential confounders—sequencing depth, glucose levels, cholesterol, and body mass index, for example—influenced associations, analyzing how these variables affect the ostensible correlation between Faecalibacterium prausnitzii abundance and a healthy gut. Overall, we propose our approach as a method to maximize confidence when prioritizing findings that emerge from microbiome association studies.

The human microbiome has been associated with many aspects of our health, but how many of these associations are truly reproducible? This study attempts to address this question by systematically testing the robustness of 581 microbial features that have been reported as being disease-associated.

The data-processing multiverse of event-related potentials (ERPs): A roadmap for the optimization and standardization of ERP processing and reduction pipelines

In studies of event-related brain potentials (ERPs), numerous decisions about data processing are required to extract ERP scores from continuous data. Unfortunately, the systematic impact of these choices on the data quality and psychometric reliability of ERP scores or even ERP scores themselves is virtually unknown, which is a barrier to the standardization of ERPs. The aim of the present study was to optimize processing pipelines for the error-related negativity (ERN) and error positivity (Pe) by considering a multiverse of data processing choices. A multiverse analysis of a data processing pipeline examines the impact of a large set of different reasonable choices to determine the robustness of effects, such as the impact of different decisions on between-trial standard deviations (i.e., data quality) and between-condition differences (i.e., experimental effects). ERN and Pe data from 298 healthy young adults were used to determine the impact of different methodological choices on data quality and experimental effects (correct vs. error trials) at several key stages: highpass filtering, lowpass filtering, ocular artifact correction, reference, baseline adjustment, scoring sensors, and measurement procedure. This multiverse analysis yielded 3,456 ERN scores and 576 Pe scores per person. An optimized pipeline for ERN included a 15 Hz lowpass filter, ICA-based ocular artifact correction, and a region of interest (ROI) approach to scoring. For Pe, the optimized pipeline included a 0.10 Hz highpass filter, 30 Hz lowpass filter, regression-based ocular artifact correction, a -200 to 0 ms baseline adjustment window, and an ROI approach to scoring. The multiverse approach can be used to optimize pipelines for eventual standardization, which would support efforts toward establishing normative ERP databases. The proposed process of analyzing the data-processing multiverse of ERP scores paves the way for better refinement, identification, and selection of data processing parameters, ultimately improving the precision and utility of ERPs.

On the Treatment of Missing Item Responses in Educational Large-Scale Assessment Data: An Illustrative Simulation Study and a Case Study Using PISA 2018 Mathematics Data

Missing item responses are prevalent in educational large-scale assessment studies such as the programme for international student assessment (PISA). The current operational practice scores missing item responses as wrong, but several psychometricians have advocated for a model-based treatment based on latent ignorability assumption. In this approach, item responses and response indicators are jointly modeled conditional on a latent ability and a latent response propensity variable. Alternatively, imputation-based approaches can be used. The latent ignorability assumption is weakened in the Mislevy-Wu model that characterizes a nonignorable missingness mechanism and allows the missingness of an item to depend on the item itself. The scoring of missing item responses as wrong and the latent ignorable model are submodels of the Mislevy-Wu model. In an illustrative simulation study, it is shown that the Mislevy-Wu model provides unbiased model parameters. Moreover, the simulation replicates the finding from various simulation studies from the literature that scoring missing item responses as wrong provides biased estimates if the latent ignorability assumption holds in the data-generating model. However, if missing item responses are generated such that they can only be generated from incorrect item responses, applying an item response model that relies on latent ignorability results in biased estimates. The Mislevy-Wu model guarantees unbiased parameter estimates if the more general Mislevy-Wu model holds in the data-generating model. In addition, this article uses the PISA 2018 mathematics dataset as a case study to investigate the consequences of different missing data treatments on country means and country standard deviations. Obtained country means and country standard deviations can substantially differ for the different scaling models. In contrast to previous statements in the literature, the scoring of missing item responses as incorrect provided a better model fit than a latent ignorable model for most countries. Furthermore, the dependence of the missingness of an item from the item itself after conditioning on the latent response propensity was much more pronounced for constructed-response items than for multiple-choice items. As a consequence, scaling models that presuppose latent ignorability should be refused from two perspectives. First, the Mislevy-Wu model is preferred over the latent ignorable model for reasons of model fit. Second, in the discussion section, we argue that model fit should only play a minor role in choosing psychometric models in large-scale assessment studies because validity aspects are most relevant. Missing data treatments that countries can simply manipulate (and, hence, their students) result in unfair country comparisons.

Leveraging vibration of effects analysis for robust discovery in observational biomedical data science

Hypothesis generation in observational, biomedical data science often starts with computing an association or identifying the statistical relationship between a dependent and an independent variable. However, the outcome of this process depends fundamentally on modeling strategy, with differing strategies generating what can be called "vibration of effects" (VoE). VoE is defined by variation in associations that often lead to contradictory results. Here, we present a computational tool capable of modeling VoE in biomedical data by fitting millions of different models and comparing their output. We execute a VoE analysis on a series of widely reported associations (e.g., carrot intake associated with eyesight) with an extended additional focus on lifestyle exposures (e.g., physical activity) and components of the Framingham Risk Score for cardiovascular health (e.g., blood pressure). We leveraged our tool for potential confounder identification, investigating what adjusting variables are responsible for conflicting models. We propose modeling VoE as a critical step in navigating discovery in observational data, discerning robust associations, and cataloging adjusting variables that impact model output.

The multiplicity of analysis strategies jeopardizes replicability: lessons learned across disciplines

For a given research question, there are usually a large variety of possible analysis strategies acceptable according to the scientific standards of the field, and there are concerns that this multiplicity of analysis strategies plays an important role in the non-replicability of research findings. Here, we define a general framework on common sources of uncertainty arising in computational analyses that lead to this multiplicity, and apply this framework within an overview of approaches proposed across disciplines to address the issue. Armed with this framework, and a set of recommendations derived therefrom, researchers will be able to recognize strategies applicable to their field and use them to generate findings more likely to be replicated in future studies, ultimately improving the credibility of the scientific process.