The role of group dynamics in scientific inconsistencies: a case study of a research consortium

The Complexities of Evaluating the Exposome in Psychiatry: A Data-Driven Illustration of Challenges and Some Propositions for Amendments

Identifying modifiable factors through environmental research may improve mental health outcomes. However, several challenges need to be addressed to optimize the chances of success. By analyzing the Netherlands Mental Health Survey and Incidence Study-2 data, we provide a data-driven illustration of how closely connected the exposures and the mental health outcomes are and how model and variable specifications produce "vibration of effects" (variation of results under multiple different model specifications). Interdependence of exposures is the rule rather than the exception. Therefore, exposure-wide systematic approaches are needed to separate genuine strong signals from selective reporting and dissect sources of heterogeneity. Pre-registration of protocols and analytical plans is still uncommon in environmental research. Different studies often present very different models, including different variables, despite examining the same outcome, even if consistent sets of variables and definitions are available. For datasets that are already collected (and often already analyzed), the exploratory nature of the work should be disclosed. Exploratory analysis should be separated from prospective confirmatory research with truly pre-specified analysis plans. In the era of big-data, where very low P values for trivial effects are detected, several safeguards may be considered to improve inferences, eg, lowering P-value thresholds, prioritizing effect sizes over significance, analyzing pre-specified falsification endpoints, and embracing alternative approaches like false discovery rates and Bayesian methods. Any claims for causality should be cautious and preferably avoided, until intervention effects have been validated. We hope the propositions for amendments presented here may help with meeting these pressing challenges.

Lack of Associations between Female Hormone Levels and Visuospatial Working Memory, Divided Attention and Cognitive Bias across Two Consecutive Menstrual Cycles

Background: Interpretation of observational studies on associations between prefrontal cognitive functioning and hormone levels across the female menstrual cycle is complicated due to small sample sizes and poor replicability.

Methods: This observational multisite study comprised data of n = 88 menstruating women from Hannover, Germany, and Zurich, Switzerland, assessed during a first cycle and n = 68 re-assessed during a second cycle to rule out practice effects and false-positive chance findings. We assessed visuospatial working memory, attention, cognitive bias and hormone levels at four consecutive time-points across both cycles. In addition to inter-individual differences we examined intra-individual change over time (i.e., within-subject effects).

Results: Estrogen, progesterone and testosterone did not relate to inter-individual differences in cognitive functioning. There was a significant negative association between intra-individual change in progesterone and change in working memory from pre-ovulatory to mid-luteal phase during the first cycle, but that association did not replicate in the second cycle. Intra-individual change in testosterone related negatively to change in cognitive bias from menstrual to pre-ovulatory as well as from pre-ovulatory to mid-luteal phase in the first cycle, but these associations did not replicate in the second cycle.

Conclusions: There is no consistent association between women's hormone levels, in particular estrogen and progesterone, and attention, working memory and cognitive bias. That is, anecdotal findings observed during the first cycle did not replicate in the second cycle, suggesting that these are false-positives attributable to random variation and systematic biases such as practice effects. Due to methodological limitations, positive findings in the published literature must be interpreted with reservation.

Subtle Scientific Fallacies Undermine the Validity of Neuroendocrinological Research: Do Not Draw Premature Conclusions on the Role of Female Sex Hormones

Major scientific flaws such as reporting and publication biases are well documented, even though acknowledgment of their importance appears to be lacking in various psychological and medical fields. Subtle and less obvious biases including selective reviews of the literature and empirically unsupported conclusions and recommendations have received even less attention. Using the literature on the association between transition to menopause, hormones and the onset of depression as a guiding example, I outline how such scientific fallacies undermine the validity of neuroendocrinological research. It is shown that in contrast to prominent claims, first, most prospective studies do not support the notion that the menopausal transition relates to increased risk for depression, second, that associations between hormone levels and depression are largely inconsistent and irreproducible, and, third, that the evidence for the efficacy of hormone therapy for the treatment of depression is very weak and at best inconclusive. I conclude that a direct and uniform association between female sex hormones and depression is clearly not supported by the literature and that more attention should be paid to the manifold scientific biases that undermine the validity of findings in psychological and medical research, with a specific focus on the behavioral neurosciences.

Personality, Psychopathology, and Psychotherapy: A Pre-specified Analysis Protocol for Confirmatory Research on Personality-Psychopathology Associations in Psychotherapy Outpatients

The role of personality trait variation in psychopathology and its influence on the outcome of psychotherapy is a burgeoning field. However, thus far most findings were based on controlled clinical trials that may only poorly represent real-world clinical settings due to highly selective samples mostly restricted to patients with major depression undergoing antidepressive medication. Focusing on personality and psychopathology in a representative naturalistic sample of psychotherapy patients is therefore worthwhile. Moreover, up to date hardly any confirmatory research has been conducted in this field. Strictly confirmatory research implies two major requirements: firstly, specific hypotheses, including expected effect sizes and statistical approaches to data analysis, must be detailed prior to inspection of the data, and secondly, corresponding protocols have to be published online and freely available. Here, we introduce a longitudinal naturalistic study aimed at examining, firstly, the prospective impact of baseline personality traits on the outcome of psychotherapy over a 6-month observation period; secondly, the stability and change in personality traits over time; thirdly, the association between longitudinal change in psychopathology and personality; fourthly, the agreement between self-reports and informant rating of personality; and fifthly, the predictive validity of personality self-reports compared to corresponding informant ratings. For it, we comprehensively state a priori hypotheses, predict the expected effect sizes and detail the statistical analyses that we intend to conduct to test these predictions. Such a stringent confirmatory design increases the transparency and objectivity of psychopathological research, which is necessary to reduce the rate of false-positive findings and to increase the yield of scientific research.

Predicting mental disorders from hypothalamic-pituitary-adrenal axis functioning: a 3-year follow-up in the TRAILS study

BACKGROUND

Hypothalamic-pituitary-adrenal axis functioning, with cortisol as its major output hormone, has been presumed to play a key role in the development of psychopathology. Predicting affective disorders from diurnal cortisol levels has been inconclusive, whereas the predictive value of stress-induced cortisol concentrations has not been studied before. The aim of this study was to predict mental disorders over a 3-year follow-up from awakening and stress-induced cortisol concentrations.

METHOD

Data were used from 561 TRAILS (TRacking Adolescents' Individual Lives Survey) participants, a prospective cohort study of Dutch adolescents. Saliva samples were collected at awakening and half an hour later and during a social stress test at age 16. Mental disorders were assessed 3 years later with the Composite International Diagnostic Interview (CIDI).

RESULTS

A lower cortisol awakening response (CAR) marginally significantly predicted new disorders [odds ratio (OR) 0.77, p = 0.06]. A flat recovery slope predicted disorders with a first onset after the experimental session (OR 1.27, p = 0.04). Recovery revealed smaller, non-significant ORs when predicting new onset affective or anxiety disorders, major depressive disorder, or dependence disorders in three separate models, corrected for all other new onsets.

CONCLUSIONS

Our results suggest that delayed recovery and possibly reduced CAR are indicators of a more general risk status and may be part of a common pathway to psychopathology. Delayed recovery suggests that individuals at risk for mental disorders perceived the social stress test as less controllable and less predictable.

Glucocorticoid receptor gene methylation and HPA-axis regulation in adolescents. The TRAILS study

Early life adversity and psychopathology are thought to be linked through HPA-axis deregulation. Changes in methylation levels of stress reactivity genes such as the glucocorticoid receptor gene (NR3C1) can be induced by adversity. Higher NR3C1 methylation levels have been associated with a reduced NR3C1 expression, possibly leading to impaired negative feedback regulation of the HPA-axis. In this study we tested whether methylation levels of NR3C1 were associated with HPA-axis regulation, operationalized as cortisol responses. In 361 adolescents (mean age 16.1, SD=0.6), salivary cortisol samples were collected before, during, and after a social stress task, from which response measures (cortisol activation and recovery) were calculated. Higher NR3C1 methylation levels were associated with a flattened cortisol recovery slope, indicating a delayed recovery time. Cortisol response activation was not associated with NR3C1 methylation. These results suggest that methylation of NR3C1 may impair negative feedback of the HPA-axis in adolescents.

Sources of error in the retracted scientific literature

Retraction of flawed articles is an important mechanism for correction of the scientific literature. We recently reported that the majority of retractions are associated with scientific misconduct. In the current study, we focused on the subset of retractions for which no misconduct was identified, in order to identify the major causes of error. Analysis of the retraction notices for 423 articles indexed in PubMed revealed that the most common causes of error-related retraction are laboratory errors, analytical errors, and irreproducible results. The most common laboratory errors are contamination and problems relating to molecular biology procedures (e.g., sequencing, cloning). Retractions due to contamination were more common in the past, whereas analytical errors are now increasing in frequency. A number of publications that have not been retracted despite being shown to contain significant errors suggest that barriers to retraction may impede correction of the literature. In particular, few cases of retraction due to cell line contamination were found despite recognition that this problem has affected numerous publications. An understanding of the errors leading to retraction can guide practices to improve laboratory research and the integrity of the scientific literature. Perhaps most important, our analysis has identified major problems in the mechanisms used to rectify the scientific literature and suggests a need for action by the scientific community to adopt protocols that ensure the integrity of the publication process.

Beyond allostatic load: Rethinking the role of stress in regulating human development

How do exposures to stress affect biobehavioral development and, through it, psychiatric and biomedical disorder? In the health sciences, the allostatic load model provides a widely accepted answer to this question: stress responses, while essential for survival, have negative long-term effects that promote illness. Thus, the benefits of mounting repeated biological responses to threat are traded off against costs to mental and physical health. The adaptive calibration model, an evolutionary–developmental theory of stress–health relations, extends this logic by conceptualizing these trade-offs as decision nodes in allocation of resources. Each decision node influences the next in a chain of resource allocations that become instantiated in the regulatory parameters of stress response systems. Over development, these parameters filter and embed information about key dimensions of environmental stress and support, mediating the organism's openness to environmental inputs, and function to regulate life history strategies to match those dimensions. Drawing on the adaptive calibration model, we propose that consideration of biological fitness trade-offs, as delineated by life history theory, is needed to more fully explain the complex relations between developmental exposures to stress, stress responsivity, behavioral strategies, and health. We conclude that the adaptive calibration model and allostatic load model are only partially complementary and, in some cases, support different approaches to intervention. In the long run, the field may be better served by a model informed by life history theory that addresses the adaptive role of stress response systems in regulating alternative developmental pathways.

Salivary cortisol and psychopathy dimensions in detained antisocial adolescents

Previous research revealed hypothalamic-pituitary-adrenal (HPA) axis abnormalities in relation to antisocial and aggressive behavior. Some evidence suggests that low cortisol levels may serve as a biological marker for a severe antisocial subgroup with pronounced callous-unemotional (CU) traits. Children displaying the combination of severe antisocial behavior and CU traits appear to be particularly at risk of developing adult psychopathy. Given the lack of studies on the relationship between cortisol levels and CU traits in antisocial adolescents, the current study investigates whether cortisol levels are uniquely associated with CU traits as compared to other psychopathy dimensions (i.e., narcissism and impulsivity). Detained antisocial adolescents (n=63) and a community comparison group (n=62) completed diaries and collected three saliva samples daily on two days, with compliance monitored electronically. Psychopathy dimensions were assessed through self-report questionnaires. Externalizing symptoms were assessed by structured clinical interview. Multilevel regression analyses indicated no differences in cortisol levels or diurnal slopes between the two groups. Overall, cortisol levels were not significantly related to psychopathy dimensions. However, greater impulsivity was associated with lower cortisol levels in the community sample, but not in the antisocial group.

CONCLUSION

Results cast doubt on the notion of low cortisol levels as a biological marker for CU traits. Low basal cortisol levels appear to be more closely related to a general deficit in behavioral regulation. Implications for future research are discussed.

Self- or parent report of (co-occurring) internalizing and externalizing problems, and basal or reactivity measures of HPA-axis functioning: a systematic evaluation of the internalizing-hyperresponsivity versus externalizing-hyporesponsivity HPA-axis hypothesis

BACKGROUND

Previous research findings on the link between adolescents' psychopathology and hypothalamic-pituitary-adrenal (HPA) axis activity have been heterogeneous.

METHOD

Adolescents (n=211) with a preadolescent DSM-IV diagnosis participated in a lab-based social stress task. Saliva cortisol was assessed at awakening and during social stress. It was investigated if continuous measures of internalizing and externalizing problems and their interaction, using both self- and parent report, were associated with basal or reactivity measures of HPA-axis functioning.

RESULTS

During social stress, an enhanced total release of cortisol was associated with self-reported internalizing problems and a blunted total release of cortisol with self-reported externalizing problems. Post hoc analyses revealed that the association between enhanced cortisol output and internalizing problems held for boys but not for girls. Associations with morning cortisol measures were overall weak.

CONCLUSIONS

Only in the context of stress, and particularly when based on self-report, blunted cortisol output was associated with externalizing and enhanced cortisol output with internalizing problems. Our broad approach demonstrates the importance of who reports on psychopathology, the use of dimensional measures of psychopathology, simultaneous analysis of internalizing and externalizing problems, and the use of awakening and social stress related measures of cortisol.