High replicability of newly discovered social-behavioural findings is achievable

Glutamate, GABA and NAA in treatment-resistant schizophrenia: A systematic review of the effect of clozapine and group differences between clozapine-responders and non-responders

Treatment-resistance in patients with schizophrenia is a major obstacle for improving outcome in patients, especially in those not gaining from clozapine. Novel research implies that glutamatergic and GABAergic abnormalities may be present in treatment-resistant patients, and preclinical research suggests that clozapine affects the GABAergic system. Moreover, clozapine may have a neuroprotective role. To investigate these issues, we conducted a systematic review to evaluate the relationship between clozapine and in vivo measures of gamma-aminobutyric acid (GABA), glutamate (glu), and N-acetylaspartate (NAA) brain levels in treatment- and ultra-treatment-resistant schizophrenia patients (TRS and UTRS). Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, we included three longitudinal and six cross sectional studies utilizing proton magnetic resonance spectroscopy (H-MRS) that explored brain metabolite levels in clozapine-treated patients. Findings were limited by a small number of studies and definite conclusions cannot be drawn, but the present studies may imply that clozapine reduces glutamate levels in striatal but not cortical areas, whereas glutamatergic metabolites and GABA levels may be increased in ACC in the combined group of TRS and UTRS. Clozapine may also increase NAA in cortical areas. Importantly, this review highlights the need for further clinical studies investigating the effect of clozapine on brain levels of glutamate, GABA, and NAA as well as metabolite group differences in patients with UTRS compared with TRS.

A standardized framework to test event-based experiments

The replication crisis in experimental psychology and neuroscience has received much attention recently. This has led to wide acceptance of measures to improve scientific practices, such as preregistration and registered reports. Less effort has been devoted to performing and reporting the results of systematic tests of the functioning of the experimental setup itself. Yet, inaccuracies in the performance of the experimental setup may affect the results of a study, lead to replication failures, and importantly, impede the ability to integrate results across studies. Prompted by challenges we experienced when deploying studies across six laboratories collecting electroencephalography (EEG)/magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI), and intracranial EEG (iEEG), here we describe a framework for both testing and reporting the performance of the experimental setup. In addition, 100 researchers were surveyed to provide a snapshot of current common practices and community standards concerning testing in published experiments' setups. Most researchers reported testing their experimental setups. Almost none, however, published the tests performed or their results. Tests were diverse, targeting different aspects of the setup. Through simulations, we clearly demonstrate how even slight inaccuracies can impact the final results. We end with a standardized, open-source, step-by-step protocol for testing (visual) event-related experiments, shared via protocols.io. The protocol aims to provide researchers with a benchmark for future replications and insights into the research quality to help improve the reproducibility of results, accelerate multicenter studies, increase robustness, and enable integration across studies.

Decatastrophizing research irreproducibility

The reported inability to replicate research findings from the published literature precipitated extensive efforts to identify and correct perceived deficiencies in the execution and reporting of biomedical research. Despite these efforts, quantification of the magnitude of irreproducible research or the effectiveness of associated remediation initiatives, across diverse biomedical disciplines, has made little progress over the last decade. The idea that science is self-correcting has been further challenged in recent years by the proliferation of unverified or fraudulent scientific content generated by predatory journals, paper mills, pre-print server postings, and the inappropriate use of artificial intelligence technologies. The degree to which the field of pharmacology has been negatively impacted by these evolving pressures is unknown. Regardless of these ambiguities, pharmacology societies and their associated journals have championed best practices to enhance the experimental rigor and reporting of pharmacological research. The value of transparent and independent validation of raw data generation and its analysis in basic and clinical research is exemplified by the discovery, development, and approval of Highly Effective Modulator Therapy (HEMT) for Cystic Fibrosis (CF) patients. This provides a didactic counterpoint to concerns regarding the current state of biomedical research. Key features of this important therapeutic advance include objective construction of basic and translational research hypotheses, associated experimental designs, and validation of experimental effect sizes with quantitative alignment to meaningful clinical endpoints with input from the FDA, which enhanced scientific rigor and transparency with real world deliverables for patients in need.

Estimating the replicability of highly cited clinical research (2004-2018)

INTRODUCTION

Previous studies about the replicability of clinical research based on the published literature have suggested that highly cited articles are often contradicted or found to have inflated effects. Nevertheless, there are no recent updates of such efforts, and this situation may have changed over time.

METHODS

We searched the Web of Science database for articles studying medical interventions with more than 2000 citations, published between 2004 and 2018 in high-impact medical journals. We then searched for replications of these studies in PubMed using the PICO (Population, Intervention, Comparator and Outcome) framework. Replication success was evaluated by the presence of a statistically significant effect in the same direction and by overlap of the replication's effect size confidence interval (CIs) with that of the original study. Evidence of effect size inflation and potential predictors of replicability were also analyzed.

RESULTS

A total of 89 eligible studies, of which 24 had valid replications (17 meta-analyses and 7 primary studies) were found. Of these, 21 (88%) had effect sizes with overlapping CIs. Of 15 highly cited studies with a statistically significant difference in the primary outcome, 13 (87%) had a significant effect in the replication as well. When both criteria were considered together, the replicability rate in our sample was of 20 out of 24 (83%). There was no evidence of systematic inflation in these highly cited studies, with a mean effect size ratio of 1.03 [95% CI (0.88, 1.21)] between initial and subsequent effects. Due to the small number of contradicted results, our analysis had low statistical power to detect predictors of replicability.

CONCLUSION

Although most studies did not have eligible replications, the replicability rate of highly cited clinical studies in our sample was higher than in previous estimates, with little evidence of systematic effect size inflation. This estimate is based on a very select sample of studies and may not be generalizable to clinical research in general.

The assessment of replicability using the sum of p-values

Statistical significance of both the original and the replication study is a commonly used criterion to assess replication attempts, also known as the two-trials rule in drug development. However, replication studies are sometimes conducted although the original study is non-significant, in which case Type-I error rate control across both studies is no longer guaranteed. We propose an alternative method to assess replicability using the sum of p -values from the two studies. The approach provides a combined p -value and can be calibrated to control the overall Type-I error rate at the same level as the two-trials rule but allows for replication success even if the original study is non-significant. The unweighted version requires a less restrictive level of significance at replication if the original study is already convincing which facilitates sample size reductions of up to 10%. Downweighting the original study accounts for possible bias and requires a more stringent significance level and larger sample sizes at replication. Data from four large-scale replication projects are used to illustrate and compare the proposed method with the two-trials rule, meta-analysis and Fisher's combination method.

Risky-Choice Framing Effects Result Partly From Mismatched Option Descriptions in Gains and Losses

Textbook psychology holds that people usually prefer a certain option over a risky one when options are framed as gains but prefer the opposite when options are framed as losses. However, this pattern can be amplified, eliminated, or reversed depending on whether option descriptions include only positive information (e.g., "200 people will be saved"), only negative information (e.g., "400 people will not be saved"), or both. Previous studies suggest that framing effects arise only when option descriptions are mismatched across frames. Using online and student samples (Ns = 906 and 521), we investigated 81 framing-effect variants created from matched and mismatched pairs of 18 option descriptions (nine in each frame). Description valence or gist explained substantial variation in risk preferences (prospect theory does not predict such variation), but a considerable framing effect remained in our balanced design. Risky-choice framing effects appear to be partly-but not completely-the result of mismatched comparisons.