SECRETS: Subject-efficient clinical randomized controlled trials using synthetic intervention

Background

The parallel-group randomized controlled trial (RCT) is commonly used in Phase-3 clinical trials to establish treatment effectiveness but requires hundreds-to-thousands of subjects, making it difficult to implement, which leads to high Phase-3 trial failure rates. One approach to increasing power of a trial is to augment data collected from an RCT with external data from prospective studies or prior RCTs. However, this requires that external data be comparable to data from the study of interest, a condition that does not hold for new interventions or populations being studied. Another approach is to lower sample size requirements by using the cross-over design, which measures individual treatment effects (ITEs) to remove inter-subject variability; however, this design is only suitable for chronic conditions and interventions with effects that wash out rapidly.

Method

We propose a novel and practical framework called SECRETS (Subject-Efficient Clinical Randomized Controlled Trials using Synthetic Intervention) to increase power of any parallel-group RCT by simulating the cross-over design using only data collected from the study. SECRETS first estimates ITEs across all subjects recruited to the RCT by using a state-of-the-art counterfactual estimation algorithm called synthetic intervention (SI). Since SI induces dependencies among the ITEs, we introduce a novel hypothesis testing strategy to test for treatment effectiveness.

Results

We show that SECRETS can increase the power of an RCT while maintaining comparable significance levels; in particular, on three real-world clinical RCTs (Phase-3 trials), SECRETS increases power over the baseline method by 6 - 54 % (average: 21.5%, standard deviation: 15.8%), thereby reducing the number of subjects needed to obtain a typically desired statistical operating point of 80% power and 5% significance level by 25 - 76 % (10-3,957 fewer subjects per arm). Our analyses show that SECRETS increases power by consistently reducing the variance of the average treatment effect, thereby mimicking the effects of a cross-over design.

Conclusion

SECRETS increases subject efficiency of an RCT by simulating the cross-over design using only data collected from the RCT; therefore, it is a feasible solution for increasing the trial's power, especially under settings where satisfying sample size requirements is difficult.

"This behavior strikes us as ideal": assessment and anticipations of Huisman (2022)

Huisman (Psychonomic Bulletin & Review, 1-10. 2022) argued that a valid measure of evidence should indicate more support in favor of a true alternative hypothesis when sample size is large than when it is small. Bayes factors may violate this pattern and hence Huisman concluded that Bayes factors are invalid as a measure of evidence. In this brief comment we call attention to the following: (1) Huisman's purported anomaly is in fact dictated by probability theory; (2) Huisman's anomaly has been discussed and explained in the statistical literature since 1939; the anomaly was also highlighted in the Psychonomic Bulletin & Review article by Rouder et al. (2009), who interpreted the anomaly as "ideal": an interpretation diametrically opposed to that of Huisman. We conclude that when intuition clashes with probability theory, chances are that it is intuition that needs schooling.

Hypothesis testing of Poisson rates in COVID-19 offspring distributions

In the present study, we undertake the task of hypothesis testing in the context of Poisson-distributed data. The primary objective of our investigation is to ascertain whether two distinct sets of discrete data share the same Poisson rate. We delve into a comprehensive review and comparative analysis of various frequentist and Bayesian methodologies specifically designed to address this problem. Among these are the conditional test, the likelihood ratio test, and the Bayes factor. Additionally, we employ the posterior predictive p-value in our analysis, coupled with its corresponding calibration procedures. As the culmination of our investigation, we apply these diverse methodologies to test both simulated datasets and real-world data. The latter consists of the offspring distributions linked to COVID-19 cases in two disparate geographies - Hong Kong and Rwanda. This allows us to provide a practical demonstration of the methodologies' applications and their potential implications in the field of epidemiology.

A method for parameter hypothesis testing in nonparametric regression with Fourier series approach

Nonparametric regression model with the Fourier series approach was first introduced by Bilodeau in 1994. In the later years, several researchers developed a nonparametric regression model with the Fourier series approach. However, these researches are limited to parameter estimation and there is no research related to parameter hypothesis testing. Parameter hypothesis testing is a statistical method used to test the significance of the parameters. In nonparametric regression model with the Fourier series approach, parameter hypothesis testing is used to determine whether the estimated parameters have significance influence on the model or not. Therefore, the purpose of this research is for parameter hypothesis testing in the nonparametric regression model with the Fourier series approach. The method that we use for hypothesis testing is the LRT method. The LRT method is a method that compares the likelihood functions under the parameter space of the null hypothesis and the hypothesis. By using the LRT method, we obtain the form of the statistical test and its distribution as well as the rejection region of the null hypothesis. To apply the method, we use ROA data from 47 go public banks that are listed on the Indonesia stock exchange in 2020. The highlights of this research are:•The Fourier series function is assumed as a non-smooth function.•The form of the statistical test is obtained using the LRT method and is distributed as F distribution.•The estimated parameters on modelling ROA data have a significant influence on the model.

Testing for a difference in means of a single feature after clustering

For many applications, it is critical to interpret and validate groups of observations obtained via clustering. A common validation approach involves testing differences in feature means between observations in two estimated clusters. In this setting, classical hypothesis tests lead to an inflated Type I error rate. To overcome this problem, we propose a new test for the difference in means in a single feature between a pair of clusters obtained using hierarchical or k-means clustering. The test based on the proposed p-value controls the selective Type I error rate in finite samples and can be efficiently computed. We further illustrate the validity and power of our proposal in simulation and demonstrate its use on single-cell RNA-sequencing data.

Ponds as experimental arenas for studying animal movement: current research and future prospects

Animal movement is a multifaceted process that occurs for multiple reasons with powerful consequences for food web and ecosystem dynamics. New paradigms and technical innovations have recently pervaded the field, providing increasingly powerful means to deliver fine-scale movement data, attracting renewed interest. Specifically in the aquatic environment, tracking with acoustic telemetry now provides integral spatiotemporal information to follow individual movements in the wild. Yet, this technology also holds great promise for experimental studies, enhancing our ability to truly establish cause-and-effect relationships. Here, we argue that ponds with well-defined borders (i.e. "islands in a sea of land") are particularly well suited for this purpose. To support our argument, we also discuss recent experiences from studies conducted in an innovative experimental infrastructure, composed of replicated ponds equipped with modern aquatic telemetry systems that allow for unparalleled insights into the movement patterns of individual animals.

happi: a hierarchical approach to pangenomics inference

Recovering metagenome-assembled genomes (MAGs) from shotgun sequencing data is an increasingly common task in microbiome studies, as MAGs provide deeper insight into the functional potential of both culturable and non-culturable microorganisms. However, metagenome-assembled genomes vary in quality and may contain omissions and contamination. These errors present challenges for detecting genes and comparing gene enrichment across sample types. To address this, we propose happi, an approach to testing hypotheses about gene enrichment that accounts for genome quality. We illustrate the advantages of happi over existing approaches using published Saccharibacteria MAGs, Streptococcus thermophilus MAGs, and via simulation.

Rapid prototyping for quantifying belief weights of competing hypotheses about emergent diseases

Emerging diseases can have devastating consequences for wildlife and require a rapid response. A critical first step towards developing appropriate management is identifying the etiology of the disease, which can be difficult to determine, particularly early in emergence. Gathering and synthesizing existing information about potential disease causes, by leveraging expert knowledge or relevant existing studies, provides a principled approach to quickly inform decision-making and management efforts. Additionally, updating the current state of knowledge as more information becomes available over time can reduce scientific uncertainty and lead to substantial improvement in the decision-making process and the application of management actions that incorporate and adapt to newly acquired scientific understanding. Here we present a rapid prototyping method for quantifying belief weights for competing hypotheses about the etiology of disease using a combination of formal expert elicitation and Bayesian hierarchical modeling. We illustrate the application of this approach for investigating the etiology of stony coral tissue loss disease (SCTLD) and discuss the opportunities and challenges of this approach for addressing emergent diseases. Lastly, we detail how our work may apply to other pressing management or conservation problems that require quick responses. We found the rapid prototyping methods to be an efficient and rapid means to narrow down the number of potential hypotheses, synthesize current understanding, and help prioritize future studies and experiments. This approach is rapid by providing a snapshot assessment of the current state of knowledge. It can also be updated periodically (e.g., annually) to assess changes in belief weights over time as scientific understanding increases. Synthesis and applications: The rapid prototyping approaches demonstrated here can be used to combine knowledge from multiple experts and/or studies to help with fast decision-making needed for urgent conservation issues including emerging diseases and other management problems that require rapid responses. These approaches can also be used to adjust belief weights over time as studies and expert knowledge accumulate and can be a helpful tool for adapting management decisions.

Selective inference for k-means clustering

We consider the problem of testing for a difference in means between clusters of observations identified via k -means clustering. In this setting, classical hypothesis tests lead to an inflated Type I error rate. In recent work, Gao et al. (2022) considered a related problem in the context of hierarchical clustering. Unfortunately, their solution is highly-tailored to the context of hierarchical clustering, and thus cannot be applied in the setting of k -means clustering. In this paper, we propose a p-value that conditions on all of the intermediate clustering assignments in the k -means algorithm. We show that the p-value controls the selective Type I error for a test of the difference in means between a pair of clusters obtained using k -means clustering in finite samples, and can be efficiently computed. We apply our proposal on hand-written digits data and on single-cell RNA-sequencing data.

Quantifying uncertainty in spikes estimated from calcium imaging data

In recent years, a number of methods have been proposed to estimate the times at which a neuron spikes on the basis of calcium imaging data. However, quantifying the uncertainty associated with these estimated spikes remains an open problem. We consider a simple and well-studied model for calcium imaging data, which states that calcium decays exponentially in the absence of a spike, and instantaneously increases when a spike occurs. We wish to test the null hypothesis that the neuron did not spike-i.e., that there was no increase in calcium-at a particular timepoint at which a spike was estimated. In this setting, classical hypothesis tests lead to inflated Type I error, because the spike was estimated on the same data used for testing. To overcome this problem, we propose a selective inference approach. We describe an efficient algorithm to compute finite-sample $p$-values that control selective Type I error, and confidence intervals with correct selective coverage, for spikes estimated using a recent proposal from the literature. We apply our proposal in simulation and on calcium imaging data from the $\texttt{spikefinder}$ challenge.

Cumulative damage for multi-type epidemics and an application to infectious diseases

A continuous time multivariate stochastic model is proposed for assessing the damage of a multi-type epidemic cause to a population as it unfolds. The instants when cases occur and the magnitude of their injure are random. Thus, we define a cumulative damage based on counting processes and a multivariate mark process. For a large population we approximate the behavior of this damage process by its asymptotic distribution. Also, we analyze the distribution of the stopping times when the numbers of cases caused by the epidemic attain levels beyond certain thresholds. We focus on introducing some tools for statistical inference on the parameters related with the epidemic. In this regard, we present a general hypothesis test for homogeneity in epidemics and apply it to data of Covid-19 in Chile.

Equivalent statistics for a one-sample t-test

Recent insights into problems with common statistical practice in psychology have motivated scientists to consider alternatives to the traditional frequentist approach that compares p-values to a significance criterion. While these alternatives have worthwhile attributes, Francis (Behavior Research Methods, 40, 1524-1538, 2017) showed that many proposed test statistics for the situation of a two-sample t-test are based on precisely the same information in a given data set; and for a given sample size, one can convert from any statistic to the others. Here, we show that the same relationship holds for the equivalent of a one-sample t-test. We derive the relationships and provide an on-line app that performs the computations. A key conclusion of this analysis is that many types of tests are based on the same information, so the choice of which approach to use should reflect the intent of the scientist and the appropriateness of the corresponding inferential framework for that intent.

Efficiency in sequential testing: Comparing the sequential probability ratio test and the sequential Bayes factor test

In a sequential hypothesis test, the analyst checks at multiple steps during data collection whether sufficient evidence has accrued to make a decision about the tested hypotheses. As soon as sufficient information has been obtained, data collection is terminated. Here, we compare two sequential hypothesis testing procedures that have recently been proposed for use in psychological research: Sequential Probability Ratio Test (SPRT; Psychological Methods, 25(2), 206-226, 2020) and the Sequential Bayes Factor Test (SBFT; Psychological Methods, 22(2), 322-339, 2017). We show that although the two methods have different philosophical roots, they share many similarities and can even be mathematically regarded as two instances of an overarching hypothesis testing framework. We demonstrate that the two methods use the same mechanisms for evidence monitoring and error control, and that differences in efficiency between the methods depend on the exact specification of the statistical models involved, as well as on the population truth. Our simulations indicate that when deciding on a sequential design within a unified sequential testing framework, researchers need to balance the needs of test efficiency, robustness against model misspecification, and appropriate uncertainty quantification. We provide guidance for navigating these design decisions based on individual preferences and simulation-based design analyses.

Assessing dynamic covariate effects with survival data

Dynamic (or varying) covariate effects often manifest meaningful physiological mechanisms underlying chronic diseases. However, a static view of covariate effects is typically adopted by standard approaches to evaluating disease prognostic factors, which can result in depreciation of some important disease markers. To address this issue, in this work, we take the perspective of globally concerned quantile regression, and propose a flexible testing framework suited to assess either constant or dynamic covariate effects. We study the powerful Kolmogorov-Smirnov (K-S) and Cramér-Von Mises (C-V) type test statistics and develop a simple resampling procedure to tackle their complicated limit distributions. We provide rigorous theoretical results, including the limit null distributions and consistency under a general class of alternative hypotheses of the proposed tests, as well as the justifications for the presented resampling procedure. Extensive simulation studies and a real data example demonstrate the utility of the new testing procedures and their advantages over existing approaches in assessing dynamic covariate effects.

Limits of epidemic prediction using SIR models

The Susceptible-Infectious-Recovered (SIR) equations and their extensions comprise a commonly utilized set of models for understanding and predicting the course of an epidemic. In practice, it is of substantial interest to estimate the model parameters based on noisy observations early in the outbreak, well before the epidemic reaches its peak. This allows prediction of the subsequent course of the epidemic and design of appropriate interventions. However, accurately inferring SIR model parameters in such scenarios is problematic. This article provides novel, theoretical insight on this issue of practical identifiability of the SIR model. Our theory provides new understanding of the inferential limits of routinely used epidemic models and provides a valuable addition to current simulate-and-check methods. We illustrate some practical implications through application to a real-world epidemic data set.

A logical analysis of null hypothesis significance testing using popular terminology

Background

Null Hypothesis Significance Testing (NHST) has been well criticised over the years yet remains a pillar of statistical inference. Although NHST is well described in terms of statistical models, most textbooks for non-statisticians present the null and alternative hypotheses (H₀ and H_A, respectively) in terms of differences between groups such as (μ₁ = μ₂) and (μ₁ ≠ μ₂) and H_A is often stated to be the research hypothesis. Here we use propositional calculus to analyse the internal logic of NHST when couched in this popular terminology. The testable H₀ is determined by analysing the scope and limits of the P-value and the test statistic’s probability distribution curve.

Results

We propose a minimum axiom set NHST in which it is taken as axiomatic that H₀ is rejected if P-value< α. Using the common scenario of the comparison of the means of two sample groups as an example, the testable H₀ is {(μ₁ = μ₂) and [(\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{x}$$\end{document}x¯₁ ≠ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{x}$$\end{document}x¯₂) due to chance alone]}. The H₀ and H_A pair should be exhaustive to avoid false dichotomies. This entails that H_A is ¬{(μ₁ = μ₂) and [(\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{x}$$\end{document}x¯₁ ≠ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{x}$$\end{document}x¯₂) due to chance alone]}, rather than the research hypothesis (H_T). To see the relationship between H_A and H_T, H_A can be rewritten as the disjunction H_A: ({(μ₁ = μ₂) ∧ [(\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{x}$$\end{document}x¯₁ ≠ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{x}$$\end{document}x¯₂) not due to chance alone]} ∨ {(μ₁ ≠ μ₂) ∧ [\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\overline{x}$$\end{document}(x¯₁ ≠ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{x}$$\end{document}x¯₂) not due to (μ₁ ≠ μ₂) alone]} ∨ {(μ₁ ≠ μ₂) ∧ [(\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{\boldsymbol{x}}$$\end{document}x¯₁≠\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{\boldsymbol{x}}$$\end{document}x¯₂) due to (μ₁ ≠ μ₂) alone]}). This reveals that H_T (the last disjunct in bold) is just one possibility within H_A. It is only by adding premises to NHST that H_T or other conclusions can be reached.

Conclusions

Using this popular terminology for NHST, analysis shows that the definitions of H₀ and H_A differ from those found in textbooks. In this framework, achieving a statistically significant result only justifies the broad conclusion that the results are not due to chance alone, not that the research hypothesis is true. More transparency is needed concerning the premises added to NHST to rig particular conclusions such as H_T. There are also ramifications for the interpretation of Type I and II errors, as well as power, which do not specifically refer to H_T as claimed by texts.

More Powerful Selective Inference for the Graph Fused Lasso

The graph fused lasso-which includes as a special case the one-dimensional fused lasso-is widely used to reconstruct signals that are piecewise constant on a graph, meaning that nodes connected by an edge tend to have identical values. We consider testing for a difference in the means of two connected components estimated using the graph fused lasso. A naive procedure such as a z-test for a difference in means will not control the selective Type I error, since the hypothesis that we are testing is itself a function of the data. In this work, we propose a new test for this task that controls the selective Type I error, and conditions on less information than existing approaches, leading to substantially higher power. We illustrate our approach in simulation and on datasets of drug overdose death rates and teenage birth rates in the contiguous United States. Our approach yields more discoveries on both datasets. Supplementary materials for this article are available online.

Evaluation of the Use of P Values in Neurosurgical Literature: from Statistical Significance to Clinical Irrelevance

The application and interpretation of P values have caused debate for several decades, and this debate has become particularly relevant in the past few years. The P value represents the probability of seeing results as extreme or more extreme than those observed in a data analysis, were the null hypothesis and other underlying assumptions to be true. While P values are useful in pointing out where an effect may be present, they have often been misused in an attempt to oversell "statistically significant" findings. As P values rely on the spread and number of measurements, a smaller P value does not necessarily imply a larger effect size, which is better assessed via an effect estimate and confidence interval interpreted in the context of the study. The clinical relevance of a computed P value is context dependent. We investigated the current use of P values in a small sample of recent neurosurgical literature. Only a minority of manuscripts that reported statistical significance described confounder adjustment, or effect sizes. A common, incorrect assumption often observed was that statistical significance equals clinical relevance. To enable correct interpretation of clinical significance, it is crucial that authors describe the clinical implications of their findings.

Expert agreement in prior elicitation and its effects on Bayesian inference

Bayesian inference requires the specification of prior distributions that quantify the pre-data uncertainty about parameter values. One way to specify prior distributions is through prior elicitation, an interview method guiding field experts through the process of expressing their knowledge in the form of a probability distribution. However, prior distributions elicited from experts can be subject to idiosyncrasies of experts and elicitation procedures, raising the spectre of subjectivity and prejudice. Here, we investigate the effect of interpersonal variation in elicited prior distributions on the Bayes factor hypothesis test. We elicited prior distributions from six academic experts with a background in different fields of psychology and applied the elicited prior distributions as well as commonly used default priors in a re-analysis of 1710 studies in psychology. The degree to which the Bayes factors vary as a function of the different prior distributions is quantified by three measures of concordance of evidence: We assess whether the prior distributions change the Bayes factor direction, whether they cause a switch in the category of evidence strength, and how much influence they have on the value of the Bayes factor. Our results show that although the Bayes factor is sensitive to changes in the prior distribution, these changes do not necessarily affect the qualitative conclusions of a hypothesis test. We hope that these results help researchers gauge the influence of interpersonal variation in elicited prior distributions in future psychological studies. Additionally, our sensitivity analyses can be used as a template for Bayesian robustness analyses that involve prior elicitation from multiple experts.

Early termination in single-parameter model phase II clinical trial designs using decreasingly informative priors

Background

To exchange the type of subjective Bayesian prior selection for assumptions more directly related to statistical decision making in clinician studies and trials, the decreasingly informative prior (DIP) is considered. We expand standard Bayesian early termination methods in one-parameter statistical models for Phase II clinical trials to include decreasingly informative priors (DIP). These priors are designed to reduce the chance of erroneously adapting trials too early by parameterize skepticism in an amount always equal to the unobserved sample size.

Method

We show how to parameterize these priors based on effective prior sample size and provide examples for common single-parameter models, include Bernoulli, Poisson, and Gaussian distributions. We use a simulation study to search through possible values of total sample sizes and termination thresholds to find the smallest total sample size (N) under admissible designs, which we define as having at least 80% power and no greater than 5% type I error rate.

Results

For Bernoulli, Poisson, and Gaussian distributions, the DIP approach requires fewer patients when admissible designs are achieved. In situations where type I error or power are not admissible, the DIP approach yields similar power and better-controlled type I error with comparable or fewer patients than other Bayesian priors by Thall and Simon.

Conclusions

The DIP helps control type I error rates with comparable or fewer patients, especially for those instances when increased type I error rates arise from erroneous termination early in a trial.

High Dimensional Change Point Inference: Recent Developments and Extensions

Change point analysis aims to detect structural changes in a data sequence. It has always been an active research area since it was introduced in the 1950s. In modern statistical applications, however, high-throughput data with increasing dimensions are ubiquitous in fields ranging from economics, finance to genetics and engineering. For those problems, the earlier works are typically no longer applicable. As a result, the problem of testing a change point for high dimensional data sequences has been an important yet challenging task. In this paper, we first focus on models for at most one change point, and review recent state-of-art techniques for change point testing of high dimensional mean vectors and compare their theoretical properties. Based on that, we provide a survey of some extensions to general high dimensional parameters beyond mean vectors as well as strategies for testing multiple change points in high dimensions. Finally, we discuss some open problems for possible future research directions.

A likelihood ratio test for the homogeneity of between-study variance in network meta-analysis

BACKGROUND

Network meta-analysis (NMA) is a statistical method used to combine results from several clinical trials and simultaneously compare multiple treatments using direct and indirect evidence. Statistical heterogeneity is a characteristic describing the variability in the intervention effects being evaluated in the different studies in network meta-analysis. One approach to dealing with statistical heterogeneity is to perform a random effects network meta-analysis that incorporates a between-study variance into the statistical model. A common assumption in the random effects model for network meta-analysis is the homogeneity of between-study variance across all interventions. However, there are applications of NMA where the single between-study assumption is potentially incorrect and instead the model should incorporate more than one between-study variances.

METHODS

In this paper, we develop an approach to testing the homogeneity of between-study variance assumption based on a likelihood ratio test. A simulation study was conducted to assess the type I error and power of the proposed test. This method is then applied to a network meta-analysis of antibiotic treatments for Bovine respiratory disease (BRD).

RESULTS

The type I error rate was well controlled in the Monte Carlo simulation. We found statistical evidence (p value = 0.052) against the homogeneous between-study variance assumption in the network meta-analysis BRD. The point estimate and confidence interval of relative effect sizes are strongly influenced by this assumption.

CONCLUSIONS

Since homogeneous between-study variance assumption is a strong assumption, it is crucial to test the validity of this assumption before conducting a network meta-analysis. Here we propose and validate a method for testing this single between-study variance assumption which is widely used for many NMA.

Meaningful associations in the adolescent brain cognitive development study

The Adolescent Brain Cognitive Development (ABCD) Study is the largest single-cohort prospective longitudinal study of neurodevelopment and children's health in the United States. A cohort of n = 11,880 children aged 9-10 years (and their parents/guardians) were recruited across 22 sites and are being followed with in-person visits on an annual basis for at least 10 years. The study approximates the US population on several key sociodemographic variables, including sex, race, ethnicity, household income, and parental education. Data collected include assessments of health, mental health, substance use, culture and environment and neurocognition, as well as geocoded exposures, structural and functional magnetic resonance imaging (MRI), and whole-genome genotyping. Here, we describe the ABCD Study aims and design, as well as issues surrounding estimation of meaningful associations using its data, including population inferences, hypothesis testing, power and precision, control of covariates, interpretation of associations, and recommended best practices for reproducible research, analytical procedures and reporting of results.

English engineer John Smeaton's experimental method(s): Optimisation, hypothesis testing and exploratory experimentation

In this paper I provide a detailed account of eighteenth-century engineer John Smeaton's experimental methods, with the aim of bringing our understanding of his work into line with recent research in the history and philosophy of science. Starting from his use of the technique of parameter variation, I identify three distinct methodological aims in the research he carried out on waterwheels, windmills and hydraulic mortars. These aims are: optimisation, hypothesis testing and maxim generation. The main claim of this paper is that Smeaton did more than merely improve engineering methods by systematising earlier artisanal approaches, which is the classic view of Smeaton's method developed by historians of technology in the 1990s. I argue instead that his approach bridged the divide between science and technology, by integrating both hypothesis testing and exploratory experimentation. This is borne out, in particular, by the way that Smeaton emphasised the exploratory side of the work he published in the Philosophical Transactions, in contrast to his account of the construction of the Eddystone lighthouse, which was aimed at a broader, non-specialist public. I contribute to recent research on exploratory experimentation by showing - in line with other work on this topic - that exploratory experimentation is not incompatible with hypothesis testing. This new perspective on Smeaton's method will hopefully lead to further research and new insights into the relation between science and technology at the start of the Industrial Revolution.

Microbial trend analysis for common dynamic trend, group comparison, and classification in longitudinal microbiome study

BACKGROUND

The human microbiome is inherently dynamic and its dynamic nature plays a critical role in maintaining health and driving disease. With an increasing number of longitudinal microbiome studies, scientists are eager to learn the comprehensive characterization of microbial dynamics and their implications to the health and disease-related phenotypes. However, due to the challenging structure of longitudinal microbiome data, few analytic methods are available to characterize the microbial dynamics over time.

RESULTS

We propose a microbial trend analysis (MTA) framework for the high-dimensional and phylogenetically-based longitudinal microbiome data. In particular, MTA can perform three tasks: 1) capture the common microbial dynamic trends for a group of subjects at the community level and identify the dominant taxa; 2) examine whether or not the microbial overall dynamic trends are significantly different between groups; 3) classify an individual subject based on its longitudinal microbial profiling. Our extensive simulations demonstrate that the proposed MTA framework is robust and powerful in hypothesis testing, taxon identification, and subject classification. Our real data analyses further illustrate the utility of MTA through a longitudinal study in mice.

CONCLUSIONS

The proposed MTA framework is an attractive and effective tool in investigating dynamic microbial pattern from longitudinal microbiome studies.

Biasing the input: A yoked-scientist demonstration of the distorting effects of optional stopping on Bayesian inference

Prior work by Michael R. Dougherty and colleagues (Yu et al., 2014) shows that when a scientist monitors the p value during data collection and uses a critical p as the signal to stop collecting data, the resulting p is distorted due to Type I error-rate inflation. They argued similarly that the use of a critical Bayes factor (BF_(crit)) for stopping distorts the obtained Bayes factor (BF), a position that has met with controversy. The present paper clarified that when BF_(crit) is used as a stopping criterion, the sample becomes biased in that data consistent with large effects have a greater chance to be included than do other data, thus biasing the input to Bayesian inference. We report simulations of yoked pairs of scientists in which Scientist A uses BF_(crit) to optionally stop, while Scientist B, sampling from the same population, stops when A stops. Thus, optional stopping is compared not to a hypothetical in which no stopping occurs, but to a situation in which B stops for reasons unrelated to the characteristics of B's sample. The results indicated that optional stopping biased the input for Bayesian inference. We also simulated the use of effect-size stabilization as a stopping criterion and found no bias in that case.

Pandemic and its effect on professional environment on the Kingdom of Saudi Arabia

The pandemic has affected the world from many different perspectives, including environmental change. This research study aims to investigate the pandemic and its associated effect on the professional environment by measuring some of the parameters that are likely to disclose the impact of the pandemic. A structural questionnaire elicits design to capture the effect of COVID-19, where 284 respondents participated and present their views on a different statement based on the Likert scale. The factor analysis reveals five factors, which were further tested by hypothesis testing and binary logistic regression-and found factors 2, 3, and 5 to be significant in both tests.

Analysis in case-control sequencing association studies with different sequencing depths

With the advent of next-generation sequencing, investigators have access to higher quality sequencing data. However, to sequence all samples in a study using next generation sequencing can still be prohibitively expensive. One potential remedy could be to combine next generation sequencing data from cases with publicly available sequencing data for controls, but there could be a systematic difference in quality of sequenced data, such as sequencing depths, between sequenced study cases and publicly available controls. We propose a regression calibration (RC)-based method and a maximum-likelihood method for conducting an association study with such a combined sample by accounting for differential sequencing errors between cases and controls. The methods allow for adjusting for covariates, such as population stratification as confounders. Both methods control type I error and have comparable power to analysis conducted using the true genotype with sufficiently high but different sequencing depths. We show that the RC method allows for analysis using naive variance estimate (closely approximates true variance in practice) and standard software under certain circumstances. We evaluate the performance of the proposed methods using simulation studies and apply our methods to a combined data set of exome sequenced acute lung injury cases and healthy controls from the 1000 Genomes project.

Why optional stopping can be a problem for Bayesians

Recently, optional stopping has been a subject of debate in the Bayesian psychology community. Rouder (Psychonomic Bulletin & Review 21(2), 301-308, 2014) argues that optional stopping is no problem for Bayesians, and even recommends the use of optional stopping in practice, as do (Wagenmakers, Wetzels, Borsboom, van der Maas & Kievit, Perspectives on Psychological Science 7, 627-633, 2012). This article addresses the question of whether optional stopping is problematic for Bayesian methods, and specifies under which circumstances and in which sense it is and is not. By slightly varying and extending Rouder's (Psychonomic Bulletin & Review 21(2), 301-308, 2014) experiments, we illustrate that, as soon as the parameters of interest are equipped with default or pragmatic priors-which means, in most practical applications of Bayes factor hypothesis testing-resilience to optional stopping can break down. We distinguish between three types of default priors, each having their own specific issues with optional stopping, ranging from no-problem-at-all (type 0 priors) to quite severe (type II priors).

Large-Scale Hypothesis Testing for Causal Mediation Effects with Applications in Genome-wide Epigenetic Studies

In genome-wide epigenetic studies, it is of great scientific interest to assess whether the effect of an exposure on a clinical outcome is mediated through DNA methylations. However, statistical inference for causal mediation effects is challenged by the fact that one needs to test a large number of composite null hypotheses across the whole epigenome. Two popular tests, the Wald-type Sobel's test and the joint significant test using the traditional null distribution are underpowered and thus can miss important scientific discoveries. In this paper, we show that the null distribution of Sobel's test is not the standard normal distribution and the null distribution of the joint significant test is not uniform under the composite null of no mediation effect, especially in finite samples and under the singular point null case that the exposure has no effect on the mediator and the mediator has no effect on the outcome. Our results explain why these two tests are underpowered, and more importantly motivate us to develop a more powerful Divide-Aggregate Composite-null Test (DACT) for the composite null hypothesis of no mediation effect by leveraging epigenome-wide data. We adopted Efron's empirical null framework for assessing statistical significance of the DACT test. We showed analytically that the proposed DACT method had improved power, and could well control type I error rate. Our extensive simulation studies showed that, in finite samples, the DACT method properly controlled the type I error rate and outperformed Sobel's test and the joint significance test for detecting mediation effects. We applied the DACT method to the US Department of Veterans Affairs Normative Aging Study, an ongoing prospective cohort study which included men who were aged 21 to 80 years at entry. We identified multiple DNA methylation CpG sites that might mediate the effect of smoking on lung function with effect sizes ranging from -0.18 to -0.79 and false discovery rate controlled at level 0.05, including the CpG sites in the genes AHRR and F2RL3. Our sensitivity analysis found small residual correlations (less than 0.01) of the error terms between the outcome and mediator regressions, suggesting that our results are robust to unmeasured confounding factors.

Violating the normality assumption may be the lesser of two evils

When data are not normally distributed, researchers are often uncertain whether it is legitimate to use tests that assume Gaussian errors, or whether one has to either model a more specific error structure or use randomization techniques. Here we use Monte Carlo simulations to explore the pros and cons of fitting Gaussian models to non-normal data in terms of risk of type I error, power and utility for parameter estimation. We find that Gaussian models are robust to non-normality over a wide range of conditions, meaning that p values remain fairly reliable except for data with influential outliers judged at strict alpha levels. Gaussian models also performed well in terms of power across all simulated scenarios. Parameter estimates were mostly unbiased and precise except if sample sizes were small or the distribution of the predictor was highly skewed. Transformation of data before analysis is often advisable and visual inspection for outliers and heteroscedasticity is important for assessment. In strong contrast, some non-Gaussian models and randomization techniques bear a range of risks that are often insufficiently known. High rates of false-positive conclusions can arise for instance when overdispersion in count data is not controlled appropriately or when randomization procedures ignore existing non-independencies in the data. Hence, newly developed statistical methods not only bring new opportunities, but they can also pose new threats to reliability. We argue that violating the normality assumption bears risks that are limited and manageable, while several more sophisticated approaches are relatively error prone and particularly difficult to check during peer review. Scientists and reviewers who are not fully aware of the risks might benefit from preferentially trusting Gaussian mixed models in which random effects account for non-independencies in the data.

Sample size estimation for biomechanical waveforms: Current practice, recommendations and a comparison to discrete power analysis

Testing a prediction is fundamental to scientific experiments. Where biomechanical experiments involve analysis of 1-Dimensional (waveform) data, sample size estimation should consider both 1D variance and hypothesised 1D effects. This study exemplifies 1D sample size estimation using typical biomechanical signals and contrasts this with 0D (discrete) power analysis. For context, biomechanics papers from 2018 and 2019 were reviewed to characterise current practice. Sample size estimation occurred in approximately 4% of 653 papers and reporting practice was mixed. To estimate sample sizes, common biomechanical signals were sourced from the literature and 1D effects were generated artificially using the open-source power1d software. Smooth Gaussian noise was added to the modelled 1D effect to numerically estimate the sample size required. Sample sizes estimated using 1D power procedures varied according to the characteristics of the dataset, requiring only small-to-moderate sample sizes of approximately 5-40 to achieve target powers of 0.8 for reported 1D effects, but were always larger than 0D sample sizes (from N + 1 to >N + 20). The importance of a priori sample size estimation is highlighted and recommendations are provided to improve the consistency of reporting. This study should enable researchers to construct 1D biomechanical effects to address adequately powered, hypothesis-driven, predictive research questions.

Shifting the focus away from binary thinking of statistical significance and towards education for key stakeholders: revisiting the debate on whether it's time to de-emphasize or get rid of statistical significance

There has been a long-standing controversy among scientists regarding the appropriate use of P-values and statistical significance in clinical research. This debate has resurfaced through recent calls to modify the threshold of P-value required to declare significance, or to retire statistical significance entirely. In this article, we revisit the issue by discussing: i) the connection between statistical thinking and evidence-based practice; ii) some history of statistical significance and P-values; iii) some practical challenges with statistical significance or P-value thresholds in clinical research; iv) the on-going debate on what to do with statistical significance; v) suggestions to shift the focus away from binary thinking of statistical significance and towards education for key stakeholders on research essentials including statistical thinking, critical thinking, good reporting, basic clinical research concepts and methods, and more. We then conclude with remarks and illustrations of the potential deleterious public health consequences of poor methods including selective choice of analysis approach and misguided reliance on binary use of P-values to report and interpret scientific findings.

Threshold-based subgroup testing in logistic regression models in two-phase sampling designs

The effect of treatment on binary disease outcome can differ across subgroups characterized by other covariates. Testing for the existence of subgroups that are associated with heterogeneous treatment effects can provide valuable insight regarding the optimal treatment recommendation in practice. Our research in this paper is motivated by the question of whether host genetics could modify a vaccine's effect on HIV acquisition risk. To answer this question, we used data from an HIV vaccine trial with a two-phase sampling design and developed a general threshold-based model framework to test for the existence of subgroups associated with the heterogeneity in disease risks, allowing for subgroups based on multivariate covariates. We developed a testing procedure based on maximum of likelihood-ratio statistics over change planes and demonstrated its advantage over alternative methods. We further developed the testing procedure to account for bias sampling of expensive (i.e. resource-intensive to measure) covariates through the incorporation of inverse probability weighting techniques. We used the proposed method to analyze the motivating HIV vaccine trial data. Our proposed testing procedure also has broad applications in epidemiological studies for assessing heterogeneity in disease risk with respect to univariate or multivariate predictors.

Artificial cognition: How experimental psychology can help generate explainable artificial intelligence

Artificial intelligence powered by deep neural networks has reached a level of complexity where it can be difficult or impossible to express how a model makes its decisions. This black-box problem is especially concerning when the model makes decisions with consequences for human well-being. In response, an emerging field called explainable artificial intelligence (XAI) aims to increase the interpretability, fairness, and transparency of machine learning. In this paper, we describe how cognitive psychologists can make contributions to XAI. The human mind is also a black box, and cognitive psychologists have over 150 years of experience modeling it through experimentation. We ought to translate the methods and rigor of cognitive psychology to the study of artificial black boxes in the service of explainability. We provide a review of XAI for psychologists, arguing that current methods possess a blind spot that can be complemented by the experimental cognitive tradition. We also provide a framework for research in XAI, highlight exemplary cases of experimentation within XAI inspired by psychological science, and provide a tutorial on experimenting with machines. We end by noting the advantages of an experimental approach and invite other psychologists to conduct research in this exciting new field.

The best predictions in experimental biology are critical and persuasive

A powerful way to evaluate scientific explanations (hypotheses) is to test the predictions that they make. In this way, predictions serve as an important bridge between abstract hypotheses and concrete experiments. Experimental biologists, however, generally receive little guidance on how to generate quality predictions. Here, we identify two important components of good predictions - criticality and persuasiveness - which relate to the ability of a prediction (and the experiment it implies) to disprove a hypothesis or to convince a skeptic that the hypothesis has merit. Using a detailed example, we demonstrate how striving for predictions that are both critical and persuasive can speed scientific progress by leading us to more powerful experiments. Finally, we provide a quality control checklist to assist students and researchers as they navigate the hypothetico-deductive method from puzzling observations to experimental tests.

Comparing experimental conditions using modern statistics

While the applied psychology community relies on statistics to assist drawing conclusions from quantitative data, the methods being used mostly today do not reflect several of the advances in statistics that have been realized over the past decades. We show in this paper how a number of issues with how statistical analyses are presently executed and reported in the literature can be addressed by applying more modern methods. Unfortunately, such new methods are not always supported by widely available statistical packages, such as SPSS, which is why we also introduce a new software platform, called ILLMO (for Interactive Log-Likelihood MOdeling), which offers an intuitive interface to such modern statistical methods. In order to limit the complexity of the material being covered in this paper, we focus the discussion on a fairly simple, but nevertheless very frequent and important statistical task, i.e., comparing two experimental conditions.

Free-viewing as experimental system to test the Temporal Correlation Hypothesis: A case of theory-generative experimental practice

Theory-free characterizations of experimental systems miss normative and conceptual components that sometimes are crucial to understanding their historical development. In the following paper, we show that these components may be part of the intrinsic capacities of experimental systems themselves. We study a case of non-exploratory and theory-oriented research in experimental neuroscience that concerns the construction of free-viewing as an experimental system to test one particular pre-existing hypothesis, the Temporal Correlation Hypothesis (TCH), at a laboratory in Santiago de Chile, during 2002-2008. We show that the system does not take well-formulated pre-existing predictions or hypotheses to test them directly, but re-creates them and re-signifies them in terms that are not implied by the theoretical background from which they originally derived. Therefore, we conclude that there is a sui generis way in which experimental systems produce proper theoretical knowledge.

P-values - a chronic conundrum

Background

In medical research and practice, the p-value is arguably the most often used statistic and yet it is widely misconstrued as the probability of the type I error, which comes with serious consequences. This misunderstanding can greatly affect the reproducibility in research, treatment selection in medical practice, and model specification in empirical analyses. By using plain language and concrete examples, this paper is intended to elucidate the p-value confusion from its root, to explicate the difference between significance and hypothesis testing, to illuminate the consequences of the confusion, and to present a viable alternative to the conventional p-value.

Main text

The confusion with p-values has plagued the research community and medical practitioners for decades. However, efforts to clarify it have been largely futile, in part, because intuitive yet mathematically rigorous educational materials are scarce. Additionally, the lack of a practical alternative to the p-value for guarding against randomness also plays a role. The p-value confusion is rooted in the misconception of significance and hypothesis testing. Most, including many statisticians, are unaware that p-values and significance testing formed by Fisher are incomparable to the hypothesis testing paradigm created by Neyman and Pearson. And most otherwise great statistics textbooks tend to cobble the two paradigms together and make no effort to elucidate the subtle but fundamental differences between them. The p-value is a practical tool gauging the “strength of evidence” against the null hypothesis. It informs investigators that a p-value of 0.001, for example, is stronger than 0.05. However, p-values produced in significance testing are not the probabilities of type I errors as commonly misconceived. For a p-value of 0.05, the chance a treatment does not work is not 5%; rather, it is at least 28.9%.

Conclusions

A long-overdue effort to understand p-values correctly is much needed. However, in medical research and practice, just banning significance testing and accepting uncertainty are not enough. Researchers, clinicians, and patients alike need to know the probability a treatment will or will not work. Thus, the calibrated p-values (the probability that a treatment does not work) should be reported in research papers.

Privacy-Aware Distributed Hypothesis Testing

A distributed binary hypothesis testing (HT) problem involving two parties, a remote observer and a detector, is studied. The remote observer has access to a discrete memoryless source, and communicates its observations to the detector via a rate-limited noiseless channel. The detector observes another discrete memoryless source, and performs a binary hypothesis test on the joint distribution of its own observations with those of the observer. While the goal of the observer is to maximize the type II error exponent of the test for a given type I error probability constraint, it also wants to keep a private part of its observations as oblivious to the detector as possible. Considering both equivocation and average distortion under a causal disclosure assumption as possible measures of privacy, the trade-off between the communication rate from the observer to the detector, the type II error exponent, and privacy is studied. For the general HT problem, we establish single-letter inner bounds on both the rate-error exponent-equivocation and rate-error exponent-distortion trade-offs. Subsequently, single-letter characterizations for both trade-offs are obtained (i) for testing against conditional independence of the observer's observations from those of the detector, given some additional side information at the detector; and (ii) when the communication rate constraint over the channel is zero. Finally, we show by providing a counter-example where the strong converse which holds for distributed HT without a privacy constraint does not hold when a privacy constraint is imposed. This implies that in general, the rate-error exponent-equivocation and rate-error exponent-distortion trade-offs are not independent of the type I error probability constraint.

A method for reducing animal use whilst maintaining statistical power in electrophysiological recordings from rodent nerves

The stimulus evoked compound action potential, recorded from ex vivo nerve trunks such as the rodent optic and sciatic nerve, is a popular model system used to study aspects of nervous system metabolism. This includes (1) the role of glycogen in supporting axon conduction, (2) the injury mechanisms resulting from metabolic insults, and (3) to test putative benefits of clinically relevant neuroprotective strategies. We demonstrate the benefit of simultaneously recording from pairs of nerves in the same superfusion chamber compared with conventional recordings from single nerves. Experiments carried out on mouse optic and sciatic nerves demonstrate that our new recording configuration decreased the relative standard deviation from samples when compared with recordings from an equivalent number of individually recorded nerves. The new method reduces the number of animals required to produce equivalent Power compared with the existing method, where single nerves are used. Adopting this method leads to increased experimental efficiency and productivity. We demonstrate that reduced animal use and increased Power can be achieved by recording from pairs of rodent nerve trunks simultaneously.

Search engines, cognitive biases and the man-computer interaction: a theoretical framework for empirical researches about cognitive biases in online search on health-related topics

The widespread use of online search engines to answer the general public's needs for information has raised concerns about possible biases and the emerging of a 'filter bubble' in which users are isolated from attitude-discordant messages. Research is split between approaches that largely focus on the intrinsic limitations of search engines and approaches that investigate user search behavior. This work evaluates the findings and limitations of both approaches and advances a theoretical framework for empirical investigations of cognitive biases in online search activities about health-related topics. We aim to investigate the interaction between the user and the search engine as a whole. Online search activity about health-related topics is considered as a hypothesis-testing process. Two questions emerge: whether the retrieved information provided by the search engines are fit to fulfill their role as evidence, and whether the use of this information by users is cognitively and epistemologically valid and unbiased.

Correlation and association analyses in microbiome study integrating multiomics in health and disease

Correlation and association analyses are one of the most widely used statistical methods in research fields, including microbiome and integrative multiomics studies. Correlation and association have two implications: dependence and co-occurrence. Microbiome data are structured as phylogenetic tree and have several unique characteristics, including high dimensionality, compositionality, sparsity with excess zeros, and heterogeneity. These unique characteristics cause several statistical issues when analyzing microbiome data and integrating multiomics data, such as large p and small n, dependency, overdispersion, and zero-inflation. In microbiome research, on the one hand, classic correlation and association methods are still applied in real studies and used for the development of new methods; on the other hand, new methods have been developed to target statistical issues arising from unique characteristics of microbiome data. Here, we first provide a comprehensive view of classic and newly developed univariate correlation and association-based methods. We discuss the appropriateness and limitations of using classic methods and demonstrate how the newly developed methods mitigate the issues of microbiome data. Second, we emphasize that concepts of correlation and association analyses have been shifted by introducing network analysis, microbe-metabolite interactions, functional analysis, etc. Third, we introduce multivariate correlation and association-based methods, which are organized by the categories of exploratory, interpretive, and discriminatory analyses and classification methods. Fourth, we focus on the hypothesis testing of univariate and multivariate regression-based association methods, including alpha and beta diversities-based, count-based, and relative abundance (or compositional)-based association analyses. We demonstrate the characteristics and limitations of each approaches. Fifth, we introduce two specific microbiome-based methods: phylogenetic tree-based association analysis and testing for survival outcomes. Sixth, we provide an overall view of longitudinal methods in analysis of microbiome and omics data, which cover standard, static, regression-based time series methods, principal trend analysis, and newly developed univariate overdispersed and zero-inflated as well as multivariate distance/kernel-based longitudinal models. Finally, we comment on current association analysis and future direction of association analysis in microbiome and multiomics studies.

Matrix decomposition for modeling lesion development processes in multiple sclerosis

Our main goal is to study and quantify the evolution of multiple sclerosis lesions observed longitudinally over many years in multi-sequence structural magnetic resonance imaging (sMRI). To achieve that, we propose a class of functional models for capturing the temporal dynamics and spatial distribution of the voxel-specific intensity trajectories in all sMRI sequences. To accommodate the hierarchical data structure (observations nested within voxels, which are nested within lesions, which, in turn, are nested within study participants), we use structured functional principal component analysis. We propose and evaluate the finite sample properties of hypothesis tests of therapeutic intervention effects on lesion evolution while accounting for the multilevel structure of the data. Using this novel testing strategy, we found statistically significant differences in lesion evolution between treatment groups.

How to understand and teach P values: a diagnostic test framework

OBJECTIVES

The aim of the tutorial is to help educators address misconceptions about P values and provide a tool that can be used to teach a more contemporary interpretation.

STUDY DESIGN AND SETTING

A scripted tutorial using problem-based learning and a diagnostic test analogy to deconstruct the misunderstandings about P values and develop a more Bayesian approach to study interpretation.

RESULTS

A diagnostic test analogy is an effective teaching tool. Learners' understanding of Bayes' theorem in diagnostic testing can be used as a bridge to the realization that the prestudy probability of a true difference is crucial for study interpretation. The analogy has several caveats and shortcomings. The limitations of this analogy and the conceptual difficulties with the Bayesian study analyses are addressed.

CONCLUSION

P values do not provide the information many assume they do-they are not equivalent to a probability of a chance finding. This tutorial helps move learners from these incorrect notions to new insights.

General Principles of Preclinical Study Design

Preclinical studies using animals to study the potential of a therapeutic drug or strategy are important steps before translation to clinical trials. However, evidence has shown that poor quality in the design and conduct of these studies has not only impeded clinical translation but also led to significant waste of valuable research resources. It is clear that experimental biases are related to the poor quality seen with preclinical studies. In this chapter, we will focus on hypothesis testing type of preclinical studies and explain general concepts and principles in relation to the design of in vivo experiments, provide definitions of experimental biases and how to avoid them, and discuss major sources contributing to experimental biases and how to mitigate these sources. We will also explore the differences between confirmatory and exploratory studies, and discuss available guidelines on preclinical studies and how to use them. This chapter, together with relevant information in other chapters in the handbook, provides a powerful tool to enhance scientific rigour for preclinical studies without restricting creativity.

Probabilistic Species Tree Distances: Implementing the Multispecies Coalescent to Compare Species Trees Within the Same Model-Based Framework Used to Estimate Them

Despite the ubiquitous use of statistical models for phylogenomic and population genomic inferences, this model-based rigor is rarely applied to post hoc comparison of trees. In a recent study, Garba et al. derived new methods for measuring the distance between two gene trees computed as the difference in their site pattern probability distributions. Unlike traditional metrics that compare trees solely in terms of geometry, these measures consider gene trees and associated parameters as probabilistic models that can be compared using standard information theoretic approaches. Consequently, probabilistic measures of phylogenetic tree distance can be far more informative than simply comparisons of topology and/or branch lengths alone. However, in their current form, these distance measures are not suitable for the comparison of species tree models in the presence of gene tree heterogeneity. Here, we demonstrate an approach for how the theory of Garba et al. (2018), which is based on gene tree distances, can be extended naturally to the comparison of species tree models. Multispecies coalescent (MSC) models parameterize the discrete probability distribution of gene trees conditioned upon a species tree with a particular topology and set of divergence times (in coalescent units), and thus provide a framework for measuring distances between species tree models in terms of their corresponding gene tree topology probabilities. We describe the computation of probabilistic species tree distances in the context of standard MSC models, which assume complete genetic isolation postspeciation, as well as recent theoretical extensions to the MSC in the form of network-based MSC models that relax this assumption and permit hybridization among taxa. We demonstrate these metrics using simulations and empirical species tree estimates and discuss both the benefits and limitations of these approaches. We make our species tree distance approach available as an R package called pSTDistanceR, for open use by the community.

Problem formulation and phenotypic characterisation for the development of novel crops

Phenotypic characterisation provides important information about novel crops that helps their developers to make technical and commercial decisions. Phenotypic characterisation comprises two activities. Product characterisation checks that the novel crop has the qualities of a viable product-the intended traits have been introduced and work as expected, and no unintended changes have been made that will adversely affect the performance of the final product. Risk assessment evaluates whether the intended and unintended changes are likely to harm human health or the environment. Product characterisation follows the principles of problem formulation, namely that the characteristics required in the final product are defined and criteria to decide whether the novel crop will have these properties are set. The hypothesis that the novel crop meets the criteria are tested during product development. If the hypothesis is corroborated, development continues, and if the hypothesis is falsified, the product is redesigned or its development is halted. Risk assessment should follow the same principles. Criteria that indicate the crop poses unacceptable risk should be set, and the hypothesis that the crop does not possess those properties should be tested. However, risk assessment, particularly when considering unintended changes introduced by new plant breeding methods such as gene editing, often ignores these principles. Instead, phenotypic characterisation seeks to catalogue all unintended changes by profiling methods and then proceeds to work out whether any of the changes are important. This paper argues that profiling is an inefficient and ineffective method of phenotypic characterisation for risk assessment. It discusses reasons why profiling is favoured and corrects some misconceptions about problem formulation.

Computing and graphing probability values of pearson distributions: a SAS/IML macro

Background

Any empirical data can be approximated to one of Pearson distributions using the first four moments of the data (Elderton WP, Johnson NL. Systems of Frequency Curves. 1969; Pearson K. Philos Trans R Soc Lond Ser A. 186:343–414 1895; Solomon H, Stephens MA. J Am Stat Assoc. 73(361):153–60 1978). Thus, Pearson distributions made statistical analysis possible for data with unknown distributions. There are both extant, old-fashioned in-print tables (Pearson ES, Hartley HO. Biometrika Tables for Statisticians, vol. II. 1972) and contemporary computer programs (Amos DE, Daniel SL. Tables of percentage points of standardized pearson distributions. 1971; Bouver H, Bargmann RE. Tables of the standardized percentage points of the pearson system of curves in terms of β₁ and β₂. 1974; Bowman KO, Shenton LR. Biometrika. 66(1):147–51 1979; Davis CS, Stephens MA. Appl Stat. 32(3):322–7 1983; Pan W. J Stat Softw. 31(Code Snippet 2):1–6 2009) available for obtaining percentage points of Pearson distributions corresponding to certain pre-specified percentages (or probability values; e.g., 1.0%, 2.5%, 5.0%, etc.), but they are little useful in statistical analysis because we have to rely on unwieldy second difference interpolation to calculate a probability value of a Pearson distribution corresponding to a given percentage point, such as an observed test statistic in hypothesis testing.

Results

The present study develops a SAS/IML macro program to identify the appropriate type of Pearson distribution based on either input of dataset or the values of four moments and then compute and graph probability values of Pearson distributions for any given percentage points.

Conclusions

The SAS macro program returns accurate approximations to Pearson distributions and can efficiently facilitate researchers to conduct statistical analysis on data with unknown distributions.

Expression Quantitative Trait Loci Analysis in Multiple Tissues

Expression quantitative trait loci (eQTL) analysis identifies genetic variants that regulate the expression level of a gene. The genetic regulation may persist or vary in different tissues. When data are available on multiple tissues, it is often desired to borrow information across tissues and conduct an integrative analysis. Here we describe a multi-tissue eQTL analysis procedure, which improves the identification of different types of eQTL and facilitates the assessment of tissue specificity.