How to Use and Report on p-values

The use of the p-value in quantitative research, particularly its threshold of "P < 0.05" for determining "statistical significance," has long been a cornerstone of statistical analysis in research. However, this standard has been increasingly scrutinized for its potential to mislead findings, especially when the practical significance, the number of comparisons, or the suitability of statistical tests are not properly considered. In response to controversy around use of p-values, the American Statistical Association published a statement in 2016 that challenged the research community to abandon the term "statistically significant". This stance has been echoed by leading scientific journals to urge a significant reduction or complete elimination in the reliance on p-values when reporting results. To provide guidance to researchers in health professions education, this paper provides a succinct overview of the ongoing debate regarding the use of p-values and the definition of p-values. It reflects on the controversy by highlighting the common pitfalls associated with p-value interpretation and usage, such as misinterpretation, overemphasis, and false dichotomization between "significant" and "non-significant" results. This paper also outlines specific recommendations for the effective use of p-values in statistical reporting including the importance of reporting effect sizes, confidence intervals, the null hypothesis, and conducting sensitivity analyses for appropriate interpretation. These considerations aim to guide researchers toward a more nuanced and informative use of p-values.

A systematic review and meta-analysis uncovering the relationship between alcohol consumption and sickness absence. When type of design, data, and sickness absence make a difference

AIM

Earlier research has revealed a strong relationship between alcohol use and sickness absence. The aim of this review was to explore and uncover this relationship by looking at differences in type of design (cross-sectional vs. longitudinal), type of data (self-reported vs. registered data), and type of sickness absence (long-term vs. short term).

METHOD

Six databases were searched through June 2020. Observational and experimental studies from 1980 to 2020, in English or Scandinavian languages reporting the results of the association between alcohol consumption and sickness absence among working population were included. Quality assessment, and statistical analysis focusing on differences in the likelihood of sickness absence on subgroup levels were performed on each association, not on each study. Differences in the likelihood of sickness absence were analyzed by means of meta-analysis. PROSPERO registration number: CRD42018112078.

RESULTS

Fifty-nine studies (58% longitudinal) including 439,209 employees (min. 43, max. 77,746) from 15 countries were included. Most associations indicating positive and statistically significant results were based on longitudinal data (70%) and confirmed the strong/causal relationship between alcohol use and sickness absence. The meta-analysis included eight studies (ten samples). The increased risk for sickness absence was likely to be found in cross-sectional studies (OR: 8.28, 95% CI: 6.33-10.81), studies using self-reported absence data (OR: 5.16, 95% CI: 3.16-8.45), and those reporting short-term sickness absence (OR: 4.84, 95% CI: 2.73-8.60).

CONCLUSION

This review supports, but also challenges earlier evidence on the association between alcohol use and sickness absence. Certain types of design, data, and types of sickness absence may produce large effects. Hence, to investigate the actual association between alcohol and sickness absence, research should produce and review longitudinal designed studies using registry data and do subgroup analyses that cover and explain variability of this association.

Hospital Characteristics and Early Enrollment Trends in the American College of Cardiology Voluntary Public Reporting Program

IMPORTANCE

Limited data exist regarding the characteristics of hospitals that do and do not participate in voluntary public reporting programs.

OBJECTIVE

To describe hospital characteristics and trends associated with early participation in the American College of Cardiology (ACC) voluntary reporting program for cardiac catheterization-percutaneous coronary intervention (CathPCI) and implantable cardioverter-defibrillator (ICD) registries.

DESIGN, SETTING, AND PARTICIPANTS

This cross-sectional study analyzed enrollment trends and characteristics of hospitals that did and did not participate in the ACC voluntary public reporting program. All hospitals reporting procedure data to the National Cardiovascular Data Registry (NCDR) CathPCI or ICD registries that were eligible for the public reporting program from July 2014 (ie, program launch date) to May 2017 were included. Stepwise logistic regression was used to identify hospital characteristics associated with voluntary participation. Enrollment trends were evaluated considering the date US News & World Report (USNWR) announced that it would credit participating hospitals. Data analysis was performed from March 2017 to January 2018.

MAIN OUTCOMES AND MEASURES

Hospital characteristics and participation in the public reporting program.

RESULTS

By May 2017, 561 of 1747 eligible hospitals (32.1%) had opted to participate in the program. Enrollment increased from 240 to 376 hospitals (56.7%) 1 month after the USNWR announcement that program participation would be considered as a component of national hospital rankings. Compared with hospitals that did not enroll, program participants had increased median (IQR) procedural volumes for PCI (481 [280-764] procedures vs 332 [186-569] procedures; P < .001) and ICD (114 [56-220] procedures vs 62 [25-124] procedures; P < .001). Compared with nonparticipating hospitals, an increased mean (SD) proportion of participating hospitals adhered to composite discharge medications after PCI (0.96 [0.03] vs 0.92 [0.07]; P < .001) and ICD (0.88 [0.10] vs 0.81 [0.12]; P < .001). Hospital factors associated with enrollment included participation in 5 or more NCDR registries (odds ratio [OR],1.98; 95% CI, 1.24-3.19; P = .005), membership in a larger hospital system (ie, 3-20 hospitals vs ≤2 hospitals in the system: OR, 2.29; 95% CI, 1.65-3.17; P = .001), participation in an NCDR pilot public reporting program of PCI 30-day readmissions (OR, 2.93; 95% CI, 2.19-3.91; P < .001), university affiliation (vs government affiliation: OR, 3.85, 95% CI, 1.03-14.29; P = .045; vs private affiliation: OR, 2.22; 95% CI, 1.35-3.57; P < .001), Midwest location (vs South: OR, 1.47; 95% CI, 1.06-2.08; P = .02), and increased comprehensive quality ranking (4 vs 1-2 performance stars in CathPCI: OR, 8.08; 95% CI, 5.07-12.87; P < .001; 4 vs 1 performance star in ICD: OR, 2.26; 95% CI, 1.48-3.44; P < .001) (C statistic = 0.829).

CONCLUSIONS AND RELEVANCE

This study found that one-third of eligible hospitals participated in the ACC voluntary public reporting program and that enrollment increased after the announcement that program participation would be considered by USNWR for hospital rankings. Several hospital characteristics, experience with public reporting, and quality of care were associated with increased odds of participation.

Reporting bias in clinical trials: Progress toward transparency and next steps

Mayookha Mitra-Majumdar and Aaron Kesselheim reflect on steps taken to combat reporting bias in clinical trials over the last two decades.

Guidelines for Designing and Evaluating Feasibility Pilot Studies

BACKGROUND

Pilot studies test the feasibility of methods and procedures to be used in larger-scale studies. Although numerous articles describe guidelines for the conduct of pilot studies, few have included specific feasibility indicators or strategies for evaluating multiple aspects of feasibility. In addition, using pilot studies to estimate effect sizes to plan sample sizes for subsequent randomized controlled trials has been challenged; however, there has been little consensus on alternative strategies.

METHODS

In Section 1, specific indicators (recruitment, retention, intervention fidelity, acceptability, adherence, and engagement) are presented for feasibility assessment of data collection methods and intervention implementation. Section 1 also highlights the importance of examining feasibility when adapting an intervention tested in mainstream populations to a new more diverse group. In Section 2, statistical and design issues are presented, including sample sizes for pilot studies, estimates of minimally important differences, design effects, confidence intervals (CI) and nonparametric statistics. An in-depth treatment of the limits of effect size estimation as well as process variables is presented. Tables showing CI around parameters are provided. With small samples, effect size, completion and adherence rate estimates will have large CI.

CONCLUSION

This commentary offers examples of indicators for evaluating feasibility, and of the limits of effect size estimation in pilot studies. As demonstrated, most pilot studies should not be used to estimate effect sizes, provide power calculations for statistical tests or perform exploratory analyses of efficacy. It is hoped that these guidelines will be useful to those planning pilot/feasibility studies before a larger-scale study.

Statistical methods for comparing test positivity rates between countries: Which method should be used and why?

The test positivity (TP) rate has emerged as an important metric for gauging the illness burden due to COVID-19. Given the importance of COVID-19 TP rates for understanding COVID-related morbidity, researchers and clinicians have become increasingly interested in comparing TP rates across countries. The statistical methods for performing such comparisons fall into two general categories: frequentist tests and Bayesian methods. Using data from Our World in Data (ourworldindata.org), we performed comparisons for two prototypical yet disparate pairs of countries: Bolivia versus the United States (large vs. small-to-moderate TP rates), and South Korea vs. Uruguay (two very small TP rates of similar magnitude). Three different statistical procedures were used: two frequentist tests (an asymptotic z-test and the 'N-1' chi-square test), and a Bayesian method for comparing two proportions (TP rates are proportions). Results indicated that for the case of large vs. small-to-moderate TP rates (Bolivia versus the United States), the frequentist and Bayesian approaches both indicated that the two rates were substantially different. When the TP rates were very small and of similar magnitude (values of 0.009 and 0.007 for South Korea and Uruguay, respectively), the frequentist tests indicated a highly significant contrast, despite the apparent trivial amount by which the two rates differ. The Bayesian method, in comparison, suggested that the TP rates were practically equivalent-a finding that seems more consistent with the observed data. When TP rates are highly similar in magnitude, frequentist tests can lead to erroneous interpretations. A Bayesian approach, on the other hand, can help ensure more accurate inferences and thereby avoid potential decision errors that could lead to costly public health and policy-related consequences.

Conduct and reporting of formula milk trials: systematic review

OBJECTIVE

To systematically review the conduct and reporting of formula trials.

DESIGN

Systematic review.

DATA SOURCES

Medline, Embase, and Cochrane Central Register of Controlled Trials (CENTRAL) were searched from 1 January 2006 to 31 December 2020.

REVIEW METHODS

Intervention trials comparing at least two formula products in children less than three years of age were included, but not trials of human breast milk or fortifiers of breast milk. Data were extracted in duplicate and primary outcome data were synthesised for meta-analysis with a random effects model weighted by the inverse variance method. Risk of bias was evaluated with Cochrane risk of bias version 2.0, and risk of undermining breastfeeding was evaluated according to published consensus guidance. Primary outcomes of the trials included in the systematic review were identified from clinical trial registries, protocols, or trial publications.

RESULTS

22 201 titles were screened and 307 trials were identified that were published between 2006 and 2020, of which 73 (24%) trials in 13 197 children were prospectively registered. Another 111 unpublished but registered trials in 17 411 children were identified. Detailed analysis was undertaken for 125 trials (23 757 children) published since 2015. Seventeen (14%) of these recently published trials were conducted independently of formula companies, 26 (21%) were prospectively registered with a clear aim and primary outcome, and authors or sponsors shared prospective protocols for 11 (9%) trials. Risk of bias was low in five (4%) and high in 100 (80%) recently published trials, mainly because of inappropriate exclusions from analysis and selective reporting. For 68 recently published superiority trials, a pooled standardised mean difference of 0.51 (range -0.43 to 3.29) was calculated with an asymmetrical funnel plot (Egger's test P<0.001), which reduced to 0.19 after correction for asymmetry. Primary outcomes were reported by authors as favourable in 86 (69%) trials, and 115 (92%) abstract conclusions were favourable. One of 38 (3%) trials in partially breastfed infants reported adequate support for breastfeeding and 14 of 87 (16%) trials in non-breastfed infants confirmed the decision not to breastfeed was firmly established before enrolment in the trial.

CONCLUSIONS

The results show that formula trials lack independence or transparency, and published outcomes are biased by selective reporting.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO 2018 CRD42018091928.

Cluster randomized trials of individual-level interventions were at high risk of bias

OBJECTIVES

To describe the prevalence of risks of bias in cluster-randomized trials of individual-level interventions, according to the Cochrane Risk of Bias tool.

STUDY DESIGN AND SETTING

Review undertaken in duplicate of a random sample of 40 primary reports of cluster-randomized trials of individual-level interventions.

RESULTS

The most common reported reasons for adopting cluster randomization were the need to avoid contamination (17, 42.5%) and practical considerations (14, 35%). Of the 40 trials all but one was assessed as being at risk of bias. A majority (27, 67.5%) were assessed as at risk due to the timing of identification and recruitment of participants; many (21, 52.5%) due to an apparent lack of adequate allocation concealment; and many due to selectively reported results (22, 55%), arising from a mixture of reasons including lack of documentation of primary outcome. Other risks mostly occurred infrequently.

CONCLUSION

Many cluster-randomized trials evaluating individual-level interventions appear to be at risk of bias, mostly due to identification and recruitment biases. We recommend that investigators carefully consider the need for cluster randomization; follow recommended procedures to mitigate risks of identification and recruitment bias; and adhere to good reporting practices including clear documentation of primary outcome and allocation concealment methods.

Bayesian Analysis Reporting Guidelines

Previous surveys of the literature have shown that reports of statistical analyses often lack important information, causing lack of transparency and failure of reproducibility. Editors and authors agree that guidelines for reporting should be encouraged. This Review presents a set of Bayesian analysis reporting guidelines (BARG). The BARG encompass the features of previous guidelines, while including many additional details for contemporary Bayesian analyses, with explanations. An extensive example of applying the BARG is presented. The BARG should be useful to researchers, authors, reviewers, editors, educators and students. Utilization, endorsement and promotion of the BARG may improve the quality, transparency and reproducibility of Bayesian analyses.

A Novel Imputation Approach for Sharing Protected Public Health Data

Objectives. To develop an imputation method to produce estimates for suppressed values within a shared government administrative data set to facilitate accurate data sharing and statistical and spatial analyses. Methods. We developed an imputation approach that incorporated known features of suppressed Massachusetts surveillance data from 2011 to 2017 to predict missing values more precisely. Our methods for 35 de-identified opioid prescription data sets combined modified previous or next substitution followed by mean imputation and a count adjustment to estimate suppressed values before sharing. We modeled 4 methods and compared the results to baseline mean imputation. Results. We assessed performance by comparing root mean squared error (RMSE), mean absolute error (MAE), and proportional variance between imputed and suppressed values. Our method outperformed mean imputation; we retained 46% of the suppressed value's proportional variance with better precision (22% lower RMSE and 26% lower MAE) than simple mean imputation. Conclusions. Our easy-to-implement imputation technique largely overcomes the adverse effects of low count value suppression with superior results to simple mean imputation. This novel method is generalizable to researchers sharing protected public health surveillance data. (Am J Public Health. 2021; 111(10):1830-1838. https://doi.org/10.2105/AJPH.2021.306432).

Association Between Lack of Blinding and Mortality Results in Critical Care Randomized Controlled Trials: A Meta-Epidemiological Study

OBJECTIVES

To investigate whether intervention effect estimates for mortality differ between blinded and nonblinded randomized controlled trials conducted in critical care. We used a meta-epidemiological approach, comparing effect estimates between blinded and nonblinded randomized controlled trials for the same research question.

DATA SOURCES

Systematic reviews and meta-analyses of randomized controlled trials evaluating a therapeutic intervention on mortality in critical care, published between January 2009 and March 2019 in high impact factor general medical or critical care journals and by Cochrane.

DATA EXTRACTION

For each randomized controlled trial included in eligible meta-analyses, we evaluated whether the trial was blinded (i.e., double-blinded and/or reporting adequate methods) or not (i.e., open-label, single-blinded, or unclear). We collected risk of bias evaluated by the review authors and extracted trial results.

DATA SYNTHESIS

Within each meta-analysis, we compared intervention effect estimates between blinded and nonblinded randomized controlled trials by using a ratio of odds ratio (< 1 indicates larger estimates in nonblinded than blinded randomized controlled trials). We then combined ratio of odds ratios across meta-analyses to obtain the average relative difference between nonblinded and blinded trials. Among 467 randomized controlled trials included in 36 meta-analyses, 267 (57%) were considered blinded and 200 (43%) nonblinded. Intervention effect estimates were statistically significantly larger in nonblinded than blinded trials (combined ratio of odds ratio, 0.91; 95% CI, 0.84-0.99). We found no heterogeneity across meta-analyses (p = 0.72; I2 = 0%; τ2 = 0). Sensitivity analyses adjusting the main analysis on risk of bias items yielded consistent results.

CONCLUSIONS

Intervention effect estimates of mortality were slightly larger in nonblinded than blinded randomized controlled trials conducted in critical care, but confounding cannot be excluded. Blinding of both patients and personnel is important to consider when possible in critical care trials, even when evaluating mortality.

Characteristics and Research Waste Among Randomized Clinical Trials in Gastric Cancer

IMPORTANCE

The results of numerous large randomized clinical trials (RCTs) have changed clinical practice in gastric cancer (GC). However, research waste (ie, unpublished data, inadequate reporting, or avoidable design limitations) is still a major challenge for evidence-based medicine.

OBJECTIVES

To determine the characteristics of GC RCTs in the past 20 years and the presence of research waste and to explore potential targets for improvement.

DESIGN, SETTING, AND PARTICIPANTS

In this cross-sectional study of GC RCTs, ClinicalTrials.gov was searched for phase 3 or 4 RCTs registered from January 2000 to December 2019 using the keyword gastric cancer. Independent investigators undertook assessments and resolved discrepancies via consensus. Data were analyzed from August through December 2020.

MAIN OUTCOMES AND MEASURES

The primary outcomes were descriptions of the characteristics of GC RCTs and the proportion of studies with signs of research waste. Research waste was defined as unpublished data, inadequate reporting, or avoidable design limitations. Publication status was determined by searching PubMed and Scopus databases. The adequacy of reporting was evaluated using the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline checklist. Avoidable design limitations were determined based on existing bias or lack of cited systematic literature reviews. In the analyses of research waste, 125 RCTs that ended after June 2016 without publication were excluded.

RESULTS

A total of 262 GC RCTs were included. The number of RCTs increased from 25 trials in 2000 to 2004 to 97 trials in 2015 to 2019, with a greater increase among RCTs of targeted therapy or immunotherapy, which increased from 0 trials in 2000 to 2004 to 36 trials in 2015 to 2019. The proportion of RCTs that were multicenter was higher in non-Asian regions than in Asian regions (50 of 71 RCTs [70.4%] vs 96 of 191 RCTs [50.3%]; P = .004). The analysis of research waste included 137 RCTs, of which 81 (59.1%) were published. Among published RCTs, 65 (80.2%) were judged to be adequately reported and 63 (77.8%) had avoidable design defects. Additionally, 119 RCTs (86.9%) had 1 or more features of research waste. Study settings that included blinding (odds ratio [OR], 0.56; 95% CI, 0.33-0.93; P = .03), a greater number of participants (ie, ≥200 participants; OR, 0.07; 95% CI, 0.01-0.51; P = .01), and external funding support (OR, 0.22; 95% CI, 0.08-0.60; P = .004) were associated with lower odds of research waste. Additionally, 35 RCTs (49.3%) were referenced in guidelines, and 18 RCTs (22.2%) had their prospective data reused.

CONCLUSIONS AND RELEVANCE

To our knowledge, this study is the first to describe the characteristics of GC RCTs in the past 20 years, and it found a research waste burden, which may provide evidence for the development of rational RCTs and reduction of waste in the future.

A systematic review and narrative synthesis of the research provisions under the Mental Capacity Act (2005) in England and Wales: Recruitment of adults with capacity and communication difficulties

BACKGROUND

The Mental Capacity Act (MCA, 2005) and its accompanying Code of Practice (2007), govern research participation for adults with capacity and communication difficulties in England and Wales. We conducted a systematic review and narrative synthesis to investigate the application of these provisions from 2007 to 2019.

METHODS AND FINDINGS

We included studies with mental capacity in their criteria, involving participants aged 16 years and above, with capacity-affecting conditions and conducted in England and Wales after the implementation of the MCA. Clinical trials of medicines were excluded. We searched seven databases: Academic Search Complete, ASSIA, MEDLINE, CINAHL, PsycArticles, PsycINFO and Science Direct. We used narrative synthesis to report our results. Our review follows Preferred Reporting Items for Systematic Reviews and is registered on PROSPERO, CRD42020195652. 28 studies of various research designs met our eligibility criteria: 14 (50.0%) were quantitative, 12 (42.9%) qualitative and 2 (7.1%) mixed methods. Included participants were adults with intellectual disabilities (n = 12), dementia (n = 9), mental health disorders (n = 2), autism (n = 3) and aphasia after stroke (n = 2). We found no studies involving adults with acquired brain injury. Diverse strategies were used in the recruitment of adults with capacity and communication difficulties with seven studies excluding individuals deemed to lack capacity.

CONCLUSIONS

We found relatively few studies including adults with capacity and communication difficulties with existing regulations interpreted variably. Limited use of consultees and exclusions on the basis of capacity and communication difficulties indicate that this group continue to be under-represented in research. If health and social interventions are to be effective for this population, they need to be included in primary research. The use of strategic adaptations and accommodations during the recruitment process, may serve to support their inclusion.

A Bayesian analysis of mortality outcomes in multicentre clinical trials in critical care

BACKGROUND

Multicentre RCTs are widely used by critical care researchers to answer important clinical questions. However, few trials evaluating mortality outcomes report statistically significant results. We hypothesised that the low proportion of trials reporting statistically significant differences for mortality outcomes is plausibly explained by lower-than-expected effect sizes combined with a low proportion of participants who could realistically benefit from studied interventions.

METHODS

We reviewed multicentre trials in critical care published over a 10-yr period in the New England Journal of Medicine, the Journal of the American Medical Association, and the Lancet. To test our hypothesis, we analysed the results using a Bayesian model to investigate the relationship between the proportion of effective interventions and the proportion of statistically significant results for prior distributions of effect size and trial participant susceptibility.

RESULTS

Five of 54 trials (9.3%) reported a significant difference in mortality between the control and the intervention groups. The median expected and observed differences in absolute mortality were 8.0% and 2.0%, respectively. Our modelling shows that, across trials, a lower-than-expected effect size combined with a low proportion of potentially susceptible participants is consistent with the observed proportion of trials reporting significant differences even when most interventions are effective.

CONCLUSIONS

When designing clinical trials, researchers most likely overestimate true population effect sizes for critical care interventions. Bayesian modelling demonstrates that that it is not necessarily the case that most studied interventions lack efficacy. In fact, it is plausible that many studied interventions have clinically important effects that are missed.

When research is me-search: How researchers' motivation to pursue a topic affects laypeople's trust in science

Research is often fueled by researchers' scientific, but also their personal interests: Sometimes, researchers decide to pursue a specific research question because the answer to that question is idiosyncratically relevant for themselves: Such "me-search" may not only affect the quality of research, but also how it is perceived by the general public. In two studies (N = 621), we investigate the circumstances under which learning about a researcher's "me-search" increases or decreases laypeople's ascriptions of trustworthiness and credibility to the respective researcher. Results suggest that participants' own preexisting attitudes towards the research topic moderate the effects of "me-search" substantially: When participants hold favorable attitudes towards the research topic (i.e., LGBTQ or veganism), "me-searchers" were perceived as more trustworthy and their research was perceived as more credible. This pattern was reversed when participants held unfavorable attitudes towards the research topic. Study 2 furthermore shows that trustworthiness and credibility perceptions generalize to evaluations of the entire field of research. Implications for future research and practice are discussed.

Pilot and feasibility studies for pragmatic trials have unique considerations and areas of uncertainty

BACKGROUND AND OBJECTIVE

Feasibility studies are increasingly being used to support the development of, and investigate uncertainties around, future large-scale trials. The future trial can be designed with either a pragmatic or explanatory mindset. Whereas pragmatic trials aim to inform the choice between different care options and thus, are designed to resemble conditions outside of a clinical trial environment, explanatory trials examine the benefit of a treatment under more controlled conditions. There is existing guidance for designing feasibility studies, but none that explicitly considers the goals of pragmatic designs. We aimed to identify unique areas of uncertainty that are relevant to planning a pragmatic trial.

RESULTS

We identified ten relevant domains, partly based on the pragmatic-explanatory continuum indicator summary-2 (PRECIS-2) framework, and describe potential questions of uncertainty within each: intervention development, research ethics, participant identification and eligibility, recruitment of individuals, setting, organization, flexibility of delivery, flexibility of adherence, follow-up, and importance of primary outcome to patients and decision-makers. We present examples to illustrate how uncertainty in these domains might be addressed within a feasibility study.

CONCLUSION

Researchers planning a feasibility study in advance of a pragmatic trial should consider feasibility objectives specifically relevant to areas of uncertainty for pragmatic trials.

Design of a Situational Judgment Test for Preclinical Interprofessional Collaboration

PROBLEM

There is an unmet need for economically feasible, valid, reliable, and contextually relevant assessments of interprofessional collaborative knowledge and skills, particularly at the early stages of health professions education. This study sought to develop and gather content and internal structure validity for an Interprofessional Situational Judgement Test (IPSJT), a tool for the measurement of students' interprofessional collaborative intentions during the early stages of their professional development.

APPROACH

After engaging in an item development and refinement process (January-June 2018), an 18-question IPSJT was administered to 953 first-year students enrolled in 10 health professions degree programs at the University of Florida Health Science Center in October 2018. The IPSJT's performance was evaluated using item-level analyses, item difficulty, test-retest reliability, and exploratory factor analysis.

OUTCOMES

Seven hundred thirty-seven (77.3%) students consented to the use of their data. Student IPSJT scores ranged from 0 to 69, averaging 42.68 (standard deviation = 12.28), with some statistically significant differences in student performance by health professions degree program. IPSJT item difficulties ranged from 0.13 to 0.92. Once one item with poor properties was excluded from analysis, the IPSJT demonstrated an overall reliability of 0.62. Students were more successful at identifying the least effective than the most effective responses. Test-retest reliability provided evidence of consistency (r = 0.50, P < .001) and similar item difficulty across administrations. An exploratory factor analysis indicated a 3-factor model with multiple cross-factor loadings.

NEXT STEPS

This work represents the first step toward the development of a valid, reliable IPSJT for early learners. The emergent 3-factor model provides evidence that multiple competencies can be assessed in early learners via this tool. Additional research is necessary to build a more robust question bank, explore different scoring and response methods, and gather additional sources of validity evidence, including relations to other variables.

Fetal alcohol spectrum disorders: an overview of current evidence and activities in the UK

Estimates for the UK suggest that alcohol consumption during pregnancy and prevalence of fetal alcohol spectrum disorder (FASD)-the most common neurodevelopmental condition-are high. Considering the significant health and social impacts of FASD, there is a public health imperative to prioritise prevention, interventions and support. In this article, we outline the current state of play regarding FASD knowledge and research in the UK, which is characterised by a lack of evidence, a lack of dedicated funding and services, and consequently little policy formulation and strategic direction. We highlight progress made to date, as well as current knowledge and service gaps to propose a way forward for UK research.

External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb

INTRODUCTION

Sample size "rules-of-thumb" for external validation of clinical prediction models suggest at least 100 events and 100 non-events. Such blanket guidance is imprecise, and not specific to the model or validation setting. We investigate factors affecting precision of model performance estimates upon external validation, and propose a more tailored sample size approach.

METHODS

Simulation of logistic regression prediction models to investigate factors associated with precision of performance estimates. Then, explanation and illustration of a simulation-based approach to calculate the minimum sample size required to precisely estimate a model's calibration, discrimination and clinical utility.

RESULTS

Precision is affected by the model's linear predictor (LP) distribution, in addition to number of events and total sample size. Sample sizes of 100 (or even 200) events and non-events can give imprecise estimates, especially for calibration. The simulation-based calculation accounts for the LP distribution and (mis)calibration in the validation sample. Application identifies 2430 required participants (531 events) for external validation of a deep vein thrombosis diagnostic model.

CONCLUSION

Where researchers can anticipate the distribution of the model's LP (eg, based on development sample, or a pilot study), a simulation-based approach for calculating sample size for external validation offers more flexibility and reliability than rules-of-thumb.

Machine Scoring of Medical Students' Written Clinical Reasoning: Initial Validity Evidence

PURPOSE

Developing medical students' clinical reasoning requires a structured longitudinal curriculum with frequent targeted assessment and feedback. Performance-based assessments, which have the strongest validity evidence, are currently not feasible for this purpose because they are time-intensive to score. This study explored the potential of using machine learning technologies to score one such assessment-the diagnostic justification essay.

METHOD

From May to September 2018, machine scoring algorithms were trained to score a sample of 700 diagnostic justification essays written by 414 third-year medical students from the Southern Illinois University School of Medicine classes of 2012-2017. The algorithms applied semantically based natural language processing metrics (e.g., coherence, readability) to assess essay quality on 4 criteria (differential diagnosis, recognition and use of findings, workup, and thought process); the scores for these criteria were summed to create overall scores. Three sources of validity evidence (response process, internal structure, and association with other variables) were examined.

RESULTS

Machine scores correlated more strongly with faculty ratings than faculty ratings did with each other (machine: .28-.53, faculty: .13-.33) and were less case-specific. Machine scores and faculty ratings were similarly correlated with medical knowledge, clinical cognition, and prior diagnostic justification. Machine scores were more strongly associated with clinical communication than were faculty ratings (.43 vs .31).

CONCLUSIONS

Machine learning technologies may be useful for assessing medical students' long-form written clinical reasoning. Semantically based machine scoring may capture the communicative aspects of clinical reasoning better than faculty ratings, offering the potential for automated assessment that generalizes to the workplace. These results underscore the potential of machine scoring to capture an aspect of clinical reasoning performance that is difficult to assess with traditional analytic scoring methods. Additional research should investigate machine scoring generalizability and examine its acceptability to trainees and educators.

Large-scale validation of the prediction model risk of bias assessment Tool (PROBAST) using a short form: high risk of bias models show poorer discrimination

OBJECTIVE

To assess whether the Prediction model Risk Of Bias ASsessment Tool (PROBAST) and a shorter version of this tool can identify clinical prediction models (CPMs) that perform poorly at external validation.

STUDY DESIGN AND SETTING

We evaluated risk of bias (ROB) on 102 CPMs from the Tufts CPM Registry, comparing PROBAST to a short form consisting of six PROBAST items anticipated to best identify high ROB. We then applied the short form to all CPMs in the Registry with at least 1 validation (n=556) and assessed the change in discrimination (dAUC) in external validation cohorts (n=1,147).

RESULTS

PROBAST classified 98/102 CPMS as high ROB. The short form identified 96 of these 98 as high ROB (98% sensitivity), with perfect specificity. In the full CPM registry, 527 of 556 CPMs (95%) were classified as high ROB, 20 (3.6%) low ROB, and 9 (1.6%) unclear ROB. Only one model with unclear ROB was reclassified to high ROB after full PROBAST assessment of all low and unclear ROB models. Median change in discrimination was significantly smaller in low ROB models (dAUC -0.9%, IQR -6.2-4.2%) compared to high ROB models (dAUC -11.7%, IQR -33.3-2.6%; P<0.001).

CONCLUSION

High ROB is pervasive among published CPMs. It is associated with poor discriminative performance at validation, supporting the application of PROBAST or a shorter version in CPM reviews.

Characteristics of Clinical Studies Used for US Food and Drug Administration Supplemental Indication Approvals of Drugs and Biologics, 2017 to 2019

IMPORTANCE

After US Food and Drug Administration (FDA) approval of a new drug, sponsors can submit additional clinical data to obtain supplemental approval for use for new indications.

OBJECTIVE

To characterize pivotal trials supporting recent supplemental new indication approvals of drugs and biologics by the FDA and to compare them with pivotal trials that supported these therapeutics' original indication approvals.

DESIGN, SETTING, AND PARTICIPANTS

This is a cross-sectional study characterizing pivotal trials supporting supplemental indication approvals by the FDA between 2017 and 2019 and pivotal trials that supported these therapeutics' original indication approvals. Data analysis was performed from August to October 2020.

MAIN OUTCOMES AND MEASURES

Number and design of pivotal trials supporting both supplemental and original indication approvals.

RESULTS

From 2017 to 2019, the FDA approved 146 supplemental indications for 107 therapeutics on the basis of 181 pivotal efficacy trials. The median (interquartile range) number of trials per supplemental indication was 1 (1-1). Most trials used either placebo (77 trials [42.5%; 95% CI, 35.6%-49.8%]) or active comparators (65 trials [35.9%; 95% CI, 29.3%-43.1%]), and most of these multigroup trials were randomized (141 trials [99.3%; 95% CI, 96.0%-100.0%]) and double-blinded (106 trials [74.5%; 95% CI, 66.6%-81.0%]); 80 trials (44.2%; 95 CI, 37.2%-51.5%) used clinical outcomes as the primary efficacy end point. There was no difference between oncology therapies and those approved for other therapeutic areas to have supplemental indication approvals be based on at least 2 pivotal trials (11.5% vs 20.6%; difference, 9.1%; 95% CI, 2.9%-21.0%; P = .10). Similarly, there was no difference in use of randomization (98.3% vs 100.0%; difference, 1.7%; 95% CI, 1.6%-5.0%; P = .43) among multigroup trials, although these trials were less likely to be double-blinded (50.8% vs 92.3%; difference, 41.5%; 95% CI, 27.4%-55.5%; P < .001); overall, these trials were less likely to use either placebo or active comparators (64.9% vs 86.7%; difference, 21.8% 95% CI, 9.8%-33.9%; P < .001) or to use clinical outcomes as their primary efficacy end point (27.5% vs 61.1%; difference, 33.6%; 95% CI, 14.1%-40.9%; P < .001) and were longer (median [interquartile range], 17 [6-48] weeks vs 95 [39-146] weeks). Original approvals were more likely than supplemental indication approvals to be based on at least 2 pivotal trials (44.0% [95% CI, 33.7%-42.6%] vs 15.8% [95% CI, 10.7%-22.5%]; difference, 28.2%; 95% CI, 17.6%-39.6%; P < .001) and less likely to be supported by at least 1 trial of 12 months' duration (27.6% [95% CI, 17.9%-35.0%] vs 54.8% [95% CI, 46.7%-62.6%]; difference, 27.2%; 95% CI, 14.5%-37.8%; P < .001). Pivotal trial designs were otherwise not significantly different.

CONCLUSIONS AND RELEVANCE

These findings suggest that the number and design of the pivotal trials supporting supplemental indication approvals by the FDA varied across therapeutic areas, with the strength of evidence for cancer indications weaker than that for other indications. There was little difference in the design characteristics of the pivotal trials supporting supplemental indication and original approvals.

A Practical Overview and Reporting Strategies for Statistical Analysis of Survival Studies

Survival (time-to-event) analysis is commonly used in clinical research. Key features of performing a survival analysis include checking proportional hazards assumptions, reporting CIs for hazards ratios and relative risks, graphically displaying the findings, and analyzing with consideration of competing risks. This article provides a brief overview of important statistical considerations for survival analysis. Censoring schemes, different methods of survival function estimation, and ways to compare survival curves are described. We also explain competing risk and how to model survival data in the presence of it. Different kinds of bias that influence survival estimation and avenues to model the data under these circumstances are also described. Several analysis techniques are accompanied by graphical representations illustrating proper reporting strategies. We provide a list of guiding statements for researchers and reviewers.

Statistical Decision Properties of Imprecise Trials Assessing Coronavirus Disease 2019 (COVID-19) Drugs

OBJECTIVES

Researchers studying treatment of coronavirus disease 2019 (COVID-19) have reported findings of randomized trials comparing standard care with care augmented by experimental drugs. Many trials have small sample sizes, so estimates of treatment effects are imprecise. Hence, clinicians may find it difficult to decide when to treat patients with experimental drugs. A conventional practice when comparing standard care and an innovation is to choose the innovation only if the estimated treatment effect is positive and statistically significant. This practice defers to standard care as the status quo. We study treatment choice from the perspective of statistical decision theory, which considers treatment options symmetrically when assessing trial findings.

METHODS

We use the concept of near-optimality to evaluate criteria for treatment choice. This concept jointly considers the probability and magnitude of decision errors. An appealing criterion from this perspective is the empirical success rule, which chooses the treatment with the highest observed average patient outcome in the trial.

RESULTS

Considering the design of some COVID-19 trials, we show that the empirical success rule yields treatment choices that are much closer to optimal than those generated by prevailing decision criteria based on hypothesis tests.

CONCLUSION

Using trial findings to make near-optimal treatment choices rather than perform hypothesis tests should improve clinical decision making.

Research Priorities for Patients with Heart Failure and Central Sleep Apnea. An Official American Thoracic Society Research Statement

Background: Central sleep apnea (CSA) is common among patients with heart failure and has been strongly linked to adverse outcomes. However, progress toward improving outcomes for such patients has been limited. The purpose of this official statement from the American Thoracic Society is to identify key areas to prioritize for future research regarding CSA in heart failure.

Methods: An international multidisciplinary group with expertise in sleep medicine, pulmonary medicine, heart failure, clinical research, and health outcomes was convened. The group met at the American Thoracic Society 2019 International Conference to determine research priority areas. A statement summarizing the findings of the group was subsequently authored using input from all members.

Results: The workgroup identified 11 specific research priorities in several key areas: 1) control of breathing and pathophysiology leading to CSA, 2) variability across individuals and over time, 3) techniques to examine CSA pathogenesis and outcomes, 4) impact of device and pharmacological treatment, and 5) implementing CSA treatment for all individuals

Conclusions: Advancing care for patients with CSA in the context of heart failure will require progress in the arenas of translational (basic through clinical), epidemiological, and patient-centered outcome research. Given the increasing prevalence of heart failure and its associated substantial burden to individuals, society, and the healthcare system, targeted research to improve knowledge of CSA pathogenesis and treatment is a priority.

Can we rely on COVID-19 data? An assessment of data from over 200 countries worldwide

To fight COVID-19, global access to reliable data is vital. Given the rapid acceleration of new cases and the common sense of global urgency, COVID-19 is subject to thorough measurement on a country-by-country basis. The world is witnessing an increasing demand for reliable data and impactful information on the novel disease. Can we trust the data on the COVID-19 spread worldwide? This study aims to assess the reliability of COVID-19 global data as disclosed by local authorities in 202 countries. It is commonly accepted that the frequency distribution of leading digits of COVID-19 data shall comply with Benford's law. In this context, the author collected and statistically assessed 106,274 records of daily infections, deaths, and tests around the world. The analysis of worldwide data suggests good agreement between theory and reported incidents. Approximately 69% of countries worldwide show some deviations from Benford's law. The author found that records of daily infections, deaths, and tests from 28% of countries adhered well to the anticipated frequency of first digits. By contrast, six countries disclosed pandemic data that do not comply with the first-digit law. With over 82 million citizens, Germany publishes the most reliable records on the COVID-19 spread. In contrast, the Islamic Republic of Iran provides by far the most non-compliant data. The author concludes that inconsistencies with Benford's law might be a strong indicator of artificially fabricated data on the spread of SARS-CoV-2 by local authorities. Partially consistent with prior research, the United States, Germany, France, Australia, Japan, and China reveal data that satisfies Benford's law. Unification of reporting procedures and policies globally could improve the quality of data and thus the fight against the deadly virus.

How fever is defined in COVID-19 publications: a disturbing lack of precision

OBJECTIVES

Fever is the single most frequently reported manifestation of COVID-19 and is a critical element of screening persons for COVID-19. The meaning of "fever" varies depending on the cutoff temperature used, the type of thermometer, the time of the day, the site of measurements, and the person's gender and race. The absence of a universally accepted definition for fever has been especially problematic during the current COVID-19 pandemic.

METHODS

This investigation determined the extent to which fever is defined in COVID-19 publications, with special attention to those associated with pregnancy.

RESULTS

Of 53 publications identified in which "fever" is reported as a manifestation of COVID-19 illness, none described the method used to measure patient's temperatures. Only 10 (19%) publications specified the minimum temperature used to define a fever with values that varied from a 37.3 °C (99.1 °F) to 38.1 °C (100.6 °F).

CONCLUSIONS

There is a disturbing lack of precision in defining fever in COVID-19 publications. Given the many factors influencing temperature measurements in humans, there can never be a single, universally accepted temperature cut-off defining a fever. This clinical reality should not prevent precision in reporting fever. To achieve the precision and improve scientific and clinical communication, when fever is reported in clinical investigations, at a minimum the cut-off temperature used in determining the presence of fever, the anatomical site at which temperatures are taken, and the instrument used to measure temperatures should each be described. In the absence of such information, what is meant by the term "fever" is uncertain.

Chest CT in COVID-19 at the ED: Validation of the COVID-19 Reporting and Data System (CO-RADS) and CT Severity Score: A Prospective, Multicenter, Observational Study

BACKGROUND

CT is thought to play a key role in coronavirus disease 2019 (COVID-19) diagnostic workup. The possibility of comparing data across different settings depends on the systematic and reproducible manner in which the scans are analyzed and reported. The COVID-19 Reporting and Data System (CO-RADS) and the corresponding CT severity score (CTSS) introduced by the Radiological Society of the Netherlands (NVvR) attempt to do so. However, this system has not been externally validated.

RESEARCH QUESTION

We aimed to prospectively validate the CO-RADS as a COVID-19 diagnostic tool at the ED and to evaluate whether the CTSS is associated with prognosis.

STUDY DESIGN AND METHODS

We conducted a prospective, observational study in two tertiary centers in The Netherlands, between March 19 and May 28, 2020. We consecutively included 741 adult patients at the ED with suspected COVID-19, who received a chest CT and severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) PCR (PCR). Diagnostic accuracy measures were calculated for CO-RADS, using PCR as reference. Logistic regression was performed for CTSS in relation to hospital admission, ICU admission, and 30-day mortality.

RESULTS

Seven hundred forty-one patients were included. We found an area under the curve (AUC) of 0.91 (CI, 0.89-0.94) for CO-RADS using PCR as reference. The optimal CO-RADS cutoff was 4, with a sensitivity of 89.4% (CI, 84.7-93.0) and specificity of 87.2% (CI, 83.9-89.9). We found a significant association between CTSS and hospital admission, ICU admission, and 30-day mortality; adjusted ORs per point increase in CTSS were 1.19 (CI, 1.09-1.28), 1.23 (1.15-1.32), 1.14 (1.07-1.22), respectively. Intraclass correlation coefficients for CO-RADS and CTSS were 0.94 (0.91-0.96) and 0.82 (0.70-0.90).

INTERPRETATION

Our findings support the use of CO-RADS and CTSS in triage, diagnosis, and management decisions for patients presenting with possible COVID-19 at the ED.

Missing at random assumption made more plausible: evidence from the 1958 British birth cohort

OBJECTIVE

Non-response is unavoidable in longitudinal surveys. The consequences are lower statistical power and the potential for bias. We implemented a systematic data-driven approach to identify predictors of non-response in the National Child Development Study (NCDS; 1958 British birth cohort). Such variables can help make the missing at random assumption more plausible, which has implications for the handling of missing data STUDY DESIGN AND SETTING: We identified predictors of non-response using data from the 11 sweeps (birth to age 55) of the NCDS (n = 17,415), employing parametric regressions and the LASSO for variable selection.

RESULTS

Disadvantaged socio-economic background in childhood, worse mental health and lower cognitive ability in early life, and lack of civic and social participation in adulthood were consistently associated with non-response. Using this information, along with other data from NCDS, we were able to replicate the "population distribution" of educational attainment and marital status (derived from external data), and the original distributions of key early life characteristics.

CONCLUSION

The identified predictors of non-response have the potential to improve the plausibility of the missing at random assumption. They can be straightforwardly used as "auxiliary variables" in analyses with principled methods to reduce bias due to missing data.

Tolerance interval testing for assessing accuracy and precision simultaneously

Tolerance intervals have been recommended for simultaneously validating both the accuracy and precision of an analytical procedure. However, statistical inferences for the corresponding hypothesis testing are scarce. The aim of this study is to establish a whole statistical inference for tolerance interval testing, including sample size determination, power analysis, and calculation of p-value. More specifically, the proposed method considers the bounds of a tolerance interval as random variables so that a bivariate distribution can be derived. Simulations confirm the theoretical properties of the method. Furthermore, an example is used to illustrate the proposed method.

Funnel plots of patient-reported outcomes to evaluate health-care quality: Basic principles, pitfalls and considerations

A funnel plot is a graphical method to evaluate health-care quality by comparing hospital performances on certain outcomes. So far, in nephrology, this method has been applied to clinical outcomes like mortality and complications. However, patient-reported outcomes (PROs; eg, health-related quality of life [HRQOL]) are becoming increasingly important and should be incorporated into this quality assessment. Using funnel plots has several advantages, including clearly visualized precision, detection of volume-effects, discouragement of ranking hospitals and easy interpretation of results. However, without sufficient knowledge of underlying methods, it is easy to stumble into pitfalls, such as overinterpretation of standardized scores, incorrect direct comparisons of hospitals and assuming a hospital to be in-control (ie, to perform as expected) based on underpowered comparisons. Furthermore, application of funnel plots to PROs is accompanied by additional challenges related to the multidimensional nature of PROs and difficulties with measuring PROs. Before using funnel plots for PROs, high and consistent response rates, adequate case mix correction and high-quality PRO measures are required. In this article, we aim to provide insight into the use and interpretation of funnel plots by presenting an overview of the basic principles, pitfalls and considerations when applied to PROs, using examples from Dutch routine dialysis care.

Assumptions About Competency-Based Medical Education and the State of the Underlying Evidence: A Critical Narrative Review

PURPOSE

As educators have implemented competency-based medical education (CBME) as a framework for training and assessment, they have made decisions based on available evidence and on the medical education community's assumptions about CBME. This critical narrative review aimed to collect, synthesize, and judge the existing evidence underpinning assumptions the community has made about CBME.

METHOD

The authors searched Ovid MEDLINE to identify empirical studies published January 2000 to February 2019 reporting on competence, competency, and CBME. The knowledge synthesis focused on "core" assumptions about CBME, selected via a survey of stakeholders who judged 31 previously identified assumptions. The authors judged, independently and in pairs, whether evidence from included studies supported, did not support, or was mixed related to each of the core assumptions. Assumptions were also analyzed to categorize their shared or contrasting purposes and foci.

RESULTS

From 8,086 unique articles, the authors reviewed 709 full-text articles and included 189 studies reporting evidence related to 15 core assumptions. Most studies (80%; n = 152) used a quantitative design. Many focused on procedural skills (48%; n = 90) and assessed behavior in clinical settings (37%; n = 69). On aggregate, the studies produced a mixed evidence base, reporting 362 data points related to the core assumptions (169 supportive, 138 not supportive, and 55 mixed). The 31 assumptions were organized into 3 categories: aspirations, conceptualizations, and assessment practices.

CONCLUSIONS

The reviewed evidence base is significant but mixed, with limited diversity in research designs and the types of competencies studied. This review pinpoints tensions to resolve (where evidence is mixed) and research questions to ask (where evidence is absent). The findings will help the community make explicit its assumptions about CBME, consider the value of those assumptions, and generate timely research questions to produce evidence about how and why CBME functions (or not).

Most randomized controlled trials for psoriasis used placebo comparators despite the availability of effective treatments

BACKGROUND

The availability of effective treatments for psoriasis raises ethical questions about the use of a placebo group in therapeutic trials. We evaluated the use of the placebo over time in such trials.

METHODS

From trials in a living Cochrane review and network meta-analysis for psoriasis, we included trials comparing a biologic to a placebo or other systemic treatment. First, we tested the changes in placebo rate from 2001 to 2019 by linear regression, then constructed networks for 2004-2019 and evaluated the contribution of the placebo to the network meta-analysis estimates per trial and per comparison.

RESULTS

We included 81 trials (36,774 patients). The placebo rate did not decrease significantly over time. The proportion contribution of trials with a placebo decreased from 100% in 2004 to 86% in 2008 and 75% in 2019. However, the proportion contribution of trials without a placebo remained low (from 0% in 2004 to 25% in 2019).

CONCLUSION

The design of future psoriasis trials should be reviewed to improve the number of patients to be included in a placebo group.

No inexplicable disagreements between real-world data-based nonrandomized controlled studies and randomized controlled trials were found

OBJECTIVES

We assessed disagreements between nonrandomized controlled studies based on real-world data (NRCS-RWDs) and randomized controlled trials (RCTs).

STUDY DESIGN AND SETTING

We systematically searched for studies that compared treatment effect estimates from NRCS-RWDs and RCTs on the same clinical question. We assessed the potential difference between NRCS-RWDs and RCTs related to internal and external validity. We calculated various meta-epidemiological measures to assess agreement. In case of disagreements, we tried to identify the probable causes of disagreements.

RESULTS

We included 12 studies comparing 15 treatment effect estimates of NRCS-RWDs and RCTs. There were many potential causes of disagreement. Ninety-five percent confidence intervals overlapped for 12 of 15 treatment effect estimates. Our analysis on predicted vs. observed overlap showed that there were no more disagreements than expected by chance. We observed only two substantial differences between the 15 treatment effect estimates. In both cases, we identified risk of bias in the NRCS-RWDs as the most probable cause of disagreement.

CONCLUSION

Our findings suggest that there are clinical questions where the difference in risk of bias between a well-conducted NRCS-RWD and an RCT is negligible. In our analysis, threats to external validity appeared to have no or only a weak impact on the disagreements of treatment effect estimates.

Improving preclinical studies through replications

The purpose of preclinical research is to inform the development of novel diagnostics or therapeutics, and the results of experiments on animal models of disease often inform the decision to conduct studies in humans. However, a substantial number of clinical trials fail, even when preclinical studies have apparently demonstrated the efficacy of a given intervention. A number of large-scale replication studies are currently trying to identify the factors that influence the robustness of preclinical research. Here, we discuss replications in the context of preclinical research trajectories, and argue that increasing validity should be a priority when selecting experiments to replicate and when performing the replication. We conclude that systematically improving three domains of validity - internal, external and translational - will result in a more efficient allocation of resources, will be more ethical, and will ultimately increase the chances of successful translation.

Energy Metabolism and Ketogenic Diets: What about the Skeletal Health? A Narrative Review and a Prospective Vision for Planning Clinical Trials on this Issue

The existence of a common mesenchymal cell progenitor shared by bone, skeletal muscle, and adipocytes cell progenitors, makes the role of the skeleton in energy metabolism no longer surprising. Thus, bone fragility could also be seen as a consequence of a “poor” quality in nutrition. Ketogenic diet was originally proven to be effective in epilepsy, and long-term follow-up studies on epileptic children undergoing a ketogenic diet reported an increased incidence of bone fractures and decreased bone mineral density. However, the causes of such negative impacts on bone health have to be better defined. In these subjects, the concomitant use of antiepileptic drugs and the reduced mobilization may partly explain the negative effects on bone health, but little is known about the effects of diet itself, and/or generic alterations in vitamin D and/or impaired growth factor production. Despite these remarks, clinical studies were adequately designed to investigate bone health are scarce and bone health related aspects are not included among the various metabolic pathologies positively influenced by ketogenic diets. Here, we provide not only a narrative review on this issue, but also practical advice to design and implement clinical studies on ketogenic nutritional regimens and bone health outcomes. Perspectives on ketogenic regimens, microbiota, microRNAs, and bone health are also included.

Comparison of a web-push survey research protocol with a mailed paper and pencil protocol in the Monitoring the Future panel survey

AIMS

The experiment tested the effects of a web-push survey research protocol, compared with the standard mailed paper-and-pencil protocol, among young adults aged 19-30 years in the 'Monitoring the Future' (MTF) longitudinal study.

DESIGN, SETTING AND PARTICIPANTS

The US-based MTF study has measured substance use trends among young adults in panel samples followed biennially, using consistent mailed survey procedures from 1977 to 2017. In 2018, young adult participants in the MTF longitudinal component scheduled to be surveyed at ages 19-30 in 2018 (from high school senior cohorts of 2006-17, n = 14 709) were randomly assigned to receive the standard mail/paper survey procedures or new web-push procedures.

MEASUREMENTS

Primary outcomes were responding to the survey and prevalence estimates for past 30-day use of alcohol, cigarettes, marijuana and illicit drugs.

FINDINGS

The web-push response rate was 39.07% [95% confidence interval (CI) = 37.889, 40.258]; this was significantly better than the standard MTF response rate of 35.12% (95% CI = 33.964, 36.285). After adjusting for covariates, the web-push condition was associated with a 19% increase in the odds of responding compared with standard MTF (adjusted odds ratio = 1.188; 95% CI = 1.096, 1.287). Substance use prevalence estimates were very similar and differences became negligible when using attrition weights and controlling for socio-demographic characteristics.

CONCLUSIONS

The web-push protocol produced a higher response rate than the mailed pencil and paper protocol in the Monitoring the Future panel study, without substantially affecting estimates of substance use once attrition weights and socio-demographic variables were factored in.

Statistical power in COVID-19 case-control host genomic study design

The identification of genetic variation that directly impacts infection susceptibility to SARS-CoV-2 and disease severity of COVID-19 is an important step towards risk stratification, personalized treatment plans, therapeutic, and vaccine development and deployment. Given the importance of study design in infectious disease genetic epidemiology, we use simulation and draw on current estimates of exposure, infectivity, and test accuracy of COVID-19 to demonstrate the feasibility of detecting host genetic factors associated with susceptibility and severity in published COVID-19 study designs. We demonstrate that limited phenotypic data and exposure/infection information in the early stages of the pandemic significantly impact the ability to detect most genetic variants with moderate effect sizes, especially when studying susceptibility to SARS-CoV-2 infection. Our insights can aid in the interpretation of genetic findings emerging in the literature and guide the design of future host genetic studies.

Ensuring the quality and specificity of preregistrations

Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of "researcher degrees of freedom" aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimated effect sizes. Preregistration is a mechanism for reducing such degrees of freedom by specifying designs and analysis plans before observing the research outcomes. The effectiveness of preregistration may depend, in part, on whether the process facilitates sufficiently specific articulation of such plans. In this preregistered study, we compared 2 formats of preregistration available on the OSF: Standard Pre-Data Collection Registration and Prereg Challenge Registration (now called "OSF Preregistration," http://osf.io/prereg/). The Prereg Challenge format was a "structured" workflow with detailed instructions and an independent review to confirm completeness; the "Standard" format was "unstructured" with minimal direct guidance to give researchers flexibility for what to prespecify. Results of comparing random samples of 53 preregistrations from each format indicate that the "structured" format restricted the opportunistic use of researcher degrees of freedom better (Cliff's Delta = 0.49) than the "unstructured" format, but neither eliminated all researcher degrees of freedom. We also observed very low concordance among coders about the number of hypotheses (14%), indicating that they are often not clearly stated. We conclude that effective preregistration is challenging, and registration formats that provide effective guidance may improve the quality of research.

Review of economic data on closed system transfer drug for preparation and administration of hazardous drugs

OBJECTIVES

The objectives of this study were to review economic data on the use of closed system drug transfer devices (CSTDs) for preparing and administering hazardous drugs, and to evaluate the quality of data reporting as defined by the Consolidated Health Economic Evaluation Reporting Standards (CHEERS).

METHODS

All references from a recent Cochrane review about CSTDs were evaluated for inclusion. A literature review was also conducted. Articles containing economic data about the use of CSTDs were retained for analysis. Two researchers independently graded the articles according to the 24-item CHEERS checklist.

RESULTS

Of the 138 articles identified initially, 12 were retained for analysis. Nine of these studies did not report acquisition costs or did not detail acquisition costs. Six studies reported economic benefits associated with the used of CSTDs, all related to extending the beyond-use date. The mean number of CHEERS criteria fulfilled by the included articles was 9.2 (SD 2.4).

CONCLUSIONS

CSTDs are costly to acquire. However, few studies have examined the economic impact of these devices, and the existing studies are incomplete. As a result, hospitals planning to implement these devices will be unable to make a sound economic evaluation. Robust economic evaluation of CSTDs is needed.

Analysis of the WHO ICTRP for novel coronavirus clinical trial registrations

Up-to-date information on the current progress made in the research and development to control the global COVID-19 pandemic is important. The study aimed to analyze the clinical trial characteristics and vaccine development progress of the new Coronavirus Disease 2019 (COVID-19) registered with the World Health Organization International Clinical Trial Registry Platform (WHO ICTRP).A comprehensive search of COVID-19 clinical trials since the establishment of the ICTRP to June 11, 2020, was conducted to record and analyze relevant characteristics. Chi-Squared test was used to compare the statistical differences between different research types, interventions, and sources.A total of 3282 COVID-19 clinical trials in 17 clinical trial registration centers were registered with the WHO ICTRP. The main research sources for the present study were ClinicalTrials.gov and ChiCTR. There were significant differences in the parameters of study location (P = .000), number of participants (P = .000), study duration (P = .001), research stage (P = .000), randomization procedure (P = .000), and blinding method (P = .000) between the 2 registration sources. There were significant differences in all the parameters between different kinds of intervention methods. Hydroxychloroquine, plasma therapy, and Xiyanping injection were the high-frequency research drugs used. Ten different vaccine studies were registered under phases I-II.Amongst the studies researched, heterogeneity existed for various parameters. Differences in the type of study, interventions, and registration sources of the studies led to significant differences in certain parameters of the COVID-19 clinical trials. The statistics of high-frequency drugs and the progress of vaccine trials may provide an informative reference for the prevention and control of COVID-19.

Clarifying assumptions in age-period-cohort analyses and validating results

BACKGROUND

Age-period-cohort (APC) models are often used to decompose health trends into period- and cohort-based sources, but their use in epidemiology and population sciences remains contentious. Central to the contention are researchers' failures to 1) clearly state their analytic assumptions and/or 2) thoroughly evaluate model results. These failures often produce varying conclusions across APC studies and generate confusion about APC methods. Consequently, scholarly exchanges about APC methods usually result in strong disagreements that rarely offer practical advice to users or readers of APC methods.

METHODS

We use research guidelines to help practitioners of APC methods articulate their analytic assumptions and validate their results. To demonstrate the usefulness of the guidelines, we apply them to a 2015 American Journal of Epidemiology study about trends in black-white differences in U.S. heart disease mortality.

RESULTS

The application of the guidelines highlights two important findings. On the one hand, some APC methods produce inconsistent results that are highly sensitive to researcher manipulation. On the other hand, other APC methods estimate results that are robust to researcher manipulation and consistent across APC models.

CONCLUSIONS

The exercise shows the simplicity and effectiveness of the guidelines in resolving disagreements over APC results. The cautious use of APC models can generate results that are consistent across methods and robust to researcher manipulation. If followed, the guidelines can likely reduce the chance of publishing variable and conflicting results across APC studies.

A Scoring Algorithm for Deriving Utility Values from the Neuro-QoL for Patients with Multiple Sclerosis

INTRODUCTION

The Neuro-QoL is a standardized approach to assessing health-related quality of life in people with neurological conditions, including multiple sclerosis (MS). Item banks were developed with item response theory (IRT) methodology so items are calibrated along a continuum of each construct. The purpose of this study was to develop a preference-based scoring algorithm for the Neuro-QoL to derive utilities that could be used in economic modeling.

METHODS

With input from neurologists, 6 Neuro-QoL domains were selected based on relevance to MS and used to define health states for a utility elicitation study in the United Kingdom. General population participants and individuals with MS valued the health states and completed questionnaires (including Neuro-QoL short forms). The Neuro-QoL Utility Scoring System (NQU) was derived based on multi-attribute utility theory using data from the general population sample. Single-attribute disutility functions for 6 Neuro-QoL domains were estimated using isotonic regression with linear interpolation and then combined with a multiplicative model. NQU validity was assessed using MS participant data.

RESULTS

Interviews were completed with 203 general population participants (50.2% female; mean age = 45.0 years) and 62 participants with MS (62.9% female; mean age = 46.1 years). Mean (SD) NQU scores were 0.94 (0.06) and 0.82 (0.13) for the general population and MS samples, respectively. The NQU demonstrated known-groups validity by differentiating among subgroups categorized based on level of disability. The NQU demonstrated convergent validity via correlations with generic measures (0.66 and 0.63 with EQ-5D-5L and Health Utilities Index Mark 3, respectively; both P < 0.001).

DISCUSSION

With the NQU, utilities can be derived from any MS treatment group, subgroup, or patient sample who completes items from 6 Neuro-QoL domains. Because the Neuro-QoL is frequently used with MS patients, the NQU greatly expands the options for quantifying outcomes in cost-utility analyses conducted to inform allocation of resources for MS treatment.

Conditional Direction Dependence Analysis: Evaluating the Causal Direction of Effects in Linear Models with Interaction Terms

Direction dependence analysis (DDA) makes use of higher than second moment information of variables (x and y) to detect potential confounding and to probe the causal direction of linear variable relations (i.e., whether x → y or y → x better approximates the underlying causal mechanism). The "true" predictor is assumed to be a continuous nonnormal exogenous variable. Existing methods compatible with DDA, however, are of limited use when the relation of a focal predictor and an outcome is affected by a moderator. This study presents a conditional direction dependence analysis (CDDA) framework which enables researchers to evaluate the causal direction of conditional regression effects. Monte-Carlo simulations were used to evaluate two different moderation scenarios: Study 1 evaluates the performance of CDDA tests when a moderator affects the strength of the causal effect x → y. Study 2 evaluates cases in which the causal direction itself (x → y vs y → x) depends on moderator values. Study 3 evaluates the robustness of DDA tests in the presence of functional model misspecifications. Results suggest that significance tests compatible with CDDA are suitable in both moderation scenarios, i.e., CDDA allows one to discern regions of a moderator in which the causal direction is uniquely identifiable. An empirical example is provided to illustrate the approach.

Exploring the Inter-observer Agreement Among the Members of the Italian Consensus for the Classification and Reporting of Thyroid Cytology

Classification schemes for reporting thyroid cytology of fine needle aspiration (FNA) of thyroid nodules are largely used in clinical practice, but the level of inter-observer agreement among cytopathologists is poorly acknowledged. The present study aimed to explore the inter-observer agreement among the experienced authors of the 2014 Italian Consensus for Classification and Reporting of Thyroid Cytology (ICCRTC). A stratified randomization was performed in order to obtain a sample homogeneously distributed and representative of all ICCRTC classes. Four high-experience raters were randomly selected among the extensors of the Italian consensus. They independently reviewed 60 FNA samples blindly of the initial cytological report and clinical features. Their overall agreement was evaluated according to Fleiss' kappa. The overall inter-observer agreement was moderate (κ 0.46). Specifically, a good agreement was found when the samples were consistent for malignancy (TIR5) or were not adequate for diagnosis (TIR1) (κ 0.67 and κ 0.73, respectively). A moderate agreement was present for suspicious-for-malignant category (TIR4), and a fair agreement was recorded in the two intermediate ones (TIR3A and TIR3B) (κ 0.36 and κ 0.35, respectively). For clinical purposes, the agreement was good (κ 0.74) in differentiating cases with surgical indication (TIR4/TIR5) from those in which surgery is not essential or requires limited extension (TIR3B/TIR3A/TIR2). In conclusion, the present study confirms the reliability of ICCRTC. These data represent a reference for cytopathologists using this system and are useful for the practice of clinicians and surgeons.

A four-step strategy for handling missing outcome data in randomised trials affected by a pandemic

BACKGROUND

The coronavirus pandemic (Covid-19) presents a variety of challenges for ongoing clinical trials, including an inevitably higher rate of missing outcome data, with new and non-standard reasons for missingness. International drug trial guidelines recommend trialists review plans for handling missing data in the conduct and statistical analysis, but clear recommendations are lacking.

METHODS

We present a four-step strategy for handling missing outcome data in the analysis of randomised trials that are ongoing during a pandemic. We consider handling missing data arising due to (i) participant infection, (ii) treatment disruptions and (iii) loss to follow-up. We consider both settings where treatment effects for a 'pandemic-free world' and 'world including a pandemic' are of interest.

RESULTS

In any trial, investigators should; (1) Clarify the treatment estimand of interest with respect to the occurrence of the pandemic; (2) Establish what data are missing for the chosen estimand; (3) Perform primary analysis under the most plausible missing data assumptions followed by; (4) Sensitivity analysis under alternative plausible assumptions. To obtain an estimate of the treatment effect in a 'pandemic-free world', participant data that are clinically affected by the pandemic (directly due to infection or indirectly via treatment disruptions) are not relevant and can be set to missing. For primary analysis, a missing-at-random assumption that conditions on all observed data that are expected to be associated with both the outcome and missingness may be most plausible. For the treatment effect in the 'world including a pandemic', all participant data is relevant and should be included in the analysis. For primary analysis, a missing-at-random assumption - potentially incorporating a pandemic time-period indicator and participant infection status - or a missing-not-at-random assumption with a poorer response may be most relevant, depending on the setting. In all scenarios, sensitivity analysis under credible missing-not-at-random assumptions should be used to evaluate the robustness of results. We highlight controlled multiple imputation as an accessible tool for conducting sensitivity analyses.

CONCLUSIONS

Missing data problems will be exacerbated for trials active during the Covid-19 pandemic. This four-step strategy will facilitate clear thinking about the appropriate analysis for relevant questions of interest.

Assessment of Trends in the Design, Accrual, and Completion of Trials Registered in ClinicalTrials.gov by Sponsor Type, 2000-2019

IMPORTANCE

ClinicalTrials.gov is a valuable resource that can be used to trace the state and nature of trials. Since its launch in 2000, more than 345 000 trials have been registered. Little is known about the characteristics and trends in clinical trials over time and how they differ by sponsor type.

OBJECTIVE

To assess trends in clinical trials registered in ClinicalTrials.gov over time and by sponsor type.

DESIGN, SETTING, AND PARTICIPANTS

This cross-sectional study included clinical trials (interventional studies) registered in ClinicalTrials.gov from January 1, 2000, through December 31, 2019. The trials were grouped by lead sponsor: National Institutes of Health (NIH) and other US government agencies, industry, and other sources (foundations, universities, hospitals, clinics, and others). A static version of the Clinical Trials Transformation Initiative Aggregate Analysis of ClinicalTrials.gov database was downloaded on January 1, 2020, for analysis.

MAIN OUTCOMES AND MEASURES

ClinicalTrials.gov registration fields, including overall status, phase, intervention, number of sites, use of masking and randomization, sample size, and time to study completion by start year and lead sponsor (organization that provided funding or support for a clinical study).

RESULTS

A total of 245 999 clinical trials (interventional studies) were started between 2000 and 2019, of which 135 144 (54.9%) were completed. Among completed trials, 5113 (3.8%) were sponsored by the NIH or a US government agency, 48 668 (36.0%) by industry, and 81 363 (60.2%) by other sources. Most trials were single center (61.3%), randomized (65.6%), and phase 1 to 2 (35.5%) or did not have a US Food and Drug Administration-defined phase (38.4%), with fewer drug trials being conducted over time. Sample sizes were small (median, 60; interquartile range [IQR], 30-160) and diminished over time. Trial median completion times varied by lead sponsor: 3.4 years (IQR, 1.9-5.0 years) for NIH- and US government-sponsored trials, 1.2 years (IQR, 0.5-2.4 years) for industry trials, and 2.1 years (IQR, 1.1-3.7) for trials sponsored by other sources.

CONCLUSIONS AND RELEVANCE

The findings suggest that the composition and design of trials changed from 2000 to 2019 and differed substantially by sponsor type. Increased funding toward larger randomized clinical trials may be warranted to inform clinical decision-making and guide future research.