1
|
Afonso J, Ramirez-Campillo R, Clemente FM, Büttner FC, Andrade R. The Perils of Misinterpreting and Misusing "Publication Bias" in Meta-analyses: An Education Review on Funnel Plot-Based Methods. Sports Med 2024; 54:257-269. [PMID: 37684502 PMCID: PMC10933152 DOI: 10.1007/s40279-023-01927-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 08/24/2023] [Indexed: 09/10/2023]
Abstract
Publication bias refers to a systematic deviation from the truth in the results of a meta-analysis due to the higher likelihood for published studies to be included in meta-analyses than unpublished studies. Publication bias can lead to misleading recommendations for decision and policy making. In this education review, we introduce, explain, and provide solutions to the pervasive misuses and misinterpretations of publication bias that afflict evidence syntheses in sport and exercise medicine, with a focus on the commonly used funnel-plot based methods. Publication bias is more routinely assessed by visually inspecting funnel plot asymmetry, although it has been consistently deemed unreliable, leading to the development of statistical tests to assess publication bias. However, most statistical tests of publication bias (i) cannot rule out alternative explanations for funnel plot asymmetry (e.g., between-study heterogeneity, choice of metric, chance) and (ii) are grossly underpowered, even when using an arbitrary minimum threshold of ten or more studies. We performed a cross-sectional meta-research investigation of how publication bias was assessed in systematic reviews with meta-analyses published in the top two sport and exercise medicine journals throughout 2021. This analysis highlights that publication bias is frequently misused and misinterpreted, even in top tier journals. Because of conceptual and methodological problems when assessing and interpreting publication bias, preventive strategies (e.g., pre-registration, registered reports, disclosing protocol deviations, and reporting all study findings regardless of direction or magnitude) offer the best and most efficient solution to mitigate the misuse and misinterpretation of publication bias. Because true publication bias is very difficult to determine, we recommend that future publications use the term "risk of publication bias".
Collapse
Affiliation(s)
- José Afonso
- Faculty of Sport, Centre of Research, Education, Innovation, and Intervention in Sport (CIFI2D), University of Porto, Porto, Portugal.
| | - Rodrigo Ramirez-Campillo
- Exercise and Rehabilitation Sciences Institute, Faculty of Rehabilitation Sciences, School of Physical Therapy, Universidad Andres Bello, Santiago, Chile
| | - Filipe Manuel Clemente
- Escola Superior de Desporto e Lazer, Instituto Politécnico de Viana do Castelo, Rua Escola Industrial Comercial de Nun'Álvares, 4900-347, Viana do Castelo, Portugal
- Instituto de Telecomunicações, Delegação da Covilhã, Covilhã, Portugal
- Research Center in Sports Performance, Recreation, Innovation and Technology (SPRINT), 4960-320, Melgaço, Portugal
| | - Fionn Cléirigh Büttner
- Pragmatic Clinical Trials Unit, Centre for Evaluation and Methods, Wolfson Centre of Population Health, Queen Mary University of London, London, UK
- Department of Medical Statistics, London School of Hygiene and Tropical Medicine, London, UK
| | - Renato Andrade
- Porto Biomechanics Laboratory (LABIOMEP), University of Porto, Porto, Portugal
- Clínica Espregueira, FIFA Medical Centre of Excellence, Porto, Portugal
- Dom Henrique Research Centre, Porto, Portugal
| |
Collapse
|
2
|
Mello AT, Kammer PV, Nascimento GM, de Lima LP, Pessini J, Valmorbida A, Page MJ, Trindade EBSM. Credibility at stake: only two-thirds of randomized trials of nutrition interventions are registered and lack transparency in outcome and treatment effect definitions. J Clin Epidemiol 2023; 161:74-83. [PMID: 37399969 DOI: 10.1016/j.jclinepi.2023.06.021] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2023] [Revised: 06/02/2023] [Accepted: 06/27/2023] [Indexed: 07/05/2023]
Abstract
OBJECTIVES This study aimed to investigate the adherence of randomized controlled trials of nutrition interventions to transparency practices informing assessments of selective reporting biases, including the availability of a trial registration entry, protocol and statistical analysis plan (SAP). STUDY DESIGN AND SETTING Retrospective observational study with cross-sectional design. We systematically searched for trials published from 1 July 2019, to 30 June 2020, and included a randomly selected sample of 400 studies. We searched for registry entries, protocols, and SAPs for all included studies. We extracted data to characterize the disclosure of sufficient information in the available materials to inform assessments of selective reporting biases, considering the definition of outcome domain, measure, metric, method of aggregation, time point, analysis population, methods to handle missing data and method of adjustment. RESULTS Most trials (69%) were registered, but these often lacked sufficient specification of outcomes and intended treatment effects. Protocols and SAPs provided more details but were less often available (14% and 3%, respectively), and even then, almost all studies presented limited information to inform the assessments of risk of bias due to the selection of the reported result. CONCLUSION Lack of full specification of outcomes and intended treatment effects hinder a full adherence of randomized controlled trials of nutrition interventions to transparency practices and may affect their credibility.
Collapse
Affiliation(s)
- Arthur T Mello
- Post-Graduate Program in Nutrition, Federal University of Santa Catarina, Florianopolis, Santa Catarina, Brazil
| | - Pedro V Kammer
- Post-Graduate Program in Dentistry, Federal University of Santa Catarina, Florianopolis, Santa Catarina, Brazil
| | - Giovanna M Nascimento
- Post-Graduate Program in Nutrition, Federal University of Santa Catarina, Florianopolis, Santa Catarina, Brazil
| | - Luana P de Lima
- Post-Graduate Program in Nutrition, Federal University of Santa Catarina, Florianopolis, Santa Catarina, Brazil
| | - Júlia Pessini
- Post-Graduate Program in Nutrition, Federal University of Santa Catarina, Florianopolis, Santa Catarina, Brazil
| | - Aline Valmorbida
- Post-Graduate Program in Nutrition, Federal University of Santa Catarina, Florianopolis, Santa Catarina, Brazil
| | - Matthew J Page
- Methods in Evidence Synthesis Unit, School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
| | - Erasmo B S M Trindade
- Post-Graduate Program in Nutrition, Federal University of Santa Catarina, Florianopolis, Santa Catarina, Brazil; Department of Nutrition, Federal University of Santa Catarina, Florianopolis, Santa Catarina, Brazil.
| |
Collapse
|
3
|
Hilton Boon M, Burns J, Craig P, Griebler U, Heise TL, Vittal Katikireddi S, Rehfuess E, Shepperd S, Thomson H, Bero L. Value and Challenges of Using Observational Studies in Systematic Reviews of Public Health Interventions. Am J Public Health 2022; 112:548-552. [PMID: 35319925 PMCID: PMC8961824 DOI: 10.2105/ajph.2021.306658] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/06/2021] [Indexed: 11/04/2022]
Affiliation(s)
- Michele Hilton Boon
- Michele Hilton Boon, Peter Craig, S. Vittal Katikireddi, and Hilary Thomson are with the Medical Research Council/Scottish Government Chief Scientist Office Social and Public Health Sciences Unit, University of Glasgow, Glasgow, UK. Jacob Burns and Eva Rehfuess are with the Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig Maximilian University of Munich, Munich, Germany. Ursula Griebler is with the Department for Evidence-Based Medicine and Evaluation, Danube University Krems, Krems an der Donau, Austria. Thomas L. Heise is with the Leibniz Institute for Prevention Research and Epidemiology-BIPS, University of Bremen, Bremen, Germany. Sasha Shepperd is with the Nuffield Department of Population Health, University of Oxford, Oxford, UK. Lisa Bero is with the School of Public Health, University of Colorado Anschutz Medical Campus, Aurora, CO
| | - Jacob Burns
- Michele Hilton Boon, Peter Craig, S. Vittal Katikireddi, and Hilary Thomson are with the Medical Research Council/Scottish Government Chief Scientist Office Social and Public Health Sciences Unit, University of Glasgow, Glasgow, UK. Jacob Burns and Eva Rehfuess are with the Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig Maximilian University of Munich, Munich, Germany. Ursula Griebler is with the Department for Evidence-Based Medicine and Evaluation, Danube University Krems, Krems an der Donau, Austria. Thomas L. Heise is with the Leibniz Institute for Prevention Research and Epidemiology-BIPS, University of Bremen, Bremen, Germany. Sasha Shepperd is with the Nuffield Department of Population Health, University of Oxford, Oxford, UK. Lisa Bero is with the School of Public Health, University of Colorado Anschutz Medical Campus, Aurora, CO
| | - Peter Craig
- Michele Hilton Boon, Peter Craig, S. Vittal Katikireddi, and Hilary Thomson are with the Medical Research Council/Scottish Government Chief Scientist Office Social and Public Health Sciences Unit, University of Glasgow, Glasgow, UK. Jacob Burns and Eva Rehfuess are with the Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig Maximilian University of Munich, Munich, Germany. Ursula Griebler is with the Department for Evidence-Based Medicine and Evaluation, Danube University Krems, Krems an der Donau, Austria. Thomas L. Heise is with the Leibniz Institute for Prevention Research and Epidemiology-BIPS, University of Bremen, Bremen, Germany. Sasha Shepperd is with the Nuffield Department of Population Health, University of Oxford, Oxford, UK. Lisa Bero is with the School of Public Health, University of Colorado Anschutz Medical Campus, Aurora, CO
| | - Ursula Griebler
- Michele Hilton Boon, Peter Craig, S. Vittal Katikireddi, and Hilary Thomson are with the Medical Research Council/Scottish Government Chief Scientist Office Social and Public Health Sciences Unit, University of Glasgow, Glasgow, UK. Jacob Burns and Eva Rehfuess are with the Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig Maximilian University of Munich, Munich, Germany. Ursula Griebler is with the Department for Evidence-Based Medicine and Evaluation, Danube University Krems, Krems an der Donau, Austria. Thomas L. Heise is with the Leibniz Institute for Prevention Research and Epidemiology-BIPS, University of Bremen, Bremen, Germany. Sasha Shepperd is with the Nuffield Department of Population Health, University of Oxford, Oxford, UK. Lisa Bero is with the School of Public Health, University of Colorado Anschutz Medical Campus, Aurora, CO
| | - Thomas L Heise
- Michele Hilton Boon, Peter Craig, S. Vittal Katikireddi, and Hilary Thomson are with the Medical Research Council/Scottish Government Chief Scientist Office Social and Public Health Sciences Unit, University of Glasgow, Glasgow, UK. Jacob Burns and Eva Rehfuess are with the Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig Maximilian University of Munich, Munich, Germany. Ursula Griebler is with the Department for Evidence-Based Medicine and Evaluation, Danube University Krems, Krems an der Donau, Austria. Thomas L. Heise is with the Leibniz Institute for Prevention Research and Epidemiology-BIPS, University of Bremen, Bremen, Germany. Sasha Shepperd is with the Nuffield Department of Population Health, University of Oxford, Oxford, UK. Lisa Bero is with the School of Public Health, University of Colorado Anschutz Medical Campus, Aurora, CO
| | - S Vittal Katikireddi
- Michele Hilton Boon, Peter Craig, S. Vittal Katikireddi, and Hilary Thomson are with the Medical Research Council/Scottish Government Chief Scientist Office Social and Public Health Sciences Unit, University of Glasgow, Glasgow, UK. Jacob Burns and Eva Rehfuess are with the Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig Maximilian University of Munich, Munich, Germany. Ursula Griebler is with the Department for Evidence-Based Medicine and Evaluation, Danube University Krems, Krems an der Donau, Austria. Thomas L. Heise is with the Leibniz Institute for Prevention Research and Epidemiology-BIPS, University of Bremen, Bremen, Germany. Sasha Shepperd is with the Nuffield Department of Population Health, University of Oxford, Oxford, UK. Lisa Bero is with the School of Public Health, University of Colorado Anschutz Medical Campus, Aurora, CO
| | - Eva Rehfuess
- Michele Hilton Boon, Peter Craig, S. Vittal Katikireddi, and Hilary Thomson are with the Medical Research Council/Scottish Government Chief Scientist Office Social and Public Health Sciences Unit, University of Glasgow, Glasgow, UK. Jacob Burns and Eva Rehfuess are with the Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig Maximilian University of Munich, Munich, Germany. Ursula Griebler is with the Department for Evidence-Based Medicine and Evaluation, Danube University Krems, Krems an der Donau, Austria. Thomas L. Heise is with the Leibniz Institute for Prevention Research and Epidemiology-BIPS, University of Bremen, Bremen, Germany. Sasha Shepperd is with the Nuffield Department of Population Health, University of Oxford, Oxford, UK. Lisa Bero is with the School of Public Health, University of Colorado Anschutz Medical Campus, Aurora, CO
| | - Sasha Shepperd
- Michele Hilton Boon, Peter Craig, S. Vittal Katikireddi, and Hilary Thomson are with the Medical Research Council/Scottish Government Chief Scientist Office Social and Public Health Sciences Unit, University of Glasgow, Glasgow, UK. Jacob Burns and Eva Rehfuess are with the Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig Maximilian University of Munich, Munich, Germany. Ursula Griebler is with the Department for Evidence-Based Medicine and Evaluation, Danube University Krems, Krems an der Donau, Austria. Thomas L. Heise is with the Leibniz Institute for Prevention Research and Epidemiology-BIPS, University of Bremen, Bremen, Germany. Sasha Shepperd is with the Nuffield Department of Population Health, University of Oxford, Oxford, UK. Lisa Bero is with the School of Public Health, University of Colorado Anschutz Medical Campus, Aurora, CO
| | - Hilary Thomson
- Michele Hilton Boon, Peter Craig, S. Vittal Katikireddi, and Hilary Thomson are with the Medical Research Council/Scottish Government Chief Scientist Office Social and Public Health Sciences Unit, University of Glasgow, Glasgow, UK. Jacob Burns and Eva Rehfuess are with the Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig Maximilian University of Munich, Munich, Germany. Ursula Griebler is with the Department for Evidence-Based Medicine and Evaluation, Danube University Krems, Krems an der Donau, Austria. Thomas L. Heise is with the Leibniz Institute for Prevention Research and Epidemiology-BIPS, University of Bremen, Bremen, Germany. Sasha Shepperd is with the Nuffield Department of Population Health, University of Oxford, Oxford, UK. Lisa Bero is with the School of Public Health, University of Colorado Anschutz Medical Campus, Aurora, CO
| | - Lisa Bero
- Michele Hilton Boon, Peter Craig, S. Vittal Katikireddi, and Hilary Thomson are with the Medical Research Council/Scottish Government Chief Scientist Office Social and Public Health Sciences Unit, University of Glasgow, Glasgow, UK. Jacob Burns and Eva Rehfuess are with the Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig Maximilian University of Munich, Munich, Germany. Ursula Griebler is with the Department for Evidence-Based Medicine and Evaluation, Danube University Krems, Krems an der Donau, Austria. Thomas L. Heise is with the Leibniz Institute for Prevention Research and Epidemiology-BIPS, University of Bremen, Bremen, Germany. Sasha Shepperd is with the Nuffield Department of Population Health, University of Oxford, Oxford, UK. Lisa Bero is with the School of Public Health, University of Colorado Anschutz Medical Campus, Aurora, CO
| |
Collapse
|
4
|
Kanukula R, Page M, Dwan K, Turner S, Loder E, Mayo-Wilson E, Li T, Misra A, McDonald S, Forbes A, McKenzie J. Development of a checklist to detect errors in meta-analyses in systematic reviews of interventions: study protocol. F1000Res 2021; 10:455. [PMID: 34249342 PMCID: PMC8258702 DOI: 10.12688/f1000research.53034.1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 05/17/2021] [Indexed: 11/20/2022] Open
Abstract
Background : Systematic reviews underpin clinical practice and policies that guide healthcare decisions. A core component of many systematic reviews is meta-analysis, which is a statistical synthesis of results across studies. Errors in the conduct and interpretation of meta-analysis can lead to incorrect conclusions regarding the benefits and harms of interventions; and studies have shown that these errors are common. Enabling peer reviewers to better detect errors in meta-analysis through the use of a checklist provides an opportunity for these errors to be rectified before publication. To our knowledge, no such checklist exists. Objective : To develop and evaluate a checklist to detect errors in pairwise meta-analyses in systematic reviews of interventions. Methods : We will undertake a four-step process to develop the checklist. First, we will undertake a systematic review of studies that have evaluated errors in the conduct and interpretation of meta-analysis to generate a bank of items to consider for the checklist. Second, we will undertake a survey of systematic review methodologists and statisticians to seek their views on which items, of the bank of items generated in step 1, are most important to include in the checklist. Third, we will hold a virtual meeting to agree upon which items to include in the checklist. Fourth, before finalising the checklist, we will pilot with editors and peer reviewers of journals. Conclusion : The developed checklist is intended to help journal editors and peer reviewers identify errors in the application and interpretation of meta-analyses in systematic reviews. Fewer errors in the conduct and improved interpretation will lead to more accurate review findings and conclusions to inform clinical practice.
Collapse
Affiliation(s)
- Raju Kanukula
- School of Public Health and Preventive Medicine, Monash University, Melbourne, VICTORIA, 3004, Australia
| | - Matthew Page
- School of Public Health and Preventive Medicine, Monash University, Melbourne, VICTORIA, 3004, Australia
| | - Kerry Dwan
- Cochrane Methods Support Unit, Cochrane, London, SW1Y 4QX, UK
| | - Simon Turner
- School of Public Health and Preventive Medicine, Monash University, Melbourne, VICTORIA, 3004, Australia
| | - Elizabeth Loder
- Division of Headache, Department of Neurology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, 02115, USA.,The BMJ, BMA House, London, WC1H 9JP, UK
| | - Evan Mayo-Wilson
- Department of Epidemiology and Biostatistics, Indiana University School of Public Health-Bloomington, Bloomington, IN, 47405, USA
| | - Tianjing Li
- Department of Ophthalmology, School of Medicine, University of Colorado Denver, Denver, CO, 80045, USA.,Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, 21205, USA
| | | | - Steve McDonald
- School of Public Health and Preventive Medicine, Monash University, Melbourne, VICTORIA, 3004, Australia
| | - Andrew Forbes
- School of Public Health and Preventive Medicine, Monash University, Melbourne, VICTORIA, 3004, Australia
| | - Joanne McKenzie
- School of Public Health and Preventive Medicine, Monash University, Melbourne, VICTORIA, 3004, Australia
| |
Collapse
|