How do authors of systematic reviews deal with research malpractice and misconduct in original studies? A cross-sectional analysis of systematic reviews and survey of their authors

Systematic reviewers used various approaches to data extraction and expressed several research needs: a survey

OBJECTIVE

Data extraction is a prerequisite for analyzing, summarizing, and interpreting evidence in systematic reviews. Yet guidance is limited, and little is known about current approaches. We surveyed systematic reviewers on their current approaches to data extraction, opinions on methods, and research needs.

STUDY DESIGN AND SETTING

We developed a 29-question online survey and distributed it through relevant organizations, social media, and personal networks in 2022. Closed questions were evaluated using descriptive statistics, and open questions were analyzed using content analysis.

RESULTS

162 reviewers participated. Use of adapted (65%) or newly developed extraction forms (62%) was common. Generic forms were rarely used (14%). Spreadsheet software was the most popular extraction tool (83%). Piloting was reported by 74% of respondents and included a variety of approaches. Independent and duplicate extraction was considered the most appropriate approach to data collection (64%). About half of respondents agreed that blank forms and/or raw data should be published. Suggested research gaps were the effects of different methods on error rates (60%) and the use of data extraction support tools (46%).

CONCLUSION

Systematic reviewers used varying approaches to pilot data extraction. Methods to reduce errors and use of support tools such as (semi-)automation tools are top research gaps.

Ethically Driven and Methodologically Tailored: Setting the Agenda for Systematic Reviews in Domestic Violence and Abuse

Purpose

Systematic reviews have an important, and growing, role to play in the global evidence eco-system of domestic violence and abuse. Alongside substantive contributions to knowledge, such reviews stimulate debates about ethical reviewing practices and the importance of tailoring methods to the nuances of the field. This paper aims to pinpoint a set of ethical and methodological priorities to guide and enhance review practices specifically in the field of domestic abuse.

Method

The five Pillars of the Research Integrity Framework (ethical guidelines for domestic abuse research) are used to interrogate the systematic review process. To do so, the Framework is retrospectively applied to a recently completed systematic review in domestic abuse. The review included a rapid systematic map and in-depth analysis of interventions aimed at creating or enhancing informal support and social networks for victim-survivors of abuse.

Results

Ethical and methodological priorities for systematic reviews in domestic abuse include (1) Safety and wellbeing: maintaining the wellbeing of researchers and stakeholders, and appraising the ethics of included studies, (2) Transparency/ accountability: transparent reporting of research funding, aims and methods together with explicit consideration of authorship of outputs, (3) Equality, human rights and social justice: developing diverse review teams/ Advisory groups, and review methods that aim to search for, and report, diverse perspectives. Considering researcher positionality/ reflexivity in the review, (4) Engagement: collaboration with non-academic stakeholders and individuals with lived experience throughout the review process, (5) Research Ethics: independent ethical scrutiny of systematic review proposals with input from researchers with expertise in systematic reviews and domestic abuse.

Conclusion

Additional research is required to comprehensively examine the ethics of each stage of the review process. In the meantime, attention should be given to the underpinning ethical framework for our systematic review practices and the wider research infrastructure that governs reviews.

Taxonomy of Interventions to Reduce Acute Care Transfers From Long-term Care Homes: A Systematic Scoping Review

OBJECTIVE

To develop a taxonomy of interventions aimed at reducing emergency department (ED) transfers and/or hospitalizations from long-term care (LTC) homes.

DESIGN

A systematic scoping review.

SETTING AND PARTICIPANTS

Permanent LTC home residents.

METHODS

Experimental and comparative observational studies were searched in MEDLINE, CINAHL, Embase Classic + Embase, the Cochrane Library, PsycINFO, Social Work Abstracts, AMED, Global Health, Health and Psychosocial Instruments, Joanna Briggs Institute EBP Database, Ovid Healthstar, and Web of Science Core Collection from inception until March 2020. Forward/backward citation tracking and gray literature searches strengthened comprehensiveness. The Mixed Methods Appraisal Tool was used to assess study quality. Intervention categories and components were identified using an inductive-deductive thematic analysis. Categories were informed by 3 intervention dimensions: (1) "when/at what point(s)" on the continuum of care they occur, (2) "for whom" (ie, intervention target resident populations), and (3) "how" these interventions effect change. Components were informed by the logistical elements of the interventions having the potential to influence outcomes. All interventions were mapped to the developed taxonomy based on their categories, components, and outcomes. Distributions of components by category and study year were graphically presented.

RESULTS

Ninety studies (25 randomized, 23 high quality) were included. Six intervention categories were identified: advance care planning; palliative and end-of-life care; onsite care for acute, subacute, or uncontrolled chronic conditions; transitional care; enhanced usual care (most prevalent, 31% of 90 interventions); and comprehensive care. Four components were identified: increasing human resource capacity (most prevalent, 93%), training or reorganization of existing staff, technology, and standardized tools. The use of technology increased over time. Potentially avoidable ED transfers and/or hospitalizations were measured infrequently as primary outcomes.

CONCLUSIONS AND IMPLICATIONS

This proposed taxonomy can guide future intervention designs. It can also facilitate systematic reviews and precise effect size estimations for homogenous interventions when outcomes are comparable.

Sexual Dysfunction in Infertile Men: A Systematic Review and Meta-Analysis

Background

According to previous studies of male infertility, we found that the association between sexual dysfunction and male infertility was reported rarely and controversially.

Aim

We carried out this 1meta-analysis to evaluate the prevalence of sexual dysfunction and the International Index of Erectile Function (IIEF) score in infertile men.

Methods

A systematic search of the target literature was conducted using PubMed, EMBASE, and Cochrane Library. Data were analyzed using Review Manager 5.4 software. Standardized mean differences (SMD) with the corresponding 95% confidence intervals (95% CIs) were implemented in 6 controlled studies as a measure of effect size to assess the relationship between sexual dysfunction and male infertility and Odds Ratio (OR) were performed for the morbidity between infertility group and fertility group.

Outcomes

Men in infertile group were found with higher prevalence of sexual dysfunction and lower IIEF values than in controls.

Results

A meta-analysis of morbidity was performed in 8 of 10 controlled studies. Meta-analysis of the 8 studies found remarkable higher prevalence of sexual dysfunction in men with infertility than in controls (OR = 2.66, 95% confidence interval = 1.69–4.19, P < .0001; I² = 67%, P for heterogeneity = 0.004). Another meta-analysis of evidence suggested that IIEF in infertile men was lower than controls (SMD = −0.47, 95% confidence interval = −0.63 to −0.31, P < .00001; I² = 64%, p for heterogeneity = 0.02).

Clinical Implications

We recommend further research based on the relevant criteria of region, sample size, rigorous statistical analysis, and research design.

Strengths & limitations

This systematic review is the first to evaluate the prevalence of sexual dysfunction and the score of sexual dysfunction in male infertility. Investigation on the topic is scarce, and only few studies used appropriate measures.

Conclusions

Male infertility was associated with an increase in the prevalence of sexual dysfunction. The areas most affected by sexual function were erectile function, orgasm and sexual desire.

Liu Y, Wang Y, Dong C, et al. Sexual Dysfunction in Infertile Men: A Systematic Review and Meta-Analysis. Sex Med 2022;10:100528.

Effectiveness of exercise for osteosarcopenia in older adults: a systematic review protocol

INTRODUCTION

Osteosarcopenia is defined as the concomitant occurrence of sarcopenia and osteopenia or osteoporosis. Older adults with this syndrome have a greater fragility and mortality risk compared with those without these conditions. Based on separate interventions with individuals with sarcopenia and osteoporosis, exercise has been recommended as a treatment for osteosarcopenia. However, there is no evidence of its efficacy. Our objective is to identify whether physical exercise can improve osteosarcopenia in older adults and lead to good health outcomes.

METHODS AND ANALYSIS

We will perform a systematic review in the following databases: PubMed, Embase, Cochrane and Scopus. The criterion of inclusion will be clinical trials involving physical exercise interventions in older adults diagnosed with osteosarcopenia. To assess the risk of bias, the Grading of Recommendations, Assessment, Development and Evaluation and Downs and Black tools will be used. For each search result, the quality of the evidence will ultimately receive one of four grades: high, moderate, low or very low. The outcome of this study is to demonstrate the effectiveness of physical exercise in improving the parameters that lead to the diagnosis of osteosarcopenia (bone mineral density, quality of muscle mass, muscle strength and physical function) in older adults. The possibility of meta-analysis will be assessed according to the homogeneity of the studies, using the methods of fixed or random effects. Sensitivity analyses will be performed, and the funnel plot will be used to assess publication bias. The proposed statistical analyses will be performed using STATA software, V.14.0.

ETHICS AND DISSEMINATION

The results of the systematic review will be disseminated via publication in a peer-reviewed journal and presented at a relevant conference. As we will not use individual patient data, ethical approval is not required.

TRIAL REGISTRATION NUMBER

CRD42020215659.

Motor performance and back pain in children and adolescents: a systematic review and meta-analysis protocol

BACKGROUND

The relationship between motor performance and back pain in children and adolescents remains unclear. This article describes the protocol for a systematic review to summarize the knowledge about the association between motor performance, such as agility, flexibility, balance, strength, muscle endurance, and cardiorespiratory fitness, and back pain. Thus, our aim is to identify the influence of motor performance on back pain among children and adolescents.

METHODS

Two independent researchers will search MEDLINE, Scopus, Embase, SPORTDiscus, and CINAHL databases, with no period or language restrictions. We will include cross-sectional, cohort, case-control, and controlled clinical trial studies based on the following criteria: (a) participants from 6 to 19 years of age, (b) assessment of motor performance, (c) assessment of back pain, and (d) report measures of associations between motor performance and back pain. Study quality and risk of bias will be assessed using an adapted version of the Downs and Black instrument. Grading of Recommendations, Assessment, Development, and Evaluations will be used to assess the strength of the body of evidence. Meta-analyses of association measures will be performed for each type of motor performance, separately for different study types. The results will be reported using forest to show the pooled effect of findings and funnel plots to assess precision of the data. If studies are not homogeneous, results from the meta-analyses will not be reported. Associations will then be synthesized descriptively using a pragmatic approach.

DISCUSSION

This systematic review will provide critical insights into the association between motor performance and back pain among children and adolescents; this information may help support clinical practice guidelines as well as public health programs.

ETHICS AND DISSEMINATION

Protocol was written according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA).

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42020178496.

Reporting of conflicts of interest by authors of primary studies on health policy and systems research: a cross-sectional survey

OBJECTIVES

The objective of this study was to assess the frequency and types of conflict of interest (COI) disclosed by authors of primary studies of health policy and systems research (HPSR).

DESIGN

We conducted a cross-sectional survey using standard systematic review methodology for study selection and data extraction. We conducted descriptive analyses.

SETTING

We collected data from papers published in 2016 in 'health policy and service journals' category in Web of Science database.

PARTICIPANTS

We included primary studies (eg, randomised controlled trials, cohort studies, qualitative studies) of HPSR published in English in 2016 peer-reviewed health policy and services journals.

OUTCOME MEASURES

Reported COI disclosures including whether authors reported COI or not, form in which COI disclosures were provided, number of authors per paper who report any type of COI, number of authors per paper who report specific types and subtypes of COI.

RESULTS

We included 200 eligible primary studies of which 132 (66%) included COI disclosure statements of authors. Of the 132 studies, 19 (14%) had at least one author reporting at least one type of COI and the most frequently reported type was individual financial COI (n=15, 11%). None of the authors reported individual intellectual COIs or personal COIs. Financial and individual COIs were reported more frequently compared with non-financial and institutional COIs.

CONCLUSION

A low percentage of HPSR primary studies included authors reporting COI. Non-financial or institutional COIs were the least reported types of COI.

Randomised controlled trials evaluating endometrial scratching: assessment of methodological issues

STUDY QUESTION

Do randomised controlled trials (RCTs) evaluating endometrial scratching suffer from methodological issues including insufficient trial registration, statistical errors or irreproducibility, randomisation errors or miscellaneous issues?

SUMMARY ANSWER

The majority of RCTs investigating endometrial scratching have methodological issues.

WHAT IS KNOWN ALREADY

A large number of small RCTs investigating the effectiveness of endometrial scratching prior to in vitro fertilisation (IVF) and intrauterine insemination (IUI)/intercourse have reported favourable findings. Subsequently, systematic reviews incorporating these RCTs yielded meta-analyses in favour of endometrial scratching. Endometrial scratching has been widely adopted by infertility specialists around the world. Recently, an international RCT including 1364 women reported no benefit from endometrial scratching before IVF.

STUDY DESIGN, SIZE, DURATION

We evaluated several methodological issues of RCTs investigating the effectiveness of endometrial scratching prior to IVF and IUI/intercourse. We identified 25 RCTs for IVF and 12 RCTs for IUI/intercourse with full-text publication.

PARTICIPANTS/MATERIALS, SETTING, METHODS

We assessed the RCTs on the following criteria: adequacy of trial registration, statistical issues (description of statistical methods and reproducibility of univariable statistical analysis), excessive similarity or difference in baseline characteristics that is not compatible with chance (Monte Carlo simulations and Kolmogorov–Smirnov test) and miscellaneous methodological issues.

MAIN RESULTS AND THE ROLE OF CHANCE

Of 25 RCTs evaluating endometrial scratching prior to IVF, only eight (32%) had adequate trial registration. In total, 10 (40%) RCTs had issues regarding statistical methods. Nine (69%, 13 applicable) RCTs had at least one inconsistency between reported and reproduced univariable statistical analysis for categorical baseline/intermediate characteristics. Statistical results of at least one outcome were not reproducible in 14 (74%, 19 applicable) RCTs. Only two (8%) RCTs had none of the above issues. Suggested by the simulations, these RCTs did not significantly violate the null hypothesis that the baseline characteristics were the results of a properly conducted randomisation process (P = 0.4395).

Of 12 IUI/intercourse RCTs, only 2 (17%) had adequate trial registration. In total, five (42%) studies had issues of statistical methods. Inconsistency between reported and reproduced univariable analysis for baseline/intermediate categorical variable(s) was found in four (57%, 7 applicable) RCTs. Statistical analysis was not reproducible for at least one outcome in eight (80%, 10 applicable) studies. All RCTs had at least one of the above issues. These RCTs were inconsistent with the null hypothesis that their baseline characteristics were the results of proper randomised allocation (P = 1.659*10−7).

LIMITATIONS, REASONS FOR CAUTION

We were unable to assess RCTs which were not published as full-text papers. We could not analyse individual participant data to investigate possible reasons for statistical inconsistencies. The method to infer the likelihood of proper random sampling rests on assumptions including independent baseline characteristics, simple randomisation and no publication bias.

WIDER IMPLICATIONS OF THE FINDINGS

The methodological issues common to RCTs evaluating endometrial scratching may have biased the results of the trials. Further development and validation of these novel methods may be helpful for the critical appraisal of RCTs.

STUDY FUNDING/COMPETING INTEREST(S)

No external funding was sought to support this work. B.W.M. is supported by a National Health Medical Research Council (NHMRC) Practitioner Fellowship (GNT1082548). B.W.M. reports consultancy for ObsEva, Merck and Guerbet. D.W. is supported by a grant from the Paracelsus Medical University Salzburg, Austria (PMU Research Fund—PMU FFF Number: L-18/02/006-WET) and by Drs Haackert Foundation, Germany. S.L. is an author of a trial included in this study, an author of an included systematic review and a Cochrane editor. All other authors have no conflicts of interest.

TRIAL REGISTRATION NUMBER

N/A

Contacting of authors modified crucial outcomes of systematic reviews but was poorly reported, not systematic, and produced conflicting results

OBJECTIVES

The objective of the study was to assess the prevalence, the reporting quality, the need, and the consequences of contacting of authors by Cochrane reviewers to obtain additional information for their reviews.

STUDY DESIGN AND SETTING

Cross-sectional study and survey on all new Cochrane intervention reviews published between January 1, 2016 and January 31, 2017.

RESULTS

The cross-sectional study found that reviewers had contacted or had tried to contact studies to obtain additional information in 73.4% (234/319) of reviews but reported poorly on the methods, outcomes, and consequences of this procedure. Most eligible studies in the reviews were poorly reported, but few reviewers 21.2% (65/306) reported that they had contacted these studies. The survey showed that risk of bias scores, Grading of Recommendations, Assessment, Development and Evaluation scores, the summary primary or secondary outcomes, and the summary effect size of the primary outcome of the review were changed as a consequence of contacting of authors. Thirty-five of one hundred and thirty (26.9%) reviews scored opposite outcomes for the same question in the cross-sectional study compared with the survey.

CONCLUSIONS

Our findings on contacting of authors by Cochrane reviewers showed relevant shortcomings in the current standards and transparency of Cochrane reviews. These shortcomings can compromise the validity and reproducibility of these reviews and affect a wide audience.

Correctable Myths About Research Misconduct in the Biomedical Sciences

A recent National Academy report on research integrity noted that policies are not evidence-based, with no formal entity responsible to attend to this deficit. Here we describe four areas of research misconduct (RM) regulations governing Public Health Service funded research that are empirically and/or ethically questionable. Policies for human subject protection, RM and conflict of interest are not harmonized, making it extremely difficult to deal with complex cases which often contain allegations in all of these areas. Second, detection of RM has depended entirely on whistleblowers in spite of evidence of significant under-reporting. Third, the scientific record is far from cleansed of the effects of falsified/fabricated work through current mechanisms of retraction. Finally, lack of fairness in the regulations may reflect lack of a Belmont Report-like document to guide ethics of RM policy. These issues are likely common in other countries. RM regulations should be harmonized with related regulations and their effectiveness tracked, open access to data for independent replication and improved statistical tests are an essential supplement to whistleblowers, correction of the scientific record will require a major effort, and further ethical analysis and guidance are as important as is empirical study for the improvement of RM regulations. Further consideration should be given to assigning current regulations for human subjects protection, RM and conflict of interest to a single authority and to the further development of a Belmont-like report of essential principles, for RM.

Reporting of conflicts of interest and of sponsorship of guidelines in anaesthesiology. A cross-sectional study

Guideline recommendations may be biased due to conflicts of interest (COI) of panel members and sponsorship of the guideline. Potential impact of COI, and their management, should be transparently reported. We analysed 110 guidelines published in ten anaesthesia journals from 2007 to June 2018. We report on the number (%) that 1) published COI disclosures; 2) in a distinct paragraph; 3) described and explained the COI of panel members, and 4) of the Chairperson; 5) reported and described the presence or absence and potential impact of a sponsor of the guideline on the recommendations; and 6) reported how COI were managed. COI were published in 70/110 (64%) guidelines; in a distinct paragraph in 25/70 (36%). Panel members reported having no COI in 27/70 (39%) guidelines, disclosed COI without describing their potential impact in 41/70 (59%), and described their potential impact in 2/70 (3%). Chairpersons were identified in 50 guidelines, 32 of which published COI disclosures; 16/32 (50%) reported having no COI, 14/32 (44%) disclosed COI without describing their potential impact, 1/32 (3%) described their impact and 1/32 (3%) made no statement regarding COI. Presence or absence of a sponsor of the guideline was reported in 40 guidelines; 12/40 (30%) declared none, 24/40 (60%) reported sponsoring without explanation of the potential impact, and 4/40 (10%) described the potential influence of the sponsor on the guideline recommendations. Seventy-five guidelines reported COI of panel members and/or sponsorship of the guideline but only seven described how the COI had been managed. Disclosures of COI of panel members and of sponsors of guidelines have increased over the 12 year period, but remain insufficiently described and their potential influence on the guidelines' recommendations is poorly documented.

Association between polymorphisms in the genes encoding toll-like receptors and dectin-1 and susceptibility to invasive aspergillosis: a systematic review

Invasive aspergillosis is a common fungal infection in immunocompromised individuals. Some studies have shown that toll-like receptor and dectin-1 genetic polymorphisms may alter signaling pathways, thus increasing an individual's susceptibility to invasive aspergillosis. We investigated the pertinent literature to determine whether polymorphisms in the genes encoding toll-like receptors and dectin-1 increase the susceptibility to invasive aspergillosis. This study systematically reviewed the literature using the databases PubMed/PMC, Scopus, and Web of Science using the keywords invasive aspergillosis, polymorphism, Toll-like, and Dectin-1. From the initial search, 415 studies were found and according to our inclusion and exclusion criteria, eight studies were selected. Several studies described single-nucleotide polymorphisms (SNPs) that are associated with a greater susceptibility to invasive aspergillosis. These SNPs were found in the genes that encode toll-like receptors 1, 3, 4, and 5 and the gene that encodes dectin-1; upon activation, both cellular receptors initiate a signaling cascade that can result in the production of cytokines and chemokines. Thus, our literature review uncovered a significant association between polymorphisms in the genes that encode toll-like receptors and dectin-1 and invasive aspergillosis. More studies should be performed to better understand the relationship between toll-like receptor and dectin-1 genetic polymorphisms and invasive aspergillosis susceptibility.

The FTO rs9939609 polymorphism and obesity risk in teens: Evidence-based meta-analysis

OBJECTIVE

This meta-analysis aims evaluate the association of the polymorphism rs9939609 of FTO with the risk of obesity among children and adolescents, based on the assessment of four genetic models: codominant, dominant, recessive alleles model.

METHODS

Case-control studies, published between the years 2011-2015, were selected from tree available databases (PubMed, Scopus and Web of Science) and were analysed using Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) recommendations. Thirteen studies were included totalling 15,613 participants, divided into 7311 cases and 8302 controls.

RESULTS

The FTO rs9939609 polymorphism was significantly associated with increased risk of obesity in children and adolescents for homozygous genotypes AA and heterozygous AT (TT vs. AT+AA: OR=0.723, 95% CI 0.629 to 0.832; p<0.0001).

CONCLUSION

This meta-analysis shows that the FTO rs9939609 polymorphism in the gene is a risk factor for obesity in children and adolescents with the presence of the A allele, both homozygous genotype AA situation, as heterozygous AT.

Effects of Real-Time (Sonification) and Rhythmic Auditory Stimuli on Recovering Arm Function Post Stroke: A Systematic Review and Meta-Analysis

Background: External auditory stimuli have been widely used for recovering arm function post-stroke. Rhythmic and real-time auditory stimuli have been reported to enhance motor recovery by facilitating perceptuomotor representation, cross-modal processing, and neural plasticity. However, a consensus as to their influence for recovering arm function post-stroke is still warranted because of high variability noted in research methods. Objective: A systematic review and meta-analysis was carried out to analyze the effects of rhythmic and real-time auditory stimuli on arm recovery post stroke. Method: Systematic identification of published literature was performed according to PRISMA guidelines, from inception until December 2017, on online databases: Web of science, PEDro, EBSCO, MEDLINE, Cochrane, EMBASE, and PROQUEST. Studies were critically appraised using PEDro scale. Results: Of 1,889 records, 23 studies which involved 585 (226 females/359 males) patients met our inclusion criteria. The meta-analysis revealed beneficial effects of training with both types of auditory inputs for Fugl-Meyer assessment (Hedge's g: 0.79), Stroke impact scale (0.95), elbow range of motion (0.37), and reduction in wolf motor function time test (-0.55). Upon further comparison, a beneficial effect of real-time auditory feedback was found over rhythmic auditory cueing for Fugl-meyer assessment (1.3 as compared to 0.6). Moreover, the findings suggest a training dosage of 30 min to 1 h for at least 3-5 sessions per week with either of the auditory stimuli. Conclusion: This review suggests the application of external auditory stimuli for recovering arm functioning post-stroke.

Bony reconstruction of hip in cerebral palsy children Gross Motor Function Classification System levels III to V: a systematic review

Hip dislocation is a common source of disability in cerebral palsy children. It has been remedied by various reconstructive procedures. This review aims at providing the best evidence for bony reconstructive procedures in cerebral palsy hip migration. The literature extraction process yielded 36 articles for inclusion in this review. There is fair evidence to indicate that the comparative effectiveness of femoral versus combined pelvifemoral reconstruction favours pelvifemoral reconstruction. All except one were retrospective articles with a significant degree of selection and performance bias and confounding variables that limited the validity and generalizability of the conclusions. The findings of this systematic review provide fair evidence for the use of adequate soft tissue and combined pelvifemoral reconstruction in the management of hip migration in none and minimally ambulatory cerebral palsy children in the short and long term. This has been shown in studies with a summed sizable patient population. There is limited evidence available that would support the use of soft-tissue and isolated femoral reconstruction. In the context of these retrospective and biased studies, it is extremely difficult to identify, with great precision, predictors of surgical success. Future studies should consider prospective designs that allow for bias control, strict patient selection criteria and incorporation of validated quality-of-life scales.

Examining publication bias-a simulation-based evaluation of statistical tests on publication bias

Background

Publication bias is a form of scientific misconduct. It threatens the validity of research results and the credibility of science. Although several tests on publication bias exist, no in-depth evaluations are available that examine which test performs best for different research settings.

Methods

Four tests on publication bias, Egger’s test (FAT), p-uniform, the test of excess significance (TES), as well as the caliper test, were evaluated in a Monte Carlo simulation. Two different types of publication bias and its degree (0%, 50%, 100%) were simulated. The type of publication bias was defined either as file-drawer, meaning the repeated analysis of new datasets, or p-hacking, meaning the inclusion of covariates in order to obtain a significant result. In addition, the underlying effect (β = 0, 0.5, 1, 1.5), effect heterogeneity, the number of observations in the simulated primary studies (N = 100, 500), and the number of observations for the publication bias tests (K = 100, 1,000) were varied.

Results

All tests evaluated were able to identify publication bias both in the file-drawer and p-hacking condition. The false positive rates were, with the exception of the 15%- and 20%-caliper test, unbiased. The FAT had the largest statistical power in the file-drawer conditions, whereas under p-hacking the TES was, except under effect heterogeneity, slightly better. The CTs were, however, inferior to the other tests under effect homogeneity and had a decent statistical power only in conditions with 1,000 primary studies.

Discussion

The FAT is recommended as a test for publication bias in standard meta-analyses with no or only small effect heterogeneity. If two-sided publication bias is suspected as well as under p-hacking the TES is the first alternative to the FAT. The 5%-caliper test is recommended under conditions of effect heterogeneity and a large number of primary studies, which may be found if publication bias is examined in a discipline-wide setting when primary studies cover different research problems.

Comparative efficacy and safety of second-line treatments for advanced non-small cell lung cancer with wild-type or unknown status for epidermal growth factor receptor: a systematic review and network meta-analysis

BACKGROUND

Docetaxel, pemetrexed, erlotinib, and gefitinib are recommended as second-line treatment for advanced non-small cell lung cancer (NSCLC) with wild-type or unknown status for epidermal growth factor receptor (EGFR). However, the number of published randomized clinical trials (RCTs) on this topic is increasing. Our objective was to assess the comparative effectiveness and tolerability of all second-line treatments for advanced NSCLC with wild-type or unknown status for EGFR by a systematic review and network meta-analysis.

METHODS

MEDLINE, EMBASE, CENTRAL, ClinicalTrials.gov, and the US Food and Drug Administration website, as well as other sources, were searched for available reports up to June 6, 2017. Two reviewers independently selected published and unpublished reports of RCTs comparing any second-line treatments, extracted data and assessed the risk of bias of all included trials. We performed a Bayesian network meta-analysis. The primary outcomes were overall survival (OS) and progression-free survival (PFS). Secondary outcomes included objective response (ObR), the number of serious adverse events, and quality of life.

RESULTS

We included 102 RCTs involving 36,058 patients (62% male, median age 61 years, 81% with stage IV cancer, 80% smokers, and 92% with performance status 0-1). We revealed a differential reporting of outcomes between efficacy and safety outcomes. Half of the trials reported safety outcomes and less than 20% quality of life. For OS, nivolumab was more effective than docetaxel (hazard ratio (HR) 0.69, 95% credible interval (CrI) 0.56-0.83), pemetrexed (0.67, 0.52-0.83), erlotinib (0.68, 0.53-0.86), and gefitinib (0.66, 0.53-0.83). Pembrolizumab, atezolizumab, and pemetrexed plus erlotinib were also significantly more effective than docetaxel, pemetrexed, erlotinib, and gefitinib. For PFS, erlotinib plus cabozantinib was more effective than docetaxel (HR 0.39, 95% CrI 0.18-0.84), pemetrexed (0.38, 0.18-0.82), erlotinib (0.37, 0.18-0.78), and gefitinib (0.38, 0.18-0.82). Cabozantinib and pemetrexed plus erlotinib were also significantly more effective than the four recommended treatments. For ObR, no treatment was significantly more effective. The effectiveness of the four recommended treatments was similar and they were ranked among the 25 less-effective treatments. For safety, evidence is insufficient to draw certain conclusions.

CONCLUSIONS

Nivolumab, pembrolizumab, atezolizumab, and pemetrexed plus erlotinib may be the most effective second-line treatments for NSCLC in terms of OS. The four recommended treatments seem to have relatively poor performance. However, the impact on life expectancy of immunotherapy versus other treatments should be further explored by future analyses, and more trials comparing the novel treatments are needed to reduce uncertainty in these results.

TRIAL REGISTRATION

Registration number: PROSPERO ( CRD42015017592 ).

Characteristics of funding of clinical trials: cross-sectional survey and proposed guidance

OBJECTIVES

To provide a detailed and current characterisation of funding of a representative sample clinical trials. We also aimed to develop guidance for standardised reporting of funding information.

METHODS

We addressed the extent to which clinical trials published in 2015 in any of the 119 Core Clinical Journals included a statement on the funding source (eg, whether a not-for-profit organisation was supported by a private-for-profit organisation), type of funding, amount and role of funder. We used a stepwise approach to develop a guidance and an instrument for standardised reporting of funding information.

RESULTS

Of 200 trials, 178 (89%) included a funding statement, of which 171 (96%) reported being funded. Funding statements in the 171 funded trials indicated the source in 100%, amount in 1% and roles of funders in 50%. The most frequent sources were governmental (58%) and private-for-profit (40%). Of 54 funding statements in which the source was a not-for-profit organisation, we found evidence of undisclosed support of those from private-for-profit organisation(s) in 26 (48%). The most frequently reported roles of funders in the 171 funded trials related to study design (42%) and data analysis, interpretation or management (41%). Of 139 randomised controlled trials (RCTs) addressing pharmacological or surgical interventions, 29 (21%) reported information on the supplier of the medication or device. The proposed guidance addresses both the funding information that RCTs should report and the reporting process. Attached to the guidance is a fillable PDF document for use as an instrument for standardised reporting of funding information.

CONCLUSION

Although the majority of RCTs report funding, there is considerable variability in the reporting of funding source, amount and roles of funders. A standardised approach to reporting of funding information would address these limitations. Future research should explore the implications of funding by not-for-profit organisations that are supported by for-profit organisations.

Synthesizing and Appraising the Quality of the Evidence on Factors Associated with Medication Adherence in Diabetes: A Systematic Review of Systematic Reviews

BACKGROUND

Nonadherence to medications is a common phenomenon in patients with diabetes. Several studies and systematic reviews have investigated the barriers to medication adherence in diabetes. However, no study has evaluated the quality of the existing literature and synthesized the plethora of evidence with a goal to design holistic conceptual frameworks and interventions.

OBJECTIVES

The aims of this review were to systematically evaluate existing systematic reviews focusing on factors associated with medication adherence in diabetes in an effort to synthesize the evidence, determine their methodological quality, and identify the gaps in the current literature.

METHODS

Fourteen databases and gray literature sources were systematically searched through June 2016. Systematic reviews reporting factors associated with medication adherence (barriers and facilitators) in patients with diabetes were selected on the basis of predetermined criteria. Studies were appraised for quality using AMSTAR (A Measurement Tool to Assess Systematic Reviews).

RESULTS

Seventeen systematic reviews including 542 primary studies, most of which were cross-sectional quantitative studies, were included. All the reviews were rated as moderate to low quality and exhibited common methodological pitfalls. Factors influencing medication adherence identified were categorized as patient-, medication-, disease-, health care provider-, health care system-, and social-related factors.

CONCLUSIONS

Factors influencing medication adherence are multifactorial with remarkably consistent findings across the existing reviews; yet, most reviews were judged to be of low to moderate quality. Further comprehensive and well-conducted original studies and systematic reviews on this topic shall be conducted taking into consideration the drawbacks of existing ones.

The influence of the team in conducting a systematic review

There is an increasing body of research documenting flaws in many published systematic reviews' methodological and reporting conduct. When good systematic review practice is questioned, attention is rarely turned to the composition of the team that conducted the systematic review. This commentary highlights a number of relevant articles indicating how the composition of the review team could jeopardise the integrity of the systematic review study and its conclusions. Key biases require closer attention such as sponsorship bias and researcher allegiance, but there may also be less obvious affiliations in teams conducting secondary evidence-syntheses. The importance of transparency and disclosure are now firmly on the agenda for clinical trials and primary research, but the meta-biases that systematic reviews may be at risk from now require further scrutiny.

Determinants of eating patterns and nutrient intake among adolescent athletes: a systematic review

Background

This review aims to update the influences of sport modalities, sport performance, and non-exercise-related determinant, on eating patterns and nutrient intake outcomes among adolescent competitive athletes.

Methods

The PubMed and Scopus databases were searched for the last 20 years. Observational and intervention studies of all languages on eating patterns and nutrient intake in adolescent (10- to 19-year-old) competitive athletes were included. Study quality and risk of bias were assessed using a Downs and Black instrument. Moreover, the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) evidence system was used to assess the strength of the body of evidence.

Results

Regarding outcomes of 21 included articles, 95.2% of studies focused on nutrient analysis, whereas few of the included articles reported eating patterns. As determinants, most studies analyzed the influences of sport-related (n = 10) and demographic factors (n = 8), among which only sport modalities were significantly associated with nutrient intake. Age and sex were not significantly associated with nutrient intake. All studies were observational, and most were cross-sectional (n = 17) and conducted in developed countries such as the United States and European nations. Most studies evaluated boys, and the sport that received the most attention was soccer.

Conclusions

Athletes do not adjust their nutrient intake or food choice to the demands of the training load or different training sessions, while sport modalities significantly influenced nutrient intake. Moreover, results concerning demographic determinants were inconsistent and thus remain inconclusive.

Trial registration

Prospero CRD42016043310.

Electronic supplementary material

The online version of this article (doi:10.1186/s12937-017-0267-0) contains supplementary material, which is available to authorized users.

Sexual dysfunction in infertile women: A systematic review and meta-analysis

OBJECTIVE

This study aimed to assess the prevalence of sexual dysfunction and Female Sexual Function Index (FSFI) score in women with infertility.

STUDY DESIGN

A systematic search of the literature was conducted using PubMed, EMBASE, IBECS, and LILACS. The search was limited to articles published from January 2000 to September 2016, without language restriction. Data were analyzed using Stata 12.0. Random effects meta-analyses in weighted mean difference (WMD) were performed for six comparative studies (infertility versus fertility). Heterogeneity was estimated using I². Moreover, to explore the heterogeneity sources among the studies, meta-regression analyses were also performed. Quality of evidence was assessed using the Grading of Recommendations Assessment, Development, and Evaluation guidelines, and risk of bias, with a graphic funnel.

RESULTS

Meta-analysis was performed in 11 of 13 comparative studies. The result indicated a significant association between an increase in sexual dysfunction and infertility in women (WMD=-0.16, 95% confidence interval=-0.254 to -0.084, p<0.001), and high heterogeneity between studies was noted (I²=98.6%, p<0.000). Meta-regression analysis did not indicate heterogeneity (I²=0.00%). We also performed a meta-analysis of individual FSFI domains in 10 studies. Infertile women had problems with lubrication, orgasm, and satisfaction. Meta-regression analysis also showed that heterogeneity had no influence on the final results of all the analyses.

CONCLUSIONS

Infertility was associated with an increase in female sexual dysfunction. The most affected areas of sexual function were lubrication, orgasm, and satisfaction.

Cochrane: the unfinished symphony of research synthesis

The NHS needs valid information on the safety and effectiveness of healthcare interventions. Cochrane systematic reviews are an important source of this information. Traditionally, Cochrane has attempted to identify and include all relevant trials in systematic reviews on the basis that if all trials are identified and included, there should be no selection bias. However, a predictable consequence of the drive to include all trials is that some studies are included that are not trials (false positives). Including such studies in reviews might increase bias. More effort is needed to authenticate trials to be included in reviews, but this task is bedevilled by the enormous increase in the number of 'trials' conducted each year. We argue that excluding small trials from reviews would release resources for more detailed appraisal of larger trials. Conducting fewer but broader reviews that contain fewer but properly validated trials might better serve patients' interests.