Retrieving randomized controlled trials from medline: a comparison of 38 published search filters

Probiotics interventions modulating gut microbiota composition in individuals with intestinal constipation: Protocol of a systemic review and meta-analysis of randomized controlled trials

INTRODUCTION

Intestinal constipation is a substantive global health concern, significantly impairing patient quality of life. An emerging view is that the gut microbiota plays a critical role in intestinal function, and probiotics could offer therapeutic benefits. This study aims to consolidate evidence from randomized controlled trials (RCTs) that assess the effectiveness of probiotics in modulating microbiota and ameliorating symptoms of constipation.

METHODS

We will execute a systematic evidence search across Medline (via PubMed), Embase, Cochrane CENTRAL, Web of Science, Scopus, and CINAHL, employing explicit search terms and further reference exploration. Two independent reviewers will ensure study selection and data integrity while assessing methodological quality via the Cochrane Collaboration's Risk of Bias-2 tool. Our primary goal is to outline changes in microbiota composition, with secondary outcomes addressing symptom relief and stool characteristics. Meta-analyses will adopt a random-effects model to quantify the effects of interventions, supplemented by subgroup analyses and publication bias assessments to fortify the rigor of our findings.

DISCUSSION

This study endeavors to provide a rigorous, synthesized overview of the probiotics interventions evidence for modulating gut microbiota in individuals with intestinal constipation. The insights derived could inform clinical guidelines, nurture the creation of novel constipation management strategies, and direct future research in this field.

ETHICS AND DISSEMINATION

As this study aggregates and analyzes existing data without direct human subject involvement, no ethical approval is required. We will disseminate the study's findings through scientific forums and seek publication in well-regarded, peer-reviewed journals.

TRIAL REGISTRATION

OSF registration number: 10.17605/OSF.IO/MEAHT.

Prophylactic ablation during cardiac surgery in patients without atrial fibrillation: a systematic review and meta-analysis of randomized trials

OBJECTIVES

Atrial fibrillation is the most common complication of cardiac surgery and occurs frequently in patients without a history of the arrhythmia. We conducted a systematic review and meta-analysis of randomized controlled trials to assess whether prophylactic ablation during cardiac surgery in patients without a history of atrial fibrillation prevents atrial fibrillation.

METHODS

We searched CENTRAL, MEDLINE and Embase from inception to August 2024. We included randomized trials of adults without a history of atrial fibrillation undergoing cardiac surgery. The intervention of interest was ablation during surgery. We pooled data using random-effects models. The primary outcome was new-onset early postoperative atrial fibrillation within 30 days following surgery. The key secondary outcome was incident clinical atrial fibrillation at follow-up (minimum 6 months). We assessed risk of bias using the Cochrane Collaboration's risk of bias tool v.2 and evidence quality using Grading of Recommendations, Assessment, Development and Evaluation (GRADE).

RESULTS

We included 7 trials (n = 687). The intervention was pulmonary vein isolation in 6 trials and ganglion plexi ablation in 1. Patients who received prophylactic ablation were less likely to have early postoperative atrial fibrillation (21% vs 37%, risk ratio [RR] 0.5, 95% confidence interval 0.3-0.8, I2 = 64%) and incident clinical atrial fibrillation at longest follow-up (range 6 months-2 years; 3% vs 10%, RR 0.3, 95% confidence interval 0.2-0.7, I2 = 0%). The quality of evidence was low.

CONCLUSIONS

Prophylactic ablation during cardiac surgery may prevent atrial fibrillation in patients without a history of the arrhythmia. A definitive randomized trial is needed to confirm effects and safety.

Ketamine versus electroconvulsive therapy for major depressive episode: An updated systematic review and non-inferiority meta-analysis

We conducted a systematic review and meta-analysis to investigate the comparative effectiveness of ketamine versus electroconvulsive therapy (ECT) for the treatment of major depressive episodes (MDEs). PubMed, EMBASE and Cochrane Library databases were systematically searched for randomized controlled trials (RCTs) comparing ketamine and ECT for MDE. The primary outcome was response rate, for which we prespecified a non-inferiority margin of -0.1, based on the largest and most recent RCT. Response was defined as a reduction of at least 50 % in the depression scale score. Six RCTs met the inclusion criteria, comprising 655 patients. In the overall population, ketamine was not non-inferior to ECT in response rate (RD -0.10; 95 % CI -0.26 to 0.05; p = 0.198; I2 = 72 %). The ECT group had a higher reduction in depression scores, but without difference in remission and relapse rates. Regarding safety outcomes, ketamine had better posttreatment cognition scores and reduced muscle pain rate compared with ECT, albeit with an increased rate of dissociative symptoms. In a subanalysis with only inpatients, ketamine was inferior to ECT in response rate (RD -0.15; 95 % CI -0.27 to -0.03; p = 0.014; I2 = 25 %), remission, and change in depression scores. These findings support the use of ECT over ketamine for inpatients. Further RCTs are warranted to clarify the comparative effect of these treatments for outpatients.

Direct Oral Anticoagulants for Stroke Prevention in Patients With Device-Detected Atrial Fibrillation: A Study-Level Meta-Analysis of the NOAH-AFNET 6 and ARTESiA Trials

BACKGROUND

Device-detected atrial fibrillation (also known as subclinical atrial fibrillation or atrial high-rate episodes) is a common finding in patients with an implanted cardiac rhythm device and is associated with an increased risk of ischemic stroke. Whether oral anticoagulation is effective and safe in this patient population is unclear.

METHODS

We performed a systematic review of MEDLINE and Embase for randomized trials comparing oral anticoagulation with antiplatelet or no antithrombotic therapy in adults with device-detected atrial fibrillation recorded by a pacemaker, implantable cardioverter defibrillator, cardiac resynchronization therapy device, or implanted cardiac monitor. We used random-effects models for meta-analysis and rated the quality of evidence using the Grading of Recommendations Assessment, Development and Evaluation framework (GRADE). The review was preregistered (PROSPERO CRD42023463212).

RESULTS

From 785 citations, we identified 2 randomized trials with relevant clinical outcome data: NOAH-AFNET 6 (Non-Vitamin K Antagonist Oral Anticoagulants in Patients With Atrial High Rate Episodes; 2536 participants) evaluated edoxaban, and ARTESiA (Apixaban for the Reduction of Thrombo-Embolism in Patients With Device-Detected Sub-Clinical Atrial Fibrillation; 4012 participants) evaluated apixaban. Meta-analysis demonstrated that oral anticoagulation with these agents reduced ischemic stroke (relative risk [RR], 0.68 [95% CI, 0.50-0.92]; high-quality evidence). The results from the 2 trials were consistent (I2 statistic for heterogeneity=0%). Oral anticoagulation also reduced a composite of cardiovascular death, all-cause stroke, peripheral arterial embolism, myocardial infarction, or pulmonary embolism (RR, 0.85 [95% CI, 0.73-0.99]; I2=0%; moderate-quality evidence). There was no reduction in cardiovascular death (RR, 0.95 [95% CI, 0.76-1.17]; I2=0%; moderate-quality evidence) or all-cause mortality (RR, 1.08 [95% CI, 0.96-1.21]; I2=0%; moderate-quality evidence). Oral anticoagulation increased major bleeding (RR, 1.62 [95% CI, 1.05-2.50]; I²=61%; high-quality evidence).

CONCLUSIONS

The results of the NOAH-AFNET 6 and ARTESiA trials are consistent with each other. Meta-analysis of these 2 large randomized trials provides high-quality evidence that oral anticoagulation with edoxaban or apixaban reduces the risk of stroke in patients with device-detected atrial fibrillation and increases the risk of major bleeding.

Statistical significance and publication reporting bias in abstracts of reproductive medicine studies

STUDY QUESTION

What were the frequency and temporal trends of reporting P-values and effect measures in the abstracts of reproductive medicine studies in 1990-2022, how were reported P-values distributed, and what proportion of articles that present with statistical inference reported statistically significant results, i.e. 'positive' results?

SUMMARY ANSWER

Around one in six abstracts reported P-values alone without effect measures, while the prevalence of effect measures, whether reported alone or accompanied by P-values, has been increasing, especially in meta-analyses and randomized controlled trials (RCTs); the reported P-values were frequently observed around certain cut-off values, notably at 0.001, 0.01, or 0.05, and among abstracts present with statistical inference (i.e. P-value, CIs, or significant terms), a large majority (77%) reported at least one statistically significant finding.

WHAT IS KNOWN ALREADY

Publishing or reporting only results that show a 'positive' finding causes bias in evaluating interventions and risk factors and may incur adverse health outcomes for patients.

Despite efforts to minimize publication reporting bias in medical research, it remains unclear whether the magnitude and patterns of the bias have changed over time.

STUDY DESIGN, SIZE, DURATION

We studied abstracts of reproductive medicine studies from 1990 to 2022. The reproductive medicine studies were published in 23 first-quartile journals under the category of Obstetrics and Gynaecology and Reproductive Biology in Journal Citation Reports and 5 high-impact general medical journals (The Journal of the American Medical Association, The Lancet, The BMJ, The New England Journal of Medicine, and PLoS Medicine). Articles without abstracts, animal studies, and non-research articles, such as case reports or guidelines, were excluded.

PARTICIPANTS/MATERIALS, SETTING, METHODS

Automated text-mining was used to extract three types of statistical significance reporting, including P-values, CIs, and text description. Meanwhile, abstracts were text-mined for the presence of effect size metrics and Bayes factors. Five hundred abstracts were randomly selected and manually checked for the accuracy of automatic text extraction. The extracted statistical significance information was then analysed for temporal trends and distribution in general as well as in subgroups of study designs and journals.

MAIN RESULTS AND THE ROLE OF CHANCE

A total of 24 907 eligible reproductive medicine articles were identified from 170 739 screened articles published in 28 journals. The proportion of abstracts not reporting any statistical significance inference halved from 81% (95% CI, 76-84%) in 1990 to 40% (95% CI, 38-44%) in 2021, while reporting P-values alone remained relatively stable, at 15% (95% CI, 12-18%) in 1990 and 19% (95% CI, 16-22%) in 2021. By contrast, the proportion of abstracts reporting effect measures alone increased considerably from 4.1% (95% CI, 2.6-6.3%) in 1990 to 26% (95% CI, 23-29%) in 2021. Similarly, the proportion of abstracts reporting effect measures together with P-values showed substantial growth from 0.8% (95% CI, 0.3-2.2%) to 14% (95% CI, 12-17%) during the same timeframe. Of 30 182 statistical significance inferences, 56% (n = 17 077) conveyed statistical inferences via P-values alone, 30% (n = 8945) via text description alone such as significant or non-significant, 9.3% (n = 2820) via CIs alone, and 4.7% (n = 1340) via both CI and P-values. The reported P-values (n = 18 417), including both a continuum of P-values and dichotomized P-values, were frequently observed around common cut-off values such as 0.001 (20%), 0.05 (16%), and 0.01 (10%). Of the 13 200 reproductive medicine abstracts containing at least one statistical inference, 77% of abstracts made at least one statistically significant statement. Among articles that reported statistical inference, a decline in the proportion of making at least one statistically significant inference was only seen in RCTs, dropping from 71% (95% CI, 48-88%) in 1990 to 59% (95% CI, 42-73%) in 2021, whereas the proportion in the rest of study types remained almost constant over the years. Of abstracts that reported P-value, 87% (95% CI, 86-88%) reported at least one statistically significant P-value; it was 92% (95% CI, 82-97%) in 1990 and reached its peak at 97% (95% CI, 93-99%) in 2001 before declining to 81% (95% CI, 76-85%) in 2021.

LIMITATIONS, REASONS FOR CAUTION

First, our analysis focused solely on reporting patterns in abstracts but not full-text papers; however, in principle, abstracts should include condensed impartial information and avoid selective reporting. Second, while we attempted to identify all types of statistical significance reporting, our text mining was not flawless. However, the manual assessment showed that inaccuracies were not frequent.

WIDER IMPLICATIONS OF THE FINDINGS

There is a welcome trend that effect measures are increasingly reported in the abstracts of reproductive medicine studies, specifically in RCTs and meta-analyses. Publication reporting bias remains a major concern. Inflated estimates of interventions and risk factors could harm decisions built upon biased evidence, including clinical recommendations and planning of future research.

STUDY FUNDING/COMPETING INTEREST(S)

No funding was received for this study. B.W.M. is supported by an NHMRC Investigator grant (GNT1176437); B.W.M. reports research grants and travel support from Merck and consultancy from Merch and ObsEva. W.L. is supported by an NHMRC Investigator Grant (GNT2016729). Q.F. reports receiving a PhD scholarship from Merck. The other author has no conflict of interest to declare.

TRIAL REGISTRATION NUMBER

N/A.

Role of lipoprotein(a) in atherosclerotic cardiovascular disease: A review of current and emerging therapies

Lipoprotein(a), or Lp(a), is structurally like low-density lipoprotein (LDL) but differs in that it contains glycoprotein apolipoprotein(a) [apo(a)]. Due to its prothrombotic and proinflammatory properties, Lp(a) is an independent risk factor for atherosclerotic cardiovascular disease (ASCVD) and aortic valve stenosis. Lp(a) levels are genetically determined, and it is estimated that 20%-25% of the global population has an Lp(a) level ≥50 mg/dL (or ≥125 nmol/L). Diet and lifestyle interventions have little to no effect on Lp(a) levels. Lipoprotein apheresis is the only approved treatment for elevated Lp(a) but is time-intensive for the patient and only modestly effective. Pharmacological approaches to reduce Lp(a) levels and its associated risks are of significant interest; however, currently available lipid-lowering therapies have limited effectiveness in reducing Lp(a) levels. Although statins are first-line agents to reduce LDL cholesterol levels, they modestly increase Lp(a) levels and have not been shown to change Lp(a)-mediated ASCVD risk. Alirocumab, evolocumab, and inclisiran reduce Lp(a) levels by 20-25%, yet the clinical implications of this reduction for Lp(a)-mediated ASCVD risk are uncertain. Niacin also lowers Lp(a) levels; however, its effectiveness in mitigating Lp(a)-mediated ASCVD risk remains unclear, and its side effects have limited its utilization. Recommendations for when to screen and how to manage individuals with elevated Lp(a) vary widely between national and international guidelines and scientific statements. Three investigational compounds targeting Lp(a), including small interfering RNA (siRNA) agents (olpasiran, SLN360) and an antisense oligonucleotide (pelacarsen), are in various stages of development. These compounds block the translation of messenger RNA (mRNA) into apo(a), a key structural component of Lp(a), thereby substantially reducing Lp(a) synthesis in the liver. The purpose of this review is to describe current recommendations for screening and managing elevated Lp(a), describe the effects of currently available lipid-lowering therapies on Lp(a) levels, and provide insight into emerging therapies targeting Lp(a).

Effectiveness of telerehabilitation and home-based falls prevention programs for community-dwelling older adults: a systematic review and meta-analysis protocol

INTRODUCTION

Falls among older adults are associated with adverse sequelae including fractures, chronic pain and disability, which can lead to loss of independence and increased risks of nursing home admissions. The COVID-19 pandemic has significantly increased the uptake of telehealth, but the effectiveness of virtual, home-based fall prevention programmes is not clearly known. We aim to synthesise the trials on telerehabilitation and home-based falls prevention programmes to determine their effectiveness in reducing falls and adverse outcomes, as well as to describe the safety risks associated with telerehabilitation.

METHODS AND ANALYSIS

This protocol was developed using the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P). Database searches from inception to August 2022 will be conducted without language restrictions of MEDLINE, EMBASE, Ovid HealthSTAR, CINAHL, SPORTDiscus, Physiotherapy EvidenceDatabase (PEDro) and the Cochrane Library. Grey literature including major geriatrics conference proceedings will be reviewed. Using Covidence software, two independent reviewers will in duplicate determine the eligibility of randomised controlled trials (RCTs). Eligible RCTs will compare telerehabilitation and home-based fall prevention programmes to usual care among community-dwelling older adults and will report at least one efficacy outcome: falls, fractures, hospitalisations, mortality or quality of life; or at least one safety outcome: pain, myalgias, dyspnoea, syncope or fatigue. Secondary outcomes include functional performance in activities of daily living, balance and endurance. Risk of bias will be assessed using the Cochrane Collaboration tool. DerSimonian-Laird random effects models will be used for the meta-analysis. Heterogeneity will be assessed using the I² statistic and Cochran's Q statistic. We will assess publication bias using the Egger's test. Prespecified subgroup analyses and univariate meta-regression will be used.

ETHICS AND DISSEMINATION

Ethics approval is not required. The results will be disseminated through peer-reviewed publications and conference presentations.

PROSPERO REGISTRATION NUMBER

CRD42022356759.

Do imaging findings modify the effect of non-surgical treatment in patients with knee and hip osteoarthritis? A systematic literature review

OBJECTIVES

To review the available evidence on diagnostic imaging findings in knee and hip osteoarthritis (OA) as treatment effect modifiers in non-surgical OA interventions.

METHODS

MEDLINE, Embase and The Cochrane Central Register of Controlled Trials were searched from the earliest records published to 22 March 2022. Studies in knee and hip OA reporting subgroup analyses in randomised controlled trials with imaging findings as potential treatment effect modifiers were included. Studies were critically appraised using the Cochrane risk of bias tool and a subgroup analysis quality assessment.

RESULTS

Of 10 014 titles and abstracts screened, eight studies met the inclusion criteria, six on knee OA and two on hip OA. The studies investigated effect modifiers in exercise therapy, intra-articular injections and unloading shoes. Imaging findings assessed as potential treatment effect modifiers were radiographic OA severity, hip effusion (ultrasound), bone marrow lesions and meniscal pathology (MRI). Two studies fulfilled the methodological quality criteria for assessing effect modification. One reported that radiographic knee OA severity modified the effect of unloading shoes on walking pain. Those with more severe radiographic knee OA had a greater response to shoe inserts. One reported no interaction between radiographic OA severity or joint effusion and the effect of intraarticular injections of corticosteroid or hyaluronic acid in hip OA, indicating no difference in response in people with greater hip joint effusion or radiographic OA severity compared with those with less severe joint disease.

CONCLUSION

Overall, methodological limitations and very few studies do not permit conclusions on diagnostic imaging findings as effect modifiers in non-surgical interventions in knee and hip OA.Radiographic severity of knee OA potentially modifies the effect of unloading shoes.

PROSPERO REGISTRATION NUMBER

CRD42020181934.

Development and validation of paired MEDLINE and Embase search filters for cost-utility studies

BACKGROUND

Search filters are standardised sets of search terms, with validated performance, that are designed to retrieve studies with specific characteristics. A cost-utility analysis (CUA) is the preferred type of economic evaluation to underpin decision-making at the National Institute for Health and Care Excellence (NICE). Until now, when searching for economic evidence for NICE guidelines, we have used a broad set of health economic-related search terms, even when the reviewer's interest is confined to CUAs alone.

METHODS

We developed search filters to retrieve CUAs from MEDLINE and Embase. Our aim was to achieve recall of 90% or better across both databases while reducing the overall yield compared with our existing broad economic filter. We used the relative recall method along with topic expert input to derive and validate 3 pairs of filters, assessed by their ability to identify a gold-standard set of CUAs that had been used in published NICE guidelines. We developed and validated MEDLINE and Embase filters in pairs (testing whether, when used together, they find target studies in at least 1 database), as this is how they are used in practice. We examined the proxy-precision of our new filters by comparing their overall yield with our previous approach using publications indexed in a randomly selected year (2010).

RESULTS

All 3 filter-pairs exceeded our target recall and led to substantial improvements in search proxy-precision. Our paired 'sensitive' filters achieved 100% recall (95% CI 99.0 to 100%) in the validation set. Our paired 'precise' filters also had very good recall (97.6% [95%CI: 95.4 to 98.9%]). We estimate that, compared with our previous search strategy, using the paired 'sensitive' filters would reduce reviewer screening burden by a factor of 5 and the 'precise' versions would do so by a factor of more than 20.

CONCLUSIONS

Each of the 3 paired cost-utility filters enable the identification of almost all CUAs from MEDLINE and Embase from the validation set, with substantial savings in screening workload compared to our previous search practice. We would encourage other researchers who regularly use multiple databases to consider validating search filters in combination as this will better reflect how they use databases in their everyday work.

Avoiding searching for outcomes called for additional search strategies: a study of Cochrane review searches

OBJECTIVES

A search strategy for a systematic review that uses the Population, Intervention, Comparison, and Outcome framework should include the population, the intervention(s), and the type(s) of study design. According to existing guidelines, outcome should generally be excluded from the search strategy unless the search is multistranded. However, a recent study found that approximately 10% (51) of recent Cochrane reviews on interventions included outcomes in their literature search strategies. This study aims to analyze the alternatives to including outcomes in a search strategy by analyzing these recent Cochrane reviews.

STUDY DESIGN AND SETTING

This study analyzes the 51 Cochrane reviews that included outcomes in their literature search strategies and analyzes the results of alternative search strategies that follow current recommendations.

RESULTS

Despite a small study sample of 51 reviews the results show that many of the reviews excluded some of the recommended elements due to very broadly defined elements (e.g., all interventions or all people). Furthermore, excluding outcomes from the search strategy is followed by an enormous increase in the number of retrieved records making it unmanageable to screen, if using a single-stranded search strategy.

CONCLUSION

Recommendations for search strategies in difficult cases are called for.

Integrating Self-Management Education and Support in Routine Care of People With Type 2 Diabetes Mellitus: A Conceptional Model Based on Critical Interpretive Synthesis and A Consensus-Building Participatory Consultation

The integration of self-management education and support into the routine diabetes care is essential in preventing complications. Currently, however, there is no consensus on how to conceptualise integration in relation to self-management education and support. Therefore, this synthesis presents a framework conceptualising integration and self-management.

Methods

Seven electronic databases (Medline, HMIC, PsycINFO, CINAHL, ERIC, Scopus and Web of Science) were searched. Twenty-one articles met the inclusion criteria. Data were synthesised using principles of critical interpretive synthesis to build the conceptual framework. The framework was presented to 49 diabetes specialist nurses working at different levels of care during a multilingual workshop.

Results

A conceptual framework is proposed in which integration is influenced by five interacting components: the programme ethos of the diabetes self-management education and support intervention (content and delivery), care system organisation (the framework in which such interventions are delivered), adapting to context (the aspects of the people receiving and delivering the interventions), interpersonal relationship (the interactions between the deliverer and receiver of the intervention), and shared learning (what deliverer and receiver gain from the interactions). The critical inputs from the workshop participants related to the different priorities given to the components according to their sociolinguistic and educational experiences, Overall, they agreed with the conceptualisation of the components and their content specific to diabetes self-management education and support.

Discussion

Integration was conceptualised in terms of the relational, ethical, learning, contextual adapting, and systemic organisational aspects of the intervention. It remains uncertain which prioritised interactions of components and to what extent these may moderate the integration of self-management education and support into routine care; in turn, the level of integration observed in each of the components may moderate the impact of these interventions, which may also apply to the impact of the professional training.

Conclusion

This synthesis provides a theoretical framework that conceptualises integration in the context of diabetes self-management education and support in routine care. More research is required to evaluate how the components identified in the framework can be addressed in clinical practice to assess whether improvements in self-management education and support can be effectively realised in this population.

Sodium-glucose cotransporter 2 inhibitors in heart failure with reduced or preserved ejection fraction: a meta-analysis

AIMS

Sodium-glucose cotransporter 2 (SGLT2) inhibitors have been shown to be an effective therapy in improving heart failure outcomes. We conducted a meta-analysis of randomized controlled trials to evaluate the efficacy of SGLT2 inhibitors in heart failure patients with either a reduced or preserved ejection fraction.

METHODS AND RESULTS

We searched MEDLINE and EMBASE for large (≥1000 patients) randomized controlled trials evaluating the effects of SGLT2 inhibitors compared with placebo in the setting of heart failure until September 2021. Our primary outcome was the composite of heart failure hospitalization and cardiovascular death, and secondary outcomes included all-cause mortality and total heart failure hospitalizations. We pooled hazard ratios and risk ratios and evaluated risk of bias with the Cochrane Collaboration tool. Four randomized controlled trials (DAPA HF, EMPEROR-Preserved, EMPEROR-Reduced, and SOLOIST-WHF) were included (n = 15 684); two of which evaluated patients with a reduced LVEF, one of which evaluated patients with a preserved LVEF, and one of which included both. Treatment with SGLT2 inhibitors resulted in a significant reduction in the composite of CV death and heart failure hospitalization (HR: 0.76, 95% CI: 0.70, 0.82, I2 : 0%, P < 0.00001). This was consistent in sub-groups of patients with LVEF ≤40% (n = 9199, HR: 0.74, 95% CI: 0.68, 0.81, I2 : 0%) and LVEF >40% (n = 6482, HR: 0.78, 95% CI: 0.68, 0.89, I2 : 0%, P-for-interaction: 0.57), as well as in sub-groups of patients with and without diabetes mellitus at baseline (P-for-interaction: 0.81). SGLT2 inhibitors were associated with a significant reduction in cardiovascular death (HR: 0.87, 95% CI: 0.79, 0.97, I2 : 0%, P < 0.00001) and total heart failure hospitalization (RR: 0.71, 95% CI: 0.67, 0.76, I2 : 0%, P < 0.00001); although a potential trend towards reduced all-cause mortality was noted with SGLT2 inhibitors, no statistically significant difference was observed (HR: 0.91, 95% CI: 0.83, 1.00, I2 : 14%, P = 0.05).

CONCLUSIONS

Sodium-glucose cotransporter 2 inhibitors reduce cardiovascular death and heart failure hospitalization among patients with heart failure, regardless of LVEF status.

Evaluation of publication type tagging as a strategy to screen randomized controlled trial articles in preparing systematic reviews

Objectives

To produce a systematic review (SR), reviewers typically screen thousands of titles and abstracts of articles manually to find a small number which are read in full text to find relevant articles included in the final SR. Here, we evaluate a proposed automated probabilistic publication type screening strategy applied to the randomized controlled trial (RCT) articles (i.e., those which present clinical outcome results of RCT studies) included in a corpus of previously published Cochrane reviews.

Materials and Methods

We selected a random subset of 558 published Cochrane reviews that specified RCT study only inclusion criteria, containing 7113 included articles which could be matched to PubMed identifiers. These were processed by our automated RCT Tagger tool to estimate the probability that each article reports clinical outcomes of a RCT.

Results

Removing articles with low predictive scores P < 0.01 eliminated 288 included articles, of which only 22 were actually typical RCT articles, and only 18 were actually typical RCT articles that MEDLINE indexed as such. Based on our sample set, this screening strategy led to fewer than 0.05 relevant RCT articles being missed on average per Cochrane SR.

Discussion

This scenario, based on real SRs, demonstrates that automated tagging can identify RCT articles accurately while maintaining very high recall. However, we also found that even SRs whose inclusion criteria are restricted to RCT studies include not only clinical outcome articles per se, but a variety of ancillary article types as well.

Conclusions

This encourages further studies learning how best to incorporate automated tagging of additional publication types into SR triage workflows.

GRADE guidance 24 optimizing the integration of randomized and non-randomized studies of interventions in evidence syntheses and health guidelines

BACKGROUND AND OBJECTIVE

This is the 24th in the ongoing series of articles describing the GRADE approach for assessing the certainty of a body of evidence in systematic reviews and health technology assessments and how to move from evidence to recommendations in guidelines.

METHODS

Guideline developers and authors of systematic reviews and other evidence syntheses use randomized controlled studies (RCTs) and non-randomized studies of interventions (NRSI) as sources of evidence for questions about health interventions. RCTs with low risk of bias are the most trustworthy source of evidence for estimating relative effects of interventions because of protection against confounding and other biases. However, in several instances, NRSI can still provide valuable information as complementary, sequential, or replacement evidence for RCTs.

RESULTS

In this article we offer guidance on the decision regarding when to search for and include either or both types of studies in systematic reviews to inform health recommendations.

CONCLUSION

This work aims to help methodologists in review teams, technology assessors, guideline panelists, and anyone conducting evidence syntheses using GRADE.

Machine Learning Approaches to Retrieve High-Quality, Clinically Relevant Evidence From the Biomedical Literature: Systematic Review

BACKGROUND

The rapid growth of the biomedical literature makes identifying strong evidence a time-consuming task. Applying machine learning to the process could be a viable solution that limits effort while maintaining accuracy.

OBJECTIVE

The goal of the research was to summarize the nature and comparative performance of machine learning approaches that have been applied to retrieve high-quality evidence for clinical consideration from the biomedical literature.

METHODS

We conducted a systematic review of studies that applied machine learning techniques to identify high-quality clinical articles in the biomedical literature. Multiple databases were searched to July 2020. Extracted data focused on the applied machine learning model, steps in the development of the models, and model performance.

RESULTS

From 3918 retrieved studies, 10 met our inclusion criteria. All followed a supervised machine learning approach and applied, from a limited range of options, a high-quality standard for the training of their model. The results show that machine learning can achieve a sensitivity of 95% while maintaining a high precision of 86%.

CONCLUSIONS

Machine learning approaches perform well in retrieving high-quality clinical studies. Performance may improve by applying more sophisticated approaches such as active learning and unsupervised machine learning approaches.

Sodium-Glucose Co-Transporter Inhibitors and Atrial Fibrillation: A Systematic Review and Meta-Analysis of Randomized Controlled Trials

Background

Sodium‐glucose co‐transporter (SGLT) inhibitors reduce cardiovascular outcomes including mortality in several populations; however, their effect on atrial fibrillation/flutter (AF) remains unclear. Our objective was to determine whether SGLT inhibitors reduce AF and whether a history of AF modifies the effect of SGLT inhibitors on the composite of heart failure hospitalization or cardiovascular death.

Methods and Results

We searched MEDLINE, Embase, and CENTRAL to March 2021. Pairs of reviewers identified randomized controlled trials that compared an SGLT inhibitor with placebo or no therapy. We pooled data using RevMan 5.4.1, assessed risk of bias using the Cochrane tool, and determined the overall quality of evidence using Grades of Recommendation, Assessment, Development and Evaluation. Thirty‐one eligible trials reported on AF events (75 279 participants, mean age 62 years, 35.0% women). Moderate quality evidence supported a lower risk of serious AF events with SGLT inhibitors (1.1% versus 1.5%; risk ratio 0.75 [95% CI, 0.66–0.86]; I²=0%). A similar reduction in total AF events was also noted with SGLT inhibitors. Three trials reported on heart failure hospitalization/cardiovascular death stratified by a baseline history of AF (18 832 participants, mean age 66 years, 38.1% women); in patients with a history of AF, SGLT inhibitors resulted in a lower risk in the composite of heart failure hospitalization or cardiovascular death (hazard ratio, 0.70 [95% CI, 0.57–0.85]; I²=0%)—similar to the effect estimate for patients without AF, P value for interaction: 1.00.

Conclusions

SGLT inhibitors may reduce AF events and likely reduce heart failure hospitalization/cardiovascular death to a similar extent in patients with and without AF.

Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews

OBJECTIVES

This study developed, calibrated, and evaluated a machine learning classifier designed to reduce study identification workload in Cochrane for producing systematic reviews.

METHODS

A machine learning classifier for retrieving randomized controlled trials (RCTs) was developed (the "Cochrane RCT Classifier"), with the algorithm trained using a data set of title-abstract records from Embase, manually labeled by the Cochrane Crowd. The classifier was then calibrated using a further data set of similar records manually labeled by the Clinical Hedges team, aiming for 99% recall. Finally, the recall of the calibrated classifier was evaluated using records of RCTs included in Cochrane Reviews that had abstracts of sufficient length to allow machine classification.

RESULTS

The Cochrane RCT Classifier was trained using 280,620 records (20,454 of which reported RCTs). A classification threshold was set using 49,025 calibration records (1,587 of which reported RCTs), and our bootstrap validation found the classifier had recall of 0.99 (95% confidence interval 0.98-0.99) and precision of 0.08 (95% confidence interval 0.06-0.12) in this data set. The final, calibrated RCT classifier correctly retrieved 43,783 (99.5%) of 44,007 RCTs included in Cochrane Reviews but missed 224 (0.5%). Older records were more likely to be missed than those more recently published.

CONCLUSIONS

The Cochrane RCT Classifier can reduce manual study identification workload for Cochrane Reviews, with a very low and acceptable risk of missing eligible RCTs. This classifier now forms part of the Evidence Pipeline, an integrated workflow deployed within Cochrane to help improve the efficiency of the study identification processes that support systematic review production.

Relevant Journals for Identifying Implementation Science Articles: Results of an International Implementation Science Expert Survey

In implementation science (IS), conducting well-targeted and reproducible literature searches is challenging due to non-specific and varying terminology that is fragmented over multiple disciplines. A list of journals that publish IS-relevant content for use in search strings can support this process. We conducted a cross-sectional online survey of 56 Australian, European, and North American IS experts to identify and prioritize relevant journals that publish IS articles. Journals' relevance was assessed by providing each with a list of 12 journals, to which they were encouraged to add additional journal names and comments as free text. We also assessed which journals had published special IS-focused issues—identified via PubMed and Google searches—over the last 20 years. Data were analyzed descriptively. Between February 28 and March 15, 2020, a purposive sample of 34/56 experts participated in the survey (response rate: 60.7%). Implementation Science and BMC Health Services Research were perceived as relevant by 97.1% of participants; other journals' relevance varied internationally. Experts proposed 50 additional journals from various clinical fields and health science disciplines. We identified 12 calls and 53 special issues on IS published within various journals and research fields. Experts' comments confirmed the described challenges in identifying IS literature. This report presents experts' ratings of IS journals, which can be included in strategies supporting searches of IS evidence. However, challenges in identifying IS evidence remain geographically and interdisciplinary. Further investment is needed to develop reproducible search strings to capture IS evidence as an important step in improving IS research quality.

PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews. J Med Libr Assoc 2021;109:174-200. [PMID: 34285662 PMCID: PMC8270366 DOI: 10.5195/jmla.2021.962]

BACKGROUND

Literature searches underlie the foundations of systematic reviews and related review types. Yet, the literature searching component of systematic reviews and related review types is often poorly reported. Guidance for literature search reporting has been diverse and, in many cases, does not offer enough detail to authors who need more specific information about reporting search methods and information sources in a clear, reproducible way. This document presents the PRISMA-S (Preferred Reporting Items for Systematic reviews and Meta-Analyses literature search extension) checklist, and explanation and elaboration.

METHODS

The checklist was developed using a three-stage Delphi survey process, followed by a consensus conference and public review process.

RESULTS

The final checklist includes sixteen reporting items, each of which is detailed with exemplar reporting and rationale.

CONCLUSIONS

The intent of PRISMA-S is to complement the PRISMA Statement and its extensions by providing a checklist that could be used by interdisciplinary authors, editors, and peer reviewers to verify that each component of a search is completely reported and, therefore, reproducible.

PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews

BACKGROUND

Literature searches underlie the foundations of systematic reviews and related review types. Yet, the literature searching component of systematic reviews and related review types is often poorly reported. Guidance for literature search reporting has been diverse, and, in many cases, does not offer enough detail to authors who need more specific information about reporting search methods and information sources in a clear, reproducible way. This document presents the PRISMA-S (Preferred Reporting Items for Systematic reviews and Meta-Analyses literature search extension) checklist, and explanation and elaboration.

METHODS

The checklist was developed using a 3-stage Delphi survey process, followed by a consensus conference and public review process.

RESULTS

The final checklist includes 16 reporting items, each of which is detailed with exemplar reporting and rationale.

CONCLUSIONS

The intent of PRISMA-S is to complement the PRISMA Statement and its extensions by providing a checklist that could be used by interdisciplinary authors, editors, and peer reviewers to verify that each component of a search is completely reported and therefore reproducible.

Natural language processing was effective in assisting rapid title and abstract screening when updating systematic reviews

BACKGROUND AND OBJECTIVE

To examine whether the use of natural language processing (NLP) technology is effective in assisting rapid title and abstract screening when updating a systematic review.

STUDY DESIGN

Using the searched literature from a published systematic review, we trained and tested an NLP model that enables rapid title and abstract screening when updating a systematic review. The model was a light gradient boosting machine (LightGBM), an ensemble learning classifier which integrates four pretrained Bidirectional Encoder Representations from Transformers (BERT) models. We divided the searched citations into two sets (ie, training and test sets). The model was trained using the training set and assessed for screening performance using the test set. The searched citations, whose eligibility was determined by two independent reviewers, were treated as the reference standard.

RESULTS

The test set included 947 citations; our model included 340 citations, excluded 607 citations, and achieved 96% sensitivity, and 78% specificity. If the classifier assessment in the case study was accepted, reviewers would lose 8 of 180 eligible citations (4%), none of which were ultimately included in the systematic review after full-text consideration, while decreasing the workload by 64.1%.

CONCLUSION

NLP technology using the ensemble learning method may effectively assist in rapid literature screening when updating systematic reviews.

Which are the most sensitive search filters to identify randomized controlled trials in MEDLINE?

OBJECTIVE

The Cochrane Handbook of Systematic Reviews contains search filters to find randomized controlled trials (RCTs) in Ovid MEDLINE: one maximizing sensitivity and another balancing sensitivity and precision. These filters were originally published in 1994 and were adapted and updated in 2008. To determine the performance of these filters, the authors tested them and thirty-six other MEDLINE filters against a large new gold standard set of relevant records.

METHODS

We identified a gold standard set of RCT reports published in 2016 from the Cochrane CENTRAL database of controlled clinical trials. We retrieved the records in Ovid MEDLINE and combined these with each RCT filter. We calculated their sensitivity, relative precision, and f-scores.

RESULTS

The gold standard comprised 27,617 records. MEDLINE searches were run on July 16, 2019. The most sensitive RCT filter was Duggan et al. (sensitivity=0.99). The Cochrane sensitivity-maximizing RCT filter had a sensitivity of 0.96 but was more precise than Duggan et al. (0.14 compared to 0.04 for Duggan). The most precise RCT filters had 0.97 relative precision and 0.83 sensitivity.

CONCLUSIONS

The Cochrane Ovid MEDLINE sensitivity-maximizing RCT filter can continue to be used by Cochrane reviewers and to populate CENTRAL, as it has very high sensitivity and a slightly better precision relative to more sensitive filters. The results of this study, which used a very large gold standard to compare the performance of all known RCT filters, allows searchers to make better informed decisions about which filters to use for their work.

Citation screening using crowdsourcing and machine learning produced accurate results: Evaluation of Cochrane's modified Screen4Me service

OBJECTIVES

To assess the feasibility of a modified workflow that uses machine learning and crowdsourcing to identify studies for potential inclusion in a systematic review.

STUDY DESIGN AND SETTING

This was a substudy to a larger randomized study; the main study sought to assess the performance of single screening search results versus dual screening. This substudy assessed the performance in identifying relevant randomized controlled trials (RCTs) for a published Cochrane review of a modified version of Cochrane's Screen4Me workflow which uses crowdsourcing and machine learning. We included participants who had signed up for the main study but who were not eligible to be randomized to the two main arms of that study. The records were put through the modified workflow where a machine learning classifier divided the data set into "Not RCTs" and "Possible RCTs." The records deemed "Possible RCTs" were then loaded into a task created on the Cochrane Crowd platform, and participants classified those records as either "Potentially relevant" or "Not relevant" to the review. Using a prespecified agreement algorithm, we calculated the performance of the crowd in correctly identifying the studies that were included in the review (sensitivity) and correctly rejecting those that were not included (specificity).

RESULTS

The RCT machine learning classifier did not reject any of the included studies. In terms of the crowd, 112 participants were included in this substudy. Of these, 81 completed the training module and went on to screen records in the live task. Applying the Cochrane Crowd agreement algorithm, the crowd achieved 100% sensitivity and 80.71% specificity.

CONCLUSIONS

Using a crowd to screen search results for systematic reviews can be an accurate method as long as the agreement algorithm in place is robust.

TRIAL REGISTRATION

Open Science Framework: https://osf.io/3jyqt.

Using the full PICO model as a search tool for systematic reviews resulted in lower recall for some PICO elements

OBJECTIVES

The use of the four-part PICO model to facilitate search strategy development for a precise answer is recommended for structuring searches for systematic reviews. Existing guidelines generally recommend that a search strategy should include the population, intervention(s), and types of study design. Consequently, comparison and outcome are not recommended as a part of the search strategy. There is evidence that comparison and particularly outcome is not represented in enough detail, but this needs to be confirmed.

STUDY DESIGN AND SETTING

The present study examines the presence of PICO elements in the records in two commonly used databases for health sciences research: Embase and PubMed. We examine the field of upper GI and pancreatic diseases as well as the field of pregnancy and childbirth by extracting the included studies as well as the related PICO elements from a random selection of Cochrane reviews within these two areas.

RESULTS

We find that the PICO elements C and O had a lower retrieval potential across the two Cochrane groups and databases also when combining text words and subject headings. In particular, we find a lower retrieval when searching for both primary and secondary outcomes.

CONCLUSION

Our results support the existing recommendation not to search for outcomes.

Quality, scope and reporting standards of randomised controlled trials in Irish Health Research: an observational study

Background

Despite efforts to improve the accuracy and transparency of the design, conduct, and reporting of randomised controlled trials (RCTs), deficiencies remain. Such deficiencies contribute to significant, avoidable waste of health research investment and impede reproducibility. This study aimed to synthesise and critically analyse changes over time in the conduct and reporting of internationally published evidence on patient and/or population health-oriented RCTs conducted in one country.

Methods

This observational study drew on systematic review methods. We searched six databases for published RCTs (database inception to December 2018) where ≥ 80% of participants were recruited in the Republic of Ireland. RCTs of interventions targeted at patients, providers and/or policy makers intended to improve health, healthcare or health research were included. For each study, screening, data extraction and methodological quality appraisal were conducted by one member of the author team.

Results

From 17,560 titles and abstracts, 752 unique RCTs were published in 745 papers between 1968 and 2018, with a steady year-on-year increase since 1968. The number of participants was in the range of 2–8628. The majority were parallel design (86%) and classified as treatment evaluation. Of the 418 RCTs published since the introduction of mandatory clinical trial registration by the International Committee of Medical Journal Editors in 2005, 32% (n = 134) provided a trial registration number. This increased to 47% when taking studies published between 2013 and 2018 (n = 232). Since the 1996 publication of the CONSORT statement, 16% of included RCTs made specific reference to a standardised reporting guideline and this increased to 31% for more recent studies published between 2013 and 2018. Overall, 7% (n = 53) of studies referred to a published study protocol, increasing to 20% for studies published between 2013 and 2018.

Conclusion

Evidence from this single-country study of RCTs published in the international literature suggests that both the number overall, the number registered and the number referencing reporting guidelines have increased steadily over time. Despite widespread endorsement of reporting standards, reporting of RCTs remains suboptimal in domains such as compliance with the CONSORT statement and prospective trial registration. Researchers, funders and journal editors, nationally and internationally, should continue to focus on improving reporting and examining avoidable waste of health research investment.

Effects of probiotics on nonalcoholic fatty liver disease: a systematic review and meta-analysis

BACKGROUND

Nonalcoholic fatty liver disease (NAFLD) has become prevalent in recent decades, especially in developed countries, and approaches for the prevention and treatment of NAFLD are not clear. The aim of this research was to analyze and summarize randomized controlled trials that investigated the effects of probiotics on NAFLD.

METHODS

Seven databases (PubMed, Embase, the Web of Science, the Cochrane Library, China National Knowledge Infrastructure, Wan Fang Data, and VIP Database) were searched. Then, eligible studies were identified. Finally, proper data extraction, synthesis and analysis were performed by trained researchers.

RESULTS

Anthropometric parameters: with use of probiotics weight was reduced by 2.31 kg, and body mass index (BMI) was reduced by 1.08 kg/m². Liver function: probiotic treatment reduced the alanine aminotransferase level by 7.22 U/l, the aspartate aminotransferase level by 7.22 U/l, the alkaline phosphatase level by 25.87 U/l, and the glutamyl transpeptidase level by -5.76 U/l. Lipid profiles: total cholesterol, low-density lipoprotein cholesterol, and triglycerides were significantly decreased after probiotic treatment. Their overall effects (shown as standard mean difference) were -0.73, -0.54, and -0.36, respectively. Plasma glucose: probiotics reduced the plasma glucose level by 4.45 mg/dl and the insulin level by 0.63. Cytokines: probiotic treatment decreased tumor necrosis factor alpha by 0.62 and leptin by 1.14. Degree of liver fat infiltration (DFI): the related risk of probiotics for restoring DFI was 2.47 (95% confidence interval, 1.61-3.81, p < 0.001).

CONCLUSION

Probiotic treatment or supplementation is a promising therapeutic method for NAFLD.

Electronic search strategies fail to identify randomized controlled trials (RCTs) in neurosurgery

Randomized controlled trials (RCTs) are the gold standard studies to evaluate the efficacy of therapeutic interventions. Although they are frequently identified through open searches in electronic databases, no studies have evaluated how easy it is to identify RCTs in neurosurgery using electronic search strategies. The present study evaluated the sensitivity and specificity of different search strategies applied to commonly used databases to identify RCTs in neurosurgery. The total number of RCTs in neurosurgery published between 1960 and 2013 was determined through a detailed search involving open keyword searches in PubMed, Cochrane Library and Center for Reviews and Dissemination (CRD) databases, a PubMed search based on clinical entity-related keywords and hand-searches on the reference list of identified articles. The sensitivity and specificity were calculated for the open keyword searches on PubMed, the Cochrane Library and the CRD database and for the Cochrane's HSSS, based on the total number of the identified RCTs. Compared to the total of 1102 RCTs identified, PubMed open search yielded 4660 articles, among which 365 were RCTs (sensitivity: 33.1%; specificity: 7.8%). Cochrane open search yielded 621 among which 36 were RCTs (sensitivity: 3.2%; specificity: 5.8%) and CRD open search returned 78 articles, among which 4 were RCTs (sensitivity: 0.4% sensitivity; specificity: 5.1%). The Cochrane HSSS retrieved 10702 results, among which 340 were RCTs (sensitivity: 30.9%; specificity: 3.2%). Most RCTs in neurosurgery cannot be identified by commonly used search strategies, which emphasizes the need to improve their indexing.

Total Versus Subtotal Hysterectomy: Systematic Review and Meta-analysis of Intraoperative Outcomes and Postoperative Short-term Events

PURPOSE

The benefits and disadvantages of cervical extraction during hysterectomy are unclear in the literature. We intended to compare total (TH) with subtotal or supracervical (SH) hysterectomy regarding intraoperative and postoperative outcomes (quality of life, sexual function, pain and cyclical bleeding).

METHODS

A systematic literature search for randomized controlled trials was conducted on MEDLINE, LILACS, Cochrane CENTRAL, SCOPUS, EMBASE, Clinicaltrials.gov databases, and conference abstracts (AAGL, AUGS, ICS) from 1970 to November 2017. Two reviewers independently searched, selected and then combined the articles. Meta-analyses were conducted using a random-effect model. The risk of bias was evaluated using the Cochrane's Collaboration tool.

FINDINGS

Eleven studies were included involving 1523 patients. The analyses showed that the events operative time (mean difference: 12.88 minutes, 95%CI [7.45, 18.30] p < 0.000001), hospital stay (MD .44 days, 95%CI [0.11, 0.77] p = 0.0008), and intraoperative blood loss (MD 81.06 ml, 95%CI [9.16, 152.97] p = 0.03) favored SH over TH, although the rate of blood transfusion did not differ between the groups. Conversely, TH group had less cyclical vaginal bleeding over SH (1.2% versus 14.1%; RR .14 95%CI [0.05, 0.43] p = 0.0006) during one-year follow up. Persistent pain and sexual satisfaction rates, and quality of life scores were similar in both total and subtotal hysterectomy groups up to 12 months follow up.

IMPLICATIONS

Overall perioperative outcomes favored the preservation of the cervix during hysterectomy but women that had SH are more susceptible to present cyclical vaginal bleeding mimicking menstruation. Those factors should be taken into account along with patient's needs and expectations prior to selecting the procedure.

Topic search filters: a systematic scoping review

BACKGROUND

Searching for topics within large biomedical databases can be challenging, especially when topics are complex, diffuse, emerging or lack definitional clarity. Experimentally derived topic search filters offer a reliable solution to effective retrieval; however, their number and range of subject foci remain unknown.

OBJECTIVES

This systematic scoping review aims to identify and describe available experimentally developed topic search filters.

METHODS

Reports on topic search filter development (1990-) were sought using grey literature sources and 15 databases. Reports describing the conception and prospective development of a database-specific topic search and including an objectively measured estimate of its performance ('sensitivity') were included.

RESULTS

Fifty-four reports met inclusion criteria. Data were extracted and thematically synthesised to describe the characteristics of 58 topic search filters.

DISCUSSION

Topic search filters are proliferating and cover a wide range of subjects. Filter reports, however, often lack clear definitions of concepts and topic scope to guide users. Without standardised terminology, filters are challenging to find. Information specialists may benefit from a centralised topic filter repository and appraisal checklists to facilitate quality assessment.

CONCLUSION

Findings will help information specialists identify existing topic search filters and assist filter developers to build on current knowledge in the field.

Study filters for non-randomized studies of interventions consistently lacked sensitivity upon external validation

BACKGROUND

Little evidence is available on searches for non-randomized studies (NRS) in bibliographic databases within the framework of systematic reviews. For instance, it is currently unclear whether, when searching for NRS, effective restriction of the search strategy to certain study types is possible. The following challenges need to be considered: 1) For non-randomized controlled trials (NRCTs): whether they can be identified by established filters for randomized controlled trials (RCTs). 2) For other NRS types (such as cohort studies): whether study filters exist for each study type and, if so, which performance measures they have. The aims of the present analysis were to identify and validate existing NRS filters in MEDLINE as well as to evaluate established RCT filters using a set of MEDLINE citations.

METHODS

Our analysis is a retrospective analysis of study filters based on MEDLINE citations of NRS from Cochrane reviews. In a first step we identified existing NRS filters. For the generation of the reference set, we screened Cochrane reviews evaluating NRS, which covered a broad range of study types. The citations of the studies included in the Cochrane reviews were identified via the reviews' bibliographies and the corresponding PubMed identification numbers (PMIDs) were extracted from PubMed. Random samples comprising up to 200 citations (i.e. 200 PMIDs) each were created for each study type to generate the test sets.

RESULTS

A total of 271 Cochrane reviews from 41 different Cochrane groups were eligible for data extraction. We identified 14 NRS filters published since 2001. The study filters generated between 660,000 and 9.5 million hits in MEDLINE. Most filters covered several study types. The reference set included 2890 publications classified as NRS for the generation of the test sets. Twelve test sets were generated (one for each study type), of which 8 included 200 citations each. None of the study filters achieved sufficient sensitivity (≥ 92%) for all of the study types targeted.

CONCLUSIONS

The performance of current NRS filters is insufficient for effective use in daily practice. It is therefore necessary to develop new strategies (e.g. new NRS filters in combination with other search techniques). The challenges related to NRS should be taken into account.

Study filters for non-randomized studies of interventions consistently lacked sensitivity upon external validation

BACKGROUND

Little evidence is available on searches for non-randomized studies (NRS) in bibliographic databases within the framework of systematic reviews. For instance, it is currently unclear whether, when searching for NRS, effective restriction of the search strategy to certain study types is possible. The following challenges need to be considered: 1) For non-randomized controlled trials (NRCTs): whether they can be identified by established filters for randomized controlled trials (RCTs). 2) For other NRS types (such as cohort studies): whether study filters exist for each study type and, if so, which performance measures they have. The aims of the present analysis were to identify and validate existing NRS filters in MEDLINE as well as to evaluate established RCT filters using a set of MEDLINE citations.

METHODS

Our analysis is a retrospective analysis of study filters based on MEDLINE citations of NRS from Cochrane reviews. In a first step we identified existing NRS filters. For the generation of the reference set, we screened Cochrane reviews evaluating NRS, which covered a broad range of study types. The citations of the studies included in the Cochrane reviews were identified via the reviews' bibliographies and the corresponding PubMed identification numbers (PMIDs) were extracted from PubMed. Random samples comprising up to 200 citations (i.e. 200 PMIDs) each were created for each study type to generate the test sets.

RESULTS

A total of 271 Cochrane reviews from 41 different Cochrane groups were eligible for data extraction. We identified 14 NRS filters published since 2001. The study filters generated between 660,000 and 9.5 million hits in MEDLINE. Most filters covered several study types. The reference set included 2890 publications classified as NRS for the generation of the test sets. Twelve test sets were generated (one for each study type), of which 8 included 200 citations each. None of the study filters achieved sufficient sensitivity (≥ 92%) for all of the study types targeted.

CONCLUSIONS

The performance of current NRS filters is insufficient for effective use in daily practice. It is therefore necessary to develop new strategies (e.g. new NRS filters in combination with other search techniques). The challenges related to NRS should be taken into account.

[Publications German-speaking countries in high impact journals: development and validation of a search filter]

Publications German-speaking countries in high impact journals: development and validation of a search filter Abstract.

BACKGROUND

The number of publications in journals with a high impact factor is an indication of a discipline's participation in international discourse. A search filter allows reliable and reproducible searches for specific publications.

AIM

Development and validation of a geographic search filter for publications by nursing scientists affiliated to German-speaking countries in nursing journals with a high impact factor.

METHODS

The search filter was objectively developed following several steps: (i) creation of a development and a validation set, each consisting of relevant and non-relevant publications, (ii) generation of the search filter by means of text analysis of the development set, (iii) internal validation based on the development set and (iv) external validation using the validation set. The validity was examined regarding several accuracy parameters, e. g. sensitivity, specificity, positive predictive value (PPV) and number needed to read (NNR).

RESULTS

The search filter correctly identified 22 of 30 relevant and 16 of 21 non-relevant publications in the development set: sensitivity 80 % (95 % CI 66 - 94), specificity 76 % (95 % CI 58 - 94), PPV 83 % (95 % CI 69 - 97). External validation yielded similar or better results: sensitivity 81 % (95 % CI 67 - 96), specificity 88 % (95 % CI 71 - 100), PPV 88 % (95 % CI 75 - 100). The NNR was 1.2 and 1.1, respectively.

CONCLUSIONS

The search filter has the potential to identify the intended publications.

Design and implementation of a tool for conversion of search strategies between PubMed and Ovid MEDLINE

BACKGROUND

Both PubMed and Ovid MEDLINE contain records from the MEDLINE database. However, there are subtle differences in content, functionality, and search syntax between the two. There are many instances in which researchers may wish to search both interfaces, such as when conducting supplementary searching for a systematic review to retrieve a unique content from PubMed or when using a previously published search strategy from a different interface, but little guidance on how to best conduct these searches. The aim of this project is to describe differences in search functionality between Ovid MEDLINE and PubMed, provide guidance for converting search strategies between the two, and develop an easy-to-use, freely available web-based tool to automate search syntax translations.

CASE PRESENTATION

In this paper, we present a custom-built freely available online tool, Medline Transpose, to streamline the process of converting search strategies between Ovid MEDLINE and PubMed. With this tool, users can paste a strategy formatted for one interface into the search box and immediately retrieve an output formatted for use in the other interface, with recommendations for changes that users can make to the strategy where an exact translation does not exist.

CONCLUSION

This novel approach has the potential to reduce time and errors that database users spend translating search strategies.

Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study

BACKGROUND

Effective study identification is essential for conducting health research, developing clinical guidance and health policy and supporting health-care decision-making. Methodological search filters (combinations of search terms to capture a specific study design) can assist in searching to achieve this.

OBJECTIVES

This project investigated the methods used to assess the performance of methodological search filters, the information that searchers require when choosing search filters and how that information could be better provided.

METHODS

Five literature reviews were undertaken in 2010/11: search filter development and testing; comparison of search filters; decision-making in choosing search filters; diagnostic test accuracy (DTA) study methods; and decision-making in choosing diagnostic tests. We conducted interviews and a questionnaire with experienced searchers to learn what information assists in the choice of search filters and how filters are used. These investigations informed the development of various approaches to gathering and reporting search filter performance data. We acknowledge that there has been a regrettable delay between carrying out the project, including the searches, and the publication of this report, because of serious illness of the principal investigator.

RESULTS

The development of filters most frequently involved using a reference standard derived from hand-searching journals. Most filters were validated internally only. Reporting of methods was generally poor. Sensitivity, precision and specificity were the most commonly reported performance measures and were presented in tables. Aspects of DTA study methods are applicable to search filters, particularly in the development of the reference standard. There is limited evidence on how clinicians choose between diagnostic tests. No published literature was found on how searchers select filters. Interviewing and questioning searchers via a questionnaire found that filters were not appropriate for all tasks but were predominantly used to reduce large numbers of retrieved records and to introduce focus. The Inter Technology Appraisal Support Collaboration (InterTASC) Information Specialists' Sub-Group (ISSG) Search Filters Resource was most frequently mentioned by both groups as the resource consulted to select a filter. Randomised controlled trial (RCT) and systematic review filters, in particular the Cochrane RCT and the McMaster Hedges filters, were most frequently mentioned. The majority indicated that they used different filters depending on the requirement for sensitivity or precision. Over half of the respondents used the filters available in databases. Interviewees used various approaches when using and adapting search filters. Respondents suggested that the main factors that would make choosing a filter easier were the availability of critical appraisals and more detailed performance information. Provenance and having the filter available in a central storage location were also important.

LIMITATIONS

The questionnaire could have been shorter and could have included more multiple choice questions, and the reviews of filter performance focused on only four study designs.

CONCLUSIONS

Search filter studies should use a representative reference standard and explicitly report methods and results. Performance measures should be presented systematically and clearly. Searchers find filters useful in certain circumstances but expressed a need for more user-friendly performance information to aid filter choice. We suggest approaches to use, adapt and report search filter performance. Future work could include research around search filters and performance measures for study designs not addressed here, exploration of alternative methods of displaying performance results and numerical synthesis of performance comparison results.

FUNDING

The National Institute for Health Research (NIHR) Health Technology Assessment programme and Medical Research Council-NIHR Methodology Research Programme (grant number G0901496).

Optimal search strategies for identifying moderators and predictors of treatment effects in PubMed

BACKGROUND

Treatment effects differ across patients. To guide selection of treatments for patients, it is essential to acknowledge these differences and identify moderators or predictors. Our aim was to generate optimal search strategies (commonly known as filters) for PubMed to retrieve papers identifying moderators and predictors of treatment effects.

METHODS

Six journals were hand-searched for articles on moderators or predictors. Selected articles were randomly allocated to a development and validation set. Search terms were extracted from the development set and tested for their performance. Search filters were created from combinations of these terms and tested in the validation set.

RESULTS

Of 4407 articles, 198 were considered to be relevant. The most sensitive filter in the development set '("Epidemiologic Methods" [MeSH] OR assign* OR control*[tiab] OR trial*[tiab]) AND therapy*[sh]' yielded in the validation set a sensitivity of 89% [88%-90%] and a specificity of 80% [79%-82%].

CONCLUSIONS

The search filters created in this study can help to efficiently retrieve evidence on moderators and predictors of treatment effect. Testing of the filters in multiple domains should reveal robustness across disciplines. These filters can facilitate the retrieval of evidence on moderators and predictors of treatment effects, helping the implementation of stratified or personalised health care.

Patients and investigators prefer measures of absolute risk in subgroups for pragmatic randomized trials

OBJECTIVES

Pragmatic randomized trials are important tools for shared decision-making, but no guidance exists on patients' preferences for types of causal information. We aimed to assess preferences of patients and investigators toward causal effects in pragmatic randomized trials.

STUDY DESIGN AND SETTING

We (a) held three focus groups with patients (n = 23) in Boston, MA; (b) surveyed (n = 12) and interviewed (n = 5) investigators with experience conducting pragmatic trials; and (c) conducted a systematic literature review of pragmatic trials (n = 63).

RESULTS

Patients were distrustful of new-to-market medications unless substantially more effective than existing choices, preferred stratified absolute risks, and valued adherence-adjusted analyses when they expected to adhere. Investigators wanted both intention-to-treat and per-protocol effects but felt methods for estimating per-protocol effects were lacking. When estimating per-protocol effects, many pragmatic trials used inappropriate methods to adjust for adherence and loss to follow-up.

CONCLUSION

We made four recommendations for pragmatic trials to improve patient centeredness: (1) focus on superiority in effectiveness or safety, rather than noninferiority; (2) involve patients in specifying a priori subgroups; (3) report absolute measures of risk; and (4) complement intention-to-treat effect estimates with valid per-protocol effect estimates.

Diretrizes para detecção precoce do câncer de mama no Brasil. I - Métodos de elaboração

Tradicionalmente, diretrizes clínicas são elaboradas a partir do consenso de opiniões de especialistas. Nos últimos anos, a magnitude dos benefícios do rastreamento mamográfico vem sendo questionada em função dos vieses detectados nos ensaios clínicos que popularizaram a disseminação dessa prática. Paralelamente, o crescente corpo de evidências sobre danos associados ao rastreamento mamográfico também demandava uma nova abordagem que considerasse as incertezas sobre os benefícios e um balanço entre ganhos e possíveis danos. O presente artigo tem por objetivo apresentar o processo de elaboração das novas diretrizes para detecção precoce do câncer de mama no Brasil, detalhando os métodos utilizados, bem como suas implicações para as novas recomendações. A nova abordagem metodológica apresenta como pilares a realização de revisões sistemáticas da literatura, a avaliação da validade das evidências e o balanço entre riscos e benefícios de cada intervenção, garantindo maior transparência, reprodutibilidade e validade no processo de elaboração. Outra inovação das novas diretrizes é a presença de recomendações dirigidas a casos com sinais e sintomas suspeitos. As vantagens da abordagem adotada frente ao modelo tradicional de consenso de especialistas são discutidas com detalhes, bem como os limites e desvantagens dos métodos utilizados. Também são discutidas as implicações de diversas decisões, como escolhas sobre desenhos de estudo, desfechos sobre efetividade do rastreamento, além da definição de sobrediagnóstico e forma de cálculo.

Advances in the treatment of anorexia nervosa: a review of established and emerging interventions

BACKGROUND

Anorexia nervosa (AN) is a disabling, deadly and costly mental disorder. Until recently, treatment recommendations were based on expert opinion and limited evidence. The aim of this systematic review is to synthesise recent evidence on established and emerging AN treatments and to forecast trends for future developments.

METHODS

We systematically review trials of established treatments and associated process outcome studies from the last 5 years, published since a previous review in this journal. 'Established' treatments were those that are widely used in AN, recommended by guidelines and/or have been tested in at least one large randomised controlled trial. Secondly, we summarise emerging treatments for AN, i.e. those that have only been (or are currently being) tested in proof-of concept, feasibility or pilot trials.

RESULTS

We identified 19 published trials of established treatments (15 of high or moderate quality), mostly assessing psychological therapies (n = 17). We also found 11 published trials of emerging treatments, and a total of 34 registered, as yet unpublished trials. Promising emerging treatments include cognitive remediation therapy, exposure therapy and non-invasive neuromodulation.

CONCLUSIONS

Evidence generation on the treatment of AN has dramatically accelerated, with our understanding of the role of family-based approaches for adolescents more nuanced and a range of psychological approaches available for the treatment of adults. Evidence on emerging treatments and from forthcoming trials suggests that there is a shift towards more targeted brain-based interventions. Future studies need to focus on elucidating mechanisms of action of treatments and what works best for whom.

Adherence to reporting guidelines and clinical trial registration policies in oncology journals: a cross-sectional review

Reporting guidelines (RG) aim to improve research transparency and ensure high-quality study reporting. Similarly, clinical trial registration policies aim to reduce bias in results reporting by ensuring prospective registration of all trial outcomes. Failure to adhere to quality standards documented in RGs may result in low-quality and irreproducible research. Herein, we investigate the adherence to common RGs and trial registration policies in 21 oncology journals. We surveyed the Instructions for Authors page for each of the included oncology journals for adherence to common reporting guidelines and trial registration policies. We corresponded with editors to determine accepted study types and cross-referenced this information with a journal's RGs and trial registration policies to calculate the per cent of journals that adhere to a specific guideline or policy. 76.2% (16/21) of oncology journals surveyed adhere to Consolidated Standards of Reporting Trials guidelines for clinical trials while only 33.3% (7/21) adhere to Strengthening the Reporting of Observational Studies in Epidemiology for observational studies. Similarly, 76.2% (16/21) of oncology journals adhere to clinical trial registration policies. We further demonstrate that journal adherence to RGs positively affects author reporting, despite adherence to trial registration policies showing no such benefit. Our results show that oncology journals adhere to RGs and trial registration policies at a higher rate than other specialties, but nonetheless show room for improvement. We conclude that oncology journal adherence to RGs and trial registration policies is encouraging, but nonetheless suboptimal. We recommend the adoption of RGs and trial registration policies by all oncology journals.

Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach

Objectives

Identifying all published reports of randomized controlled trials (RCTs) is an important aim, but it requires extensive manual effort to separate RCTs from non-RCTs, even using current machine learning (ML) approaches. We aimed to make this process more efficient via a hybrid approach using both crowdsourcing and ML.

Methods

We trained a classifier to discriminate between citations that describe RCTs and those that do not. We then adopted a simple strategy of automatically excluding citations deemed very unlikely to be RCTs by the classifier and deferring to crowdworkers otherwise.

Results

Combining ML and crowdsourcing provides a highly sensitive RCT identification strategy (our estimates suggest 95%-99% recall) with substantially less effort (we observed a reduction of around 60%-80%) than relying on manual screening alone.

Conclusions

Hybrid crowd-ML strategies warrant further exploration for biomedical curation/annotation tasks.

Machine learning for identifying Randomized Controlled Trials: An evaluation and practitioner's guide

Machine learning (ML) algorithms have proven highly accurate for identifying Randomized Controlled Trials (RCTs) but are not used much in practice, in part because the best way to make use of the technology in a typical workflow is unclear. In this work, we evaluate ML models for RCT classification (support vector machines, convolutional neural networks, and ensemble approaches). We trained and optimized support vector machine and convolutional neural network models on the titles and abstracts of the Cochrane Crowd RCT set. We evaluated the models on an external dataset (Clinical Hedges), allowing direct comparison with traditional database search filters. We estimated area under receiver operating characteristics (AUROC) using the Clinical Hedges dataset.

We demonstrate that ML approaches better discriminate between RCTs and non‐RCTs than widely used traditional database search filters at all sensitivity levels; our best‐performing model also achieved the best results to date for ML in this task (AUROC 0.987, 95% CI, 0.984‐0.989). We provide practical guidance on the role of ML in (1) systematic reviews (high‐sensitivity strategies) and (2) rapid reviews and clinical question answering (high‐precision strategies) together with recommended probability cutoffs for each use case. Finally, we provide open‐source software to enable these approaches to be used in practice.

Roles for librarians in systematic reviews: a scoping review

OBJECTIVE

What roles do librarians and information professionals play in conducting systematic reviews? Librarians are increasingly called upon to be involved in systematic reviews, but no study has considered all the roles librarians can perform. This inventory of existing and emerging roles aids in defining librarians' systematic reviews services.

METHODS

For this scoping review, the authors conducted controlled vocabulary and text-word searches in the PubMed; Library, Information Science & Technology Abstracts; and CINAHL databases. We separately searched for articles published in the Journal of the European Association for Health Information and Libraries, Evidence Based Library and Information Practice, the Journal of the Canadian Heath Libraries Association, and Hypothesis. We also text-word searched Medical Library Association annual meeting poster and paper abstracts.

RESULTS

We identified 18 different roles filled by librarians and other information professionals in conducting systematic reviews from 310 different articles, book chapters, and presented papers and posters. Some roles were well known such as searching, source selection, and teaching. Other less documented roles included planning, question formulation, and peer review. We summarize these different roles and provide an accompanying bibliography of references for in-depth descriptions of these roles.

CONCLUSION

Librarians play central roles in systematic review teams, including roles that go beyond searching. This scoping review should encourage librarians who are fulfilling roles that are not captured here to document their roles in journal articles and poster and paper presentations.

The efficiency of database searches for creating systematic reviews was improved by search filters

OBJECTIVES

To compare Clinical Queries (CQs) for randomized trials of therapy 'methods' and 'NOT' limits search filters with Cochrane methods filters.

STUDY DESIGN AND SETTING

Analytic survey of Cochrane reviews as the reference standard for retrieving studies included in the reviews ("included studies [ISs]"). The sensitivity and precision of Cochrane content terms + Cochrane methods terms were compared in MEDLINE and Embase with Cochrane content terms + CQs maximally sensitive filter for therapy studies, without and with additional 'NOT' limits (CQ-S [CQ sensitive]; CQ-S + limits) and a balanced filter without and with additional NOT limits (CQ-B [CQ balanced]; CQ-B + limits).

RESULTS

Cochrane or CQ methods terms reduced, by 64-96%, the overall retrieval of articles with minimal loss of ISs. Sensitivity was high and similar for the 4 filters. However, CQ-B + limits had the highest precision (2.64%, number needed to be read to find one eligible study [NNR] 38) followed by the CQ-B (1.05%, NNR 95), Cochrane search (0.51%, NNR 198), CQ-S + limits (0.34%, NNR 296), and CQ-S filters (0.31%, NNR 325).

CONCLUSION

For systematic reviews of therapeutic interventions, the efficiency of searches in MEDLINE and Embase was better served by the CQs for therapy studies with balanced methods filter and NOT limits.

Trial registration and adherence to reporting guidelines in cardiovascular journals

Objective

This study investigated the policies of cardiac and cardiovascular system journals concerning clinical trial registration and guideline adoption to understand how frequently journals use these mechanisms to improve transparency, trial reporting and overall study quality.

Methods

We selected the top 20 (by impact factor) journals cited in the subcategory ‘Cardiac and Cardiovascular Systems’ of the Expanded Science Citation Index of the 2014 Journal Citation Reports to extract journal policies concerning the 17 guidelines we identified. In addition, trial and systematic review registration adherence statements were extracted. 300 randomised controlled trials published in 2016 in the top 20 journals were searched for clinical trial registry numbers and CONSORT diagrams.

Results

Of the 19 cardiac and cardiovascular system journals included in our analysis, eight journals (42%) did not require or recommend trial or review registration. Seven (37%) did not recommend or require a single guideline within their instructions to authors. Consolidated Standards for Reporting Trials guidelines (10/19, 53%) were recommended or required most often. Of the trials surveyed, 122/285 (42.8%) published a CONSORT diagram in their manuscript, while 236/292 (80.8%) published a trial registry number.

Discussion

Cardiac and cardiovascular system journals infrequently require, recommend or enforce the use of reporting guidelines. Furthermore, too few require or enforce the use of clinical trial registration. Cardiology journal editors should consider guideline adoption due to their potential to limit bias and increase transparency.

Locating sex- and gender-specific data in health promotion research: evaluating the sensitivity and precision of published filters

Objective

This study explored the effectiveness of search filters in identifying sex- and gender-specific data in health promotion studies that are indexed in MEDLINE.

Methods

Literature searches were conducted to identify studies on patient or consumer attitudes and behaviors toward colorectal cancer screening, nutritional labeling, and influenza vaccination. Publications reporting sex- or gender-specific outcome data constituted the gold standards for this study. The sensitivity and precision of previously published gender-specific filters, as well as individual filter component terms, were calculated and compared with values identified in prior studies.

Results

The sensitivity and precision of published sex or gender filters varied across topics. Sensitivity values ranged from 14.3% to 92.5%, while precision varied from 17.9% to 51.4%. These filters were less sensitive and less precise in their identification of relevant studies than has been reported in previous studies. Further, while the MEDLINE Medical Subject Headings (MeSH) term “Sex Factors” achieved the greatest average precision (59.3%) of any individual filter term, the MEDLINE check tag “Female” returned the highest average sensitivity (90.1%), with an average precision of 25.0% across topics.

Conclusions

Although search filters can facilitate the identification of research evidence to enable decision making, variability in study abstracting and indexing can limit the generalizability and usability of these filters. This potential for variability should be considered when deciding to incorporate a search filter into any literature search. This research highlights the importance of this awareness when developing strategies for searching the published literature and the potential value of supplementing database searching with other methods of study identification.