Machine learning for identifying Randomized Controlled Trials: An evaluation and practitioner's guide

The effects of step-count monitoring interventions on physical activity: systematic review and meta-analysis of community-based randomised controlled trials in adults

Background

Step-count monitors (pedometers, body-worn trackers and smartphone applications) can increase walking, helping to tackle physical inactivity. We aimed to assess the effect of step-count monitors on physical activity (PA) in randomised controlled trials (RCTs) amongst community-dwelling adults; including longer-term effects, differences between step-count monitors, and between intervention components.

Methods

Systematic literature searches in seven databases identified RCTs in healthy adults, or those at risk of disease, published between January 2000–April 2020. Two reviewers independently selected studies, extracted data and assessed risk of bias. Outcome was mean differences (MD) with 95% confidence intervals (CI) in steps at follow-up between treatment and control groups. Our preferred outcome measure was from studies with follow-up steps adjusted for baseline steps (change studies); but we also included studies reporting follow-up differences only (end-point studies). Multivariate-meta-analysis used random-effect estimates at different time-points for change studies only. Meta-regression compared effects of different step-count monitors and intervention components amongst all studies at ≤4 months.

Results

Of 12,491 records identified, 70 RCTs (at generally low risk of bias) were included, with 57 trials (16,355 participants) included in meta-analyses: 32 provided change from baseline data; 25 provided end-point only. Multivariate meta-analysis of the 32 change studies demonstrated step-counts favoured intervention groups: MD of 1126 steps/day 95%CI [787, 1466] at ≤4 months, 1050 steps/day [602, 1498] at 6 months, 464 steps/day [301, 626] at 1 year, 121 steps/day [− 64, 306] at 2 years and 434 steps/day [191, 676] at 3–4 years. Meta-regression of the 57 trials at ≤4 months demonstrated in mutually-adjusted analyses that: end-point were similar to change studies (+ 257 steps/day [− 417, 931]); body-worn trackers/smartphone applications were less effective than pedometers (− 834 steps/day [− 1542, − 126]); and interventions providing additional counselling/incentives were not better than those without (− 812 steps/day [− 1503, − 122]).

Conclusions

Step-count monitoring leads to short and long-term step-count increases, with no evidence that either body-worn trackers/smartphone applications, or additional counselling/incentives offer further benefit over simpler pedometer-based interventions. Simple step-count monitoring interventions should be prioritised to address the public health physical inactivity challenge.

Systematic review registration

PROSPERO number CRD42017075810.

Household interventions for secondary prevention of domestic lead exposure in children

BACKGROUND

Lead exposure is a serious health hazard, especially for children. It is associated with physical, cognitive and neurobehavioural impairment in children. There are many potential sources of lead in the environment, therefore trials have tested many household interventions to prevent or reduce lead exposure. This is an update of a previously published review.

OBJECTIVES

To assess the effects of household interventions intended to prevent or reduce further lead exposure in children on improvements in cognitive and neurobehavioural development, reductions in blood lead levels and reductions in household dust lead levels.

SEARCH METHODS

In March 2020, we updated our searches of CENTRAL, MEDLINE, Embase, 10 other databases and ClinicalTrials.gov. We also searched Google Scholar, checked the reference lists of relevant studies and contacted experts to identify unpublished studies.

SELECTION CRITERIA

Randomised controlled trials (RCTs) and quasi-RCTs of household educational or environmental interventions, or combinations of interventions to prevent lead exposure in children (from birth to 18 years of age), where investigators reported at least one standardised outcome measure.

DATA COLLECTION AND ANALYSIS

Two authors independently reviewed all eligible studies for inclusion, assessed risk of bias and extracted data. We contacted trialists to obtain missing information. We assessed the certainty of the evidence using the GRADE approach.

MAIN RESULTS

We included 17 studies (three new to this update), involving 3282 children: 16 RCTs (involving 3204 children) and one quasi-RCT (involving 78 children). Children in all studies were under six years of age. Fifteen studies took place in urban areas of North America, one in Australia and one in China. Most studies were in areas with low socioeconomic status. Girls and boys were equally represented in those studies reporting this information. The duration of the intervention ranged from three months to 24 months in 15 studies, while two studies performed interventions on a single occasion. Follow-up periods ranged from three months to eight years. Three RCTs were at low risk of bias in all assessed domains. The other 14 studies were at unclear or high risk of bias; for example, we considered two RCTs and one quasi-RCT at high risk of selection bias and six RCTs at high risk of attrition bias. National or international research grants or governments funded 15 studies, while the other two did not report their funding sources. Education interventions versus no intervention None of the included studies in this comparison assessed effects on cognitive or neurobehavioural outcomes, or adverse events. All studies reported data on blood lead level outcomes. Educational interventions showed there was probably no evidence of a difference in reducing blood lead levels (continuous: mean difference (MD) -0.03, 95% confidence interval (CI) -0.13 to 0.07; I² = 0%; 5 studies, 815 participants; moderate-certainty evidence; log-transformed data), or in reducing floor dust levels (MD -0.07, 95% CI -0.37 to 0.24; I² = 0%; 2 studies, 318 participants; moderate-certainty evidence). Environmental interventions versus no intervention Dust control: one study in this comparison reported data on cognitive and neurobehavioural outcomes, and on adverse events in children. The study showed numerically there may be better neurobehavioural outcomes in children of the intervention group. However, differences were small and the CI included both a beneficial and non-beneficial effect of the environmental intervention (e.g. mental development (Bayley Scales of Infant Development-II): MD 0.1, 95% CI -2.1 to 2.4; 1 study, 302 participants; low-certainty evidence). The same study did not observe any adverse events related to the intervention during the eight-year follow-up, but observed two children with adverse events in the control group (1 study, 355 participants; very low-certainty evidence). Meta-analysis also found no evidence of effectiveness on blood lead levels (continuous: MD -0.02, 95% CI -0.09 to 0.06; I² = 0%; 4 studies, 565 participants; moderate-certainty evidence; log-transformed data). We could not pool the data regarding floor dust levels, but studies reported that there may be no evidence of a difference between the groups (very low-certainty evidence). Soil abatement: the two studies assessing this environmental intervention only reported on the outcome of 'blood lead level'. One study showed a small effect on blood lead level reduction, while the other study showed no effect. Therefore, we deem the current evidence insufficient to draw conclusions about the effectiveness of soil abatement (very low-certainty evidence). Combination of educational and environmental interventions versus standard education Studies in this comparison only reported on blood lead levels and dust lead levels. We could not pool the studies in a meta-analysis due to substantial differences between the studies. Since the studies reported inconsistent results, the evidence is currently insufficient to clarify whether a combination of interventions reduces blood lead levels and floor dust levels (very low-certainty evidence).

AUTHORS' CONCLUSIONS

Based on available evidence, household educational interventions and environmental interventions (namely dust control measures) show no evidence of a difference in reducing blood lead levels in children as a population health measure. The evidence of the effects of environmental interventions on cognitive and neurobehavioural outcomes and adverse events is uncertain too. Further trials are required to establish the most effective intervention for reducing or even preventing further lead exposure. Key elements of these trials should include strategies to reduce multiple sources of lead exposure simultaneously using empirical dust clearance levels. It is also necessary for trials to be carried out in low- and middle-income countries and in differing socioeconomic groups in high-income countries.

Comparing the quantity and quality of randomised placebo-controlled trials of antibiotics for acute respiratory, urinary, and skin and soft tissue infections: a scoping review

BACKGROUND

The management of acute respiratory infections (ARIs), urinary tract infections (UTIs), and skin and soft tissue infections (SSTIs) should be guided by high quality evidence.

AIM

To compare the quantity and quality of randomised placebo-controlled trials of antibiotics for ARIs, UTIs, and SSTIs.

DESIGN & SETTING

A scoping review of the literature was performed using comprehensive search strategies.

METHOD

PubMed and the Cochrane Central Register of Controlled Trials (CENTRAL) were searched for published studies from inception until 17 April 2019. Randomised controlled trials (RCTs) that compared participants in primary care or in the community who had uncomplicated acute ARI, UTI, or studies, and were randomised to antibiotic or placebo (or no active treatment), were eligible for inclusion. Two groups of researchers independently screened articles for inclusion, extracted data, and assessed the quality of included studies.

RESULTS

A total of 108 eligible studies were identified: 80 on ARI, eight on UTI, and 20 on SSTI. The quality of studies varied with unclear risk of bias (RoB) prevalent in many domains. There was a gradual improvement in the quality of trials investigating ARIs over time, which could not be assessed in SSTI and UTI studies.

CONCLUSION

This review highlights a sparsity of trials assessing the effectiveness of antibiotics in people with UTIs and SSTIs, compared to trials targeting ARIs. This gap in the evidence needs to be addressed by conducting further high quality trials on the effects of antibiotics in patients with UTI and SSTI.

Citation screening using crowdsourcing and machine learning produced accurate results: Evaluation of Cochrane's modified Screen4Me service

OBJECTIVES

To assess the feasibility of a modified workflow that uses machine learning and crowdsourcing to identify studies for potential inclusion in a systematic review.

STUDY DESIGN AND SETTING

This was a substudy to a larger randomized study; the main study sought to assess the performance of single screening search results versus dual screening. This substudy assessed the performance in identifying relevant randomized controlled trials (RCTs) for a published Cochrane review of a modified version of Cochrane's Screen4Me workflow which uses crowdsourcing and machine learning. We included participants who had signed up for the main study but who were not eligible to be randomized to the two main arms of that study. The records were put through the modified workflow where a machine learning classifier divided the data set into "Not RCTs" and "Possible RCTs." The records deemed "Possible RCTs" were then loaded into a task created on the Cochrane Crowd platform, and participants classified those records as either "Potentially relevant" or "Not relevant" to the review. Using a prespecified agreement algorithm, we calculated the performance of the crowd in correctly identifying the studies that were included in the review (sensitivity) and correctly rejecting those that were not included (specificity).

RESULTS

The RCT machine learning classifier did not reject any of the included studies. In terms of the crowd, 112 participants were included in this substudy. Of these, 81 completed the training module and went on to screen records in the live task. Applying the Cochrane Crowd agreement algorithm, the crowd achieved 100% sensitivity and 80.71% specificity.

CONCLUSIONS

Using a crowd to screen search results for systematic reviews can be an accurate method as long as the agreement algorithm in place is robust.

TRIAL REGISTRATION

Open Science Framework: https://osf.io/3jyqt.

Adaptive design methods in dialysis clinical trials: a systematic review protocol

INTRODUCTION

Adaptive design methods are a potential solution to improve efficiency of clinical trials but their uptake in dialysis is unknown. We aim to investigate the use of adaptive design methods in dialysis clinical trials and to cultivate further adoption of adaptive design methods by the nephrology community.

METHODS AND ANALYSIS

We will adhere to the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols guidelines and the Cochrane Collaboration Handbook. We will perform a literature search through MEDLINE (PubMed), EMBASE and CENTRAL, a detailed data extraction of trial characteristics and a narrative synthesis of the data. There will be no language restrictions. We will estimate the percentage of adaptive clinical trials per year in dialysis. Subgroup analysis will be performed by dialysis modality, funder and geographical location.

ETHICS AND DISSEMINATION

Ethical approval will not be required for this study as data will be obtained from publicly available clinical trials. We will disseminate our results in a peer-reviewed publication. PROSPERO REGISTRATION NUMBER.

Drug treatments for covid-19: living systematic review and network meta-analysis

OBJECTIVE

To compare the effects of treatments for coronavirus disease 2019 (covid-19).

DESIGN

Living systematic review and network meta-analysis.

DATA SOURCES

WHO covid-19 database, a comprehensive multilingual source of global covid-19 literature, up to 3 December 2021 and six additional Chinese databases up to 20 February 2021. Studies identified as of 1 December 2021 were included in the analysis.

STUDY SELECTION

Randomised clinical trials in which people with suspected, probable, or confirmed covid-19 were randomised to drug treatment or to standard care or placebo. Pairs of reviewers independently screened potentially eligible articles.

METHODS

After duplicate data abstraction, a bayesian network meta-analysis was conducted. Risk of bias of the included studies was assessed using a modification of the Cochrane risk of bias 2.0 tool, and the certainty of the evidence using the grading of recommendations assessment, development, and evaluation (GRADE) approach. For each outcome, interventions were classified in groups from the most to the least beneficial or harmful following GRADE guidance.

RESULTS

463 trials enrolling 166 581 patients were included; 267 (57.7%) trials and 89 814 (53.9%) patients are new from the previous iteration; 265 (57.2%) trials evaluating treatments with at least 100 patients or 20 events met the threshold for inclusion in the analyses. Compared with standard care, three drugs reduced mortality in patients with mostly severe disease with at least moderate certainty: systemic corticosteroids (risk difference 23 fewer per 1000 patients, 95% credible interval 40 fewer to 7 fewer, moderate certainty), interleukin-6 receptor antagonists when given with corticosteroids (23 fewer per 1000, 36 fewer to 7 fewer, moderate certainty), and Janus kinase inhibitors (44 fewer per 1000, 64 fewer to 20 fewer, high certainty). Compared with standard care, two drugs probably reduce hospital admission in patients with non-severe disease: nirmatrelvir/ritonavir (36 fewer per 1000, 41 fewer to 26 fewer, moderate certainty) and molnupiravir (19 fewer per 1000, 29 fewer to 5 fewer, moderate certainty). Remdesivir may reduce hospital admission (29 fewer per 1000, 40 fewer to 6 fewer, low certainty). Only molnupiravir had at least moderate quality evidence of a reduction in time to symptom resolution (3.3 days fewer, 4.8 fewer to 1.6 fewer, moderate certainty); several others showed a possible benefit. Several drugs may increase the risk of adverse effects leading to drug discontinuation; hydroxychloroquine probably increases the risk of mechanical ventilation (moderate certainty).

CONCLUSION

Corticosteroids, interleukin-6 receptor antagonists, and Janus kinase inhibitors probably reduce mortality and confer other important benefits in patients with severe covid-19. Molnupiravir and nirmatrelvir/ritonavir probably reduce admission to hospital in patients with non-severe covid-19.

SYSTEMATIC REVIEW REGISTRATION

This review was not registered. The protocol is publicly available in the supplementary material.

READERS' NOTE

This article is a living systematic review that will be updated to reflect emerging evidence. Updates may occur for up to two years from the date of original publication. This is the fifth version of the original article published on 30 July 2020 (BMJ 2020;370:m2980), and previous versions can be found as data supplements. When citing this paper please consider adding the version number and date of access for clarity.

Trialstreamer: Mapping and Browsing Medical Evidence in Real-Time

We introduce Trialstreamer, a living database of clinical trial reports. Here we mainly describe the evidence extraction component; this extracts from biomedical abstracts key pieces of information that clinicians need when appraising the literature, and also the relations between these. Specifically, the system extracts descriptions of trial participants, the treatments compared in each arm (the interventions), and which outcomes were measured. The system then attempts to infer which interventions were reported to work best by determining their relationship with identified trial outcome measures. In addition to summarizing individual trials, these extracted data elements allow automatic synthesis of results across many trials on the same topic. We apply the system at scale to all reports of randomized controlled trials indexed in MEDLINE, powering the automatic generation of evidence maps, which provide a global view of the efficacy of different interventions combining data from all relevant clinical trials on a topic. We make all code and models freely available alongside a demonstration of the web interface.

Effectiveness of adult community-based physical activity interventions with objective physical activity measurements and long-term follow-up: a systematic review and meta-analysis

OBJECTIVE

To identify randomised controlled trials (RCTs) of physical activity (PA) interventions with objective PA outcomes in adults and to evaluate whether intervention effects were sustained beyond 12 months.

DESIGN

Systematic review and meta-analysis.

DATA SOURCES

Seven databases (Medline, Embase, PsycINFO, Web of Science, Cochrane library, CINAHL (Cumulative Index of Nursing and Allied Health Literature) and ASSIA (Applied Social Sciences Index and Abstracts)) were searched from January 2000 until December 2019.

ELIGIBILITY CRITERIA

RCTs reporting objective PA outcomes beyond 12 months with community-based participants aged ≥18 years were included; those where controls received active interventions, including advice to increase PA levels, were excluded.

DATA EXTRACTION AND SYNTHESIS

Two independent reviewers completed extraction of aggregate data and assessed risk of bias. Meta-analyses used random-effects models at different follow-up points. Primary outcomes were daily steps and weekly minutes of moderate-to-vigorous PA (MVPA).

RESULTS

Of 33 282 records identified, nine studies (at generally low risk of bias) were included, five in meta-analyses with 12 months to 4 year follow-up. We observed 12 month increases for intervention vs control participants in steps/day (mean difference (MD)=554 (95% CIs: 384 to 724) p<0.0001, I²=0%; 2446 participants; four studies) and weekly MVPA minutes (MD=35 (95% CI: 27 to 43) p<0.0001, I²=0%; 2647 participants; four studies). Effects were sustained up to 4 years for steps/day (MD=494 (95% CI: 251 to 738) p<0.0001, I²=0%; 1944 participants; four studies) and weekly MVPA minutes (MD=25 (95% CI: 13 to 37) p<0.0001, I²=0%; 1458 participants; three studies).

CONCLUSIONS

There are few PA interventions with objective follow-up beyond 12 months, more studies are needed. However, this review provided evidence of PA intervention effects beyond 12 months and sustained up to 4 years for both steps/day and MVPA. These findings have important implications for potential long-term health benefits.

PROSPERO REGISTRATION NUMBER

CRD42017075753.

A systematic review of effectiveness of interventions applicable to radiotherapy that are administered to improve patient comfort, increase patient compliance, and reduce patient distress or anxiety

OBJECTIVES

The aim of this review was to search existing literature to identify comfort interventions that can be used to assist an adult patient to undergo complex radiotherapy requiring positional stability for periods greater than 10 min. The objectives of this review were to; 1) identify comfort interventions used for clinical procedures that involve sustained inactivity similar to radiotherapy; 2) define characteristics of comfort interventions for future practice; and 3) determine the effectiveness of identified comfort interventions. The Preferred Reporting Items for Systematic Reviews and meta-analyses statement and the Template-for-Intervention-Description-and Replication guide were used.

KEY FINDINGS

The literature search was performed using PICO criteria with five databases (AMED, CINAHL EMBASE, MEDLINE, PsycINFO) identifying 5269 titles. After screening, 46 randomised controlled trials met the inclusion criteria. Thirteen interventions were reported and were grouped into four categories: Audio-visual, Psychological, Physical, and Other interventions (education/information and aromatherapy). The majority of aromatherapy, one audio-visual and one educational intervention were judged to be clinically significant for improving patient comfort based on anxiety outcome measures (effect size ≥ 0.4, mean change is greater than the Minimal-Important-Difference and low-risk-of-bias). Medium to large effect sizes were reported in many interventions where differences did not exceed the Minimal-Important-Difference for the measure. These interventions were deemed worthy of further investigation.

CONCLUSION

Several interventions were identified that may improve comfort during radiotherapy assisting patients to sustain and endure the same position over time. This is crucial for the continual growth of complex radiotherapy requiring a need for comfort to ensure stability for targeted treatment.

IMPLICATIONS FOR PRACTICE

Further investigation of comfort interventions is warranted, including tailoring interventions to patient choice and determining if multiple interventions can be used concurrently to improve effectiveness.

PROTOCOL: A comparison of the effectiveness of cognitive behavioural interventions based on delivery features for elevated symptoms of depression in adolescents

This is the protocol for a Campbell review. The primary aim is to estimate the relative efficacy of different modes of CBT delivery compared with control conditions for reducing depressive symptoms in adolescents. The secondary aim is to compare the different modes of delivery with regards to intervention completion/attrition (used as a proxy for intervention acceptability). The review will provide relative effect estimates and ranking probabilities for each outcome based on intervention delivery.

Increased fluid intake to prevent urinary tract infections: systematic review and meta-analysis

BACKGROUND

Approximately 15% of community-prescribed antibiotics are used in treating urinary tract infections (UTIs). Increase in antibiotic resistance necessitates considering alternatives.

AIM

To assess the impact of increased fluid intake in individuals at risk for UTIs, for impact on UTI recurrence (primary outcome), antimicrobial use, and UTI symptoms (secondary outcomes).

DESIGN AND SETTING

A systematic review.

METHOD

The authors searched PubMed, Cochrane CENTRAL, EMBASE, two trial registries, and conducted forward and backward citation searches of included studies in January 2019. Randomised controlled trials of individuals at risk for UTIs were included; comparisons with antimicrobials were excluded. Different time-points (≤6 months and 12 months) were compared for the primary outcome. Risk of bias was assessed using Cochrane Risk of Bias tool. Meta-analyses were undertaken where ≥3 studies reported the same outcome.

RESULTS

Eight studies were included; seven were meta-analysed. There was a statistically non-significant reduction in the number of patients with any UTI recurrence in the increased fluid intake group compared with control after 12 months (odds ratio [OR] 0.39, 95% confidence interval [CI] = 0.15 to 1.03, P = 0.06); reduction was significant at ≤6 months (OR 0.13, 95% CI = 0.07 to 0.25, P<0.001). Excluding studies with low volume of fluid (<200 ml) significantly favoured increased fluid intake (OR 0.25, 95% CI = 0.11 to 0.59, P = 0.001). Increased fluid intake reduced the overall rate of all recurrent UTIs (rate ratio [RR] 0.46, 95% CI = 0.40 to 0.54, P<0.001); there was no difference in antimicrobial use (OR 0.52, 95% CI = 0.25 to 1.07, P = 0.08). Paucity of data precluded meta-analysing symptoms.

CONCLUSION

Given the minimal potential for harm, patients with recurrent UTIs could be advised to drink more fluids to reduce recurrent UTIs. Further research is warranted to establish the optimal volume and type of increased fluid.

Biologics for chronic rhinosinusitis

BACKGROUND

This living systematic review is one of several Cochrane Reviews evaluating the medical management of patients with chronic rhinosinusitis. Chronic rhinosinusitis is common. It is characterised by inflammation of the nasal and sinus linings, nasal blockage, rhinorrhoea, facial pressure/pain and loss of sense of smell. It occurs with or without nasal polyps.   'Biologics' are medicinal products produced by a biological process. Monoclonal antibodies are one type, already evaluated in related inflammatory conditions (e.g. asthma and atopic dermatitis).

OBJECTIVES

To assess the effects of biologics for the treatment of chronic rhinosinusitis.

SEARCH METHODS

The Cochrane ENT Information Specialist searched the Cochrane ENT Register; CENTRAL (2019, Issue 9); Ovid MEDLINE; Ovid Embase; Web of Science; ClinicalTrials.gov; ICTRP and additional sources for published and unpublished trials. The date of the search was 16 September 2019.

SELECTION CRITERIA

Randomised controlled trials (RCTs) with at least three months follow-up comparing biologics (currently, monoclonal antibodies) against placebo/no treatment in patients with chronic rhinosinusitis.

DATA COLLECTION AND ANALYSIS

We used standard Cochrane methodological procedures. Our primary outcomes were disease-specific health-related quality of life (HRQL), disease severity and serious adverse events (SAEs). The secondary outcomes were avoidance of surgery, extent of disease (measured by endoscopic or computerised tomography (CT) score), generic HRQL and adverse events (nasopharyngitis, including sore throat). We used GRADE to assess the certainty of the evidence for each outcome.

MAIN RESULTS

We included eight RCTs. Of 986 adult participants, 984 had severe chronic rhinosinusitis with nasal polyps; 43% to 100% of participants also had asthma. Three biologics, with different targets, were evaluated: dupilumab, mepolizumab and omalizumab. All the studies were sponsored or supported by industry. Anti-IL-4Rα mAb (dupilumab) versusplacebo/no treatment (all receiving intranasal steroids) Three studies (784 participants) evaluated dupilumab. Disease-specific HRQL was measured with the SNOT-22 (score 0 to 110; minimal clinically important difference (MCID) 8.9 points). At 24 weeks, the SNOT-22 score was 19.61 points lower (better) in participants receiving dupilumab (mean difference (MD) -19.61, 95% confidence interval (CI) -22.54 to -16.69; 3 studies; 784 participants; high certainty). Symptom severity measured on a 0- to 10-point visual analogue scale (VAS) was 3.00 lower in those receiving dupilumab (95% CI -3.47 to -2.53; 3 studies; 784 participants; moderate certainty). The risk of serious adverse events may be lower in the dupilumab group (risk ratio (RR) 0.45, 95% CI 0.28 to 0.75; 3 studies; 782 participants; low certainty).  The number of participants requiring nasal polyp surgery (actual or planned) during the treatment period is probably lower in those receiving dupilumab (RR 0.17, 95% CI 0.05 to 0.52; 2 studies; 725 participants; moderate certainty). Change in the extent of disease using the Lund Mackay computerised tomography (CT) score (0 to 24, higher = worse) was -7.00 (95% CI -9.61 to -4.39; 3 studies; 784 participants; high certainty), a large effect favouring the dupilumab group.  The EQ-5D visual analogue scale (0 to 100, higher = better; MCID 8 points) was used to measure change in generic quality of life. The mean difference favouring dupilumab was 8.59 (95% CI 5.31 to 11.86; 2 studies; 706 participants; moderate certainty). There may be little or no difference in the risk of nasopharyngitis (RR 0.95, 95% CI 0.72 to 1.25; 3 studies; 783 participants; low certainty). Anti-IL-5 mAb (mepolizumab) versusplacebo/no treatment (all receiving intranasal steroids) Two studies (137 participants) evaluated mepolizumab. Disease-specific HRQL measured with the SNOT-22 at 25 weeks was 13.26 points lower (better) in participants receiving mepolizumab (95% CI -22.08 to -4.44; 1 study; 105 participants; low certainty; MCID 8.9).  It is very uncertain whether there is a difference in s ymptom severity: on a 0- to 10-point VAS symptom severity was -2.03 lower in those receiving mepolizumab (95% CI -3.65 to -0.41; 1 study; 72 participants; very low certainty). It is very uncertain if there is difference in the risk of serious adverse events (RR 1.57, 95% CI 0.07 to 35.46; 2 studies; 135 participants, very low certainty). It is very uncertain whether or not the overall risk that patients still need surgery at trial end is lower in the mepolizumab group (RR 0.78, 95% CI 0.64 to 0.94; 2 studies; 135 participants; very low certainty). It is very uncertain whether mepolizumab reduces the extent of disease as measured by endoscopic nasal polyps score (scale range 0 to 8). The mean difference was 1.23 points lower in the mepolizumab group (MD -1.23, 95% -1.79 to -0.68; 2 studies; 137 participants; very low certainty). The difference in generic quality of life (EQ-5D) was 5.68 (95% CI -1.18 to 12.54; 1 study; 105 participants; low certainty), favouring the mepolizumab group. This difference is smaller than the MCID of 8 points. There may be little or no difference in the risk of nasopharyngitis (RR 0.73, 95% 0.36 to 1.47; 2 studies; 135 participants; low certainty). Anti-IgE mAb (omalizumab) versus placebo/no treatment (all receiving intranasal steroids) Three very small studies (65 participants) evaluated omalizumab. We are very uncertain about the effect of omalizumab on disease-specific HRQL, severe adverse events, extent of disease (CT scan scores), generic HRQL and adverse effects.

AUTHORS' CONCLUSIONS

In adults with severe chronic rhinosinusitis and nasal polyps, using regular topical nasal steroids, dupilumab improves disease-specific HRQL compared to placebo, and reduces the extent of the disease as measured on a CT scan. It probably also improves symptoms and generic HRQL and there is no evidence of an increased risk of serious adverse events. It may reduce the need for further surgery. There may be little or no difference in the risk of nasopharyngitis. In similar patients, mepolizumab may improve both disease-specific and generic HRQL. It is uncertain whether it reduces the need for surgery or improves nasal polyp scores. There may be little or no difference in the risk of nasopharyngitis. It is uncertain if there is a difference in symptom severity and the risk of serious adverse events. We are uncertain about the effects of omalizumab.

Semi-Automated evidence synthesis in health psychology: current methods and future prospects

The evidence base in health psychology is vast and growing rapidly. These factors make it difficult (and sometimes practically impossible) to consider all available evidence when making decisions about the state of knowledge on a given phenomenon (e.g., associations of variables, effects of interventions on particular outcomes). Systematic reviews, meta-analyses, and other rigorous syntheses of the research mitigate this problem by providing concise, actionable summaries of knowledge in a given area of study. Yet, conducting these syntheses has grown increasingly laborious owing to the fast accumulation of new evidence; existing, manual methods for synthesis do not scale well. In this article, we discuss how semi-automation via machine learning and natural language processing methods may help researchers and practitioners to review evidence more efficiently. We outline concrete examples in health psychology, highlighting practical, open-source technologies available now. We indicate the potential of more advanced methods and discuss how to avoid the pitfalls of automated reviews.

A full systematic review was completed in 2 weeks using automation tools: a case study

BACKGROUND AND OBJECTIVES

Systematic reviews (SRs) are time and resource intensive, requiring approximately 1 year from protocol registration to submission for publication. Our aim was to describe the process, facilitators, and barriers to completing the first 2-week full SR.

STUDY DESIGN AND SETTING

We systematically reviewed evidence of the impact of increased fluid intake, on urinary tract infection (UTI) recurrence, in individuals at risk for UTIs. The review was conducted by experienced systematic reviewers with complementary skills (two researcher clinicians, an information specialist, and an epidemiologist), using Systematic Review Automation tools, and blocked off time for the duration of the project. The outcomes were time to complete the SR, time to complete individual SR tasks, facilitators and barriers to progress, and peer reviewer feedback on the SR manuscript. Times to completion were analyzed quantitatively (minutes and calendar days); facilitators and barriers were mapped onto the Theoretical Domains Framework; and peer reviewer feedback was analyzed quantitatively and narratively.

RESULTS

The SR was completed in 61 person-hours (9 workdays; 12 calendar days); accepted version of the manuscript required 71 person-hours. Individual SR tasks ranged from 16 person-minutes (deduplication of search results) to 461 person-minutes (data extraction). The least time-consuming SR tasks were obtaining full-texts, searches, citation analysis, data synthesis, and deduplication. The most time-consuming tasks were data extraction, write-up, abstract screening, full-text screening, and risk of bias. Facilitators and barriers mapped onto the following domains: knowledge; skills; memory, attention, and decision process; environmental context and resources; and technology and infrastructure. Two sets of peer reviewer feedback were received on the manuscript: the first included 34 comments requesting changes, 17 changes were made, requiring 173 person-minutes; the second requested 13 changes, and eight were made, requiring 121 person-minutes.

CONCLUSION

A small and experienced systematic reviewer team using Systematic Review Automation tools who have protected time to focus solely on the SR can complete a moderately sized SR in 2 weeks.

Machine learning with the hierarchy-of-hypotheses (HoH) approach discovers novel pattern in studies on biological invasions

Research synthesis on simple yet general hypotheses and ideas is challenging in scientific disciplines studying highly context-dependent systems such as medical, social, and biological sciences. This study shows that machine learning, equation-free statistical modeling of artificial intelligence, is a promising synthesis tool for discovering novel patterns and the source of controversy in a general hypothesis. We apply a decision tree algorithm, assuming that evidence from various contexts can be adequately integrated in a hierarchically nested structure. As a case study, we analyzed 163 articles that studied a prominent hypothesis in invasion biology, the enemy release hypothesis. We explored if any of the nine attributes that classify each study can differentiate conclusions as classification problem. Results corroborated that machine learning can be useful for research synthesis, as the algorithm could detect patterns that had been already focused in previous narrative reviews. Compared with the previous synthesis study that assessed the same evidence collection based on experts' judgement, the algorithm has newly proposed that the studies focusing on Asian regions mostly supported the hypothesis, suggesting that more detailed investigations in these regions can enhance our understanding of the hypothesis. We suggest that machine learning algorithms can be a promising synthesis tool especially where studies (a) reformulate a general hypothesis from different perspectives, (b) use different methods or variables, or (c) report insufficient information for conducting meta-analyses.

Biologics for chronic rhinosinusitis

This is a protocol for a Cochrane Review (Intervention). The objectives are as follows:

Main objective

To assess the effects of biologics for the treatment of chronic rhinosinusitis.

Secondary objective

To maintain the currency of the evidence, using a living systematic review approach.

Screening PubMed abstracts: is class imbalance always a challenge to machine learning?

BACKGROUND

The growing number of medical literature and textual data in online repositories led to an exponential increase in the workload of researchers involved in citation screening for systematic reviews. This work aims to combine machine learning techniques and data preprocessing for class imbalance to identify the outperforming strategy to screen articles in PubMed for inclusion in systematic reviews.

METHODS

We trained four binary text classifiers (support vector machines, k-nearest neighbor, random forest, and elastic-net regularized generalized linear models) in combination with four techniques for class imbalance: random undersampling and oversampling with 50:50 and 35:65 positive to negative class ratios and none as a benchmark. We used textual data of 14 systematic reviews as case studies. Difference between cross-validated area under the receiver operating characteristic curve (AUC-ROC) for machine learning techniques with and without preprocessing (delta AUC) was estimated within each systematic review, separately for each classifier. Meta-analytic fixed-effect models were used to pool delta AUCs separately by classifier and strategy.

RESULTS

Cross-validated AUC-ROC for machine learning techniques (excluding k-nearest neighbor) without preprocessing was prevalently above 90%. Except for k-nearest neighbor, machine learning techniques achieved the best improvement in conjunction with random oversampling 50:50 and random undersampling 35:65.

CONCLUSIONS

Resampling techniques slightly improved the performance of the investigated machine learning techniques. From a computational perspective, random undersampling 35:65 may be preferred.

Improving reference prioritisation with PICO recognition

BACKGROUND

Machine learning can assist with multiple tasks during systematic reviews to facilitate the rapid retrieval of relevant references during screening and to identify and extract information relevant to the study characteristics, which include the PICO elements of patient/population, intervention, comparator, and outcomes. The latter requires techniques for identifying and categorising fragments of text, known as named entity recognition.

METHODS

A publicly available corpus of PICO annotations on biomedical abstracts is used to train a named entity recognition model, which is implemented as a recurrent neural network. This model is then applied to a separate collection of abstracts for references from systematic reviews within biomedical and health domains. The occurrences of words tagged in the context of specific PICO contexts are used as additional features for a relevancy classification model. Simulations of the machine learning-assisted screening are used to evaluate the work saved by the relevancy model with and without the PICO features. Chi-squared and statistical significance of positive predicted values are used to identify words that are more indicative of relevancy within PICO contexts.

RESULTS

Inclusion of PICO features improves the performance metric on 15 of the 20 collections, with substantial gains on certain systematic reviews. Examples of words whose PICO context are more precise can explain this increase.

CONCLUSIONS

Words within PICO tagged segments in abstracts are predictive features for determining inclusion. Combining PICO annotation model into the relevancy classification pipeline is a promising approach. The annotations may be useful on their own to aid users in pinpointing necessary information for data extraction, or to facilitate semantic search.

Best Practice Guidelines and Essential Methodological Steps to Conduct Rigorous and Systematic Meta-Reviews

BACKGROUND

A growing body of primary study and systematic review literature evaluates interventions and phenomena in applied and health psychology. Reviews of reviews (i.e., meta-reviews) systematically synthesise and utilise this vast and often overwhelming literature; yet, currently there are few practical guidelines for meta-review authors to follow.

OBJECTIVE

The aim of this article is to provide an overview of the best practice guidelines for all research synthesis and to detail additional specific considerations and methodological details for the best practice of conducting a rigorous meta-review.

METHODS

This article provides readers with six systematic and practical steps along with accompanying examples to address with rigor the unique challenges that arise when authors familiar with systematic review methods begin a meta-review: (a) detailing a clear scope, (b) identifying synthesis literature through strategic searches, (c) considering datedness of the literature, (d) addressing overlap among included reviews, (e) choosing and applying review quality tools, and (f) appropriate options for handling the synthesis and reporting of the vast amount of data collected in a meta-review.

CONCLUSIONS

We have curated best practice recommendations and practical tips for conducting a meta-review. We anticipate that assessments of meta-review quality will ultimately formalise best-method guidelines.

What Does the Evidence Say? Models to Help Make Sense of the Biomedical Literature

Ideally decisions regarding medical treatments would be informed by the totality of the available evidence. The best evidence we currently have is in published natural language articles describing the conduct and results of clinical trials. Because these are unstructured, it is difficult for domain experts (e.g., physicians) to sort through and appraise the evidence pertaining to a given clinical question. Natural language technologies have the potential to improve access to the evidence via semi-automated processing of the biomedical literature. In this brief paper I highlight work on developing tasks, corpora, and models to support semi-automated evidence retrieval and extraction. The aim is to design models that can consume articles describing clinical trials and automatically extract from these key clinical variables and findings, and estimate their reliability. Completely automating 'machine reading' of evidence remains a distant aim given current technologies; the more immediate hope is to use such technologies to help domain experts access and make sense of unstructured biomedical evidence more efficiently, with the ultimate aim of improving patient care. Aside from their practical importance, these tasks pose core NLP challenges that directly motivate methodological innovation.

Toward systematic review automation: a practical guide to using machine learning tools in research synthesis

Technologies and methods to speed up the production of systematic reviews by reducing the manual labour involved have recently emerged. Automation has been proposed or used to expedite most steps of the systematic review process, including search, screening, and data extraction. However, how these technologies work in practice and when (and when not) to use them is often not clear to practitioners. In this practical guide, we provide an overview of current machine learning methods that have been proposed to expedite evidence synthesis. We also offer guidance on which of these are ready for use, their strengths and weaknesses, and how a systematic review team might go about using them in practice.

Working with a new kind of team: harnessing the wisdom of the crowd in trial identification

Background: At a time when research output is expanding exponentially, citizen science, the process of engaging willing volunteers in scientific research activities, has an important role to play in helping to manage the information overload. It also creates a model of contribution that enables anyone with an interest in health to contribute meaningfully and in a way that is flexible. Citizen science models have been shown to be extremely effective in other domains such as astronomy and ecology. Methods: Cochrane Crowd (crowd.cochrane.org) is a citizen science platform that offers contributors a range of microtasks, designed to help identify and describe health research. The platform enables contributors to dive into needed tasks that capture and describe health evidence. Brief interactive training modules and agreement algorithms help to ensure accurate collective decision making. Contributors can work online or offline; they can view their activity and performance in detail. They can choose to work in topic areas of interest to them such dementia or diabetes, and as contributors progress, they unlock milestone rewards and new tasks. Cochrane Crowd was launched in May 2016. It now hosts a range of microtasks which help to identify health evidence and then describe it according to a PICO (Population; Intervention; Comparator; Outcome) ontology. The microtasks are either at ‘citation level’ in which a contributor is presented with a title and abstract to classify or annotate, or at the full‐text level in which a whole or a portion of a full paper is displayed. Results: To date (March 2019), the Cochrane Crowd community comprises over 12,000 contributors from more than 180 countries. Almost 3 million individual classifications have been made, and around 70,000 reports of randomised trials have been identified for Cochrane's Central Register of Controlled Trials. Performance evaluations to assess crowd accuracy have shown crowd sensitivity is 99.1%, and crowd specificity is 99%. Main motivations for involvement are that people want to help Cochrane, and people want to learn. Conclusion: This model of contribution is now an established part of Cochrane's effort to manage the deluge of information produced in a way that offers contributors a chance to get involved, learn and play a crucial role in evidence production. Our experience has shown that people want to be involved and that, with little or no prior experience, can do certain tasks to a very high degree of collective accuracy. Using a citizen science approach effectively has enabled Cochrane to better support its expert community through better use of human effort. It has also generated large, high‐quality data sets on a scale not carried out before which has provided training material for machine learning routines. Citizen science is not an easy option, but performed well it brings a wealth of advantages to both the citizen and the organisation.

Systematic reviews and meta-analyses in the health sciences: Best practice methods for research syntheses

RATIONALE

The journal Social Science & Medicine recently adopted the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA; Moher et al., 2009) as guidelines for authors to use when disseminating their systematic reviews (SRs).

APPROACH

After providing a brief history of evidence synthesis, this article describes why reporting standards are important, summarizes the sequential steps involved in conducting SRs and meta-analyses, and outlines additional methodological issues that researchers should address when conducting and reporting results from their SRs.

RESULTS AND CONCLUSIONS

Successful SRs result when teams of reviewers with appropriate expertise use the highest scientific rigor in all steps of the SR process. Thus, SRs that lack foresight are unlikely to prove successful. We advocate that SR teams consider potential moderators (M) when defining their research problem, along with Time, Outcomes, Population, Intervention, Context, and Study design (i.e., TOPICS + M). We also show that, because the PRISMA reporting standards only partially overlap dimensions of methodological quality, it is possible for SRs to satisfy PRISMA standards yet still have poor methodological quality. As well, we discuss limitations of such standards and instruments in the face of the assumptions of the SR process, including meta-analysis spanning the other SR steps, which are highly synergistic: Study search and selection, coding of study characteristics and effects, analysis, interpretation, reporting, and finally, re-analysis and criticism. When a SR targets an important question with the best possible SR methods, its results can become a definitive statement that guides future research and policy decisions for years to come.

Bringing stroke clinical guidelines to life

Clinical practice guidelines are essential for driving evidence-based clinical care to patients. In an era of ever-increasing research evidence, keeping guidelines up to date is a challenging and resource-intensive process. Advances in technological platforms provide opportunities to develop new models of guideline development that will allow for continuous, rapid updates to recommendations as new evidence emerges. As Australia and other countries begin to develop these models, we have an opportunity to work more closely together to ensure the most efficient use of resources.

Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error

BACKGROUND

Here, we outline a method of applying existing machine learning (ML) approaches to aid citation screening in an on-going broad and shallow systematic review of preclinical animal studies. The aim is to achieve a high-performing algorithm comparable to human screening that can reduce human resources required for carrying out this step of a systematic review.

METHODS

We applied ML approaches to a broad systematic review of animal models of depression at the citation screening stage. We tested two independently developed ML approaches which used different classification models and feature sets. We recorded the performance of the ML approaches on an unseen validation set of papers using sensitivity, specificity and accuracy. We aimed to achieve 95% sensitivity and to maximise specificity. The classification model providing the most accurate predictions was applied to the remaining unseen records in the dataset and will be used in the next stage of the preclinical biomedical sciences systematic review. We used a cross-validation technique to assign ML inclusion likelihood scores to the human screened records, to identify potential errors made during the human screening process (error analysis).

RESULTS

ML approaches reached 98.7% sensitivity based on learning from a training set of 5749 records, with an inclusion prevalence of 13.2%. The highest level of specificity reached was 86%. Performance was assessed on an independent validation dataset. Human errors in the training and validation sets were successfully identified using the assigned inclusion likelihood from the ML model to highlight discrepancies. Training the ML algorithm on the corrected dataset improved the specificity of the algorithm without compromising sensitivity. Error analysis correction leads to a 3% improvement in sensitivity and specificity, which increases precision and accuracy of the ML algorithm.

CONCLUSIONS

This work has confirmed the performance and application of ML algorithms for screening in systematic reviews of preclinical animal studies. It has highlighted the novel use of ML algorithms to identify human error. This needs to be confirmed in other reviews with different inclusion prevalence levels, but represents a promising approach to integrating human decisions and automation in systematic review methodology.

Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error

BACKGROUND

Here, we outline a method of applying existing machine learning (ML) approaches to aid citation screening in an on-going broad and shallow systematic review of preclinical animal studies. The aim is to achieve a high-performing algorithm comparable to human screening that can reduce human resources required for carrying out this step of a systematic review.

METHODS

We applied ML approaches to a broad systematic review of animal models of depression at the citation screening stage. We tested two independently developed ML approaches which used different classification models and feature sets. We recorded the performance of the ML approaches on an unseen validation set of papers using sensitivity, specificity and accuracy. We aimed to achieve 95% sensitivity and to maximise specificity. The classification model providing the most accurate predictions was applied to the remaining unseen records in the dataset and will be used in the next stage of the preclinical biomedical sciences systematic review. We used a cross-validation technique to assign ML inclusion likelihood scores to the human screened records, to identify potential errors made during the human screening process (error analysis).

RESULTS

ML approaches reached 98.7% sensitivity based on learning from a training set of 5749 records, with an inclusion prevalence of 13.2%. The highest level of specificity reached was 86%. Performance was assessed on an independent validation dataset. Human errors in the training and validation sets were successfully identified using the assigned inclusion likelihood from the ML model to highlight discrepancies. Training the ML algorithm on the corrected dataset improved the specificity of the algorithm without compromising sensitivity. Error analysis correction leads to a 3% improvement in sensitivity and specificity, which increases precision and accuracy of the ML algorithm.

CONCLUSIONS

This work has confirmed the performance and application of ML algorithms for screening in systematic reviews of preclinical animal studies. It has highlighted the novel use of ML algorithms to identify human error. This needs to be confirmed in other reviews with different inclusion prevalence levels, but represents a promising approach to integrating human decisions and automation in systematic review methodology.

Prioritising references for systematic reviews with RobotAnalyst: A user study

Screening references is a time-consuming step necessary for systematic reviews and guideline development. Previous studies have shown that human effort can be reduced by using machine learning software to prioritise large reference collections such that most of the relevant references are identified before screening is completed. We describe and evaluate RobotAnalyst, a Web-based software system that combines text-mining and machine learning algorithms for organising references by their content and actively prioritising them based on a relevancy classification model trained and updated throughout the process. We report an evaluation over 22 reference collections (most are related to public health topics) screened using RobotAnalyst with a total of 43 610 abstract-level decisions. The number of references that needed to be screened to identify 95% of the abstract-level inclusions for the evidence review was reduced on 19 of the 22 collections. Significant gains over random sampling were achieved for all reviews conducted with active prioritisation, as compared with only two of five when prioritisation was not used. RobotAnalyst's descriptive clustering and topic modelling functionalities were also evaluated by public health analysts. Descriptive clustering provided more coherent organisation than topic modelling, and the content of the clusters was apparent to the users across a varying number of clusters. This is the first large-scale study using technology-assisted screening to perform new reviews, and the positive results provide empirical evidence that RobotAnalyst can accelerate the identification of relevant studies. The results also highlight the issue of user complacency and the need for a stopping criterion to realise the work savings.

Extending PubMed searches to ClinicalTrials.gov through a machine learning approach for systematic reviews

OBJECTIVES

Despite their essential role in collecting and organizing published medical literature, indexed search engines are unable to cover all relevant knowledge. Hence, current literature recommends the inclusion of clinical trial registries in systematic reviews (SRs). This study aims to provide an automated approach to extend a search on PubMed to the ClinicalTrials.gov database, relying on text mining and machine learning techniques.

STUDY DESIGN AND SETTING

The procedure starts from a literature search on PubMed. Next, it considers the training of a classifier that can identify documents with a comparable word characterization in the ClinicalTrials.gov clinical trial repository. Fourteen SRs, covering a broad range of health conditions, are used as case studies for external validation. A cross-validated support-vector machine (SVM) model was used as the classifier.

RESULTS

The sensitivity was 100% in all SRs except one (87.5%), and the specificity ranged from 97.2% to 99.9%. The ability of the instrument to distinguish on-topic from off-topic articles ranged from an area under the receiver operator characteristic curve of 93.4% to 99.9%.

CONCLUSION

The proposed machine learning instrument has the potential to help researchers identify relevant studies in the SR process by reducing workload, without losing sensitivity and at a small price in terms of specificity.

A Deep Learning Method to Automatically Identify Reports of Scientifically Rigorous Clinical Research from the Biomedical Literature: Comparative Analytic Study

Background

A major barrier to the practice of evidence-based medicine is efficiently finding scientifically sound studies on a given clinical topic.

Objective

To investigate a deep learning approach to retrieve scientifically sound treatment studies from the biomedical literature.

Methods

We trained a Convolutional Neural Network using a noisy dataset of 403,216 PubMed citations with title and abstract as features. The deep learning model was compared with state-of-the-art search filters, such as PubMed’s Clinical Query Broad treatment filter, McMaster’s textword search strategy (no Medical Subject Heading, MeSH, terms), and Clinical Query Balanced treatment filter. A previously annotated dataset (Clinical Hedges) was used as the gold standard.

Results

The deep learning model obtained significantly lower recall than the Clinical Queries Broad treatment filter (96.9% vs 98.4%; P<.001); and equivalent recall to McMaster’s textword search (96.9% vs 97.1%; P=.57) and Clinical Queries Balanced filter (96.9% vs 97.0%; P=.63). Deep learning obtained significantly higher precision than the Clinical Queries Broad filter (34.6% vs 22.4%; P<.001) and McMaster’s textword search (34.6% vs 11.8%; P<.001), but was significantly lower than the Clinical Queries Balanced filter (34.6% vs 40.9%; P<.001).

Conclusions

Deep learning performed well compared to state-of-the-art search filters, especially when citations were not indexed. Unlike previous machine learning approaches, the proposed deep learning model does not require feature engineering, or time-sensitive or proprietary features, such as MeSH terms and bibliometrics. Deep learning is a promising approach to identifying reports of scientifically rigorous clinical research. Further work is needed to optimize the deep learning model and to assess generalizability to other areas, such as diagnosis, etiology, and prognosis.

Machine learning for identifying Randomized Controlled Trials: An evaluation and practitioner's guide

Machine learning (ML) algorithms have proven highly accurate for identifying Randomized Controlled Trials (RCTs) but are not used much in practice, in part because the best way to make use of the technology in a typical workflow is unclear. In this work, we evaluate ML models for RCT classification (support vector machines, convolutional neural networks, and ensemble approaches). We trained and optimized support vector machine and convolutional neural network models on the titles and abstracts of the Cochrane Crowd RCT set. We evaluated the models on an external dataset (Clinical Hedges), allowing direct comparison with traditional database search filters. We estimated area under receiver operating characteristics (AUROC) using the Clinical Hedges dataset.

We demonstrate that ML approaches better discriminate between RCTs and non‐RCTs than widely used traditional database search filters at all sensitivity levels; our best‐performing model also achieved the best results to date for ML in this task (AUROC 0.987, 95% CI, 0.984‐0.989). We provide practical guidance on the role of ML in (1) systematic reviews (high‐sensitivity strategies) and (2) rapid reviews and clinical question answering (high‐precision strategies) together with recommended probability cutoffs for each use case. Finally, we provide open‐source software to enable these approaches to be used in practice.