Heterogeneity in application, design, and analysis characteristics was found for controlled before-after and interrupted time series studies included in Cochrane reviews

Sex as a prognostic factor for mortality in adults with acute symptomatic pulmonary embolism

BACKGROUND

Pulmonary embolism (PE) is relatively common worldwide. It is a serious condition that can be life-threatening. Studies on the relationship between adverse outcomes of this condition and whether a patient is male or female have yielded inconsistent results. Determining whether there is an association between sex and short-term mortality in patients with acute PE is important as this information may help guide different approaches to PE monitoring and treatment.

OBJECTIVES

To determine whether sex (i.e. being a male or a female patient) is an independent prognostic factor for predicting mortality in adults with acute symptomatic pulmonary embolism.

SEARCH METHODS

The Cochrane Vascular Information Specialist searched the Cochrane Vascular Specialised Register, CENTRAL, MEDLINE, Embase, and CINAHL databases, and the World Health Organization International Clinical Trials Registry Platform and ClinicalTrials.gov trials register up to 17 February 2023. We scanned conference abstracts and reference lists of included studies and systematic reviews. We also contacted experts to identify additional studies. There were no restrictions with respect to language or date of publication.

SELECTION CRITERIA

We included phase 2-confirmatory prognostic studies, that is, any longitudinal study (prospective or retrospective) evaluating the independent association between sex (male or female) and mortality in adults with acute PE.

DATA COLLECTION AND ANALYSIS

We followed the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of prognostic factor studies (CHARMS-PF) and the Cochrane Prognosis Methods Group template for prognosis reviews. Two review authors independently screened the studies, extracted data, assessed the risk of bias according to the Quality in Prognosis Studies (QUIPS) tool, and assessed the certainty of the evidence (GRADE). Meta-analyses were performed by pooling adjusted estimates. When meta-analysis was not possible, we reported the main results narratively.

MAIN RESULTS

We included seven studies (726,293 participants), all of which were retrospective cohort studies with participants recruited and managed in hospitals between 2000 and 2018. Studies took place in the USA, Spain, and Japan. Most studies were multicentre. None were conducted in low- or middle-income countries. The participants' mean age ranged from 62 to 69 years, and the proportion of females was higher in six of the seven studies, ranging from 46% to 60%. Sex and gender terms were used inconsistently. Participants received different PE treatments: reperfusion, inferior vena cava filter, anticoagulation, and haemodynamic/respiratory support. The prognostication time (the point from which the outcome was predicted) was frequently omitted. The included studies provided data for three of our outcomes of interest. We did not consider any of the studies to be at an overall low risk of bias for any of the outcomes analysed. We judged the certainty of the evidence as moderate to low due to imprecision and risk of bias. We found moderate-certainty evidence (due to imprecision) that for female patients there is likely a small but clinically important reduction in all-cause mortality at 30 days (odds ratio (OR) 0.81, 95% confidence interval (CI) 0.72 to 0.92; I2 = 0%; absolute risk difference (ARD) 24 fewer deaths in women per 1000 participants, 95% CI 35 to 10 fewer; 2 studies, 17,627 participants). However, the remaining review outcomes do not indicate lower mortality in female patients. There is low-certainty evidence (due to serious risk of bias and imprecision) indicating that for females with PE, there may be a small but clinically important increase in all-cause hospital mortality (OR 1.11, 95% CI 1.00 to 1.22; I2 = 21.7%; 95% prediction interval (PI) 0.76 to 1.61; ARD 13 more deaths in women per 1000 participants, 95% CI 0 to 26 more; 3 studies, 611,210 participants). There is also low-certainty evidence (due to very serious imprecision) indicating that there may be little to no difference between males and females in PE-related mortality at 30 days (OR 1.08, 95% CI 0.55 to 2.12; I2 = 0%; ARD 4 more deaths in women per 1000 participants, 95% CI 22 fewer to 50 more; 2 studies, 3524 participants). No study data was found for the other outcomes, including sex-specific mortality data at one year. Moreover, due to insufficient studies, many of our planned methods were not implemented. In particular, we were unable to conduct assessments of heterogeneity or publication bias or subgroup and sensitivity analyses.

AUTHORS' CONCLUSIONS

The evidence is uncertain about sex (being male or female) as an independent prognostic factor for predicting mortality in adults with PE. We found that, for female patients with PE, there is likely a small but clinically important reduction in all-cause mortality at 30 days relative to male patients. However, this result should be interpreted cautiously, as the remaining review outcomes do not point to an association between being female and having a lower risk of death. In fact, the evidence in the review also suggested that, in female patients, there may be a small but clinically important increase in all-cause hospital mortality. It also showed that there may be little to no difference in PE-related mortality at 30 days between male and female patients. There is currently no study evidence from longitudinal studies for our other review outcomes. Although the available evidence is conflicting and therefore cannot support a recommendation for or against routinely considering sex to quantify prognosis or to guide personalised therapeutic approaches for patients with PE, this Cochrane review offers information to guide future primary research and systematic reviews.

"Start low, go slow," a strategy to tailor treatment dosing in older or vulnerable adults with advanced solid cancer: A systematic review and meta-analysis

INTRODUCTION

The use of standard-dose cancer treatment can result in a decline in the functional abilities of older adults with cancer. The "start-low, go-slow" (SLGS) strategy involves initiating cancer treatment at lower-than-standard doses in selected patients who are vulnerable to excess toxicity and escalating based on tolerance. We performed a systematic review and meta-analysis to assess the available data and the effectiveness of the SLGS strategy in the treatment of cancer in older adults with incurable solid cancer.

MATERIALS AND METHODS

The review was registered with PROSPERO. Two independent reviewers (GA and TP) conducted a comprehensive search across multiple databases (PubMed/Medline, Journal of Geriatric Oncology, American Society of Clinical Oncology abstracts, and EMBASE) of prospective studies involving patients with solid tumors who received SLGS. SLGS was defined as starting cancer therapy with a lower than standard dose and dose-escalating, if possible. The main objective of this study was to evaluate overall survival (OS) in patients treated with the SLGS strategy. Secondary objectives were to analyze treatment discontinuation and toxicity in patients treated with the SLGS strategy. Additionally, we aimed to compile a comprehensive report on studies employing the SLGS strategy in solid oncology. We utilized a random-effects meta-analysis model to consider the diversity among patient populations with different cancer stages, types, and treatments. Two researchers independently employed the Newcastle-Ottawa Quality (NOQ) assessment for cohort analysis to evaluate the methodological quality and standard of outcomes reporting in the included studies. The quality of evidence was appraised using the Grading recommendations assessment, development and evaluation GRADE summary of findings tool.

RESULTS

The systematic search identified a total of 12,690 articles. Thirteen studies met criteria for inclusion in the systematic review, totaling 8546 patients. Twelve studies evaluated OS. However, only five studies focused solely on older adults, and the studies involved different types of cancer without following a specific pattern. In meta-analysis of survival among three studies, patients who underwent the SLGS approach had lower mortality (hazar ratio 0.91, 95 % confidence interval [CI] 0.85-0.98, p = 0.01, i2 = 0 %). Toxicity ranged from 5 % to 89 % across studies; SLGS had lower grade 3 and 4 toxicity compared to the standard dose (six studies, meta-analysis relative risk 0.86, 95 % CI 0.75-0.98, p < 0.02, i2 = 30 %). Treatment discontinuation was not different for SLGS vs. standard dose (seven studies, meta-analysis RR 0.96, 95 % CI 0.87-1.05, p = 0.37 i2 = 50 %).

DISCUSSION

This systematic review and meta-analysis suggests that a SLGS approach to systemic therapy dosing may reduce toxicity without affecting survival among older patients with solid tumors, although results are limited by a limited number of prospective studies. Additional research is needed to understand better the effects of SLGS in older adults receiving palliative chemotherapy.

Development of a search filter to retrieve reports of interrupted time series studies from MEDLINE and PubMed

BACKGROUND

Interrupted time series (ITS) studies contribute importantly to systematic reviews of population-level interventions. We aimed to develop and validate search filters to retrieve ITS studies in MEDLINE and PubMed.

METHODS

A total of 1017 known ITS studies (published 2013-2017) were analysed using text mining to generate candidate terms. A control set of 1398 time-series studies were used to select differentiating terms. Various combinations of candidate terms were iteratively tested to generate three search filters. An independent set of 700 ITS studies was used to validate the filters' sensitivities. The filters were test-run in Ovid MEDLINE and the records randomly screened for ITS studies to determine their precision. Finally, all MEDLINE filters were translated to PubMed format and their sensitivities in PubMed were estimated.

RESULTS

Three search filters were created in MEDLINE: a precision-maximising filter with high precision (78%; 95% CI 74%-82%) but moderate sensitivity (63%; 59%-66%), most appropriate when there are limited resources to screen studies; a sensitivity-and-precision-maximising filter with higher sensitivity (81%; 77%-83%) but lower precision (32%; 28%-36%), providing a balance between expediency and comprehensiveness; and a sensitivity-maximising filter with high sensitivity (88%; 85%-90%) but likely very low precision, useful when combined with specific content terms. Similar sensitivity estimates were found for PubMed versions.

CONCLUSION

Our filters strike different balances between comprehensiveness and screening workload and suit different research needs. Retrieval of ITS studies would be improved if authors identified the ITS design in the titles.

A Systematic Review of the Use of Routine Versus Selective Episiotomy for Vaginal Birth

Episiotomy is a common obstetric surgical procedure involving an incision to enlarge the vaginal opening, facilitating the delivery of the fetus during the second stage of labor. Hence, the current review was conducted to assess the impact of using selective versus routine episiotomy during vaginal birth on birth outcomes. This systematic review used the Joanna Briggs Institute (JBI) methodology for systematic reviews and research syntheses. PubMed, the Cochrane Library, Cumulative Index to Nursing and Allied Health Literature (CINAHL), and SCOPUS were searched for relevant studies. Two reviewers independently screened and extracted data from relevant studies. Four studies met the eligibility criteria and were included in this review. The findings suggest selective episiotomy is associated with better maternal and fetal outcomes than routine episiotomy in certain contexts. However, results varied, indicating the importance of tailoring episiotomy practices to specific patient populations and healthcare settings. To conclude, the review supports the use of selective episiotomy over routine episiotomy during vaginal birth, as it is associated with fewer adverse maternal outcomes.

PROTOCOL: Key characteristics of effective preschool-based interventions to promote self-regulation: A systematic review and meta-analysis

This is the protocol for a Cochrane Review. The objectives are as follows: The aim of this systematic review is to advance our understanding of the key characteristics of effective preschool-based interventions designed to foster self-regulation. To accomplish this, the review addresses the following questions: 1. What types of preschool-based interventions have been developed to promote self-regulation? 2. What is the average effect of these preschool-based interventions on self-regulation, focusing on four key constructs: integrative effortful control, integrative executive function, self-regulation, and self-regulated learning? 3. What characteristics-such as Resource Allocation, Activity Type, and Instruction Method-could potentially contribute to the effects of preschool-based interventions in promoting self-regulation?

Evaluation of statistical methods used to meta-analyse results from interrupted time series studies: A simulation study

Interrupted time series (ITS) are often meta-analysed to inform public health and policy decisions but examination of the statistical methods for ITS analysis and meta-analysis in this context is limited. We simulated meta-analyses of ITS studies with continuous outcome data, analysed the studies using segmented linear regression with two estimation methods [ordinary least squares (OLS) and restricted maximum likelihood (REML)], and meta-analysed the immediate level- and slope-change effect estimates using fixed-effect and (multiple) random-effects meta-analysis methods. Simulation design parameters included varying series length; magnitude of lag-1 autocorrelation; magnitude of level- and slope-changes; number of included studies; and, effect size heterogeneity. All meta-analysis methods yielded unbiased estimates of the interruption effects. All random effects meta-analysis methods yielded coverage close to the nominal level, irrespective of the ITS analysis method used and other design parameters. However, heterogeneity was frequently overestimated in scenarios where the ITS study standard errors were underestimated, which occurred for short series or when the ITS analysis method did not appropriately account for autocorrelation. The performance of meta-analysis methods depends on the design and analysis of the included ITS studies. Although all random effects methods performed well in terms of coverage, irrespective of the ITS analysis method, we recommend the use of effect estimates calculated from ITS methods that adjust for autocorrelation when possible. Doing so will likely to lead to more accurate estimates of the heterogeneity variance.

Quantifying the impact of COVID-19 on essential health services: a comparison of interrupted time series analysis using Prophet and Poisson regression models

BACKGROUND

Coronavirus disease 2019 (COVID-19) altered healthcare utilization patterns. However, there is a dearth of literature comparing methods for quantifying the extent to which the pandemic disrupted healthcare service provision in sub-Saharan African countries.

OBJECTIVE

To compare interrupted time series analysis using Prophet and Poisson regression models in evaluating the impact of COVID-19 on essential health services.

METHODS

We used reported data from Uganda's Health Management Information System from February 2018 to December 2020. We compared Prophet and Poisson models in evaluating the impact of COVID-19 on new clinic visits, diabetes clinic visits, and in-hospital deliveries between March 2020 to December 2020 and across the Central, Eastern, Northern, and Western regions of Uganda.

RESULTS

The models generated similar estimates of the impact of COVID-19 in 10 of the 12 outcome-region pairs evaluated. Both models estimated declines in new clinic visits in the Central, Northern, and Western regions, and an increase in the Eastern Region. Both models estimated declines in diabetes clinic visits in the Central and Western regions, with no significant changes in the Eastern and Northern regions. For in-hospital deliveries, the models estimated a decline in the Western Region, no changes in the Central Region, and had different estimates in the Eastern and Northern regions.

CONCLUSIONS

The Prophet and Poisson models are useful in quantifying the impact of interruptions on essential health services during pandemics but may result in different measures of effect. Rigor and multimethod triangulation are necessary to study the true effect of pandemics on essential health services.

Methodological systematic review recommends improvements to conduct and reporting when meta-analyzing interrupted time series studies

OBJECTIVES

Interrupted Time Series (ITS) are a type of nonrandomized design commonly used to evaluate public health policy interventions, and the impact of exposures, at the population level. Meta-analysis may be used to combine results from ITS across studies (in the context of systematic reviews) or across sites within the same study. We aimed to examine the statistical approaches, methods, and completeness of reporting in reviews that meta-analyze results from ITS.

STUDY DESIGN AND SETTINGS

Eight electronic databases were searched to identify reviews (published 2000-2019) that meta-analyzed at least two ITS. Characteristics of the included reviews, the statistical methods used to analyze the ITS and meta-analyze their results, effect measures, and risk of bias assessment tools were extracted.

RESULTS

Of the 4213 identified records, 54 reviews were included. Nearly all reviews (94%) used two-stage meta-analysis, most commonly fitting a random effects model (69%). Among the 41 reviews that re-analyzed the ITS, linear regression (39%) and ARIMA (20%) were most commonly used; 38% adjusted for autocorrelation. The most common effect measure meta-analyzed was an immediate level-change (46/54). Reporting of the statistical methods and ITS characteristics was often incomplete.

CONCLUSION

Improvement is needed in the conduct and reporting of reviews that meta-analyze results from ITS.

Interrupted Time Series Analysis of Pediatric Infectious Diseases and the Consumption of Antibiotics in an Atlantic European Region during the SARS-CoV-2 Pandemic

The increasing concern about bacterial resistance has made the rational prescription of antibiotics even more urgent. The non-pharmacological measures established to reduce the impact of the SARS-CoV-2 pandemic have modified the epidemiology of pediatric infections and, consequently, the use of antibiotics. Interrupted time series (ITS) analyses are quasi-experimental studies that allow for the estimation of causal effects with observational data in "natural experiments", such as changes in health policies or pandemics. The effect of the SARS-CoV-2 pandemic on the incidence of infectious diseases and the use of antibiotics between 2018 and 2020 in the Health Area of Vigo (Galicia, Spain) was quantified and analyzed. This paper outlines a real-world data study with administrative records from primary care services provided for the pediatric population. The records were related to episodes classified as infectious by the International Classification of Primary Care (ICPC-2) and oral medication in the therapeutic subgroup J01, corresponding to antibiotics for systemic use, according to the World Health Organization's Anatomical Therapeutic Chemical (ATC) classification system. The records were classified according to incident episodes, age, dose per inhabitant, and year. Segmented regression models were applied using an algorithm that automatically identifies the number and position of the change points. During the SARS-CoV-2 pandemic, the number of infectious diseases being transmitted between individuals, through the air and through the fecal-oral route, significantly decreased, and a slight decrease in infections transmitted via other mechanisms (urinary tract infections) was also found. In parallel, during the months of the pandemic, there has been a marked and significant reduction in antibacterial agent utilization, mainly of penicillins, cephalosporins, and macrolides.

Methodological considerations for interrupted time series analysis in radiation epidemiology: an overview

Interrupted time series analysis (ITSA) is a method that can be applied to evaluate health outcomes in populations exposed to ionizing radiation following major radiological events. Using aggregated time series data, ITSA evaluates whether the time trend of a health indicator shows a change associated with the radiological event. That is, ITSA checks whether there is a statistically significant discrepancy between the projection of a pre-event trend and the data empirically observed after the event. Conducting ITSA requires one to consider specific methodological issues due to unique threats to internal validity that make ITSA prone to bias. We here discuss the strengths and limitations of ITSA with respect to bias and confounding, data quality, and statistical aspects. We provide recommendations to strengthen the robustness of ITSA studies and reduce their susceptibility to producing spurious results as a consequence of arbitrary modelling decisions.

Effectiveness of Meningococcal Vaccines at Reducing Invasive Meningococcal Disease and Pharyngeal Neisseria meningitidis Carriage: A Systematic Review and Meta-analysis

BACKGROUND

Invasive meningococcal disease (IMD), caused by Neisseria meningitidis, leads to significant morbidity and mortality worldwide. This review aimed to establish the effectiveness of meningococcal vaccines at preventing IMD and N. meningitidis pharyngeal carriage.

METHODS

A search within PubMed, Embase, Scopus, and unpublished studies up to 1 February 2020 was conducted.

RESULTS

After removal of duplicates, 8565 studies were screened and 27 studies included. Protection was provided by meningococcal C vaccines for group C IMD (odds ratio [OR], 0.13 [95% confidence interval {CI}, .07-.23]), outer membrane vesicle (OMV) vaccines against group B IMD (OR, 0.35 [95% CI, .25-.48]), and meningococcal A, C, W, Y (MenACWY) vaccines against group ACWY IMD (OR, 0.31 [95% CI, .20-.49]). A single time series analysis found a reduction following an infant 4CMenB program (incidence rate ratio, 0.25 [95% CI, .19-.36]). Multivalent MenACWY vaccines did not reduce carriage (relative risk [RR], 0.88 [95% CI, .66-1.18]), unlike monovalent C vaccines (RR, 0.50 [95% CI, .26-.97]). 4CMenB vaccine had no effect on group B carriage (RR, 1.12 [95% CI, .90-1.40]). There was also no reduction in group B carriage following MenB-FHbp vaccination (RR, 0.98 [95% CI, .53-1.79]).

CONCLUSIONS

Meningococcal conjugate C, ACWY, and OMV vaccines are effective at reducing IMD. A small number of studies demonstrate that monovalent C conjugate vaccines reduce pharyngeal N. meningitidis carriage. There is no evidence of carriage reduction for multivalent MenACWY, OMV, or recombinant MenB vaccines, which has implications for immunization strategies.

CLINICAL TRIALS REGISTRATION

CRD42018082085 (PROSPERO).

Current Practices in Missing Data Handling for Interrupted Time Series Studies Performed on Individual-Level Data: A Scoping Review in Health Research

OBJECTIVE

Missing data can produce biased estimates in interrupted time series (ITS) analyses. We reviewed recent ITS investigations on health topics for determining 1) the data management strategies and statistical analysis performed, 2) how often missing data were considered and, if so, how they were evaluated, reported and handled.

STUDY DESIGN AND SETTING

This was a scoping review following standard recommendations from the PRISMA Extension for Scoping Reviews. We included a random sample of all ITS studies that assessed any intervention relevant to health care (eg, policies or programmes) with individual-level data, published in 2019, with abstracts indexed on MEDLINE.

RESULTS

From 732 studies identified, we finally reviewed 60. Reporting of missing data was rare. Data aggregation, statistical tools for modelling population-level data and complete case analyses were preferred, but these can lead to bias when data are missing at random. Seasonality and other time-dependent confounders were rarely accounted for and, when they were, missing data implications were typically ignored. Very few studies reflected on the consequences of missing data.

CONCLUSION

Handling and reporting of missing data in recent ITS studies performed for health research have many shortcomings compared with best practice.

The dangers of ignoring underlying trends in before-and-after studies - A cautionary tale using hip fracture mortality data

BACKGROUND

Before-and-after studies are a valuable study design in situations where randomization is not feasible. These studies measure an outcome both before and after an intervention and compare the outcome rates in both time periods to determine the effectiveness of the intervention. Before-and-after studies do not involve a contemporaneous control group and must, therefore, take into account any underlying secular trends to separate the effect of the intervention from any pre-existing trend.

METHODS

To illustrate the importance of accounting for underlying trends, we performed a before-and-after study assessing 30-day mortality in hip fracture patients without any actual intervention, and instead designated an arbitrarily chosen time point as our 'intervention'. We then analysed the data first disregarding and then incorporating the pre-existing underlying trend. We did this to show that even intervention of nothing may be spuriously interpreted to have an effect if the before-and-after study design is incorrectly analysed. Our study involved a secondary analysis of routinely collected data on 30-day mortality following hip fracture in our institution.

RESULTS

We found a secular trend in our data showing improving 30-day mortality in hip fracture patients in our institution. We then demonstrated that disregarding this underlying trend showed that our intervention of nothing 'resulted' in a significant 54% decrease in mortality, from 6.7% in the 'before' period to 3.1% in the 'after' period (p < 0.0008). Though the 30-day mortality rate decreased during the 'after' period, the decrease was not significantly different from the underlying trend in the 'before' period, projected onto the 'after' period. When we accounted for the underlying trend in our analysis, the impact of the intervention (nothing) on 30-day mortality was no longer apparent (incidence rate ratio 0.75, 95% confidence interval 0.32-1.78; p = 0.5).

CONCLUSION

Our study highlights the importance of appropriate measurement and consideration of underlying trends when analysing data from before-and-after studies and illustrates what can happen should researchers neglect this important step.

Interrupted time series study found mixed effects of the impact of the Bavarian smoke-free legislation on pregnancy outcomes

In 2007 the German government passed smoke-free legislation, leaving the details of implementation to the individual federal states. In January 2008 Bavaria implemented one of the strictest laws in Germany. We investigated its impact on pregnancy outcomes and applied an interrupted time series (ITS) study design to assess any changes in preterm birth, small for gestational age (primary outcomes), and low birth weight, stillbirth and very preterm birth. We included 1,236,992 singleton births, comprising 83,691 preterm births and 112,143 small for gestational age newborns. For most outcomes we observed unclear effects. For very preterm births, we found an immediate drop of 10.4% (95%CI - 15.8, - 4.6%; p = 0.0006) and a gradual decrease of 0.5% (95%CI - 0.7, - 0.2%, p = 0.0010) after implementation of the legislation. The majority of subgroup and sensitivity analyses confirm these results. Although we found no statistically significant effect of the Bavarian smoke-free legislation on most pregnancy outcomes, a substantial decrease in very preterm births was observed. We cannot rule out that despite our rigorous methods and robustness checks, design-inherent limitations of the ITS study as well as country-specific factors, such as the ambivalent German policy context have influenced our estimation of the effects of the legislation.

Impact of Meningococcal B Vaccine on Invasive Meningococcal Disease in Adolescents

Background

From 2017, a statewide cluster randomized trial was conducted in South Australia to assess the impact of the meningococcal B vaccine 4CMenB on pharyngeal Neisseria meningitidis carriage in adolescents. Senior schools were randomized to receive the vaccine in 2017 (intervention) or 2018 (control). In this study we report the vaccine impact of 4CMenB on serogroup B invasive meningococcal disease (IMD) in 16- to 19-year-old adolescents in South Australia.

Methods

This observational time series analysis of serogroup B IMD cases compares the 14 years prior to the commencement of the trial (2003–2016) with the 2 years following 4CMenB vaccination of the 2017 adolescent cohort.

Results

Approximately 62% of year 10 and 11 students (15–16 years old) in South Australia enrolled in the trial. A total of 30 522 year 10–12 students received at least 1 dose of 4CMenB. The number of serogroup B IMD cases in 16- to 19-year old adolescents in South Australia increased on average by 10% per year from 2003 to 2016 (95% confidence interval [CI], 6%–15%, P &lt; .001), peaking with 10 cases in 2015. Serogroup B IMD cases reduced to 5 in 2017–2018 and 1 in 2018–2019, below the expected numbers of 9.9 (95% prediction interval [PI], 3.9–17.5) and 10.9 (95% PI, 4.4–19.1), respectively. This translated to an overall reduction in the number of serogroup B IMD cases of 71% (95% CI, 15%–90%, P = .02). There were no serogroup B IMD cases in vaccinated adolescents.

Conclusions

Vaccinating adolescents with 4CMenB was associated with a reduction in group B meningococcal disease in South Australia.

Clinical Trials Registration

NCT03089086.

Use of interrupted time series methods in the evaluation of health system quality improvement interventions: a methodological systematic review

BACKGROUND

When randomisation is not possible, interrupted time series (ITS) design has increasingly been advocated as a more robust design to evaluating health system quality improvement (QI) interventions given its ability to control for common biases in healthcare QI. However, there is a potential risk of producing misleading results when this rather robust design is not used appropriately. We performed a methodological systematic review of the literature to investigate the extent to which the use of ITS has followed best practice standards and recommendations in the evaluation of QI interventions.

METHODS

We searched multiple databases from inception to June 2018 to identify QI intervention studies that were evaluated using ITS. There was no restriction on date, language and participants. Data were synthesised narratively using appropriate descriptive statistics. The risk of bias for ITS studies was assessed using the Cochrane Effective Practice and Organisation of Care standard criteria. The systematic review protocol was registered in PROSPERO (registration number: CRD42018094427).

RESULTS

Of 4061 potential studies and 2028 unique records screened for inclusion, 120 eligible studies assessed eight QI strategies and were from 25 countries. Most studies were published since 2010 (86.7%), reported data using monthly interval (71.4%), used ITS without a control (81%) and modelled data using segmented regression (62.5%). Autocorrelation was considered in 55% of studies, seasonality in 20.8% and non-stationarity in 8.3%. Only 49.2% of studies specified the ITS impact model. The risk of bias was high or very high in 72.5% of included studies and did not change significantly over time.

CONCLUSIONS

The use of ITS in the evaluation of health system QI interventions has increased considerably over the past decade. However, variations in methodological considerations and reporting of ITS in QI remain a concern, warranting a need to develop and reinforce formal reporting guidelines to improve its application in the evaluation of health system QI interventions.

Taxation of the fat content of foods for reducing their consumption and preventing obesity or other adverse health outcomes

BACKGROUND

Overweight and obesity are increasing worldwide and are considered to be a major public health issue of the 21st century. Introducing taxation of the fat content in foods is considered a potentially powerful policy tool to reduce consumption of foods high in fat or saturated fat, or both.

OBJECTIVES

To assess the effects of taxation of the fat content in food on consumption of total fat and saturated fat, energy intake, overweight, obesity, and other adverse health outcomes in the general population.

SEARCH METHODS

We searched CENTRAL, Cochrane Database of Systematic Reviews, MEDLINE, Embase, and 15 other databases and trial registers on 12 September 2019. We handsearched the reference lists of all records of included studies, searched websites of international organizations and institutions (14 October 2019), and contacted review advisory group members to identify planned, ongoing, or unpublished studies (26 February 2020).

SELECTION CRITERIA

In line with Cochrane Effective Practice and Organisation of Care Group (EPOC) criteria, we included the following study types: randomized controlled trials (RCTs), cluster-randomized controlled trials (cRCTs), non-randomized controlled trials (nRCTs), controlled before-after (CBA) studies, and interrupted time series studies. We included studies that evaluated the effects of taxes on the fat content in foods. Such a tax could be expressed as sales, excise, or special value added tax (VAT) on the final product or an intermediary product. Eligible interventions were taxation at any level, with no restriction on the duration or the implementation level (i.e. local, regional, national, or multinational). Eligible study populations were children (zero to 17 years) and adults (18 years or older) from any country and setting. We excluded studies that focused on specific subgroups only (e.g. people receiving pharmaceutical intervention; people undergoing a surgical intervention; ill people who are overweight or obese as a side effect, such as those with thyroiditis and depression; and people with chronic illness). Primary outcomes were total fat consumption, consumption of saturated fat, energy intake through fat, energy intake through saturated fat, total energy intake, and incidence/prevalence of overweight or obesity. We did not exclude studies based on country, setting, comparison, or population.

DATA COLLECTION AND ANALYSIS

We used standard Cochrane methods for all phases of the review. Risk of bias of the included studies was assessed using the criteria of Cochrane's 'Risk of bias' tool and the EPOC Group's guidance. Results of the review are summarized narratively and the certainty of the evidence was assessed using the GRADE approach. These steps were done by two review authors, independently.

MAIN RESULTS

We identified 23,281 records from searching electronic databases and 1173 records from other sources, leading to a total of 24,454 records. Two studies met the criteria for inclusion in the review. Both included studies investigated the effect the Danish tax on saturated fat contained in selected food items between 2011 and 2012. Both studies used an interrupted time series design. Neither included study had a parallel control group from another geographic area. The included studies investigated an unbalanced panel of approximately 2000 households in Denmark and the sales data from a specific Danish supermarket chain (1293 stores). Therefore, the included studies did not address individual participants, and no restriction regarding age, sex, and socioeconomic characteristics were defined. We judged the overall risk of bias of the two included studies as unclear. For the outcome total consumption of fat, a reduction of 41.8 grams per week per person in a household (P < 0.001) was estimated. For the consumption of saturated fat, one study reported a reduction of 4.2% from minced beef sales, a reduction of 5.8% from cream sales, and an increase of 0.5% to sour cream sales (no measures of statistical precision were reported for these estimates). These estimates are based on a restricted number of food types and derived from sales data; they do not measure individual intake. Moreover, these estimates do not account for other relevant sources of fat intake (e.g. packaged or processed food) or other food outlets (e.g. restaurants or cafeterias); hence, we judged the evidence on the effect of taxation on total fat consumption or saturated fat consumption to be very uncertain. We did not identify evidence on the effect of the intervention on energy intake or the incidence or prevalence of overweight or obesity.

AUTHORS' CONCLUSIONS

Given the very low quality of the evidence currently available, we are unable to reliably establish whether a tax on total fat or saturated fat is effective or ineffective in reducing consumption of total fat or saturated fat. There is currently no evidence on the effect of a tax on total fat or saturated fat on total energy intake or energy intake through saturated fat or total fat, or preventing the incidence or reducing the prevalence of overweight or obesity.

Methods used to meta-analyse results from interrupted time series studies: A methodological systematic review protocol

Background: Systematic reviews are used to inform healthcare decision making. In reviews that aim to examine the effects of organisational, policy change or public health interventions, or exposures, evidence from interrupted time series (ITS) studies may be included. A core component of many systematic reviews is meta-analysis, which is the statistical synthesis of results across studies. There is currently a lack of guidance informing the choice of meta-analysis methods for combining results from ITS studies, and there have been no studies examining the meta-analysis methods used in practice. This study therefore aims to describe current meta-analysis methods used in a cohort of reviews of ITS studies. Methods: We will identify 100 reviews that include meta-analyses of ITS studies from a search of eight electronic databases covering several disciplines (public health, psychology, education, economics). Study selection will be undertaken independently by two authors. Data extraction will be undertaken by one author, and for a random sample of the reviews, two authors. From eligible reviews we will extract details at the review level including discipline and type of interruption; at the meta-analytic level we will extract type of outcome, effect measure(s), meta-analytic methods, and any methods used to re-analyse the individual ITS studies. Descriptive statistics will be used to summarise the data. Conclusions: This review will describe the methods used to meta-analyse results from ITS studies. Results from this review will inform future methods research examining how different meta-analysis methods perform, and ultimately, the development of guidance.

Methods used to meta-analyse results from interrupted time series studies: A methodological systematic review protocol

Background: Systematic reviews are used to inform healthcare decision making. In reviews that aim to examine the effects of organisational, policy change or public health interventions, or exposures, evidence from interrupted time series (ITS) studies may be included. A core component of many systematic reviews is meta-analysis, which is the statistical synthesis of results across studies. There is currently a lack of guidance informing the choice of meta-analysis methods for combining results from ITS studies, and there have been no studies examining the meta-analysis methods used in practice. This study therefore aims to describe current meta-analysis methods used in a cohort of reviews of ITS studies. Methods: We will identify the 100 most recent reviews (published between 1 January 2000 and 11 October 2019) that include meta-analyses of ITS studies from a search of eight electronic databases covering several disciplines (public health, psychology, education, economics). Study selection will be undertaken independently by two authors. Data extraction will be undertaken by one author, and for a random sample of the reviews, two authors. From eligible reviews we will extract details at the review level including discipline, type of interruption and any tools used to assess the risk of bias / methodological quality of included ITS studies; at the meta-analytic level we will extract type of outcome, effect measure(s), meta-analytic methods, and any methods used to re-analyse the individual ITS studies. Descriptive statistics will be used to summarise the data. Conclusions: This review will describe the methods used to meta-analyse results from ITS studies. Results from this review will inform future methods research examining how different meta-analysis methods perform, and ultimately, the development of guidance.

Methods used to meta-analyse results from interrupted time series studies: A methodological systematic review protocol

Background: Systematic reviews are used to inform healthcare decision making. In reviews that aim to examine the effects of organisational, policy change or public health interventions, or exposures, evidence from interrupted time series (ITS) studies may be included. A core component of many systematic reviews is meta-analysis, which is the statistical synthesis of results across studies. There is currently a lack of guidance informing the choice of meta-analysis methods for combining results from ITS studies, and there have been no studies examining the meta-analysis methods used in practice. This study therefore aims to describe current meta-analysis methods used in a cohort of reviews of ITS studies. Methods: We will identify the 100 most recent reviews (published between 1 January 2000 and 11 October 2019) that include meta-analyses of ITS studies from a search of eight electronic databases covering several disciplines (public health, psychology, education, economics). Study selection will be undertaken independently by two authors. Data extraction will be undertaken by one author, and for a random sample of the reviews, two authors. From eligible reviews we will extract details at the review level including discipline, type of interruption and any tools used to assess the risk of bias / methodological quality of included ITS studies; at the meta-analytic level we will extract type of outcome, effect measure(s), meta-analytic methods, and any methods used to re-analyse the individual ITS studies. Descriptive statistics will be used to summarise the data. Conclusions: This review will describe the methods used to meta-analyse results from ITS studies. Results from this review will inform future methods research examining how different meta-analysis methods perform, and ultimately, the development of guidance.

Myosteatosis and prognosis in cancer: Systematic review and meta-analysis

BACKGROUND

The evidence that body composition parameters influence multiple cancer outcomes is rapidly expanding. Excess adiposity deposits in muscle tissue, termed myosteatosis, can be detected in CT scans through variations in the density of muscle tissues (Hounsfield Units). Patients with similar muscle mass but different amounts of intramuscular adipose infiltration have increased chemotherapy toxicity, time to tumor progression and other adverse outcomes among different cancer types. Our review examines the impact of myosteatosis on overall survival (OS) in patients with cancer.

METHODS

A systematic search of the literature was conducted on PubMed/ MEDLINE, Cochrane CENTRAL, and EMBASE. Meta-analysis was conducted using a random-effects model. Risk of bias was evaluated using the Newcastle-Ottawa Quality assessment for cohort studies, funnel plot (publication bias), and GRADE summary of findings tool from Cochrane.

RESULTS

A total of 4880 articles were screened from which 40 articles selected, including 21,222 patients. The overall mean proportion of patients with myosteatosis was 48 % (range 11-85 %). Using skeletal muscle density (SMD), patients classified as having myosteatosis had 75 % greater mortality risk compared to non-myosteatosis patients (HR 1.75 95 % CI 1.60-1.92, 40 studies) (p < .00001) (i2 = 62 %). Specifically, myosteatosis was prognostic for worse OS in patients with gynecological, renal, periampullary/pancreatic, hepatocellular, gastroesophageal, and colorectal carcinoma, and lymphomas.

CONCLUSION

Our analysis of the literature shows that cancer patients with myosteatosis have shorter survival. Our findings suggest that in oncological practice, muscle density assessment is valuable as a prognostic parameter.

Health Technology Assessment of Public Health Interventions Published 2012 to 2016: An Analysis of Characteristics and Comparison of Methods

Objectives

The aim of this study was to provide an overview of the methodological characteristics and compare the assessment methods applied in health technology assessments (HTAs) of public health interventions (PHIs).

Methods

We defined a PHI as a population-based intervention on health promotion or for primary prevention of chronic or nonchronic diseases. HTAs on PHIs were identified by systematically searching the Web pages of members of international HTA networks. We included only full HTA reports published between 2012 and 2016. Two reviewers extracted data on the methods used to assess effectiveness/safety, as well as on economic, social, cultural, ethical, and legal aspects using a-priori standardized tables.

Results

We included ten HTAs provided by four different organizations. Of these, all reports assessed the effectiveness of the interventions and conducted economic evaluations, seven investigated social/cultural aspects, and four each considered legal and ethical aspects, respectively. Some reports addressed applicability, context/setting, and intervention fidelity issues in different ways. We found that most HTAs adapted their methods to some extent, for example, by including nonrandomized studies, expanding the search strategy, involving stakeholders, or applying a framework to guide the HTA process.

Conclusions

Our analysis provides a comprehensive overview of methods applied in HTAs on public health interventions. We found that a heterogeneous set of approaches is used to deal with the challenges of evaluating complex public health interventions.

Muscle composition and outcomes in patients with breast cancer: meta-analysis and systematic review

PURPOSE

Breast cancer is the most common cancer and leading cause of cancer death in women. Body composition parameters, especially those related to muscle, have become a growing focus of cancer research. In this review, we summarize the literature on breast cancer and muscle parameters as well as combine their outcomes for overall survival (OS), time to tumor progression (TTP), and chemotherapy toxicity in a meta-analysis.

METHODS

A systematic search of the literature for randomized controlled trials and observational studies was conducted on MEDLINE, Cochrane CENTRAL, and EMBASE through May 1, 2019. Two reviewers independently searched and selected. Meta-analysis was conducted using a random-effects model. The risk of bias was evaluated using the Newcastle-Ottawa quality assessment for cohorts and GRADE summary of findings tool from Cochrane.

RESULTS

A total of 754 articles were screened from which 6 articles and one abstract were selected. Using skeletal muscle index (SMI), patients classified as sarcopenic had a 68% greater mortality risk compared to non-sarcopenic patients (HR 1.68 95% CI 1.09-2.59, 5 studies) (p = .02) (i² = 70%). Low muscle density was not predictive of OS (HR 1.44 95% CI 0.77-2.68, 2 studies) (p = .25) (i² = 87%). Patients with sarcopenia (56%) had more grade 3-5 toxicity compared to non-sarcopenic (25%) (RR 2.17 95% CI 1.4-3.34, 3 studies) (p = .0005) (i² = 0%). TTP was nearly 71 days longer in advanced/metastatic patients classified as non-sarcopenic compared to patients with sarcopenia (MD - 70.75 95% CI - 122.32 to - 19.18) (p = .007) (i² = 0%).

CONCLUSION

Our synthesis of the literature shows that patients with sarcopenia have more severe chemotherapy toxicity as well as shorter OS and TTP, and that low muscle density is prognostic of OS for women with metastatic breast cancer. Our findings suggest that in clinical practice, body composition assessment is valuable as a prognostic parameter in breast cancer.

Methodology and reporting characteristics of studies using interrupted time series design in healthcare

BACKGROUND

Randomised controlled trials (RCTs) are considered the gold standard when evaluating the causal effects of healthcare interventions. When RCTs cannot be used (e.g. ethically difficult), the interrupted time series (ITS) design is a possible alternative. ITS is one of the strongest quasi-experimental designs. The aim of this methodological study was to describe how ITS designs were being used, the design characteristics, and reporting in the healthcare setting.

METHODS

We searched MEDLINE for reports of ITS designs published in 2015 which had a minimum of two data points collected pre-intervention and one post-intervention. There was no restriction on participants, language of study, or type of outcome. Data were summarised using appropriate summary statistics.

RESULTS

One hundred and sixteen studies were included in the study. Interventions evaluated were mainly programs 41 (35%) and policies 32 (28%). Data were usually collected at monthly intervals, 74 (64%). Of the 115 studies that reported an analysis, the most common method was segmented regression (78%), 55% considered autocorrelation, and only seven reported a sample size calculation. Estimation of intervention effects were reported as change in slope (84%) and change in level (70%) and 21% reported long-term change in levels.

CONCLUSIONS

This methodological study identified problems in the reporting of design features and results of ITS studies, and highlights the need for future work in the development of formal reporting guidelines and methodological work.

A systematic review of interventions to improve uptake of pertussis vaccination in pregnancy

BACKGROUND

Maternal pertussis vaccination has been introduced in several countries to prevent pertussis morbidity and mortality in infants too young to be vaccinated. Our review aimed to systematically collect and summarize the available evidence on the effectiveness of interventions used to improve pertussis vaccination uptake in pregnant women.

METHODS

We conducted a systematic search of MEDLINE/PubMed, PMC and CINAHL. Before and after studies and those with a concurrent control group were considered for inclusion. Standardized effect sizes were described as the ratio of the odds to be vaccinated in the intervention group compared with the standard care group and absolute benefit increase (ABI) were calculated.

RESULTS

Six studies were included in the review, of which three were randomized controlled trials (RCTs). Strategies to improve uptake were focused on healthcare providers, pregnant women, or enhancing vaccine access. Healthcare provider interventions included provider reminder, education, feedback and standing orders. Interventions directed at pregnant women focused solely on education. Observational studies showed: (1) the provision of maternal pertussis vaccination by midwives at the place of antenatal care has improved uptake of pertussis vaccine during pregnancy from 20% to 90%; (2) introduction of an automated reminder within the electronic medical record was associated with an improvement in the pertussis immunization rate from 48% to 97%; (3) an increase in prenatal pertussis vaccine uptake from 36% to 61% after strategies to increase provider awareness of recommendations were introduced. In contrast to these findings, interventions in all three RCTs (2 involved education of pregnant women, 1 had multi-component interventions) did not demonstrate improved vaccination uptake.

CONCLUSIONS

Based on the existing research, we recommend incorporating midwife delivered maternal immunization programs at antenatal clinics, use of a provider reminder system to target unvaccinated pregnant women and include maternal pertussis immunization as part of standard antenatal care.

Total Versus Subtotal Hysterectomy: Systematic Review and Meta-analysis of Intraoperative Outcomes and Postoperative Short-term Events

PURPOSE

The benefits and disadvantages of cervical extraction during hysterectomy are unclear in the literature. We intended to compare total (TH) with subtotal or supracervical (SH) hysterectomy regarding intraoperative and postoperative outcomes (quality of life, sexual function, pain and cyclical bleeding).

METHODS

A systematic literature search for randomized controlled trials was conducted on MEDLINE, LILACS, Cochrane CENTRAL, SCOPUS, EMBASE, Clinicaltrials.gov databases, and conference abstracts (AAGL, AUGS, ICS) from 1970 to November 2017. Two reviewers independently searched, selected and then combined the articles. Meta-analyses were conducted using a random-effect model. The risk of bias was evaluated using the Cochrane's Collaboration tool.

FINDINGS

Eleven studies were included involving 1523 patients. The analyses showed that the events operative time (mean difference: 12.88 minutes, 95%CI [7.45, 18.30] p < 0.000001), hospital stay (MD .44 days, 95%CI [0.11, 0.77] p = 0.0008), and intraoperative blood loss (MD 81.06 ml, 95%CI [9.16, 152.97] p = 0.03) favored SH over TH, although the rate of blood transfusion did not differ between the groups. Conversely, TH group had less cyclical vaginal bleeding over SH (1.2% versus 14.1%; RR .14 95%CI [0.05, 0.43] p = 0.0006) during one-year follow up. Persistent pain and sexual satisfaction rates, and quality of life scores were similar in both total and subtotal hysterectomy groups up to 12 months follow up.

IMPLICATIONS

Overall perioperative outcomes favored the preservation of the cervix during hysterectomy but women that had SH are more susceptible to present cyclical vaginal bleeding mimicking menstruation. Those factors should be taken into account along with patient's needs and expectations prior to selecting the procedure.

Low dose naltrexone: Effects on medication in rheumatoid and seropositive arthritis. A nationwide register-based controlled quasi-experimental before-after study

In recent years, low dose naltrexone (LDN) has been used as an off-label therapy for several chronic diseases. Results from small laboratory and clinical studies indicate some beneficial effects of LDN in autoimmune diseases, but clinical research on LDN in rheumatic disease is limited. Using a pharmacoepidemiological approach, we wanted to test the hypothesis that starting LDN leads to reduced dispensing of medicines used in the treatment of rheumatic disease. We performed a controlled before-after study based on the Norwegian Prescription Database (NorPD) to compare prescriptions to patients one year before and one year after starting LDN in 2013. The identified patients (n = 360) were stratified into three groups based on LDN exposure. Outcomes were differences in dispensing of medicines used in rheumatic disease. In persistent LDN users, there was a 13% relative reduction in cumulative defined daily doses (DDD) of all medicines examined corresponding to -73.3 DDD per patient (95% CI -120,2 to -26.4, p = 0.003), and 23% reduction of analgesics (-21.6 DDD (95% CI -35.5 to -7.6, p<0.009)). There was no significant DDD change in patients with lower LDN exposure. Persistent LDN users had significantly reduced DDDs of NSAID and opioids, and a lower proportion of users of DMARDs (-6.7 percentage points, 95% CI -12.3 to-1.0, p = 0.028), TNF-α antagonists and opioids. There was a decrease in the number of NSAID users among patients with the least LDN exposure. Important limitations are that prescription data are proxies for clinical effects and that a control group unexposed to LDN is lacking. The results support the hypothesis that persistent use of LDN reduces the need for medication used in the treatment of rheumatic and seropositive arthritis. Randomised clinical trials on LDN in rheumatic disease are warranted.

Considerations of complexity in rating certainty of evidence in systematic reviews: a primer on using the GRADE approach in global health

Public health interventions and health technologies are commonly described as 'complex', as they involve multiple interacting components and outcomes, and their effects are largely influenced by contextual interactions and system-level processes. Systematic reviewers and guideline developers evaluating the effects of these complex interventions and technologies report difficulties in using existing methods and frameworks, such as the Grading of Recommendations Assessment, Development and Evaluation (GRADE). As part of a special series of papers on implications of complexity in the WHO guideline development, this paper serves as a primer on how to consider sources of complexity when using the GRADE approach to rate certainty of evidence. Relevant sources of complexity in systematic reviews, health technology assessments and guidelines of public health are outlined and mapped onto the reported difficulties in rating the estimates of the effect of these interventions. Recommendations on how to address these difficulties are further outlined, and the need for an integrated use of GRADE from the beginning of the review or guideline development is emphasised. The content of this paper is informed by the existing GRADE guidance, an ongoing research project on considering sources of complexity when applying the GRADE approach to rate certainty of evidence in systematic reviews and the review authors' own experiences with using GRADE.

Study filters for non-randomized studies of interventions consistently lacked sensitivity upon external validation

BACKGROUND

Little evidence is available on searches for non-randomized studies (NRS) in bibliographic databases within the framework of systematic reviews. For instance, it is currently unclear whether, when searching for NRS, effective restriction of the search strategy to certain study types is possible. The following challenges need to be considered: 1) For non-randomized controlled trials (NRCTs): whether they can be identified by established filters for randomized controlled trials (RCTs). 2) For other NRS types (such as cohort studies): whether study filters exist for each study type and, if so, which performance measures they have. The aims of the present analysis were to identify and validate existing NRS filters in MEDLINE as well as to evaluate established RCT filters using a set of MEDLINE citations.

METHODS

Our analysis is a retrospective analysis of study filters based on MEDLINE citations of NRS from Cochrane reviews. In a first step we identified existing NRS filters. For the generation of the reference set, we screened Cochrane reviews evaluating NRS, which covered a broad range of study types. The citations of the studies included in the Cochrane reviews were identified via the reviews' bibliographies and the corresponding PubMed identification numbers (PMIDs) were extracted from PubMed. Random samples comprising up to 200 citations (i.e. 200 PMIDs) each were created for each study type to generate the test sets.

RESULTS

A total of 271 Cochrane reviews from 41 different Cochrane groups were eligible for data extraction. We identified 14 NRS filters published since 2001. The study filters generated between 660,000 and 9.5 million hits in MEDLINE. Most filters covered several study types. The reference set included 2890 publications classified as NRS for the generation of the test sets. Twelve test sets were generated (one for each study type), of which 8 included 200 citations each. None of the study filters achieved sufficient sensitivity (≥ 92%) for all of the study types targeted.

CONCLUSIONS

The performance of current NRS filters is insufficient for effective use in daily practice. It is therefore necessary to develop new strategies (e.g. new NRS filters in combination with other search techniques). The challenges related to NRS should be taken into account.

Study filters for non-randomized studies of interventions consistently lacked sensitivity upon external validation

BACKGROUND

Little evidence is available on searches for non-randomized studies (NRS) in bibliographic databases within the framework of systematic reviews. For instance, it is currently unclear whether, when searching for NRS, effective restriction of the search strategy to certain study types is possible. The following challenges need to be considered: 1) For non-randomized controlled trials (NRCTs): whether they can be identified by established filters for randomized controlled trials (RCTs). 2) For other NRS types (such as cohort studies): whether study filters exist for each study type and, if so, which performance measures they have. The aims of the present analysis were to identify and validate existing NRS filters in MEDLINE as well as to evaluate established RCT filters using a set of MEDLINE citations.

METHODS

Our analysis is a retrospective analysis of study filters based on MEDLINE citations of NRS from Cochrane reviews. In a first step we identified existing NRS filters. For the generation of the reference set, we screened Cochrane reviews evaluating NRS, which covered a broad range of study types. The citations of the studies included in the Cochrane reviews were identified via the reviews' bibliographies and the corresponding PubMed identification numbers (PMIDs) were extracted from PubMed. Random samples comprising up to 200 citations (i.e. 200 PMIDs) each were created for each study type to generate the test sets.

RESULTS

A total of 271 Cochrane reviews from 41 different Cochrane groups were eligible for data extraction. We identified 14 NRS filters published since 2001. The study filters generated between 660,000 and 9.5 million hits in MEDLINE. Most filters covered several study types. The reference set included 2890 publications classified as NRS for the generation of the test sets. Twelve test sets were generated (one for each study type), of which 8 included 200 citations each. None of the study filters achieved sufficient sensitivity (≥ 92%) for all of the study types targeted.

CONCLUSIONS

The performance of current NRS filters is insufficient for effective use in daily practice. It is therefore necessary to develop new strategies (e.g. new NRS filters in combination with other search techniques). The challenges related to NRS should be taken into account.