Agreement among reviewers of review articles

Effects of task shifting from primary care physicians to nurses: a protocol for an overview of systematic reviews

INTRODUCTION

Task-shifting from primary care physicians (PCPs) to nurses is one option to better and more efficiently meet the needs of the population in primary care and to overcome PCP shortages. This protocol outlines an overview of systematic reviews to assess the effects of delegation or substitution by nurses of PCPs' activities regarding clinical, patient-relevant, professional and health services-related outcomes.

METHODS AND ANALYSIS

We will conduct a systematic literature search for secondary literature in PubMed/MEDLINE, EMBASE, CINAHL and Cochrane databases. Systematic reviews, meta-analyses and Health Technology Assessments in German and English comprising randomised controlled trials and prospective controlled trials will be considered for inclusion. Search terms will include Medical Subject Headings combined with free text words. At least one-third of abstracts and full-text articles are reviewed by two independent reviewers. Methodological quality will be assessed using the Overview Quality Assessment Questionnaire. We will only consider reviews if they include controlled trials, if the profession that substituted or delegated tasks was a nurse, if the profession of the control was a PCP, if the assessed intervention was the same in the intervention and control group and if the Overview Quality Assessment Questionnaire score is ≥5. The corrected covered area will be calculated to describe the degree of overlap of studies in the reviews included in the study. We will report the overview according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

ETHICS AND DISSEMINATION

The overview of secondary literature does not require the approval of an Ethics Committee and will be published in a peer-reviewed journal.

PROSPERO REGISTRATION NUMBER

CRD42020183327.

Are We Sure We Fully Understand What an Infodemic Is? A Global Perspective on Infodemiological Problems

Infodemic is defined as an information epidemic that can lead to engaging in dangerous behavior. Although the most striking manifestations of the latter occurred on social media, some studies show that dismisinformation is significantly influenced by numerous additional factors, both web-based and offline. These include social context, age, education, personal knowledge and beliefs, mood, psychological defense mechanisms, media resonance, and how news and information are presented to the public. Moreover, various incorrect scientific practices related to disclosure, publication, and training can also fuel such a phenomenon. Therefore, in this opinion article, we seek to provide a comprehensive overview of the issues that need to be addressed to bridge the gap between science and the public and build resilience to the infodemic. In particular, we stress that the infodemic cannot be curbed by simply disproving every single false or misleading information since the belief system and the cultural or educational background are chief factors regarding the success of fake news. For this reason, we believe that the process of forming a critical sense should begin with children in schools (ie, when the mind is more receptive to new ways of learning). Furthermore, we also believe that themes such as scientific method and evidence should be at the heart of the university education of a future scientist. Indeed, both the public and scientists must be educated on the concepts of evidence and validity of sources, as well as learning how to dialogue appropriately with each other. Finally, we believe that the scientific publishing process could be greatly improved by paying reviewers for their work and by ceasing to pursue academic success at all costs.

A Methodological Quality Assessment of Meta-Analysis Studies in Dance Therapy Using AMSTAR and AMSTAR 2

Although earlier meta-analysis studies have provided evidence-based information useful for decision-making, debate regarding their quality continues. This study aimed to evaluate the quality of meta-analysis studies in the field of dance therapy (DT) using the Assessment of Multiple Systematic Reviews (AMSTAR) and AMSTAR 2 assessment tools. Meta-analysis studies on DT were collected from various databases. Seven meta-analysis studies were selected for this study. Our findings showed that the quality level of the meta-analysis studies related to DT was “High” on the AMSTAR evaluation, but their quality decreased to “Low” on the AMSTAR 2 evaluation. Moreover, using AMSTAR 2, 71.43% of the studies fell within the category of “Moderate” or below. There was no statistically significant difference in the quality scores of the characteristics of these studies. Our results suggest that (1) education on meta-analysis guidelines is required to improve the quality of DT-related meta-analysis studies, and (2) methodological caution is warranted, since different outcomes in evaluation scores for each tool may be obtained when using AMSTAR and AMSTAR 2. Based on this study, it is expected that common and specific guidelines for meta-analysis in DT can be established.

How is AMSTAR applied by authors - a call for better reporting

BACKGROUND

The assessment of multiple systematic reviews (AMSTAR) tool is widely used for investigating the methodological quality of systematic reviews (SR). Originally, AMSTAR was developed for SRs of randomized controlled trials (RCTs). Its applicability to SRs of other study designs remains unclear. Our objectives were to: 1) analyze how AMSTAR is applied by authors and (2) analyze whether the authors pay attention to the original purpose of AMSTAR and for what it has been validated.

METHODS

We searched MEDLINE (via PubMed) from inception through October 2016 to identify studies that applied AMSTAR. Full-text studies were sought for all retrieved hits and screened by one reviewer. A second reviewer verified the excluded studies (liberal acceleration). Data were extracted into structured tables by one reviewer and were checked by a second reviewer. Discrepancies at any stage were resolved by consensus or by consulting a third person. We analyzed the data descriptively as frequencies or medians and interquartile ranges (IQRs). Associations were quantified using the risk ratio (RR), with 95% confidence intervals.

RESULTS

We identified 247 studies. They included a median of 17 reviews (interquartile range (IQR): 8 to 47) per study. AMSTAR was modified in 23% (57/247) of studies. In most studies, an AMSTAR score was calculated (200/247; 81%). Methods for calculating an AMSTAR score varied, with summing up all yes answers (yes = 1) being the most frequent option (102/200; 51%). More than one third of the authors failed to report how the AMSTAR score was obtained (71/200; 36%). In a subgroup analysis, we compared overviews of reviews (n = 154) with the methodological publications (n = 93). The overviews of reviews were much less likely to mention both limitations with respect to study designs (if other studies other than RCTs were included in the reviews) (RR 0.27, 95% CI 0.09 to 0.75) and overall score (RR 0.08, 95% CI 0.02 to 0.35).

CONCLUSIONS

Authors, peer reviewers, and editors should pay more attention to the correct use and reporting of assessment tools in evidence synthesis. Authors of overviews of reviews should ensure to have a methodological expert in their review team.

How is AMSTAR applied by authors - a call for better reporting

Background

The assessment of multiple systematic reviews (AMSTAR) tool is widely used for investigating the methodological quality of systematic reviews (SR). Originally, AMSTAR was developed for SRs of randomized controlled trials (RCTs). Its applicability to SRs of other study designs remains unclear. Our objectives were to: 1) analyze how AMSTAR is applied by authors and (2) analyze whether the authors pay attention to the original purpose of AMSTAR and for what it has been validated.

Methods

We searched MEDLINE (via PubMed) from inception through October 2016 to identify studies that applied AMSTAR. Full-text studies were sought for all retrieved hits and screened by one reviewer. A second reviewer verified the excluded studies (liberal acceleration). Data were extracted into structured tables by one reviewer and were checked by a second reviewer. Discrepancies at any stage were resolved by consensus or by consulting a third person. We analyzed the data descriptively as frequencies or medians and interquartile ranges (IQRs). Associations were quantified using the risk ratio (RR), with 95% confidence intervals.

Results

We identified 247 studies. They included a median of 17 reviews (interquartile range (IQR): 8 to 47) per study. AMSTAR was modified in 23% (57/247) of studies. In most studies, an AMSTAR score was calculated (200/247; 81%). Methods for calculating an AMSTAR score varied, with summing up all yes answers (yes = 1) being the most frequent option (102/200; 51%). More than one third of the authors failed to report how the AMSTAR score was obtained (71/200; 36%). In a subgroup analysis, we compared overviews of reviews (n = 154) with the methodological publications (n = 93). The overviews of reviews were much less likely to mention both limitations with respect to study designs (if other studies other than RCTs were included in the reviews) (RR 0.27, 95% CI 0.09 to 0.75) and overall score (RR 0.08, 95% CI 0.02 to 0.35).

Conclusions

Authors, peer reviewers, and editors should pay more attention to the correct use and reporting of assessment tools in evidence synthesis. Authors of overviews of reviews should ensure to have a methodological expert in their review team.

Understanding and Appraising Systematic Reviews and Meta-Analysis

Systematic reviews and meta-analysis seek to answer a pre-framed research question to lead to a valid answer through a systematic, explicit and reproducible method of locating; identifying, including and appraising appropriate trials. The results are synthesized considering the methodological rigor of included trials. While the meta-analysis quantitatively pools the results from individual included studies, the systematic review summarizes the findings as qualitative conclusions. These reviews are crux of evidence based dentistry for various stake-holders, i.e., clinicians, researchers and policy-makers. Although the meticulous methodology of systematic review and meta-analysis minimizes the elements of bias, yet the validity and reliability of their findings should be explored prior to translating their conclusions to practice. The goal of this paper is to familiarize readers with rationale, conduct and appraisal of systematic review and meta-analysis. Further, guidance is provided on tracing potential elements of bias in the review to enable readers to judge the quality of evidence generated from the review.

Methodological overview of systematic reviews to establish the evidence base for emergency general surgery

BACKGROUND

The evidence for treatment decision-making in emergency general surgery has not been summarized previously. The aim of this overview was to review the quantity and quality of systematic review evidence for the most common emergency surgical conditions.

METHODS

Systematic reviews of the most common conditions requiring unplanned admission and treatment managed by general surgeons were eligible for inclusion. The Centre for Reviews and Dissemination databases were searched to April 2014. The number and type (randomized or non-randomized) of included studies and patients were extracted and summarized. The total number of unique studies was recorded for each condition. The nature of the interventions (surgical, non-surgical invasive or non-invasive) was documented. The quality of reviews was assessed using the AMSTAR checklist.

RESULTS

The 106 included reviews focused mainly on bowel conditions (42), appendicitis (40) and gallstone disease (17). Fifty-one (48·1 per cent) included RCTs alone, 79 (74·5 per cent) included at least one RCT and 25 (23·6 per cent) summarized non-randomized evidence alone. Reviews included 727 unique studies, of which 30·3 per cent were RCTs. Sixty-five reviews compared different types of surgical intervention and 27 summarized trials of surgical versus non-surgical interventions. Fifty-seven reviews (53·8 per cent) were rated as low risk of bias.

CONCLUSION

This overview of reviews highlights the need for more and better research in this field.

Industry sponsorship and research outcome

BACKGROUND

Clinical research affecting how doctors practice medicine is increasingly sponsored by companies that make drugs and medical devices. Previous systematic reviews have found that pharmaceutical-industry sponsored studies are more often favorable to the sponsor's product compared with studies with other sources of sponsorship. A similar association between sponsorship and outcomes have been found for device studies, but the body of evidence is not as strong as for sponsorship of drug studies. This review is an update of a previous Cochrane review and includes empirical studies on the association between sponsorship and research outcome.

OBJECTIVES

To investigate whether industry sponsored drug and device studies have more favorable outcomes and differ in risk of bias, compared with studies having other sources of sponsorship.

SEARCH METHODS

In this update we searched MEDLINE (2010 to February 2015), Embase (2010 to February 2015), the Cochrane Methodology Register (2015, Issue 2) and Web of Science (June 2015). In addition, we searched reference lists of included papers, previous systematic reviews and author files.

SELECTION CRITERIA

Cross-sectional studies, cohort studies, systematic reviews and meta-analyses that quantitatively compared primary research studies of drugs or medical devices sponsored by industry with studies with other sources of sponsorship. We had no language restrictions.

DATA COLLECTION AND ANALYSIS

Two assessors screened abstracts and identified and included relevant papers. Two assessors extracted data, and we contacted authors of included papers for additional unpublished data. Outcomes included favorable results, favorable conclusions, effect size, risk of bias and whether the conclusions agreed with the study results. Two assessors assessed risk of bias of included papers. We calculated pooled risk ratios (RR) for dichotomous data (with 95% confidence intervals (CIs)).

MAIN RESULTS

Twenty-seven new papers were included in this update and in total the review contains 75 included papers. Industry sponsored studies more often had favorable efficacy results, RR: 1.27 (95% CI: 1.17 to 1.37) (25 papers) (moderate quality evidence), similar harms results RR: 1.37 (95% CI: 0.64 to 2.93) (four papers) (very low quality evidence) and more often favorable conclusions RR: 1.34 (95% CI: 1.19 to 1.51) (29 papers) (low quality evidence) compared with non-industry sponsored studies. Nineteen papers reported on sponsorship and efficacy effect size, but could not be pooled due to differences in their reporting of data and the results were heterogeneous. We did not find a difference between drug and device studies in the association between sponsorship and conclusions (test for interaction, P = 0.98) (four papers). Comparing industry and non-industry sponsored studies, we did not find a difference in risk of bias from sequence generation, allocation concealment, follow-up and selective outcome reporting. However, industry sponsored studies more often had low risk of bias from blinding, RR: 1.25 (95% CI: 1.05 to 1.50) (13 papers), compared with non-industry sponsored studies. In industry sponsored studies, there was less agreement between the results and the conclusions than in non-industry sponsored studies, RR: 0.83 (95% CI: 0.70 to 0.98) (six papers).

AUTHORS' CONCLUSIONS

Sponsorship of drug and device studies by the manufacturing company leads to more favorable efficacy results and conclusions than sponsorship by other sources. Our analyses suggest the existence of an industry bias that cannot be explained by standard 'Risk of bias' assessments.

Resuming the discussion of AMSTAR: What can (should) be made better?

BACKGROUND

Evidence syntheses, and in particular systematic reviews (SRs), have become one of the cornerstones of evidence-based health care. The Assessment of Multiple Systematic Reviews (AMSTAR) tool has become the most widely used tool for investigating the methodological quality of SRs and is currently undergoing revision. The objective of this paper is to present insights, challenges and potential solutions from the point of view of a group of assessors, while referring to earlier methodological discussions and debates with respect to AMSTAR.

DISCUSSION

One major drawback of AMSTAR is that it relies heavily on reporting quality rather than on methodological quality. This can be found in several items. Furthermore, it should be acknowledged that there are now new methods and procedures that did not exist when AMSTAR was developed. For example, the note to item 1 should now refer to the International Prospective Register of Ongoing Systematic Reviews (PROSPERO). Furthermore, item 3 should consider the definition of hand-searching, as the process of reviewing conference proceedings using the search function (e.g. in Microsoft Word or in a PDF file) does not meet the definition set out by the Cochrane Collaboration. Moreover, methods for assessing the quality of the body of evidence have evolved since AMSTAR was developed and should be incorporated into a revised AMSTAR tool. Potential solutions are presented for each AMSTAR item with the aim of allowing a more thorough assessment of SRs. As the AMSTAR tool is currently undergoing further development, our paper hopes to add to preceding discussions and papers regarding this tool and stimulate further discussion.

Weighing up the evidence: The contribution of critical literature reviews to the development of practice

Over the past decade there has been increasing awareness of the importance of using sound research evidence to assist clinicians, patients and clients to make informed decisions about aspects of care. The recent drive towards promoting evidence-based healthcare has provided an added impetus, highlighting the important and unique contribution of systematic reviews to this decision-making process. This paper argues that healthcare professionals should also draw on evidence and insights derived from a broader range of research traditions rather than relying on synthesised evidence from randomised controlled trials. The potential of other types of literature is explored.

Does Stroke Volume Increase During an Incremental Exercise? A Systematic Review

INTRODUCTION

Cardiac output increases during incremental-load exercise to meet metabolic skeletal muscle demand. This response requires a fast adjustment in heart rate and stroke volume. The heart rate is well known to increase linearly with exercise load; however, data for stroke volume during incremental-load exercise are unclear. Our objectives were to (a) review studies that have investigated stroke volume on incremental load exercise and (b) summarize the findings for stroke volume, primarily at maximal-exercise load.

METHODS

A comprehensive review of the Cochrane Library's, Embase, Medline, SportDiscus, PubMed, and Web of Sci-ence databases was carried out for the years 1985 to the present. The search was performed between February and June 2014 to find studies evaluating changes in stroke volume during incremental-load exercise. Controlled and uncontrolled trials were evaluated for a quality score.

RESULTS

The stroke volume data in maximal-exercise load are inconsistent. There is evidence to hypothesis that stroke volume increases during maximal-exercise load, but other lines of evidence indicate that stroke volume reaches a plateau under these circumstances, or even decreases.

CONCLUSION

The stroke volume are unclear, include contradictory evidence. Additional studies with standardized reporting for subjects (e.g., age, gender, physical fitness, and body position), exercise test protocols, and left ventricular function are required to clarify the characteristics of stroke volume during incremental maximal-exercise load.

Effects of eHealth for patients and informal caregivers confronted with cancer: A meta-review

BACKGROUND

eHealth can be defined as information provision about illness or health care and/or support for patients and/or informal caregivers, using the computer or related technologies. eHealth interventions are increasingly being used in cancer care, e.g. to support patients and informal caregivers in managing symptoms and problems in daily life.

OBJECTIVES

To synthesize evidence from systematic reviews on the effects of eHealth for cancer patients or their informal caregivers.

MATERIALS AND METHODS

A systematic meta-review, in the sense of a systematic review of reviews, was conducted. Searches were performed in PubMed, Embase, CINAHL, PsycINFO, and the Cochrane Library. All steps in the review process were either performed by two reviewers independently or checked by a second reviewer. Disagreements were resolved by consensus.

RESULTS

Ten systematic reviews were included. All reviews focused on the effects of eHealth for patients and none on effects for informal caregivers. Except for one review of high methodological quality, all reviews were of moderate methodological quality. Evidence was found for effects on perceived support, knowledge levels, and information competence of cancer patients. Indications of evidence were found for health status and healthcare participation. Findings were inconsistent for outcomes related to decision-making, psychological wellbeing, depression and anxiety, and quality of life. No evidence was found for effects on physical and functional wellbeing.

CONCLUSION

There is evidence for positive effects of eHealth on perceived support, knowledge, and information competence of cancer patients. For effects on other outcomes in cancer patients, findings are mainly inconsistent or lacking. This meta-review did not find relevant reviews focusing on or including the effects of eHealth on informal caregivers, which seems a rather unexplored area.

Controversies in epidemiology of intracranial aneurysms and SAH

Rupture of an intracranial aneurysm is the most common cause of subarachnoid haemorrhage (SAH), which is a life-threatening acute cerebrovascular event that typically affects working-age people. The exact prevalence of unruptured intracranial aneurysms (UIAs) is unknown, but at least one in 20 to 30 adults is likely to carry an asymptomatic UIA. Approximately one quarter of these UIAs rupture in a lifetime. Complex methodological challenges in conducting studies of epidemiology and risk factors for UIAs and SAH might have led to conclusions being drawn on the basis of epidemiological data of variable quality. We believe that, as a result, misconceptions about UIAs and SAH may have arisen. In this Perspectives article, we discuss three possible misconceptions about the epidemiology of UIAs and SAH, and suggest how the quality of future research could be improved.

[Practical experience with overviews of reviews--valuable decision aid or academic exercise?]

The "overview of reviews" has evolved as a method to aggregate information from systematic reviews. Based on research projects conducted by two Austrian institutions, this article aims to point out methods and perceived strengths and limitations of overviews of reviews and to discuss their application and constraints for different healthcare settings. The six analysed projects differed in their objectives as well as in the corresponding methodology. We identified the following strengths of the overviews of reviews performed: the overview of the evidence base on an issue, the rapid detection of the results of numerous reviews, the demonstration of evidence gaps and potential savings in time and resources. At the same time, the methodology could lead to a loss of information, limited relevance and to uncertainties regarding the robustness of the overall results. However, the heterogeneity of the methods used shows that the development of methods for overviews of reviews is still ongoing. Whether overviews of reviews provide valuable decision support depends on the research question and realistic expectations towards the method.

Systematic review found AMSTAR, but not R(evised)-AMSTAR, to have good measurement properties

OBJECTIVES

To summarize all available evidence on measurement properties in terms of reliability, validity, and feasibility of the Assessment of Multiple Systematic Reviews (AMSTAR) tool, including R(evised)-AMSTAR.

STUDY DESIGN AND SETTING

MEDLINE, EMBASE, Psycinfo, and CINAHL were searched for studies containing information on measurement properties of the tools in October 2013. We extracted data on study characteristics and measurement properties. These data were analyzed following measurement criteria.

RESULTS

We included 13 studies, four of them were labeled as validation studies. Nine articles dealt with AMSTAR, two articles dealt with R-AMSTAR, and one article dealt with both instruments. In terms of interrater reliability, most items showed a substantial agreement (>0.6). The median intraclass correlation coefficient (ICC) for the overall score of AMSTAR was 0.83 (range 0.60-0.98), indicating a high agreement. In terms of validity, ICCs were very high with all but one ICC lower than 0.8 when the AMSTAR score was compared with scores from other tools. Scoring AMSTAR takes between 10 and 20 minutes.

CONCLUSION

AMSTAR seems to be reliable and valid. Further investigations for systematic reviews of other study designs than randomized controlled trials are needed. R-AMSTAR should be further investigated as evidence for its use is limited and its measurement properties have not been studied sufficiently. In general, test-retest reliability should be investigated in future studies.

[Review of meta-analysis research on exercise in South Korea]

PURPOSE

The purpose of this study was to evaluate the quality of meta-analysis regarding exercise using Assessment of Multiple Systematic Reviews (AMSTAR) as well as to compare effect size according to outcomes.

METHODS

Electronic databases including the Korean Studies Information Service System (KISS), the National Assembly Library and the DBpia, HAKJISA and RISS4U for the dates 1990 to January 2014 were searched for 'meta-analysis' and 'exercise' in the fields of medical, nursing, physical therapy and physical exercise in Korea. AMSTAR was scored for quality assessment of the 33 articles included in the study. Data were analyzed using descriptive statistics, t-test, ANOVA and χ²-test.

RESULTS

The mean score for AMSTAR evaluations was 4.18 (SD=1.78) and about 67% were classified at the low-quality level and 30% at the moderate-quality level. The scores of quality were statistically different by field of research, number of participants, number of databases, financial support and approval by IRB. The effect size that presented in individual studies were different by type of exercise in the applied intervention.

CONCLUSION

This critical appraisal of meta-analysis published in various field that focused on exercise indicates that a guideline such as the PRISMA checklist should be strongly recommended for optimum reporting of meta-analysis across research fields.

Factors underlying the success of behavioral HIV-prevention interventions for adolescents: a meta-review

The purpose of this meta-review was to identify characteristics of successful HIV prevention interventions for adolescents based on quantitative (i.e., meta-analyses) and qualitative reviews published to date, and to inform intervention utilization and future development. To that end, we were guided by principles of triangulation. Searches of seven electronic bibliographic databases yielded five meta-analyses and six qualitative reviews that satisfied the selection criteria. Reviews were subjected to careful content analysis. All reviews reported that behavioral interventions had positive outcomes on at least one of the following outcomes: HIV-related knowledge, subjective cognitions and beliefs enabling safer sex, abstinence, delaying next sexual intercourse, decreasing number of sexual partners, and actual condom use. Four categories, suggesting factors more prominently linked to intervention success, emerged: behavior change techniques (e.g., cognitive-behavior and motivation enhancement skills training); recipient characteristics (e.g., age, vulnerability to contracting STIs/HIV); prominent design features (e.g., use of theory, formative research); and socio-ecological features (e.g., supportive school environment). Future interventions would benefit from conducting preliminary formative research in order to enable optimal implementation of all these factors.

Up-to-dateness of reviews is often neglected in overviews: a systematic review

BACKGROUND AND OBJECTIVE

As systematic reviews may run out of date, it might be necessary to update them. Out-of-date reviews may jeopardize the comparability when used in the context of overviews (review of reviews).

METHODS

Seven electronic databases were searched for overviews up to November 2012. We first aimed to analyze whether the authors of overviews additionally searched for primary studies or alternatively explained why they did not. Second, we sought to analyze the actual publication lag (publication date of the overview - publication date of the review) in overviews and to develop recommendations for authors of overviews.

RESULTS

We included 147 overviews. The mean publication lag in overviews was more than 5 years. A median of 36% of the reviews were published more than 6 years ago. Only one in four reviews considered up-to-dateness. The methods for updating reviews were heterogeneous. We found no overview that systematically investigated whether an update was necessary.

CONCLUSION

The issue of up-to-dateness when conducting overviews seems to be neglected by most authors of overviews. Authors should assess the quality of evidence, based on their included reviews first.

Can AMSTAR also be applied to systematic reviews of non-randomized studies?

BACKGROUND

There is a lack of an instrument to evaluate systematic reviews of non-randomized studies in epidemiological research. The Assessment of Multiple Systematic Reviews (AMSTAR) is widely used to evaluate the scientific quality of systematic reviews, but it has not been validated for SRs of non-randomized studies. The objective of this paper is to report our experience in applying AMSTAR to systematic reviews of non-randomized studies in terms of applicability, reliability and feasibility. Thus, we applied AMSTAR to a recently published review of 32 systematic reviews of non-randomized studies investigating the hospital volume-outcome relationship in surgery.

RESULTS

The inter-rater reliability was high (0.76), albeit items 8 (scientific quality used in formulating conclusions), 9 (appropriate method to combine studies), and 11 (conflicts of interest) scored moderate (≤0.58). However, there was a high heterogeneity between the two pairs of reviewers. In terms of feasibility, AMSTAR proved easy to apply to systematic reviews of non-randomized studies, each review taking 5-10 minutes to complete. We faced problems in applying three items, mainly related to scientific quality of the included studies.

CONCLUSIONS

AMSTAR showed good psychometric properties, comparable to prior findings in systematic reviews of randomized controlled trials. AMSTAR can be applied to systematic reviews of non-randomized studies, although there are some item specific issues users should be aware of. Revisions and extensions of AMSTAR might be helpful.

WITHDRAWN: Non-invasive physical treatments for chronic/recurrent headache

This review is out of date, but is correct at the date of publication. The review has been withdrawn from The Cochrane Library, but readers can still access previous versions in the 'Other versions' tab. The original author team is preparing three new protocols which will serve to update and replace this review: Manual treatment and spinal rehabilitative exercise for the prevention of migraine attacks in adults, Manual treatment and spinal rehabilitative exercise for the prevention of TTH in adults, and Manual treatment and spinal rehabilitative exercise for the prevention of cervicogenic headaches in adults. For further information, please contact the PaPaS CRG (details here ).

July 2017

At July 2017, the series of three new reviews intended to replace this review were withdrawn as they were not able to be completed within the available editorial resource.

The editorial group responsible for this previously published document have withdrawn it from publication.

[SR/PS-method for using data of primary studies from systematic reviews in the evaluation of health technologies]

In general, the evaluation of health technologies is a time-consuming process being performed on the basis of systematic reviews of clinical (primary) studies. In order to save time, health technologies may be assessed based on previously published systematic reviews. However, this assessment method can be associated with a high risk of bias of the obtained results. We developed, therefore, the "Systematic Reviews for assessment based on Primary Studies" (SR/PS) method to enable a transparent, valid and time-saving evaluation of the technologies. Using the SR/PS method the evaluation of the hits that were identified through the literature search for systematic reviews and, if appropriate, through additional search for primary studies is being performed in three stages, namely identification, quality assessment and information synthesis. This process results in the ascertainment of the three most important sets of information: the pool of identified studies, the pool of methodologically sound studies and the results of the information synthesis. Each stage of the evaluation comprises the comprehensive use of relevant data on primary studies from the systematic reviews. At each stage, the corresponding systematic reviews will be selected from the identified hits using certain quality criteria. If information obtained from the systematic reviews is insufficient it will be completed by additionally incorporating the primary studies. The SR/PS method can be especially helpful in performing projects with many primary studies (e. g., guidelines development).

Impact of choice of quality appraisal tool for systematic reviews in overviews

OBJECTIVE

The question whether the choice of a critical appraisal tool has an impact on the result of the evidence synthesis in systematic reviews has been neglected by research. This is also true for psychometric properties of critical appraisal tools. The objective of the study is to exemplify that in the context of overviews (reviews of reviews).

METHODS

Based on a published overview investigating the hospital volume-outcome relationship in surgery, 32 therein included systematic reviews were independently evaluated with four critical appraisal tools by two reviewers. We rated the relationship on a five-point rating scale using qualitative evidence synthesis. Measures of reliability and correlation coefficients were calculated.

RESULTS

The result of the evidence synthesis was not dependent on the choice of a critical appraisal tool. Inter-rater reliability differed depending on the tool, Cohens Kappa ranging from 0.47 to 0.76. There was a high heterogeneity between the two pairs of reviewers.

CONCLUSION

The choice of a critical appraisal tool has no impact on the result of the evidence synthesis, despite differences in the covered components by each CAT. Further studies should concentrate on investigating psychometric properties and the impact of choice of CATs on the evidence synthesis in other contexts. The high heterogeneity between the two pairs of reviewers, all of them experienced in appraising systematic reviews, indicates a degree of interpretability in the items.

A systematic assessment of the quality of systematic reviews/meta-analyses in radiofrequency ablation versus hepatic resection for small hepatocellular carcinoma

OBJECTIVES

The systematic reviews (SRs) of radiofrequency ablation (RFA) versus hepatic resection (HR) for early hepatocellular carcinoma (HCC) are increasing with varies qualities. The aim of this study is to evaluate quality and their impacts on outcomes of these studies.

METHODS

We searched six databases and five official websites to find the SRs of RFA versus HR for early HCC. The Overview Quality Assessment Questionnaire (OQAQ), the Cochrane Collaboration's tool, and modified MINORS score were applied to assess their quality for SRs, randomized (RCTs) and nonrandomized controlled trials (NRCTs), respectively.

RESULTS

Nineteen SRs were included. The results showed that the overall quality was poor, with a mean OQAQ score of 3.3 and 95%CI 2.6 to 4.1, only five (26.3%) SRs were good quality, six (31.6%) misused the statistical models, and three of them changed outcome direction after modification. Five SRs taken retrospective studies as RCT. In addition, a total of 39 primary studies referenced by these 19 SRs were included. The results showed that 3 RCTs were leveled grade B, and 35 NRCTs were of moderate quality, with an estimated mean MINORS score of 15.0 and 95%CI 14.6 to 15.4.

CONCLUSIONS

The overall quality of SRs comparing the effects between RFA and HR for early HCC was poor. There was high heterogeneity and low evidence level. Physicians should take caution when applying the results from these studies to their clinical practice.

Telemonitoring can assist in managing cardiovascular disease in primary care: a systematic review of systematic reviews

Background

There has been growing interest regarding the impact of telemonitoring and its ability to reduce the increasing burden of chronic diseases, including chronic cardiovascular disease (CVD), on healthcare systems. A number of randomised trials have been undertaken internationally and synthesised into various systematic reviews to establish an evidence base for this model of care. This study sought to synthesise and critically evaluate this large body of evidence to inform clinicians, researchers and policy makers.

Methods

A systematic review of systematic reviews investigating the impact of telemonitoring interventions in the primary care management of CVD was conducted. Reviews were included if they explored primary care based telemonitoring in either CVD, heart failure or hypertension, were reported in the English language and were published between 2000 and 2013. Data was extracted by one reviewer and checked by a second reviewer using a standardised form. Two assessors then rated the quality of each review using the Overview Quality Assessment Questionnaire (OQAQ).

Results

Of the 13 included reviews, four focused on telemonitoring interventions in hypertension or CVD management and the remaining 9 reviews investigated telemonitoring in HF management. Seven reviews scored a five or above on the OQAQ evidencing good quality reviews. Findings suggest that telemonitoring can contribute to significant reductions in blood pressure, decreased all-cause and HF related hospitalisations, reduced all-cause mortality and improved quality of life. Telemonitoring was also demonstrated to reduce health care costs and appears acceptable to patients.

Conclusion

Telemonitoring has the potential to enhance primary care management of CVD by improving patient outcomes and reducing health costs. However, further research needs to explore the specific elements of telemonitoring interventions to determine the relative value of the various elements. Additionally, the ways in which telemonitoring care improves health outcomes needs to be further explored to understand the nature of these interventions.

Self-management of oral anticoagulation

BACKGROUND

Properly dosed oral anticoagulation effectively prevents thromboembolic events. It is unclear whether adult patients with an indication for long-term oral anticoagulation can benefit from self-management in terms of patient-oriented endpoints and improved coagulation values.

METHOD

We selectively searched the Medline database for high-quality systematic reviews based on randomized controlled trials of self-measurement or self-management of oral anticoagulation, compared to standard treatment.

RESULTS

We identified eight review articles based on overlapping sets of ran - domized clinical trials. In all of these systematic reviews, patients who performed self-measurement or self-management had a 40% to 50% lower rate of thromboembolic events; in six of them, the mortality was also significantly lower, by 30% to 50%. Subgroup analysis revealed that these effects were present exclusively in patients who performed self-management, and not in those who only performed self-measurement. None of the review articles revealed any difference in the frequency of severe hemorrhagic events. Quality of life and patient satisfaction were rated in five reviews, which, however, used different instruments, with the result that no clear conclusions could be drawn. All of the review articles documented an improvement in coagulation values, but information on statistical significance was mostly lacking.

CONCLUSION

Adults with an indication for long-term oral anticoagulation benefit from self-management, as compared to standard treatment with management of dosing by a physician. A limitation of this study is that the multiple review articles on which it is based were largely analyses of the same group of clinical trials.

Methodology and reporting of meta-analyses in the neurosurgical literature

OBJECT

Neurosurgeons are inundated with vast amounts of new clinical research on a daily basis, making it difficult and time-consuming to keep up with the latest literature. Meta-analysis is an extension of a systematic review that employs statistical techniques to pool the data from the literature in order to calculate a cumulative effect size. This is done to answer a clearly defined a priori question. Despite their increasing popularity in the neurosurgery literature, meta-analyses have not been scrutinized in terms of reporting and methodology.

METHODS

The authors performed a literature search using PubMed/MEDLINE to locate all meta-analyses that have been published in the JNS Publishing Group journals (Journal of Neurosurgery, Journal of Neurosurgery: Pediatrics, Journal of Neurosurgery: Spine, and Neurosurgical Focus) or Neurosurgery. Accepted checklists for reporting (PRISMA) and methodology (AMSTAR) were applied to each meta-analysis, and the number of items within each checklist that were satisfactorily fulfilled was recorded. The authors sought to answer 4 specific questions: Are meta-analyses improving 1) with time; 2) when the study met their definition of a meta-analysis; 3) when clinicians collaborated with a potential expert in meta-analysis; and 4) when the meta-analysis was the only focus of the paper?

RESULTS

Seventy-two meta-analyses were published in the JNS Publishing Group journals and Neurosurgery between 1990 and 2012. The number of published meta-analyses has increased dramatically in the last several years. The most common topics were vascular, and most were based on observational studies. Only 11 papers were prepared using an established checklist. The average AMSTAR and PRISMA scores (proportion of items satisfactorily fulfilled divided by the total number of eligible items in the respective instrument) were 31% and 55%, respectively. Major deficiencies were identified, including the lack of a comprehensive search strategy, study selection and data extraction, assessment of heterogeneity, publication bias, and study quality. Almost one-third of the papers did not meet our basic definition of a meta-analysis. The quality of reporting and methodology was better 1) when the study met our definition of a meta-analysis; 2) when one or more of the authors had experience or expertise in conducting a meta-analysis; 3) when the meta-analysis was not conducted alongside an evaluation of the authors' own data; and 4) in more recent studies.

CONCLUSIONS

Reporting and methodology of meta-analyses in the neurosurgery literature is excessively variable and overall poor. As these papers are being published with increasing frequency, neurosurgical journals need to adopt a clear definition of a meta-analysis and insist that they be created using checklists for both reporting and methodology. Standardization will ensure high-quality publications.

GAP-REACH: a checklist to assess comprehensive reporting of race, ethnicity, and culture in psychiatric publications

Growing awareness of health and health care disparities highlights the importance of including information about race, ethnicity, and culture (REC) in health research. Reporting of REC factors in research publications, however, is notoriously imprecise and unsystematic. This article describes the development of a checklist to assess the comprehensiveness and the applicability of REC factor reporting in psychiatric research publications. The 16-item GAP-REACH checklist was developed through a rigorous process of expert consensus, empirical content analysis in a sample of publications (N = 1205), and interrater reliability (IRR) assessment (N = 30). The items assess each section in the conventional structure of a health research article. Data from the assessment may be considered on an item-by-item basis or as a total score ranging from 0% to 100%. The final checklist has excellent IRR (κ = 0.91). The GAP-REACH may be used by multiple research stakeholders to assess the scope of REC reporting in a research article.

A systematic scoping review of adherence to reporting guidelines in health care literature

Background

Reporting guidelines have been available for the past 17 years since the inception of the Consolidated Standards of Reporting Trials statement in 1996. These guidelines were developed to improve the quality of reporting of studies in medical literature. Despite the widespread availability of these guidelines, the quality of reporting of medical literature remained suboptimal. In this study, we assess the current adherence practice to reporting guidelines; determine key factors associated with better adherence to these guidelines; and provide recommendations to enhance adherence to reporting guidelines for future studies.

Methods

We undertook a systematic scoping review of systematic reviews of adherence to reporting guidelines across different clinical areas and study designs. We searched four electronic databases (Cumulative Index to Nursing and Allied Health Literature, Web of Science, Embase, and Medline) from January 1996 to September 2012. Studies were included if they addressed adherence to one of the following guidelines: Consolidated Standards of Reporting Trials (CONSORT), Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), Quality of Reporting of Meta-analysis (QUOROM), Transparent Reporting of Evaluations with Nonrandomized Designs (TREND), Meta-analysis Of Observational Studies in Epidemiology (MOOSE) and Strengthening the Reporting of Observational Studies in Epidemiology (STROBE). A protocol for this study was devised. A literature search, data extraction, and quality assessment were performed independently by two authors in duplicate. This study reporting follows the PRISMA guidelines.

Results

Our search retrieved 5159 titles, of which 50 were eligible. Overall, 86.0% of studies reported suboptimal levels of adherence to reporting guidelines. Factors associated with better adherence included journal impact factor and endorsement of guidelines, publication date, funding source, multisite studies, pharmacological interventions and larger studies.

Conclusion

Reporting guidelines in the clinical literature are important to improve the standards of reporting of clinical studies; however, adherence to these guidelines remains suboptimal. Action is therefore needed to enhance the adherence to these standards. Strategies to enhance adherence include journal editorial policies endorsing these guidelines.

Acute traumatic brain injury: is current management evidence based? An empirical analysis of systematic reviews

Traumatic brain injury (TBI) is a major health and socioeconomic problem worldwide with a high rate of death and long-term disability. Previous studies have summarized evidence from large-scale randomized trials, finding no intervention showing convincing efficacy for acute TBI management. The present empirical study set out to assess another crucial component of evidence base-systematic review, which contributes a lot to evidence-based health care, in terms of clinical issues, methodological aspects, and implication for practice and research. A total of 44 systematic reviews pertaining to therapeutic interventions for acute TBI were identified through electronic database searching, clinical guideline retrieval, and expert consultation, of which 21 were published in Cochrane Library and 23 in peer-reviewed journals. Their methodological quality was generally satisfactory, with the median Overview Quality Assessment Questionnaire score of 5.5 (interquartile range 2-7). Cochrane reviews are of better quality than regular journal reviews. Twenty-nine high-quality reviews provided no conclusive evidence for the investigated 22 interventions except for an adverse effect of corticosteroids. Less than one-third of the component trials were reported with adequate allocation concealment. Additionally other methodological flaws in design-for example, ignoring heterogeneity among the TBI population-also contributed to the failure of past clinical research. Based on the above findings, evidence from both systematic reviews and clinical trials does not fully support current management of acute TBI. Translating from laboratory success to clinical effect remains an unique challenge. Accordingly it may be the time to rethink the way in future practice and clinical research in TBI.

F-18-fluoro-2-deoxyglucose positron emission tomography (PET) and PET/computed tomography imaging in primary staging of patients with malignant melanoma: a systematic review

PURPOSE

The aim of this systematic review was to systematically assess the potential patient-relevant benefit (primary aim) and diagnostic and prognostic accuracy (secondary aim) of positron emission tomography (PET) and PET/computed tomography (CT) in primary staging of malignant melanoma. This systematic review updates the previous evidence for PET(/CT) in malignant melanoma.

MATERIALS AND METHODS

For the first aim, randomized controlled trials (RCTs) investigating patient-relevant outcomes and comparing PET and PET(/CT) with each other or with conventional imaging were considered. For the secondary aim, a review of reviews was conducted, which was amended by an update search for primary studies. MEDLINE, EMBASE and four databases of the Cochrane Library were searched. The risk of bias was assessed using a modified QUADAS tool.

RESULTS

No RCTs investigating the patient-relevant benefit of PET(/CT) and no prognostic accuracy studies were found. Seventeen diagnostic accuracy studies of varying quality were identified. For patients with American Joint Committee on Cancer (AJCC) stages I and II, sensitivity mostly ranged from 0 to 67%. Specificity ranged from 77 to 100%. For AJCC stages III and IV, sensitivity ranged from 68 to 87% and specificity from 92 to 98%.

CONCLUSION

There is currently no evidence of a patient-relevant benefit of PET(/CT) in the primary staging of malignant melanoma. RCTs investigating patient-relevant outcomes are therefore required. The diagnostic accuracy of PET(/CT) appears to increase with higher AJCC stages.

Industry sponsorship and research outcome

BACKGROUND

Clinical research affecting how doctors practice medicine is increasingly sponsored by companies that make drugs and medical devices. Previous systematic reviews have found that pharmaceutical industry sponsored studies are more often favorable to the sponsor's product compared with studies with other sources of sponsorship. This review is an update using more stringent methodology and also investigating sponsorship of device studies.

OBJECTIVES

To investigate whether industry sponsored drug and device studies have more favorable outcomes and differ in risk of bias, compared with studies having other sources of sponsorship.

SEARCH METHODS

We searched MEDLINE (1948 to September 2010), EMBASE (1980 to September 2010), the Cochrane Methodology Register (Issue 4, 2010) and Web of Science (August 2011). In addition, we searched reference lists of included papers, previous systematic reviews and author files.

SELECTION CRITERIA

Cross-sectional studies, cohort studies, systematic reviews and meta-analyses that quantitatively compared primary research studies of drugs or medical devices sponsored by industry with studies with other sources of sponsorship. We had no language restrictions.

DATA COLLECTION AND ANALYSIS

Two assessors identified potentially relevant papers, and a decision about final inclusion was made by all authors. Two assessors extracted data, and we contacted authors of included papers for additional unpublished data. Outcomes included favorable results, favorable conclusions, effect size, risk of bias and whether the conclusions agreed with the study results. Two assessors assessed risk of bias of included papers. We calculated pooled risk ratios (RR) for dichotomous data (with 95% confidence intervals).

MAIN RESULTS

Forty-eight papers were included. Industry sponsored studies more often had favorable efficacy results, risk ratio (RR): 1.24 (95% confidence interval (CI): 1.14 to 1.35), harms results RR: 1.87 (95% CI: 1.54 to 2.27) and conclusions RR: 1.31 (95% CI: 1.20 to 1.44) compared with non-industry sponsored studies. Ten papers reported on sponsorship and effect size, but could not be pooled due to differences in their reporting of data. The results were heterogeneous; five papers found larger effect sizes in industry sponsored studies compared with non-industry sponsored studies and five papers did not find a difference in effect size. Only two papers (including 120 device studies) reported separate data for devices and we did not find a difference between drug and device studies on the association between sponsorship and conclusions (test for interaction, P = 0.23). Comparing industry and non-industry sponsored studies, we did not find a difference in risk of bias from sequence generation, allocation concealment and follow-up. However, industry sponsored studies more often had low risk of bias from blinding, RR: 1.32 (95% CI: 1.05 to 1.65), compared with non-industry sponsored studies. In industry sponsored studies, there was less agreement between the results and the conclusions than in non-industry sponsored studies, RR: 0.84 (95% CI: 0.70 to 1.01).

AUTHORS' CONCLUSIONS

Sponsorship of drug and device studies by the manufacturing company leads to more favorable results and conclusions than sponsorship by other sources. Our analyses suggest the existence of an industry bias that cannot be explained by standard 'Risk of bias' assessments.

Review of systematic reviews about the efficacy of non-pharmacological interventions to improve sleep quality in insomnia

Background  Insomnia is a very common condition in various populations. Non-pharmacological interventions might offer (safe) alternatives for hypnotics. Aim  To evaluate the evidence for efficacy from systematic reviews about non-pharmacological interventions to improve sleep quality in insomnia by a systematic review of systematic reviews and meta-analyses. Search strategy  Search strategies were conducted in the Database of Abstracts of Reviews of Effects (2002-July 2008), The Cochrane Database of Systematic Reviews (2000-July 2008) and PubMed (1950-July 2008). Sleep quality was the outcome measure of interest. Selection criteria  Systematic reviews about the efficacy of one or more non-pharmacological interventions for insomnia, concerning both adult and elderly populations, were included. Reviews that included studies performed among populations suffering with severe neurological or cognitive impairments or with addictive disorders were excluded. Data analysis  Relevant data were extracted. The quality of the reviews found was appraised by using the Overview Quality Assessment Questionnaire. The evidence was appraised and divided into six classes. Results and conclusions  Sixteen reviews about 17 interventions were included. Six reviews were of adequate methodological quality. Of these, only one provided an effect size: a moderate effect was found for music-assisted relaxation. Weak evidence indicating a large effect was found for multicomponent cognitive behavioural therapy, progressive muscle relaxation, stimulus control and 'behavioural only'. Weak evidence indicating a moderate effect was found for paradoxical intention. Finally, weak evidence indicating a moderate to large effect was found for relaxation training. Because of the lack of sufficient methodological quality and the lack of calculated effect sizes, most of the included reviews were not suitable for drawing rigorous conclusions about the effect of non-pharmacological interventions on sleep quality in insomniacs. The non-pharmacological treatment of insomnia would benefit from renewed reviews based on a rigorous methodological approach.

Peer review of grant applications: criteria used and qualitative study of reviewer practices

BACKGROUND

Peer review of grant applications has been criticized as lacking reliability. Studies showing poor agreement among reviewers supported this possibility but usually focused on reviewers' scores and failed to investigate reasons for disagreement. Here, our goal was to determine how reviewers rate applications, by investigating reviewer practices and grant assessment criteria.

METHODS AND FINDINGS

We first collected and analyzed a convenience sample of French and international calls for proposals and assessment guidelines, from which we created an overall typology of assessment criteria comprising nine domains relevance to the call for proposals, usefulness, originality, innovativeness, methodology, feasibility, funding, ethical aspects, and writing of the grant application. We then performed a qualitative study of reviewer practices, particularly regarding the use of assessment criteria, among reviewers of the French Academic Hospital Research Grant Agencies (Programmes Hospitaliers de Recherche Clinique, PHRCs). Semi-structured interviews and observation sessions were conducted. Both the time spent assessing each grant application and the assessment methods varied across reviewers. The assessment criteria recommended by the PHRCs were listed by all reviewers as frequently evaluated and useful. However, use of the PHRC criteria was subjective and varied across reviewers. Some reviewers gave the same weight to each assessment criterion, whereas others considered originality to be the most important criterion (12/34), followed by methodology (10/34) and feasibility (4/34). Conceivably, this variability might adversely affect the reliability of the review process, and studies evaluating this hypothesis would be of interest.

CONCLUSIONS

Variability across reviewers may result in mistrust among grant applicants about the review process. Consequently, ensuring transparency is of the utmost importance. Consistency in the review process could also be improved by providing common definitions for each assessment criterion and uniform requirements for grant application submissions. Further research is needed to assess the feasibility and acceptability of these measures.

Neuraxial anesthesia compared to general anesthesia for procedures on the lower half of the body: systematic review of systematic reviews

BACKGROUND AND OBJECTIVES

Systematic reviews organize literature data by combining results from published studies in order to resolve conflicts in the area of medical knowledge describing the interventions. The inadequate reporting of systematic reviews can damage the credibility and interfere in the results' quality. The objective of this study was to determine the frequency of good quality systematic reviews comparing neuraxial anesthesia with general anesthesia for procedures on the lower half of the body.

METHODS

Systematic review of systematic reviews. Primary variable: The frequency of good quality systematic reviews. The information was analyzed from the following databases: LILACS (January 1982 to December 2010); PubMed (January 1950 to December 2010); The Cochrane Database of Systematic Review and Database of Abstracts of Reviews of Effects (volume 10, 2010); and SciELO (December 2010). The quality of systematic reviews was determined by the Overview Quality Assessment Questionnaire. The sample size calculation showed that it was necessary to analyze eight systematic reviews, taking into account that the frequency of good quality systematic reviews was 5%, an absolute precision of 15%, and a significance level of 5%.

RESULTS

Were identified 1,995 articles. The selection process eliminated 1,968 articles. Twenty-seven articles of systematic reviews were read in full, 9 were excluded due to incompatibility with the inclusion criteria, and 8 were duplicate publications. Ten systematic reviews were assessed for their quality. The frequency of good quality systematic reviews was 40% (4/10; 95% CI 9.6 to 70.4%).

CONCLUSION

The frequency of good quality systematic reviews was 40%.