The judgement of biases included in the category "other bias" in Cochrane systematic reviews of interventions: a systematic survey

Sevoflurane and its metabolic byproduct compound A induce nephrotoxicity: a systematic review and meta-analysis of animal studies

Animal models investigating sevoflurane or compound A and renal function serve as the initial basis for concerns regarding renal injury following sevoflurane anesthesia and subsequent recommendations of minimum fresh gas flow, but this evidence basis has not been critically appraised. Primary literature searches were performed in MEDLINE OVID, PubMed, EMBASE, the Cochrane Library), the Cochrane Central Register of Controlled Trials, the International HTA Database, CINAHL, and Web of Science to identify randomized controlled trials and quasi-experimental studies in animals utilizing sevoflurane or compound A. The primary outcomes included renal function as determined by blood urea nitrogen, serum creatinine, creatinine clearance, and urine volume. The secondary outcomes included the serum fluoride concentration and histopathological findings. A total of 2537 records were screened, and 21 randomized controlled trials and 9 quasi-experimental animal studies were identified. No associations between sevoflurane exposure and subsequent changes in renal function (blood urea nitrogen, serum creatinine or changes in urine volume) were noted. A similar effect on renal function was observed following compound A exposure, but urine volume was elevated following compound A exposure. In addition, the histopathological damage following compound A exposure was observed only at concentrations that are unachievable in clinical practice. Our review of evidence from animal models revealed that sevoflurane usage was not associated with changes in renal function tests or urine volume. Histopathologic changes after sevoflurane exposure were either nonexistent or minor. Studies on compound A did not reveal an alteration in renal function, although histopathological evidence of injury was present when compound A was administered at very high, unphysiologic concentrations. In light of the existing evidence, the initial concerns of sevoflurane-related nephrotoxicity based on animal studies that leads to minimum fresh gas flow recommendations are called into question.

Diagnostic ability and adverse events of mucosal incision-assisted biopsy for gastric subepithelial tumors: Systematic review and meta-analysis

OBJECTIVES

This systematic review and meta-analysis aimed to evaluate the diagnostic ability and examine the efficacy of countermeasures to adverse events of mucosal incision-assisted biopsy (MIAB) for gastric subepithelial tumors (SETs).

METHODS

We performed a literature search and identified 533 relevant articles. Eleven articles, including 339 lesions, were ultimately used in the meta-analysis. The primary end-point was the pathological diagnostic rate of MIAB for gastric SETs, and the secondary end-point was the incidence of adverse events. The efficacy of acid secretion inhibitors in preventing postoperative bleeding and that of local injection before incision to prevent perforation were also examined.

RESULTS

Nine studies were conducted in Japan and two in South Korea, of which only two were prospective studies. The pooled pathological diagnostic rate of MIAB for gastric SETs was 87.8% (95% confidence interval [CI] 80.2-94.0; I2 = 68.7%). The adverse event rate of the pooled population was 0.2% (95% CI 0-1.4; I2 = 0%). The acid secretion inhibitors significantly reduced postoperative bleeding (odds ratio 0.06, 95% CI 0.01-0.66, P = 0.02). Perforation occurred in 0% and 2.6% of the local and nonlocal injection cohorts, respectively, and the pathological diagnostic rates were 50% and 66.7%, respectively.

CONCLUSIONS

MIAB is a reliable technique with a favorable diagnostic rate and few adverse events. Acid secretion inhibitors may effectively prevent postoperative bleeding; however, the efficacy of local injection remains unclear. This technique could be an option for tissue sampling in gastric SETs.

A scoping review shows that no single existing risk of bias assessment tool considers all sources of bias for cross-sectional studies

OBJECTIVES

Different tools to assess the potential risk of bias (RoB) for cross-sectional studies have been developed, but it is unclear whether all pertinent bias concepts are addressed. We aimed to identify RoB concepts applicable to cross-sectional research validity and to explore coverage for each in existing appraisal tools.

STUDY DESIGN AND SETTING

This scoping review followed the Joanna Briggs Institute methodology. We included records of any study design describing or reporting methods, concepts or tools used to consider RoB in health research reported to be descriptive/prevalence survey or analytic/association (cross-sectional) study designs. Synthesis included quantitative and qualitative analysis.

RESULTS

Of the 4556 records screened, 90 were selected for inclusion; 67 (74%) described the development of, or validation process for, appraisal tools, 15 (17%) described methodological content or theory relevant to RoB for cross-sectional studies and 8 (9%) records of methodological systematic reviews. Review of methodological reports identified important RoB concepts for both descriptive/prevalence and analytic/association studies. Tools identified (n = 64 unique tools) were either intended to appraise quality or assess RoB in multiple study designs including cross-sectional studies (n = 21; 33%) or cross-sectional designs alone (n = 43; 67%). Several existing tools were modified (n = 17; 27%) for application to cross-sectional studies. The RoB items most frequently addressed in the RoB tools were validity and reliability of the exposure (53%) or outcome (65%) measurement and representativeness of the study population (59%). Most tools did not consider nonresponse or missingness appropriately or at all.

CONCLUSION

Assessing cross-sectional studies involve unique RoB considerations. We identified RoB tools designed for broad applicability across various study designs as well as those specifically tailored for cross-sectional studies. However, none of the identified tools comprehensively address all potential biases pertinent to cross-sectional studies. Our findings indicate a need for continued improvement of RoB tools and suggest that the development of context-specific or more precise tools for this study design may be necessary.

Disagreements in risk of bias assessment for randomized controlled trials in hypertension-related Cochrane reviews

BACKGROUND

The inter-reviewer reliability of the risk of bias (RoB) assessment lacked agreement in previous studies. It is important to analyse these disagreements to improve the repeatability of RoB assessment. The objective of the study was to evaluate the frequency and reasons for disagreements in RoB assessments for randomised controlled trials (RCTs) that were included in multiple Cochrane reviews in the field of hypertension.

METHODS

A cross-sectional study was employed. We retrieved any RCTs that had been included in multiple Cochrane reviews in the field of hypertension from ARCHIE. The results of the RoB assessments were extracted, and the distributions of agreements and possible reasons for disagreement were analyzed.

RESULTS

Twenty-six Cochrane reviews were included in this study. A total of 78 RCTs appeared in more than one Cochrane review. The level of agreement ranged from domain to domain. "Blinding of outcome assessment" showed a reasonably high level of agreement (94.9%), while "incomplete outcome data", "selective outcome reporting" and "other sources of bias" showed moderate levels of agreement (74.6%, 79.2% and 75.6%, respectively). However, the domains of "allocation concealment", "random sequence generation" and "blinding of participants and personnel" showed low levels of agreement (24.4%, 23.5%, and 47.4%, respectively). In the domains of "allocation concealment" and "blinding of participants and personnel", the agreement group had higher proportion of publication year ≤ 1996 than the disagreement group (P = 0.008 and P < 0.001, respectively). In the "blinding of participants and personnel", the impact factor was higher in the agreement group (P < 0.001). By analyzing the support text, we found that the most likely reason for disagreement was extracting different information from the same RCT.

CONCLUSION

For Cochrane reviews in the field of hypertension using the 2011 version of the RoB tool, there was a large disagreement in the RoB assessment. It is suggested that the results of RoB assessments in systematic reviews that used the 2011 version of the RoB tool need to be interpreted with caution. More accurate information from RCTs needs to be collected when we synthesize clinical evidence.

The Cochrane risk of bias assessment tool 2 (RoB 2) versus the original RoB: A perspective on the pros and cons

Background and Aims

Critical appraisal or risk of bias assessment is a fundamental part of systematic reviews that clarifies the degree to which included research articles are qualified and reliable. Version 2 of the Cochrane tool for assessing the risk of bias in randomized trials (RoB 2), the updated version of the first tool, was released in 2019. Here, we have compared these two versions of Cochrane risk of bias assessment tools and highlighted the pros and cons of RoB 2.

Methods

Statistical analysis and methodology is not applicable to this article as no new data were created or analyzed in this study.

Results

The overall approach in RoB 2 is that by answering some signaling questions after the specification of results, effects of interest, and sources of information, an overall judgment for the quality of each study is reached. Accordingly, in the original version of the Cochrane RoB tool, the judgment can be in three different conclusions, including low, unclear, and high risk of bias. The most prominent difference in bias domains is the removal of "other bias" domain being replaced by "overall bias" judgment. Also, the most common presentation types of Cochrane risk of bias assessments are the "summary" and "graph" which are generated by Review Manager, web-based applications, or packages in R software.

Conclusion

The RoB 2 tool, compared to the original RoB, has improved and is the recommended version by the Cochrane Collaboration for quality assessment of randomized controlled trials. It is recommended to consider funding source, duration of follow-up, declaration of data availability, the status of baseline characteristics between groups, and sample size calculation methods in further revisions of the Cochrane risk of bias assessment tools.

Frequency of use and adequacy of Cochrane risk of bias tool 2 in non-Cochrane systematic reviews published in 2020: Meta-research study

Risk of bias (RoB) assessment is essential to the systematic review methodology. The new version of the Cochrane RoB tool for randomized trials (RoB 2) was published in 2019 to address limitations identified since the first version of the tool was published in 2008 and to increase the reliability of assessments. This study analyzed the frequency of usage of the RoB 2 and the adequacy of reporting the RoB 2 assessments in non-Cochrane reviews published in 2020. This meta-research study included non-Cochrane systematic reviews of interventions published in 2020. For the reviews that used the RoB 2 tool, we analyzed the reporting of the RoB 2 assessment. Among 3880 included reviews, the Cochrane RoB 1 tool was the most frequently used (N = 2228; 57.4%), followed by the Cochrane RoB 2 tool (N = 267; 6.9%). From 267 reviews that reported using the RoB 2 tool, 213 (79.8%) actually used it. In 26 (12.2%) reviews, erroneous statements were used to indicate the RoB 2 assessment. Only 20 (9.4%) reviews presented a complete RoB 2 assessment with a detailed table of answers to all signaling questions. The judgment of risk of bias by the RoB 2 tool was not justified by a comment in 158 (74.2%) reviews. Only in 33 (14.5%) of reviews the judgment in all domains was justified in the accompanying comment. In most reviews (81.7%), the RoB was inadequately assessed at the study level. In conclusion, the majority of non-Cochrane reviews published in 2020 still used the Cochrane RoB 1 tool. Many reviews used the RoB 2 tool inadequately. Further studies about the uptake and the use of the RoB 2 tool are needed.

Author instructions in biomedical journals infrequently address systematic review reporting and methodology: a cross-sectional study

OBJECTIVES

We aimed to analyze how instructions for authors in journals indexed in MEDLINE address systematic review (SR) reporting and methodology.

STUDY DESIGN AND SETTING

We analyzed instructions for authors in 20% of MEDLINE-indexed journals listed in the online catalog of the National Library of Medicine on July 27, 2021. We extracted data only from the instructions published in English. We extracted data on the existence of instructions for reporting and methodology of SRs.

RESULTS

Instructions from 1,237 journals mentioned SRs in 45% (n = 560) of the cases. Systematic review (SR) registration was mentioned in 104/1,237 (8%) of instructions. Guidelines for reporting SR protocols were found in 155/1,237 (13%) of instructions. Guidelines for reporting SRs were explicitly mentioned in 461/1,237 (37%), whereas the EQUATOR (Enhancing the Quality and Transparency of Health Research) network was referred to in 474/1,237 (38%) of instructions. Less than 2% (n = 20) of instructions mentioned risk of bias and meta-analyses; less than 1% mentioned certainty of evidence assessment, methodological expectations, updating of SRs, overviews of SRs, or scoping reviews.

CONCLUSION

Journals indexed in MEDLINE rarely provide instructions for authors regarding SR reporting and methodology. Such instructions could potentially raise authors' awareness and improve how SRs are prepared and reported.

Utilization of Mirror Visual Feedback for Upper Limb Function in Poststroke Patients: A Systematic Review and Meta-Analysis

Mirror visual feedback (MVF), a noninvasive treatment method, is attracting attention as a possibility to promote the recovery of upper limb function in stroke patients. However, the cognitive effects of this therapy have received limited attention in the existing literature. To address this gap, we conducted a systematic review and meta-analysis to investigate the relationship between upper limb function and cognition in stroke patients and to evaluate the effect of MVF on improving upper limb function. A comprehensive search was performed on the Embase, MEDLINE, and PubMed databases to identify original articles and clinical studies published between 2013 and 2022. Qualitative analysis was performed using the Cochrane Risk of Bias tool, and in the quantitative analysis, a random-effects model was used as the effect model, and standard mean difference (SMD) was used as the effect measure. Eight studies that met the inclusion criteria were entered in the analysis. Data extraction included an assessment tool for upper extremity function. Results of the quantitative analysis demonstrate that MVF was effective in improving upper extremity function in stroke patients (SMD = 0.94, 95% CI 0.69 to 1.20). In conclusion, this systematic review and meta-analysis provides evidence supporting the effectiveness of MVF in improving upper limb function in stroke patients. However, further studies are needed to investigate the cognitive effects of MVF and elucidate the underlying mechanisms.

A Systematic Review and Meta-Analysis of Exercise Beneficial for Locomotion in Community-Dwelling Elderly People with Sarcopenia

Sarcopenia, in addition to aging and reduced physical activity, is a progressive skeletal muscle disorder that causes the loss of muscle mass and strength. The most prominent functional change is mobility, which contributes to a decrease in the quality of life. Therefore, we aimed to perform qualitative and quantitative analyses by synthesizing randomized controlled trials (RCTs) that evaluated exercises that affected locomotion in patients with sarcopenia. The RCTs were retrieved in April 2023 from three international electronic databases (Embase, MEDLINE, and PubMed). RCTs published after 2013 were compared with a control group that did not include exercise. Qualitative and quantitative analyses were performed on the identified studies using RevMan 5.4 and risk of bias assessment provided by Cochrane. RCTs involving 594 patients with sarcopenia were included in this study. The analysis model was synthesized as a random effects model, and the standard mean difference (SMD) was used as the effect measure. Exercise interventions were found to not change muscle mass in individuals with sarcopenia (SMD = 0.04; 95% CI: -0.15 to 0.22). However, they had positive effects on lower extremity muscle strength (SMD = 0.34; 95% CI: 0.02 to 0.66) and walking speed (SMD = 0.42; 95% CI: 0.11 to 0.72). For community-dwelling elderly people with sarcopenia, exercise intervention did not lead to an increase in reduced muscle mass, but it brought positive improvements in lower extremity strength and gait speed to improve locomotion.

Interventions to facilitate the successful eruption of impacted maxillary incisor teeth due to the presence of a supernumerary: A systematic review and meta-analysis

INTRODUCTION

A failure of maxillary incisor eruption is commonly attributed to the presence of a supernumerary tooth. This systematic review aimed to assess the percentage of impacted maxillary incisors that successfully erupt after surgical removal of supernumerary teeth with or without other interventions.

METHODS

Systematic literature searches without restrictions were undertaken in 8 databases for studies reporting any intervention aimed at facilitating incisor eruption, including surgical removal of the supernumerary alone or in conjunction with additional interventions published up to September 2022. After duplicate study selection, data extraction, and risk of bias assessment according to the risk of bias in nonrandomized studies of interventions and Newcastle-Ottawa scale, random-effects meta-analyses of aggregate data were conducted.

RESULTS

Fifteen studies (14 retrospective and 1 prospective) were included with 1058 participants (68.9% male; mean age, 9.1 years). The pooled eruption prevalence for removal of the supernumerary tooth with space creation or removal of the supernumerary tooth with orthodontic traction was significantly higher at 82.4% (95% confidence interval [CI], 65.5-93.2) and 96.9% (95% CI, 83.8-99.9) respectively, compared with removal of an associated supernumerary only (57.6%; 95% CI, 47.8-67.0). The odds of successful eruption of an impacted maxillary incisor after removal of a supernumerary were more favorable if the obstruction was removed in the deciduous dentition (odds ratio [OR], 0.42; 95% CI, 0.20-0.90; P = 0.02); if the supernumeraries were conical (OR, 2.91; 95% CI, 1.98-4.28; P <0.001); if the incisor was in the correct position (OR, 2.19; 95% CI, 1.14-4.20; P = 0.02), at the level of the gingival third (OR 0.07; 95% CI, <0.01-0.97; P = 0.04) and had incomplete root formation (OR, 9.02; 95% CI, 2.04-39.78; P = 0.004). Delaying removal of the supernumerary tooth 12 months after the expected eruption time of the maxillary incisor (OR, 0.33; 95% CI, 0.10-1.03; P = 0.05) and waiting >6 months for spontaneous eruption after removal of the obstacle (OR, 0.13; 95% CI, 0.03-0.50; P = 0.003) was associated with worse odds for eruption.

CONCLUSIONS

Limited evidence indicated that the adjunctive use of orthodontic measures and removal of supernumerary teeth might be associated with greater odds of successfull impacted incisor eruption than removal of the supernumerary tooth alone. Certain characteristics related to supernumerary type and the position or developmental stage of the incisor may also influence successful eruption after removal of the supernumerary. However, these findings should be viewed with caution as our certainty is very low to low because of bias and heterogeneity. Further well-conducted and reported studies are required. The results of this systematic review have been used to inform and justify the iMAC Trial.

From critical appraisal to risk of bias assessment: clarifying the terminology for study evaluation in JBI systematic reviews

The foundations for critical appraisal of literature have largely progressed through the development of epidemiologic research methods and the use of research to inform medical teaching and practice. This practical application of research is referred to as evidence-based medicine and has delivered a standard for the health care profession where clinicians are equally as engaged in conducting scientific research as they are in the practice of delivering treatments. Evidence-based medicine, now referred to as evidence-based health care, has generally been operationalized through empirically supported treatments, whereby the choice of treatments is substantiated by scientific support, usually by means of an evidence synthesis. As evidence synthesis methodology has advanced, guidance for the critical appraisal of primary research has emphasized a distinction from the assessment of internal validity required for synthesized research. This assessment is conceptualized and branded in various ways in the literature, such as risk of bias, critical appraisal, study validity, methodological quality, and methodological limitations. This paper provides a discussion of the definitions and characteristics of these terms, concluding with a recommendation for JBI to adopt the term "risk of bias" assessment.

Adherence of systematic reviews to Cochrane RoB2 guidance was frequently poor: a meta epidemiological study

OBJECTIVES

To assess whether the use of the revised Cochrane risk of bias tool for randomized trials (RoB2) in systematic reviews (SRs) adheres to RoB2 guidance.

METHODS

We searched MEDLINE, Embase, Cochrane Library from 2019 to May 2021 to identify SRs using RoB2. We analyzed methods and results sections to see whether risk of bias was assessed at outcome measure level and applied to primary outcomes of the SR as per RoB2 guidance. The relation between SR characteristics and adequacy of RoB2 use was examined by logistic regression analysis.

RESULTS

Two hundred-eight SRs were included. We could assess adherence in 137 SRs as 12 declared using RoB2 but actually used RoB1 and 59 did not report the number of primary outcomes. The tool usage was adherent in 69.3% SRs. Considering SRs with multiple primary outcomes, adherence dropped to 28.8%. We found a positive association between RoB2 guidance adherence and the methodological quality of the reviews assessed by AMSTAR2 (p-for-trend 0.007). Multivariable regression analysis suggested journal impact factor [first quartile vs. other quartiles] was associated with RoB2 adherence (OR 0.34; 95% CI: 0.16-0.72).

CONCLUSIONS

Many SRs did not adhere to RoB2 guidance as they applied the tool at the study level rather than at the outcome measure level. Lack of adherence was more likely among low and very low quality reviews.

Therapeutic Application of Virtual Reality in the Rehabilitation of Mild Cognitive Impairment: A Systematic Review and Meta-Analysis

This review aimed to quantify the effect of therapeutic application of virtual reality (VR) on cognitive function in individuals with mild cognitive impairment (MCI). We searched for randomized controlled trials involving VR in the interventions provided to individuals with MCI. After searching four international electronic databases, we analyzed six studies involving 279 individuals with MCI. RevMan 5.4 was used for quality assessment and quantitative analysis. Therapeutic application of VR in individuals with MCI resulted in a significant improvement in cognitive function (mean difference = -1.46; 95% confidence interval: -2.53 to -0.39; heterogeneity: χ² = 970.56, df = 18, I² = 98%; and overall effect: Z = 2.67, p = 0.008). However, there was no significant improvement in the subcategories such as global cognition, working memory, executive function, memory function, and attention. In conclusion, feedback stimulation through VR has a potential value in improving cognitive function in individuals with MCI. However, on the basis of the results of the subcategories, a personalized VR program is required for the individual subcategories of cognitive function.

Effect of Joint Mobilization in Individuals with Chronic Ankle Instability: A Systematic Review and Meta-Analysis

Sensorimotor and range of motion deficits due to chronic ankle instability (CAI) are abnormalities of the movement system that make postural control difficult. This review aimed to quantify the effect of joint mobilization on the range of motion, dynamic balance, and function in individuals with CAI. Randomized controlled trials in which joint mobilization was performed in individuals with CAI were searched for in five international databases (CENTRAL, CINAHL, Embase, MEDLINE, PEDro). Qualitative and quantitative analyses were performed using the risk of bias tool and RevMan 5.4 provided by the Cochrane Library. Nine studies with 364 individuals with CAI were included in this study. This meta-analysis reported that joint mobilization showed significant improvement in the dorsiflexion range of motion (standardized mean difference [SMD] = 1.02, 95% confidence interval [CI]: 0.41 to 1.63) and dynamic balance (SMD = 0.49, 95% CI: 0.06 to 0.78) in individuals with CAI. However, there was no significant improvement in function (patient-oriented outcomes) (SMD = 0.76, 95% CI: -0.00 to 1.52). For individuals with CAI, joint mobilization has limited function but has positive benefits for the dorsiflexion range of motion and dynamic balance.

Effects of tongue strengthening exercises on tongue muscle strength: a systematic review and meta-analysis of randomized controlled trials

Tongue strengthening exercise (TSE) has been proposed as an intervention to increase tongue strength and improve swallowing. However, clinical evidence of its effectiveness is lacking. In this review, seven databases were searched from inception to September 30, 2021 for randomized controlled trials that compared tongue strengths between the TSE intervention and control groups, obtained from maximal tongue elevation peak force in kilopascals (kPa). The Cochrane risk of bias tool was used for quality assessment. In total, 12 studies with 388 participants were included. The pooled meta-analysis demonstrated that the anterior tongue strength (ATS) (MD = 5.34 kPa; 95% CI 3.28–7.40; I² = 71%) and posterior tongue strength (MD = 8.12; 95% CI 3.45–12.79; I² = 90%) were significantly higher in the TSE intervention than that in the control group. Among healthy participants, subgroup analysis showed that TSE had improvements on ATS in all age groups, with the greatest improvement in old people (≥ 65 years) (MD = 8.01; 95% CI 4.39–11.64; I² = 30%). Meta-regression analysis revealed a nonsignificant trend toward greater improvement on tongue strength with increasing TSE duration. This study provides positive evidence that TSE may be beneficial in improving tongue strength and could be applied for adults, especially healthy older adults.

Effects of Repetitive Transcranial Magnetic Stimulation on the Primary Motor Cortex of Individuals with Fibromyalgia: A Systematic Review and Meta-Analysis

The purpose of this study was to quantify the effect of repetitive transcranial magnetic stimulation (rTMS), which is recommended for the improvement of some pain-related symptoms and for antidepressant treatment, on the primary motor cortex (M1) in patients with fibromyalgia (FM). We searched for studies comparing rTMS and sham rTMS in the M1 of FM patients. Pain intensity, quality of life, health status, and depression were compared with or without rTMS for at least 10 sessions. We searched four databases. Quality assessment and quantitative analysis were performed using RevMan 5.4. After screening, five randomized controlled trials of 170 patients with FM were included in the analysis. As a result of the meta-analysis of rTMS on the M1 of individuals with FM, high-frequency rTMS resulted in a significant improvement on quality of life (MD = -2.50; 95% CI: -3.99 to -1.01) compared with sham rTMS. On the other hand, low-frequency rTMS resulted in a significant improvement on health status (MD = 15.02; 95% CI: 5.59 to 24.45). The application of rTMS to the M1 is proposed as an adjunctive measure in the treatment of individuals with FM. Because rTMS has various effects depending on each application site, it is necessary to classify sites or set frequencies as variables.

Assessing the risk of performance and detection bias in Cochrane reviews as a joint domain is less accurate compared to two separate domains

Background

Initially, the Cochrane risk of bias (RoB) tool had a domain for “blinding of participants, personnel and outcome assessors”. In the 2011 tool, the assessment of blinding was split into two domains: blinding of participants and personnel (performance bias) and blinding of outcome assessors (detection bias). The aims of this study were twofold; first, to analyze the frequency of usage of the joint blinding domain (a single domain for performance and detection bias), and second, to assess the proportion of adequate assessments made in the joint versus single RoB domains for blinding by comparing whether authors’ RoB judgments were supported by explanatory comments in line with the Cochrane Handbook recommendations.

Methods

We extracted information about the assessment of blinding from RoB tables (judgment, comment, and whether it was specified which outcome type; e.g., objective, subjective) of 729 Cochrane reviews published in 2015-2016. In the Cochrane RoB tool, judgment (low, unclear or high risk) needs to be accompanied by a transparent comment, in which authors provide a summary justifying RoB judgment, to ensure transparency in how these judgments were reached. We reassessed RoB based on the supporting comments reported in Cochrane RoB tables, in line with instructions from the Cochrane Handbook. Then, we compared our new assessments to judgments made by Cochrane authors. We compared the frequency of adequate judgments in reviews with two separate domains for blinding versus those with a joint domain for blinding.

Results

The total number of assessments for performance bias was 6918, with 8656 for detection bias and 3169 for the joint domain. The frequency of adequate assessments was 74% for performance bias, 78% for detection bias, and 59% for the joint domain. The lowest frequency of adequate assessments was found when Cochrane authors judged low risk – 47% in performance bias, 62% in detection bias, and 31% in the joint domain. The joint domain and detection bias domain had a similar proportion of specified outcome types (17% and 18%, respectively).

Conclusions

Splitting joint RoB assessment about blinding into two domains was justified because the frequency of adequate judgments was higher in separate domains. Specification of outcome types in RoB domains should be further scrutinized.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01339-1.

Enhanced access to recommendations from the Cochrane Handbook for improving authors' judgments about risk of bias: A randomized controlled trial

This randomized controlled trial (RCT) aimed to test the efficacy of enhanced access to Cochrane Handbook (Handbook) recommendations for judging the 2011 Cochrane risk of bias (RoB) domains for improving the adequacy of RoB judgments. Parallel-group RCT with a 1:1 allocation ratio (N = 2271 per group) was conducted. Eligible participants were corresponding authors of all published Cochrane reviews and protocols. After allocation by a random number generator, participants received 20 scenarios for assessing RoB. The intervention group was shown tables from the Handbook with instructions for assessing 2011 RoB tool together with scenarios they were supposed to assess-enhanced access to the Handbook. The control group was shown only a general link to the Handbook. The primary outcome was the proportion of participants that made an adequate judgment of RoB scenarios for analyzed domains. There were 240 responses out of 2020 delivered e-mail invitations in the intervention and 197/2254 in the control group. Only five participants from the intervention group judged RoB adequately in all the 20 scenarios and no one in the control group. The proportion of participants who adequately assessed all the scenarios within a domain was significantly higher in the intervention than in the control group. The frequency of adequate RoB judgments was 7.1% (95% CI: 5.0-9.3%, p < 0.001) higher in the intervention group (76.2%) than in the control group (69.0%). The enhanced access yields more adequate RoB assessments and could be incorporated in software supporting the RoB tool.

Virtual Reality Exergames for Improving Older Adults' Cognition and Depression: A Systematic Review and Meta-Analysis of Randomized Control Trials

OBJECTIVES

Virtual reality (VR) exergames are an innovative approach to promote older adults' mental health. VR exergames are active video games controlled by bodily movements in a safe surrounding with advantages of physical activity engagement and interactions. The purpose of this study was to explore the effectiveness of VR exergames in improving older adults' cognition and ameliorating depressive outcome by a systematic review, meta-analysis, subgroup analysis, and meta-regression.

DESIGN

This is a systematic review and meta-analysis.

SETTING AND PARTICIPANTS

Adults over 60 years old.

METHODS

Four electronic databases, including PubMed, Embase, Medline, and Cochrane, were searched for related articles. After evaluation, 18 randomized control trials were selected for qualitative and quantitative synthesis.

RESULTS

The meta-analytical results found moderate effects of VR exergames on overall cognitive function and memory, and a large effect on depressive outcomes in older adults. Commercial VR games had more-significant and larger effects on depressive outcomes than exergames with VR devices. The meta-regression results revealed that the total intervention duration had a significant effect on depressive outcomes. A higher intervention duration had greater effects on depressive outcomes.

CONCLUSIONS AND IMPLICATIONS

This meta-analytical study suggests that VR exergames can provide potential positive influences on cognition, memory, and depression in older adult populations. VR exergames could be an interesting strategy for active aging and a good mental health status.

Methodological quality and risk of bias in orthodontic systematic reviews using AMSTAR and ROBIS

BACKGROUND

This study aimed to compare the methodological quality and risk bias of orthodontic systematic reviews (SRs) using the AMSTAR (Assessing the Methodological Quality of Systematic Reviews) and ROBIS (Risk of Bias in Systematic Review) tools.

MATERIALS AND METHODS

A search of electronic databases (OVID and Medline) was undertaken to identify orthodontic SRs published in five primary orthodontic journals (January 2015 to December 2018) and the Cochrane Library of Systematic Reviews (January 2000 to January 2018). Full articles were reviewed by two assessors against the eligibility criteria. Methodological quality of each SR was gauged using the AMSTAR tool with a score of 0 or 1 given for each of the 11 items. Cumulative totals were calculated and scores between 4 and 8 represented poor to fair methodological quality and 9 or greater deemed to be good. As per the ROBIS tool, the risk of bias (ROB) for each domain was assessed and the overall ROB was classified as low, high, or unclear.

RESULTS

A total of 91 SRs were included. The median AMSTAR score was 8 (IQR = 3). The methodological quality of 47.3% SRs was rated good. SRs without protocol registration (Coef: -3.00, 95% CI: -3.72, -2.28, P < 0.001) and American continent SRs (Coef: -1.00, 95% CI: -1.72, -0.21, P = 0.007) were associated with lower AMSTAR scores. A total of 56.0% SRs were rated a low ROB, with a lower ROB apparent in multicentre SRs (OR: 0.27, 95% CI: 0.11, 0.64, P = 0.003) and a higher ROB evident in SRs without a registered protocol (OR: 111.81, 95% CI: 22.34, 559.62, P < 0.001). When adjusted for the effect of AMSTAR score on ROB, a higher ROB was associated with SRs without protocol registration (OR: 32.24, 95% CI: 6.03, 172.44, P ≤ 0.001). As the AMSTAR score (per unit) increased, the odds of having a high ROB rating decreased (OR: 0.31, 95% CI: 0.21, 0.45, P ≤ 0.001).

CONCLUSIONS

As the methodological quality rating of orthodontic SRs increases, a reduction in the ROB is evident.

Adequacy of risk of bias assessment in surgical vs non-surgical trials in Cochrane reviews: a methodological study

Background

Bias in randomized controlled trials (RCTs) can lead to underestimation or overestimation of the true effects of interventions. Surgical RCTs may suffer from the risk of bias (RoB) that is avoidable in trials of other interventions, and vice versa. We aimed to compare the adequacy of RoB assessments in surgical versus non-surgical RCTs included in Cochrane reviews and to assess the most common differences in those RoB assessments. Due to specificities of surgical trials, i.e. difficulties associated with blinding of surgical interventions, we hypothesized that assessments of surgical trials may be more adequate, compared to RCTs of non-surgical interventions.

Methods

This was a methodological study, analyzing methods of published Cochrane systematic reviews. Data were extracted from RoB tables in Cochrane reviews (judgments and accompanying explanatory comment) for the following four RoB domains used in the 2011 Cochrane RoB tool: randomization, allocation concealment, blinding of participants and personnel, and blinding of outcome assessors. We defined adequate assessments as those that were in line with instructions from the Cochrane Handbook for Systematic Reviews of Interventions. The prevalence of adequate assessments was compared in surgical versus non-surgical trials. The most common differences in both groups of reviews were presented.

Results

In 729 analyzed Cochrane reviews, there were 10,537 included trials. The prevalence of adequate RoB judgments made by Cochrane authors ranged from 87.9, 95%CI (87.3 to 88.6%) for randomization to 70.7, 95%CI (69.8 to 71.5%) for blinding of participants and personnel. For all analyzed RoB domains, the prevalence of adequate RoB domains was higher in surgical trials than in non-surgical trials. For two RoB domains assessing blinding, this difference between surgical and non-surgical trials was statistically significant (P < 0.001), while the difference was not significant for the RoB domain regarding randomization (P = 0.124) and allocation concealment (P = 0.039, β < 0.8).

Conclusions

RoB judgments were more in line with instructions from the Cochrane Handbook when Cochrane reviews assessed surgical trials, compared to those that analyzed non-surgical interventions. However, further steps are warranted to scrutinize RoB assessment in trials of both surgical and non-surgical interventions.

Agreement of Risk-of -Bias varied in systematic reviews on acupuncture and was associated with methodological quality

OBJECTIVES

The objective of the study was to evaluate the consistency of risk of bias assessments for overlapping randomized controlled trials (RCTs) included in systematic reviews (SRs) on acupuncture.

STUDY DESIGN AND SETTING

Databases were searched for acupuncture SRs. A weighted kappa (κ) statistic was calculated, and logistic regression was used to explore the factors of disagreements.

RESULTS

We included 241 RCTs from 109 SRs on acupuncture. The percentage disagreements ranged from 25% to 44%, with moderate agreement for random sequence generation (κ = 0.57), allocation concealment (κ = 0.50), and incomplete outcome data (κ = 0.50), besides fair agreement for blinding of participants and personnel (κ = 0.44), blinding of outcome assessment (κ = 0.31), and selective reporting (κ = 0.39). Only 19% RCTs were evaluated completely consistent. Methodological quality (random sequence generation, odds ratio (OR) = 3.46), international cooperation (allocation concealment, OR = 0.14; incomplete outcome data, OR = 0.14; selective reporting, OR = 0.05), and risk of bias reporting completeness score (selective reporting, OR = 0.53) significantly affected the relative odds of disagreements.

CONCLUSION

The level of agreement varied from fair to moderate agreement depending on the risk of bias domain. Methodological quality appears to be an overarching factor to account for disagreements.

Assessing risk of bias judgments for blinding of outcome assessors in Cochrane reviews

Aim: Adequate judging of risk of bias (RoB) for blinding of outcome assessors (detection bias) is important for supporting highest level of evidence. Materials & methods: Judgments and supporting comments for detection bias were retrieved from RoB tables reported in Cochrane reviews. We categorized comments, and then compared judgment and supporting comment with instructions from the Cochrane Handbook. Results: We analyzed 8656 judgments for detection bias from 7626 trials included in 575 reviews. Overall, 1909 judgments (22%) were not in line with the Cochrane Handbook. In 9% of trials, the authors split the detection bias domain according to outcomes. Here, prevalence of inadequate judgments was 19%. Conclusion: Interventions to improve RoB assessments in systematic reviews should be explored.

Methodological tools and sensitivity analysis for assessing quality or risk of bias used in systematic reviews published in the high-impact anesthesiology journals

Background

A crucial element in the systematic review (SR) methodology is the appraisal of included primary studies, using tools for assessment of methodological quality or risk of bias (RoB). SR authors can conduct sensitivity analyses to explore whether their results are sensitive to exclusion of low quality studies or a high RoB. However, it is unknown which tools do SR authors use for assessing quality/RoB, and how they set threshold for quality/RoB in sensitivity analyses. The aim of this study was to assess quality/RoB assessment tools, the types of sensitivity analyses and quality/RoB thresholds for sensitivity analyses used within SRs published in high-impact pain/anesthesiology journals.

Methods

This was a methodological study. We analyzed SRs published from January 2005 to June 2018 in the 25% highest-ranking journals within the Journal Citation Reports (JCR) “Anesthesiology” category. We retrieved the SRs from PubMed. Two authors independently screened records, full texts, and extracted data on quality/RoB tools and sensitivity analyses. We extracted data about quality/RoB tools, types of sensitivity analyses and the thresholds for quality/RoB used in them.

Results

Out of 678 analyzed SRs, 513 (76%) reported the use of quality/RoB assessments. The most commonly reported tools for assessing quality/RoB in the studies were the Cochrane tool for risk of bias assessment (N = 251; 37%) and Jadad scale (N = 99; 15%). Meta-analysis was conducted in 451 (66%) of SRs and sensitivity analysis in 219/451 (49%). Most commonly, sensitivity analysis was conducted to explore the influence of study quality/RoB (90/219; 41%) on the results. Quality/RoB thresholds used for sensitivity analysis for those studies were clearly reported in 47 (52%) articles that used them. The quality/RoB thresholds used for sensitivity analyses were highly heterogeneous and inconsistent, even when the same tool was used.

Conclusions

A quarter of SRs reported using quality/RoB assessments, and some of them cited tools that are not meant for assessing quality/RoB. Authors who use quality/RoB to explore the robustness of their results in meta-analyses use highly heterogeneous quality/RoB thresholds in sensitivity analyses. Better methodological consistency for quality/RoB sensitivity analyses is needed.

Cochrane risk of bias tool was used inadequately in the majority of non-Cochrane systematic reviews

OBJECTIVES

To analyze how many non-Cochrane systematic reviews (NCSRs) used Cochrane's risk of bias (RoB) tool, domains they used, and whether judgments and comments about RoB were in line with Cochrane Handbook.

METHODS

This was a methodological (research-on-research) study. We retrieved NCSRs from PubMed, extracted information about methods used for RoB assessment, and if they used 2011 Cochrane RoB tool, we analyzed their RoB methods and compared them with Cochrane Handbook guidance.

RESULTS

We included 508 NCSRs; 431 (85%) reported they analyzed RoB, and 269 (53%) used Cochrane RoB tool. Only 16 of those 269 (5.9%) reported both a judgment and a supporting comment in the Cochrane RoB table in the manuscript (N = 4) or in a supplementary file (N = 12). Fifteen reviews, with 158 included trials, used judgments low/high/unclear; 41% of analyzed available judgments were inadequate, either because judgment was not in line with comment or comment was missing.

CONCLUSIONS

Most NCSRs use Cochrane RoB tool to assess RoB, but most of them reported it incompletely, with high prevalence of inadequate judgments. Authors, editors, and peer-reviewers should make an effort to improve completeness and adequacy of Cochrane RoB assessment in non-Cochrane reviews.

Inter-review agreement of risk-of-bias judgments varied in Cochrane reviews

OBJECTIVES

The objective of the study was to measure the level of agreement between Cochrane reviews of overlapping randomized controlled trials (RCTs) regarding risk-of-bias (RoB) judgments.

STUDY DESIGN AND SETTING

On November 5, 2017, the Cochrane Database of Systematic Reviews was searched for Cochrane reviews on tobacco. Reviews that included overlapping RCTs were included. RoB judgments were extracted from RoB tables using automated data scraping with manual verification and adjustments. Agreement between the reviews was calculated using Conger's generalized kappa coefficient (κ) and raw agreement (a).

RESULTS

We included 53 Cochrane reviews of 376 RCTs. For the RoB domain "random sequence generation," the level of agreement between the reviews was substantial with κ = 0.63 (95% confidence interval: 0.56 to 0.71; a = 0.80). There was slight-to-moderate agreement between the reviews regarding the domains "allocation concealment": κ = 0.51 (0.41 to 0.61), a = 0.75; "blinding": κ = 0.19 (0.02 to 0.37), a = 0.52; "blinding of outcome assessment": κ = 0.43 (0.14 to 0.72) a = 0.67; and "incomplete outcome data": κ = 0.15 (-0.03 to 0.32), a = 0.64. For "blinding of participants and personnel" and "selective reporting", κ could not be calculated. The raw agreement was 0.40 and 0.42, respectively.

CONCLUSION

The level of agreement between Cochrane reviews regarding RoB judgments ranged from slight to substantial depending on the RoB domain. Further investigations regarding reasons for variation and interventions to improve agreement are needed.

Overall bias methods and their use in sensitivity analysis of Cochrane reviews were not consistent

OBJECTIVE

The objective of the study was to analyze methods of assessing "overall bias" in Cochrane reviews of interventions published in the Cochrane Database of Systematic Reviews and sensitivity analyses related to overall risk of bias (RoB).

STUDY DESIGN AND SETTING

From Cochrane reviews published within 3 years, from July 2015 to June 2018, we extracted data regarding methods of judging overall bias for a single trial, as well as details regarding methods used in frequency of RoB in sensitivity analyses.

RESULTS

Of the 1,452 analyzed Cochrane reviews, 409 mentioned assessment of overall RoB on a study level. In 107 reviews, authors clearly specified key domains that determined the overall RoB, whereas in the remaining reviews, assessment of overall bias was not in line with the Cochrane Handbook. Among 268 Cochrane reviews that had any RoB-related sensitivity analysis, in 56 (21%) reviews, the authors reported a significant change for at least one outcome compared with the initial analysis.

CONCLUSION

Highly heterogeneous approaches to summarizing overall RoB on a study level and using RoB for sensitivity analyses may yield inconsistent and incomparable results across Cochrane reviews.

Efficacy and safety of modified-release paracetamol for acute and chronic pain: a systematic review protocol

INTRODUCTION

Paracetamol (acetaminophen) is widely used for management of mild-to-moderate pain and reduction of fever. It is available as immediate release (IR) and modified-release (MR) formulation. In 2017, European Medicines Agency recommended a suspension of marketing of MR paracetamol in the European Union. Benefit-risk balance of these products has been assessed as negative as data showed that existing procedures for overdose management may not be efficient. Since MR paracetamol is still available in other countries (Australia and USA) and there is no available systematic review (SR) of efficacy and safety of MR paracetamol in the literature, we have decided to perform one to evaluate available data from randomised clinical trials (RCTs).

METHODS AND ANALYSIS

Using predefined search criteria, we will search EMBASE, MEDLINE, Cochrane Central Register of Controlled Trials, ClinicalTrials.gov and WHO International Clinical Trials Registry Platform to identify RCTs evaluating efficacy and safety of MR paracetamol alone in any dose or duration for any pain. Participants are defined as adults and adolescents (over 12 years). Primary efficacy outcomes will be pain intensity, pain relief and sleep. Primary safety outcomes will be the number of patients experiencing any (serious) adverse event, the number of patients withdrawn due to adverse events and the number of patients with gastrointestinal and hepatic adverse events. Data analysis will be subdivided based on different clinical syndromes. Meta-analysis will be conducted if possible. Cochrane risk of bias (RoB) tool with seven dimensions will be used to assess RoB of individual studies.

ETHICS AND DISSEMINATION

This SR will include only data collected from trial reports; therefore, an ethical approval will not be sought. We will publish the protocol and our findings in peer-reviewed journals.

PROSPERO REGISTRATION NUMBER

CRD42018115769.

Risk of bias judgments for random sequence generation in Cochrane systematic reviews were frequently not in line with Cochrane Handbook

BACKGROUND

Assessing the risk of bias (RoB) in included studies is one of the key methodological aspects of systematic reviews. Cochrane systematic reviews appraise RoB of randomised controlled trials (RCTs) with the Cochrane RoB tool. Detailed instructions for using the Cochrane RoB tool are provided in the Cochrane Handbook for Systematic Reviews of Interventions (The Cochrane Handbook). The purpose of this study was to analyse whether Cochrane authors use adequate judgments about the RoB for random sequence generation of RCTs included in Cochrane reviews.

METHODS

We extracted authors' judgments (high, low or unclear RoB) and supports for judgments (comments accompanying judgments which explain the rationale for a judgment) for random sequence generation of included RCTs from RoB tables of Cochrane reviews using automated data scraping. We categorised all supporting comments, analysed the number and type of various supporting comments and assessed adequacy of RoB judgment for randomisation in line with recommendations from the Cochrane Handbook.

RESULTS

We analysed 10,103 RCTs that were included in 704 Cochrane reviews. For 5,706 RCTs, randomisation was not described, but for the remaining RCTs, it was indicated that randomisation was performed using computer/software/internet (N = 2,850), random number table (N = 883), mechanical method (N = 359) or it was incomplete/inappropriate (N = 305). Overall, 1,220/10,103 trials (12%) did not have a RoB judgment in line with Cochrane Handbook guidance about randomisation. The highest proportion of misjudgements was found for trials with high RoB (28%), followed by those with low (20%) or unclear (3%). Therefore, one in eight judgments for the analysed domain in Cochrane reviews was not in line with Cochrane Handbook, and one in four if the judgment was "high risk".

CONCLUSION

Authors of Cochrane reviews often make judgments about the RoB related to random sequence generation that are not in line with instructions given in the Cochrane Handbook, which compromises the reliability of the systematic reviews. Our results can help authors of both Cochrane and non-Cochrane reviews which use Cochrane RoB tool to avoid making common mistakes when assessing RoB in included trials.