Inter-rater reliability and concurrent validity of ROBINS-I: protocol for a cross-sectional study

Integrating large language models in systematic reviews: a framework and case study using ROBINS-I for risk of bias assessment

Large language models (LLMs) may facilitate and expedite systematic reviews, although the approach to integrate LLMs in the review process is unclear. This study evaluates GPT-4 agreement with human reviewers in assessing the risk of bias using the Risk Of Bias In Non-randomised Studies of Interventions (ROBINS-I) tool and proposes a framework for integrating LLMs into systematic reviews. The case study demonstrated that raw per cent agreement was the highest for the ROBINS-I domain of 'Classification of Intervention'. Kendall agreement coefficient was highest for the domains of 'Participant Selection', 'Missing Data' and 'Measurement of Outcomes', suggesting moderate agreement in these domains. Raw agreement about the overall risk of bias across domains was 61% (Kendall coefficient=0.35). The proposed framework for integrating LLMs into systematic reviews consists of four domains: rationale for LLM use, protocol (task definition, model selection, prompt engineering, data entry methods, human role and success metrics), execution (iterative revisions to the protocol) and reporting. We identify five basic task types relevant to systematic reviews: selection, extraction, judgement, analysis and narration. Considering the agreement level with a human reviewer in the case study, pairing artificial intelligence with an independent human reviewer remains required.

Does the evidence support in utero influences on later health and disease? A systematic review of highly cited Barker studies on developmental origins

The developmental origins of health and disease hypothesis proposes that early exposure to adverse conditions during fetal development and early life have strong detrimental consequences on long-term health and susceptibility to chronic diseases. We conducted a systematic review to critically appraise Barker's highest cited publications using the risk-of-bias assessment tool (ROBINS-I) and investigate effects of overadjustment by later body weight. Our findings revealed that all included studies displayed high risks of bias, with particular concerns regarding confounding (8/8), selection of reported results (8/8), classification of exposure (7/8), selection of participants (5/8) and high rates of missing data (ranged from 15 to 87%). Later body weight was over-adjusted in most (6/8) of the studies. As all studies displayed high bias risk due to confounding, missing data and overadjustment, evidence is insufficient to support causal relationships between low birthweight and adult disease, warranting caution in clinical application. PROTOCOL REGISTRATION: PROSPERO CRD42023433179.

Racial and ethnic differences in the effects of state firearm laws: a systematic review subgroup analysis

BACKGROUND

Despite growing evidence about how state-level firearm regulations affect overall rates of injury and death, little is known about whether potential harms or benefits of firearm laws are evenly distributed across demographic subgroups. In this systematic review, we synthesized available evidence on the extent to which firearm policies produce differential effects by race and ethnicity on injury, recreational or defensive gun use, and gun ownership or purchasing behaviors.

MAIN BODY

We searched 13 databases for English-language studies published between 1995 and February 28, 2023 that estimated a relationship between firearm policy in the USA and one of eight outcomes, included a comparison group, evaluated time series data, and provided estimated policy effects differentiated by race or ethnicity. We used pre-specified criteria to evaluate the quality of inference and causal effect identification. By policy and outcome, we compared policy effects across studies and across racial/ethnic groups using two different ways to express effect sizes: incidence rate ratios (IRRs) and rate differences. Of 182 studies that used quasi-experimental methods to evaluate firearm policy effects, only 15 estimated policy effects differentiated by race or ethnicity. These 15 eligible studies provided 57 separate policy effect comparisons across race/ethnicity, 51 of which evaluated interpersonal violence. In IRR terms, there was little consistent evidence that policies produced significantly different effects for different racial/ethnic groups. However, because of different baseline homicide rates, similar relative effects for some policies (e.g., universal background checks) translated into significantly greater absolute differences in homicide rates among Black compared to white victims.

CONCLUSIONS

The current literature does not support strong conclusions about whether state firearm policies differentially benefit or harm particular racial/ethnic groups. This largely reflects limited attention to these questions in the literature and challenges with detecting such effects given existing data availability and statistical power. Findings also emphasize the need for additional rigorous research that adopts a more explicit focus on testing for racial differences in firearm policy effects and that assesses the quality of race/ethnicity information in firearm injury and crime datasets.

Impact of Human Papillomavirus Vaccination on Male Disease: A Systematic Review

Human papillomavirus (HPV)-related diseases are highly prevalent in men worldwide, comprising external anogenital condyloma, anal intraepithelial neoplasia (AIN), penile intraepithelial neoplasia (PIN), and anogenital and oropharyngeal cancers. There is exceptionally low vaccine coverage in the male population. Only 4% of men were fully vaccinated, worldwide, as of 2019. The aim of this review is to assess the impact of HPV vaccination on male disease. Three databases (MEDLINE, Web of Science, Scopus) and Clinical Trials.gov were searched. We included thirteen studies, eight randomized controlled trials (RCTs), and five cohorts, comprising a total of 14,239 participants. Regarding anal disease, seven studies reported HPV vaccine efficacy ranging from 91.1% to 93.1% against AIN1, and ranging from 89.6% to 91.7% against AIN2|3 and anal cancer. Five studies showed an efficacy against genital condyloma of 89.9% in HPV-naïve males, varying between 66.7% and 67.2% in intention-to-treat populations. Studies reporting no efficacy have included older participants. These results support vaccination of young men previously infected, beyond HPV-naïve males. The evidence quality was moderate to low for most outcomes, namely genital diseases. RCTs are needed to assess the efficacy of HPV vaccination on male oropharyngeal cancer.

Effects of music therapy intervention on gait disorders in persons with multiple sclerosis: A systematic review of clinical trials

BACKGROUND

Music Therapy (MT) is a unique treatment method for Persons with Multiple Sclerosis (PwMS) that can accelerate their functional recovery. MT has been proven to adjust the gait performance of PwMS in a short period. Its therapeutic effects in gait disorders of PwMS for long-term intervention are also starting to draw interest, but it has yet to be investigated.

AIM

This review aimed to systematically examine the outcomes of PwMS with gait disorders after receiving MT intervention.

METHODS

A systematic review has been performed using several academic databases with keywords such as music therapy, multiple sclerosis, and gait. The study protocol was registered on PROSPERO (CRD42022365668).

RESULTS

A total of 405 studies were initially identified. After applying the inclusion and exclusion criteria, twelve studies were finally included. The results showed that all PwMS received MT intervention with different strategies, and ten studies confirmed that gait disorders of PwMS were effectively improved by MT intervention.

CONCLUSION

Most previous studies focused on the transient effects of MT on the gait performance of PwMS. This review bridges gaps in the long-term intervention of MT on gait disorders of PwMS and offers references for therapists to design treatment plans. According to this review, MT intervention has positive therapeutic effects on gait disorders in PwMS.

Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training

Assessing the risk of bias (ROB) of studies is an important part of the conduct of systematic reviews and meta-analyses in clinical medicine. Among the many existing ROB tools, the Prediction Model Risk of Bias Assessment Tool (PROBAST) is a rather new instrument specifically designed to assess the ROB of prediction studies. In our study we analyzed the inter-rater reliability (IRR) of PROBAST and the effect of specialized training on the IRR. Six raters independently assessed the risk of bias (ROB) of all melanoma risk prediction studies published until 2021 (n = 42) using the PROBAST instrument. The raters evaluated the ROB of the first 20 studies without any guidance other than the published PROBAST literature. The remaining 22 studies were assessed after receiving customized training and guidance. Gwet's AC₁ was used as the primary measure to quantify the pairwise and multi-rater IRR. Depending on the PROBAST domain, results before training showed a slight to moderate IRR (multi-rater AC₁ ranging from 0.071 to 0.535). After training, the multi-rater AC₁ ranged from 0.294 to 0.780 with a significant improvement for the overall ROB rating and two of the four domains. The largest net gain was achieved in the overall ROB rating (difference in multi-rater AC₁: 0.405, 95%-CI 0.149-0.630). In conclusion, without targeted guidance, the IRR of PROBAST is low, questioning its use as an appropriate ROB instrument for prediction studies. Intensive training and guidance manuals with context-specific decision rules are needed to correctly apply and interpret the PROBAST instrument and to ensure consistency of ROB ratings.

An Empirical Comparative Assessment of Inter-Rater Agreement of Binary Outcomes and Multiple Raters

Background: Many methods under the umbrella of inter-rater agreement (IRA) have been proposed to evaluate how well two or more medical experts agree on a set of outcomes. The objective of this work was to assess key IRA statistics in the context of multiple raters with binary outcomes. Methods: We simulated the responses of several raters (2–5) with 20, 50, 300, and 500 observations. For each combination of raters and observations, we estimated the expected value and variance of four commonly used inter-rater agreement statistics (Fleiss’ Kappa, Light’s Kappa, Conger’s Kappa, and Gwet’s AC1). Results: In the case of equal outcome prevalence (symmetric), the estimated expected values of all four statistics were equal. In the asymmetric case, only the estimated expected values of the three Kappa statistics were equal. In the symmetric case, Fleiss’ Kappa yielded a higher estimated variance than the other three statistics. In the asymmetric case, Gwet’s AC1 yielded a lower estimated variance than the three Kappa statistics for each scenario. Conclusion: Since the population-level prevalence of a set of outcomes may not be known a priori, Gwet’s AC1 statistic should be favored over the three Kappa statistics. For meaningful direct comparisons between IRA measures, transformations between statistics should be conducted.

COVID-19-Associated Encephalopathy: Systematic Review of Case Reports

Background and Purpose

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) primarily attacks the respiratory system, but there are also several reports of the involvement of the central nervous system, with one of the manifestations being encephalopathy. The relatively new emergence of COVID-19 means that few studies have investigated the clinical profile of encephalopathy associated with this disease. This study aimed to determine the clinical profile, laboratory, and imaging results of encephalopathy associated with COVID-19.

Methods

Three databases, namely PubMed/MEDLINE, Embase, and Scopus, were systematically searched for case reports and case series related to COVID-19-associated encephalopathy published from January 1, 2019 to July 20, 2020.

Results

This review included 24 studies involving 33 cases. The most-reported neurological symptoms were disorientation/confusion (72.72%), decreased consciousness (54.54%), and seizures (27.27%). Laboratory examinations revealed increases in the C-reactive protein level (48.48%), the lactate dehydrogenase level (30.30%), and lymphopenia (27.27%). Brain imaging did not produce any pathological findings in 51.51% of the cases. Electroencephalography showed generalized slowing in 45.45% of the cases. Elevated protein (42.42%) and lymphocytosis (24.24%) were found in the cerebrospinal fluid. Fifteen patients were reportedly discharged from the hospital in a stable condition, while four cases of mortality were recorded.

Conclusions

The clinical, laboratory, and imaging findings in this review support the hypothesis that cerebral damage in COVID-19-associated encephalopathy is caused by cytokine-immune-mediated inflammation rather than by direct invasion.

Selecting Risk of Bias Tools for Observational Studies for a Systematic Review of Anthropometric Measurements and Dental Caries among Children

In conducting a systematic review, assessing the risk of bias of the included studies is a vital step; thus, choosing the most pertinent risk of bias (ROB) tools is crucial. This paper determined the most appropriate ROB tools for assessing observational studies in a systematic review assessing the association between anthropometric measurements and dental caries among children. First, we determined the ROB tools used in previous reviews on a similar topic. Subsequently, we reviewed articles on ROB tools to identify the most recommended ROB tools for observational studies. Of the twelve ROB tools identified from the previous steps, three ROB tools that best fit the eight criteria of a good ROB tool were the Newcastle-Ottawa Scale (NOS) for cohort and case-control studies, and Agency for Healthcare Research and Quality (AHRQ) and the Effective Public Health Practice Project (EPHPP) for a cross-sectional study. We further assessed the inter-rater reliability for all three tools by analysing the percentage agreement, inter-class correlation coefficient (ICC) and kappa score. The overall percentage agreements and reliability scores of these tools ranged from good to excellent. Two ROB tools for the cross-sectional study were further evaluated qualitatively against nine of a tool's advantages and disadvantages. Finally, the AHRQ and NOS were selected as the most appropriate ROB tool to assess cross-sectional and cohort studies in the present review.

Customized guidance/training improved the psychometric properties of methodologically rigorous risk of bias instruments for non-randomized studies

OBJECTIVES

To evaluate the impact of guidance and training on the inter-rater reliability (IRR), inter-consensus reliability (ICR) and evaluator burden of the Risk of Bias (RoB) in Non-randomized Studies (NRS) of Interventions (ROBINS-I) tool, and the RoB instrument for NRS of Exposures (ROB-NRSE).

STUDY DESIGN AND SETTING

In a before-and-after study, seven reviewers appraised the RoB using ROBINS-I (n = 44) and ROB-NRSE (n = 44), before and after guidance and training. We used Gwet's AC₁ statistic to calculate IRR and ICR.

RESULTS

After guidance and training, the IRR and ICR of the overall bias domain of ROBINS-I and ROB-NRSE improved significantly; with many individual domains showing either a significant (IRR and ICR of ROB-NRSE; ICR of ROBINS-I), or nonsignificant improvement (IRR of ROBINS-I). Evaluator burden significantly decreased after guidance and training for ROBINS-I, whereas for ROB-NRSE there was a slight nonsignificant increase.

CONCLUSION

Overall, there was benefit for guidance and training for both tools. We highly recommend guidance and training to reviewers prior to RoB assessments and that future research investigate aspects of guidance and training that are most effective.

Cost-effectiveness of prehabilitation prior to elective surgery compared to usual preoperative care: protocol for a systematic review of economic evaluations

INTRODUCTION

Preoperative functional capacity is an important predictor of postoperative outcomes. Prehabilitation aims to optimise patients' functional capacity before surgery to improve postoperative outcomes. As prolonged hospital stay and postoperative complications present an avoidable use of healthcare resources, prehabilitation might also save costs.The aim of this systematic review is to investigate the cost-effectiveness of prehabilitation programmes for patients awaiting elective surgery compared with usual preoperative care. The results will be useful to inform decisions about the implementation of prehabilitation programmes and the design of future economic evaluations of prehabilitation programmes.

METHODS AND ANALYSIS

We will search PubMed, Embase, the Centre for Reviews and Dissemination Database, the WHO International Clinical Trials Registry Platform and ClinicalTrials.gov for full or partial economic evaluations of preoperative prehabilitation programmes conducted in any population compared with usual preoperative care. Studies will be included regardless of the type, design and perspective of the economic evaluation, and their publication year, language or status. Initial searches were performed between 30 April and 4 May 2020.Study selection, data extraction and assessment of the included studies' risk of bias and methodological quality will initially be performed by two independent reviewers and, if agreement was sufficiently high, by one reviewer. We will extract data regarding the included studies' basic characteristics, economic evaluation methods and cost-effectiveness results.A narrative synthesis will be performed. The primary endpoint will be cost-effectiveness based on cost-utility analyses. We will discuss heterogeneity between the studies and assess the risk of publication bias. The certainty of the evidence will be determined using the Grading of Recommendations, Assessment, Development and Evaluation approach.

ETHICS AND DISSEMINATION

Ethics approval is not required as the systematic review will not involve human participants. We plan to present our findings at scientific conferences, pass them on to relevant stakeholder organisations and publish them in a peer-reviewed journal.

PROSPERO REGISTRATION NUMBER

CRD42020182813.

Observational studies assessing the pharmacological treatment of obstructive lung disease: strengths, challenges and considerations for study design

Randomised controlled trials (RCTs) are the gold standard for evaluating treatment efficacy in patients with obstructive lung disease. However, due to strict inclusion criteria and the conditions required for ascertaining statistical significance, the patients included typically represent as little as 5% of the general obstructive lung disease population. Thus, studies in broader patient populations are becoming increasingly important. These can be randomised effectiveness trials or observational studies providing data on real-world treatment effectiveness and safety data that complement efficacy RCTs. In this review we describe the features associated with the diagnosis of asthma and chronic obstructive pulmonary disease (COPD) in the real-world clinical practice setting. We also discuss how RCTs and observational studies have reported opposing outcomes with several treatments and inhaler devices due to differences in study design and the variations in patients recruited by different study types. Whilst observational studies are not without weaknesses, we outline recently developed tools for defining markers of quality of observational studies. We also examine how observational studies are capable of providing valuable insights into disease mechanisms and management and how they are a vital component of research into obstructive lung disease. As we move into an era of personalised medicine, recent observational studies, such as the NOVEL observational longiTudinal studY (NOVELTY), have the capacity to provide a greater understanding of the value of a personalised healthcare approach in patients in clinical practice by focussing on standardised outcome measures of patient-reported outcomes, physician assessments, airway physiology, and blood and airway biomarkers across both primary and specialist care.

Inter-rater reliability and validity of risk of bias instrument for non-randomized studies of exposures: a study protocol

BACKGROUND

A new tool, "risk of bias (ROB) instrument for non-randomized studies of exposures (ROB-NRSE)," was recently developed. It is important to establish consistency in its application and interpretation across review teams. In addition, it is important to understand if specialized training and guidance will improve the reliability in the results of the assessments. Therefore, the objective of this cross-sectional study is to establish the inter-rater reliability (IRR), inter-consensus reliability (ICR), and concurrent validity of the new ROB-NRSE tool. Furthermore, as this is a relatively new tool, it is important to understand the barriers to using this tool (e.g., time to conduct assessments and reach consensus-evaluator burden).

METHODS

Reviewers from four participating centers will apprise the ROB of a sample of NRSE publications using ROB-NRSE tool in two stages. For IRR and ICR, two pairs of reviewers will assess the ROB for each NRSE publication. In the first stage, reviewers will assess the ROB without any formal guidance. In the second stage, reviewers will be provided customized training and guidance. At each stage, each pair of reviewers will resolve conflicts and arrive at a consensus. To calculate the IRR and ICR, we will use Gwet's AC₁ statistic. For concurrent validity, reviewers will appraise a sample of NRSE publications using both the Newcastle-Ottawa Scale (NOS) and ROB-NRSE tool. We will analyze the concordance between the two tools for similar domains and for the overall judgments using Kendall's tau coefficient. To measure evaluator burden, we will assess the time taken to apply ROB-NRSE tool (without and with guidance), and the NOS. To assess the impact of customized training and guidance on the evaluator burden, we will use the generalized linear models. We will use Microsoft Excel and SAS 9.4, to manage and analyze study data, respectively.

DISCUSSION

The quality of evidence from systematic reviews that include NRSE depends partly on the study-level ROB assessments. The findings of this study will contribute to an improved understanding of ROB-NRSE and how best to use it.