Review of quality assessment tools for the evaluation of pharmacoepidemiological safety studies

The ROBINS-I and the NOS had similar reliability but differed in applicability: A random sampling observational studies of systematic reviews/meta-analysis

OBJECTIVE

There is a lack of evidence on the usage of the quality assessment tool-the Risk Of Bias In Nonrandomized Studies-of Interventions (ROBINS-I). This article aimed to measure the reliability, criterion validity, and feasibility of the ROBINS-I and the Newcastle-Ottawa Scale (NOS).

METHODS

A sample of systematic reviews or meta-analyses of observational studies were selected from Medline (2013-2017) and assessed by two reviewers using ROBINS-I and the NOS. We reported on reliability in terms of the first-order agreement coefficient (AC1) statistic. Correlation coefficient statistic was used to explore the criterion validity of the ROBINS-I. We compared the feasibility of the ROBINS-I and NOS by recording the time to complete an assessment and the instances where assessing was difficult.

RESULTS

Five systematic reviews containing 41 cohort studies were finally included. Interobserver agreement on the individual domain of the ROBINS-I as well as the NOS was substantial with a mean AC1 statistic of 0.67 (95% CI: 0.50-0.83) and 0.73 (95% CI: 0.65-0.81), respectively. The criterion validity of the ROBNS-I was moderate (K = 0.52) against NOS. The time in assessing a single study by ROBINS-I varied from 7 hours initially to 3 hours compared with 30 minutes for the NOS. Both reviewers rated "bias due to departure from the intended interventions" the most time-consuming domain in the ROBINS-I, items in the NOS were equal.

CONCLUSIONS

The ROBINS-I and the NOS seem to provide the same reliability but vary in applicability. The over-complicated feature of ROBINS-I may limit its usage and a simplified version is needed.

How well can we assess the validity of non-randomised studies of medications? A systematic review of assessment tools

OBJECTIVE

To determine whether assessment tools for non-randomised studies (NRS) address critical elements that influence the validity of NRS findings for comparative safety and effectiveness of medications.

DESIGN

Systematic review and Delphi survey.

DATA SOURCES

We searched PubMed, Embase, Google, bibliographies of reviews and websites of influential organisations from inception to November 2019. In parallel, we conducted a Delphi survey among the International Society for Pharmacoepidemiology Comparative Effectiveness Research Special Interest Group to identify key methodological challenges for NRS of medications. We created a framework consisting of the reported methodological challenges to evaluate the selected NRS tools.

STUDY SELECTION

Checklists or scales assessing NRS.

DATA EXTRACTION

Two reviewers extracted general information and content data related to the prespecified framework.

RESULTS

Of 44 tools reviewed, 48% (n=21) assess multiple NRS designs, while other tools specifically addressed case-control (n=12, 27%) or cohort studies (n=11, 25%) only. Response rate to the Delphi survey was 73% (35 out of 48 content experts), and a consensus was reached in only two rounds. Most tools evaluated methods for selecting study participants (n=43, 98%), although only one addressed selection bias due to depletion of susceptibles (2%). Many tools addressed the measurement of exposure and outcome (n=40, 91%), and measurement and control for confounders (n=40, 91%). Most tools have at least one item/question on design-specific sources of bias (n=40, 91%), but only a few investigate reverse causation (n=8, 18%), detection bias (n=4, 9%), time-related bias (n=3, 7%), lack of new-user design (n=2, 5%) or active comparator design (n=0). Few tools address the appropriateness of statistical analyses (n=15, 34%), methods for assessing internal (n=15, 34%) or external validity (n=11, 25%) and statistical uncertainty in the findings (n=21, 48%). None of the reviewed tools investigated all the methodological domains and subdomains.

CONCLUSIONS

The acknowledgement of major design-specific sources of bias (eg, lack of new-user design, lack of active comparator design, time-related bias, depletion of susceptibles, reverse causation) and statistical assessment of internal and external validity is currently not sufficiently addressed in most of the existing tools. These critical elements should be integrated to systematically investigate the validity of NRS on comparative safety and effectiveness of medications. SYSTEMATIC REVIEW PROTOCOL AND REGISTRATION: https://osf.io/es65q.

Risk of bias tools in systematic reviews of health interventions: an analysis of PROSPERO-registered protocols

BACKGROUND

Systematic reviews of health interventions are increasingly incorporating evidence outside of randomized controlled trials (RCT). While non-randomized study (NRS) types may be more prone to bias compared to RCT, the tools used to evaluate risk of bias (RoB) in NRS are less straightforward and no gold standard tool exists. The objective of this study was to evaluate the planned use of RoB tools in systematic reviews of health interventions, specifically for reviews that planned to incorporate evidence from RCT and/or NRS.

METHODS

We evaluated a random sample of non-Cochrane protocols for systematic reviews of interventions registered in PROSPERO between January 1 and October 12, 2018. For each protocol, we extracted data on the types of studies to be included (RCT and/or NRS) as well as the name and number of RoB tools planned to be used according to study design. We then conducted a longitudinal analysis of the most commonly reported tools in the random sample. Using keywords and name variants for each tool, we searched PROSPERO records by year since the inception of the database (2011 to December 7, 2018), restricting the keyword search to the "Risk of bias (quality) assessment" field.

RESULTS

In total, 471 randomly sampled PROSPERO protocols from 2018 were included in the analysis. About two-thirds (63%) of these planned to include NRS, while 37% restricted study design to RCT or quasi-RCT. Over half of the protocols that planned to include NRS listed only a single RoB tool, most frequently the Cochrane RoB Tool. The Newcastle-Ottawa Scale and ROBINS-I were the most commonly reported tools for NRS (39% and 33% respectively) for systematic reviews that planned to use multiple RoB tools. Looking at trends over time, the planned use of the Cochrane RoB Tool and ROBINS-I seems to be increasing.

CONCLUSIONS

While RoB tool selection for RCT was consistent, with the Cochrane RoB Tool being the most frequently reported in PROSPERO protocols, RoB tools for NRS varied widely. Results suggest a need for more education and awareness on the appropriate use of RoB tools for NRS. Given the heterogeneity of study designs comprising NRS, multiple RoB tools tailored to specific designs may be required.

Patterns and Determinants of Prescribing for Parkinson's Disease: A Systematic Literature Review

Since the discovery of levodopa (L-dopa) in 1967, the range of medications available to treat Parkinson's disease has increased significantly and guidance on the use, efficacy, and safety of these medications has evolved. To assess levels of adherence to national prescribing guidelines and awareness of changes in the efficacy and safety data published in the profiles of medications for the treatment of PD, we have reviewed studies on patterns and determinants of prescribing PD medications conducted in the last 50 years (since the discovery of L-dopa). A systematic literature review was conducted using EMBASE (1967 to March, 2018), Ovid MEDLINE(R) ALL (1967 to March 16, 2018), PsycINFO (1967 to the 2nd week of March, 2018), and PubMed to identify all studies measuring prescribing patterns of PD medication between 1967 and 2017. Study design, source of data, country, year of study, number of patients and/or prescriptions, unit of analysis, prescribing determinants, and percentage utilisation of PD medications were extracted where possible. 44 studies examining prescribing patterns and/or prescribing determinants across 17 countries were identified. Unsurprisingly, L-dopa was the most commonly prescribed medication in all studies, accounting for 46.50% to 100% of all prescriptions for PD. In several studies, the prescribing rate of ergot-derived dopamine agonists (DAs) decreased over time in concordance with guidance. In contrast, the prescribing rates of non-ergot DAs increased over the last ten years in most of the included studies. In examining prescribing factors, two major categories were exemplified, patients' factors and prescribers' factors, with patients' age being the most common factor that affected the prescription in most studies. In conclusion, L-dopa is now the most commonly prescribed medication for cases of PD but there is large variation in the prescribing rates of catechol-O-methyltransferase (COMT) inhibitors, monoamine oxidase B (MAO-B) inhibitors, amantadine, and anticholinergics between countries. New studies examining the effects of recent clinical trials and measuring the prescribing rates of newly approved medications are warranted.

American Geriatrics Society 2019 Updated AGS Beers Criteria® for Potentially Inappropriate Medication Use in Older Adults

The American Geriatrics Society (AGS) Beers Criteria® (AGS Beers Criteria®) for Potentially Inappropriate Medication (PIM) Use in Older Adults are widely used by clinicians, educators, researchers, healthcare administrators, and regulators. Since 2011, the AGS has been the steward of the criteria and has produced updates on a 3-year cycle. The AGS Beers Criteria® is an explicit list of PIMs that are typically best avoided by older adults in most circumstances or under specific situations, such as in certain diseases or conditions. For the 2019 update, an interdisciplinary expert panel reviewed the evidence published since the last update (2015) to determine if new criteria should be added or if existing criteria should be removed or undergo changes to their recommendation, rationale, level of evidence, or strength of recommendation. J Am Geriatr Soc 67:674-694, 2019.

Exosomal microRNAs as potential circulating biomarkers in gastrointestinal tract cancers: a systematic review protocol

BACKGROUND

Metastasis is the most frequent type of recurrence in gastrointestinal (GI) cancers, and there is an emerging potential for new diagnostic and therapeutic approaches, especially in the cases of metastatic GI carcinomas. The expression profiles of circulating exosomal microRNAs are of particular interest as novel non-invasive diagnostic and prognostic biomarkers for improved detection of GI cancers in body fluids, especially in the serum of patients with recurrent cancers. The aim of this study is to systematically review primary studies and identify the miRNA profiles of serum exosomes of GI cancers.

METHODS AND DESIGN

This systematic review will be reported in line with the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidance. Relevant studies will be identified through a comprehensive search of the following main electronic databases: PubMed, Web of Science, Embase, Scopus, and Google Scholar, with no language restrictions (up to July 2017). Full copies of articles will be identified by a defined search strategy and will be considered for inclusion against pre-defined criteria. The quality assessment of the included studies will be performed by the Newcastle-Ottawa Scale (NOS). Data will be analyzed using Stata software V.12. Publication bias will be assessed by funnel plots, Beggs' and Eggers' tests. The levels of evidence for primary outcomes will be evaluated using the GRADE criteria.

DISCUSSION

The analysis of circulating exosomal miRNA profiles provides attractive screening and non-invasive diagnostic tools for the majority of solid tumors including GI cancers. There is limited information regarding the relationship between serum exosomal miRNA profiles and the pathological condition of patients with different GI cancers. Since there is no specific biomarker for GI cancers, we aim to suggest a number of circulating exosomal miRNA candidates as potential multifaceted GI cancer biomarkers for clinical utility.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42017057129.

A new risk of bias checklist applicable to randomized trials, observational studies, and systematic reviews was developed and validated to be used for systematic reviews focusing on drug adverse events

OBJECTIVES

The objective of the study was to develop and validate an adequate tool to evaluate the risk of bias of randomized controlled trials, observational studies, and systematic reviews assessing drug adverse events.

STUDY DESIGN AND SETTING

We developed a structured risk of bias checklist applicable to randomized trials, cohort, case-control and nested case-control studies, and systematic reviews focusing on drug safety. Face and content validity was judged by three experienced reviewers. Interrater and intrarater reliability were determined using 20 randomly selected studies, assessed by three other independent reviewers including one performing a 3-week retest.

RESULTS

The developed checklist examines eight domains: study design and objectives, selection bias, attrition, adverse events information bias, other information bias, statistical methods to control confounding, other statistical methods, and conflicts of interest. The total number of questions varied from 10 to 32 depending on the study design. Interrater and intrarater agreements were fair with Kendall's W of 0.70 and 0.74, respectively. Median time to complete the checklist was 8.5 minutes.

CONCLUSION

The developed checklist showed face and content validity and acceptable reliability to assess the risk of bias for studies analyzing drug adverse events. Hence, it might be considered as a novel useful tool for systematic reviews and meta-analyses focusing on drug safety.

Anticholinergic medication use and dementia: latest evidence and clinical implications

Use of medications with anticholinergic activity is widespread in older adults. Several studies have highlighted that anticholinergic use may be associated with an increased risk of dementia. The objective of this narrative review is to describe and evaluate studies of anticholinergic medication use and dementia and provide practical suggestions for avoiding use of these medications in older adults. A comprehensive review of the literature, citations from recent reviews and the author's personal files was conducted. Four studies were found that evaluated anticholinergic use and dementia as the primary outcome. Three studies focused on overall anticholinergic medication use and reported a statistically significantly increased risk of Alzheimer's disease or dementia. In one study, dementia risk was primarily found with higher cumulative doses; people using anticholinergic medications at the minimum effective dose recommended for older adults for at least 3 years were at highest risk. In contrast, a study conducted in nursing-home residents with depression did not find that paroxetine [a highly anticholinergic selective serotonin reuptake inhibitor antidepressant, (SSRI)] increased risk for dementia compared with other SSRIs (without anticholinergic activity). Further study is needed to understand the mechanism by which anticholinergic medications may increase risk. In conclusion, there is evidence from three observational studies suggesting that anticholinergic medications may increase dementia risk. Given this potential risk and the myriad of other well-known adverse effects (i.e. constipation, blurred vision, urinary retention, and delirium) associated with anticholinergic medications, it is prudent for prescribers and older adults to minimize use of these medications and consider alternatives when possible.

Non-psychotropic medication and risk of suicide or attempted suicide: a systematic review

OBJECTIVES

To establish which non-psychotropic medications have been assessed in relation to risk of suicide or attempted suicide in observational studies, document reported associations and consider study strengths and limitations.

DESIGN

Systematic review.

METHODS

Four databases (Embase, Medline, PsycINFO and International Pharmaceutical Abstracts) were searched from 1990 to June 2014, and reference lists of included articles were hand-searched. Case-control, cohort and case only studies which reported suicide or attempted suicide in association with any non-psychotropic medication were included.

OUTCOME MEASURES

The outcomes eligible for inclusion were suicide and attempted suicide, as defined by the authors of the included study.

RESULTS

Of 11,792 retrieved articles, 19 were eligible for inclusion. Five studies considered cardiovascular medication and antiepileptics; two considered leukotriene receptor antagonists, isotretinoin and corticosteroids; one assessed antibiotics and another assessed varenicline. An additional study compared multiple medications prescribed to suicide cases versus controls. There was marked heterogeneity in study design, outcome and exposure classification, and control for confounding factors; particularly comorbid mental and physical illness. No increased risk was associated with cardiovascular medications, but associations with other medications remained inconclusive and meta-analysis was inappropriate due to study heterogeneity.

CONCLUSIONS

Whether non-psychotropic medications are associated with increased risk of suicide or attempted suicide remains largely unknown. Robust identification of suicide outcomes and control of comorbidities could improve quantification of risk associated with non-psychotropic medication, beyond that conferred by underlying physical and mental illnesses.

Quality assessment of observational studies in a drug-safety systematic review, comparison of two tools: the Newcastle-Ottawa Scale and the RTI item bank

BACKGROUND

The study objective was to compare the Newcastle-Ottawa Scale (NOS) and the RTI item bank (RTI-IB) and estimate interrater agreement using the RTI-IB within a systematic review on the cardiovascular safety of glucose-lowering drugs.

METHODS

We tailored both tools and added four questions to the RTI-IB. Two reviewers assessed the quality of the 44 included studies with both tools, (independently for the RTI-IB) and agreed on which responses conveyed low, unclear, or high risk of bias. For each question in the RTI-IB (n=31), the observed interrater agreement was calculated as the percentage of studies given the same bias assessment by both reviewers; chance-adjusted interrater agreement was estimated with the first-order agreement coefficient (AC1) statistic.

RESULTS

The NOS required less tailoring and was easier to use than the RTI-IB, but the RTI-IB produced a more thorough assessment. The RTI-IB includes most of the domains measured in the NOS. Median observed interrater agreement for the RTI-IB was 75% (25th percentile [p25] =61%; p75 =89%); median AC1 statistic was 0.64 (p25 =0.51; p75 =0.86).

CONCLUSION

The RTI-IB facilitates a more complete quality assessment than the NOS but is more burdensome. The observed agreement and AC1 statistic in this study were higher than those reported by the RTI-IB's developers.