Evaluation of inter-rater agreement between three causality assessment methods used in pharmacovigilance

Off-the-Shelf Large Language Models for Causality Assessment of Individual Case Safety Reports: A Proof-of-Concept with COVID-19 Vaccines

BACKGROUND

This study evaluated the feasibility of ChatGPT and Gemini, two off-the-shelf large language models (LLMs), to automate causality assessments, focusing on Adverse Events Following Immunizations (AEFIs) of myocarditis and pericarditis related to COVID-19 vaccines.

METHODS

We assessed 150 COVID-19-related cases of myocarditis and pericarditis reported to the Vaccine Adverse Event Reporting System (VAERS) in the United States of America (USA). Both LLMs and human experts conducted the World Health Organization (WHO) algorithm for vaccine causality assessments, and inter-rater agreement was measured using percentage agreement. Adherence to the WHO algorithm was evaluated by comparing LLM responses to the expected sequence of the algorithm. Statistical analyses, including descriptive statistics and Random Forest modeling, explored case complexity (e.g., string length measurements) and factors affecting LLM performance and adherence.

RESULTS

ChatGPT showed higher adherence to the WHO algorithm (34%) compared to Gemini (7%) and had moderate agreement (71%) with human experts, whereas Gemini had fair agreement (53%). Both LLMs often failed to recognize listed AEFIs, with ChatGPT and Gemini incorrectly identifying 6.7% and 13.3% of AEFIs, respectively. ChatGPT showed inconsistencies in 8.0% of cases and Gemini in 46.7%. For ChatGPT, adherence to the algorithm was associated with lower string complexity in prompt sections. The random forest analysis achieved an accuracy of 55% (95% confidence interval: 35.7-73.5) for predicting adherence to the WHO algorithm for ChatGPT.

CONCLUSION

Notable limitations of ChatGPT and Gemini have been identified in their use for aiding causality assessments in vaccine safety. ChatGPT performed better, with higher adherence and agreement with human experts. In the investigated scenario, both models are better suited as complementary tools to human expertise.

Development of Real-Time Surveillance for Serious Adverse Events in a Pragmatic Clinical Trial Using National Registers in Finland

Purpose

We developed a hybrid safety surveillance approach for a large, pragmatic clinical trial of a high-dose quadrivalent influenza vaccine (QIV-HD), using both active and passive data collection methods. Here, we present the methods and results for the passive register-based surveillance of serious adverse events (SAEs), which replaced conventional SAE reporting during the trial.

Patients and Methods

The trial recruited over 33,000 older adults of whom 50% received the QIV-HD while the rest received a standard-dose vaccine (QIV-SD) as a control vaccine. We collected diagnoses related to all acute hospitalizations during the six months following vaccination from national registers. During the blinded phase of the trial, we utilized a cohort study design and compared the incidences of 1811 ICD10 diagnosis groups (SAE categories) between the trial population and older adults vaccinated with the QIV-SD outside the trial, either during the study or the previous influenza season. Based on a real-time probabilistic comparison, we flagged SAE categories with higher incidence in the trial population and then evaluated possible causal associations between each flagged category and the trial intervention.

Results

Our novel approach to safety surveillance provided information, which we could evaluate in real-time during the trial. The trial participants experienced 1217 hospitalizations related to any SAE categories, contributed by 941 patients. We flagged 10 SAE categories for further analysis during the study but based on further data review, none presented strong evidence of causality with vaccination.

Conclusion

Safety signals can be detected and evaluated in real-time during a pragmatic vaccine trial with register-based follow-up, utilizing passive data collection and population level comparison. Compared to conventional methods of safety follow-up, this method is likely to be more comprehensive, objective and resource effective.

A scoping review of the clinical utility of adverse drug reaction causality analysis tools for use in the hospital setting

INTRODUCTION

Identification and monitoring of adverse drug reactions (ADRs) and interventions to reduce ADRs are essential for patient safety in hospitals. Causality analysis (CA) is an approach that helps to determine a causal link between medication and patient harm (i.e. an ADR). While numerous CA tools exist, there is no gold standard.

AREAS COVERED

Five online databases were searched to identify studies that evaluated the potential clinical utility of CA tools for ADRs. CA tools were mapped against the Bradford Hill (BH) criteria and included if they adhered to the first seven criteria proposed by BH. Upon the database search, 550 studies were identified, with 41 studies being selected that looked at tools mapped to BH. Thirty-four different CA tools were identified in the included studies.

EXPERT OPINION

Naranjo and WHO-UMC were the most reported CA tools for studies examining inter-rater and intra-rater reliability. Naranjo commonly received a 'fair' agreement level while WHO-UMC received a 'substantial' agreement level between raters. Along with kappa statistics, time using the CA tool was also analyzed, with WHO-UMC being the most time-efficient. There does not appear to be one CA tool that can be applied universally to pharmacovigilance efforts in hospital in-patient settings.

Causality Assessment Between Drugs and Fatal Cerebral Haemorrhage Using Electronic Medical Records: Comparative Evaluation of Disease-Specific and Conventional Methods

INTRODUCTION

A new algorithm for causality assessment of drugs and fatal cerebral haemorrhage (ACAD-FCH) was published in 2021. However, its use in clinical practice has not been verified.

OBJECTIVES

This study aimed to explore the practical value of the ACAD-FCH when applying information available in clinical practice.

METHODS

The medical records of patients who died at the University of Tokyo Hospital in 2020 were reviewed, and cases with intracranial haemorrhage were selected. Two evaluators independently assessed these cases using three methods (the ACAD-FCH, Naranjo algorithm, and WHO-UMC scale). The number of 'Yes', 'No', and 'No information/Do not know' responses to each question by both evaluators were summed and compared. Inter-rater reliability was evaluated for each method using agreement rates and kappa coefficients with 95% confidence intervals (CI).

RESULTS

Among 316 deaths, 24 cases with intracranial haemorrhage were evaluated. The proportion of ‛No information/Do not know' responses for each question was 35.6% (95% CI 31.4-40.6%) for the ACAD-FCH and 66.9% (95% CI 62.5-71.1%) for the Naranjo algorithm. The respective agreement rates and kappa coefficients were 0.917 (0.798-1.00) and 0.867 (0.675-1.00) for the ACAD-FCH, 0.708 (0.512-0.904) and 0.139 (-0.236 to 0.513) for the Naranjo algorithm, and 0.50 (0.284-0.716) and 0.326 (0.110-0.541) for the WHO-UMC scale, respectively.

CONCLUSION

Our findings suggest the utility of the ACAD-FCH when assessing death cases with intracranial haemorrhage. However, larger studies including intra-rater assessments are warranted for further validation of this algorithm.

Adverse Drug Reactions at Nonelective Hospital Admission in Children and Adolescents: Comparison of 4 Causality Assessment Methods

OBJECTIVES

This study aimed to compare assessment methods to determine adverse drug reactions (ADRs) at nonelective hospital admission in pediatric patients, to investigate the interrater reliability of assessment methods in pediatric care, and to analyze symptoms related to ADRs and (suicidal) drug intoxications.

METHODS

For 1 year, the medical records of nonelective patients admitted to a university pediatric department were evaluated for potential ADRs using 4 assessments methods by 1 experienced rater. Krippendorff α was calculated from a sample of 14 patients evaluated by 4 experienced raters to determine interrater reliability.

RESULTS

In 1831 nonelective hospital admissions, 63.4% (1161 of 1831) of patients had received at least one drug before admission. We found a potential causal relationship between drugs and symptoms documented at admission and thus potential ADRs according to Naranjo in 23.3% (271 of 1161) of those patients, World Health Organization - Uppsala Monitoring Centre (WHO-UMC) in 22.5% (261 of 1161), Koh in 21.7% (252 of 1161), and Begaud in 16.5% (192 of 1161). The probability rating of the potential causal relationships varied considerably between the methods (Naranjo-Begaud, P < 0.01; Naranjo-Koh, P < 0.001; Koh-Begaud, P < 0.01; Begaud-WHO-UMC, P < 0.01). Acceptable interrater reliability (α ≥ 0.667) was only obtained for WHO-UMC (α = 0.7092). The most frequently identified definite ADR was sedation in 1.5% of all nonelective patients with medication before hospital admission. In 1.2% (22 of 1831) of all nonelective admissions, we found drug intoxications with suicidal intent.

CONCLUSIONS

The assessment methods showed a high variability in the determination of a potential causal relationship between drug and documented symptom, in the classification of the probability of ADRs, and suboptimal interrater reliability. Thus, their feasibility in pediatric patients is limited.

Leveraging Machine Learning to Facilitate Individual Case Causality Assessment of Adverse Drug Reactions

INTRODUCTION

Causality assessment of individual case safety reports (ICSRs) is an important step in pharmacovigilance case-level review and aims to establish a position on whether a patient's exposure to a drug is causally related to the patient experiencing an untoward adverse event. There are many different approaches for case causality adjudication, including the use of expert opinions and algorithmic frameworks; however, a great deal of variability exists between assessment methods, products, therapeutic classes, individual physicians, change of process and conventions over time, and other factors.

OBJECTIVE

The objective of this study was to develop a machine learning-based model that can predict the likelihood of a causal association of an observed drug-reaction combination in an ICSR.

METHODS

In this study, we used a set of annotated solicited ICSRs (50K cases) from a company post-marketing database. These data were enriched with novel supplementary features from external and internal data sources that aim to capture facets such as temporal plausibility, scientific validity, and confoundedness that have been shown to contribute to causality adjudication. Using these features, we constructed a Bayesian network (BN) model to predict drug-event pair causality assessment. BN topology was driven by an internally developed ICSR causality decision support tool. Performance of the model was evaluated through examination of sensitivity, positive predictive value (PPV), and the area under the receiver operating characteristic curve (AUC) on an independent set of data from a temporally adjacent interval (20K cases). No external validation was performed because of a lack of publicly available ICSRs with causality assessments for drug-event pairs.

RESULTS

The model demonstrated high performance in predicting the causality assessment of drug-event pairs compared with clinical judgment using global introspection (AUC 0.924; 95% confidence interval [CI] 0.922-0.927). The sensitivity of the model was 0.900 (95% CI 0.896-0.904), and the PPV of the model was 0.778 (95% CI 0.773-0.783).

CONCLUSION

These results show that robust probabilistic modeling of ICSR causality is feasible, and the approach used in the development of the model can serve as a framework for such causality assessments, leading to improvements in safety decision making.

Adverse drug reactions with oral anticoagulants: data from sicilian spontaneous reporting system database

OBJECTIVE

Direct oral anticoagulants (DOACs) were developed to avoid the limitations of vitamin K antagonists (VKAs). DOACs are associated with a greater incidence of gastrointestinal bleeding and a smaller number of intracranial haemorrhages than VKAs. Therefore, it is important to deepen our knowledge of their safety profiles. The aim of this study was thus to analyse adverse drug reaction (ADR) reports on DOACs and VKAs using the Sicilian Spontaneous Reporting System (SRS) database.

METHODS

All ADR reports with DOACs and VKAs as suspected drugs that were entered into the Sicilian SRS database during the period 2001-2019 were selected. In detail, all reports with the following single active substances were included: dabigatran etexilate, rivaroxaban, apixaban and edoxaban; acenocoumarol and warfarin were included as a comparator group. Descriptive statistical methodology was used to evaluate characteristics of the reported cases with a case-by-case assessment.

RESULTS AND DISCUSSION

Out of 521 reports related to anticoagulants, 444 (85.2%) and 77 (14.8%) involved DOACs and VKAs, respectively. DOAC-related reports were mainly of gastrointestinal disorders. In contrast, VKAs were mostly associated with blood and lymphatic system disorders, injury, investigations and vascular disorders. Many more cases of ADRs in the form of gastrointestinal disorders concerned dabigatran etexilate (n = 179, 73.7%) than the other DOACs, while ADRs in the form of blood disorders were mainly associated with acenocoumarol (n = 27, 57.4%). The most commonly reported Preferred Terms for DOACs were dyspepsia (n = 89, 17.1%), upper abdominal pain (n = 41, 9.2%) and pruritus (n = 26, 5.8%), whereas for VKAs, they were anaemia (n = 21, 27.3%) and hypocoagulable state (n = 18, 3.5%). Potentially interacting concomitant medications particularly included antithrombotic agents (n = 19, 4.3%) for DOACs and proton-pump inhibitors (PPIs) (n = 37, 48.1%) and antithrombotic agents (n = 13, 16.9%) for VKAs.

CONCLUSION

The ADRs most commonly associated with DOACs, especially dabigatran, were gastrointestinal disorders, particularly gastrointestinal bleeding. Our study also highlights the potential role of drug-drug interactions in the ADRs. The cases of gastrointestinal bleeding highlight the need for careful prescribing of DOACs and use of potentially interacting concomitant drugs.

The use of metformin as an add-on therapy to insulin in the treatment of poorly controlled type 1 diabetes mellitus in adolescents

The aim of this integrative literature review was to evaluate the efficacy and safety of Metformin as an add-on therapy to insulin in poorly controlled overweight adolescents with type 1 diabetes mellitus. The research problem centered on providing optimum disease management during the most critical growth period and reducing the potential for cardiovascular-related morbidity and mortality in the future. The findings suggested that Metformin, in conjunction with insulin therapy, helped patients to achieve better metabolic control. The quality of metabolic control varied between studies according to differences in study design, exclusion and inclusion criteria, and methods. Adjunctive Metformin therapy has a positive effect on diabetes management, treatment, and prevention of cardiovascular-related complications with a minimal risk of side effects. Suggestions for further exploration of the research results and clinical implications are included in the review.

[Recommendations for Drug Treatment in Patients with Multimorbidity]

Applying guidelines in patients with multimorbidity can result in dangerous or contraindicated drug-drug and drug-disease-interactions. A representative working group of medical scientific associations identifies such therapeutic conflicts and develops management strategies that will be published as a formally consensus based (S2K) guideline. Rational, aims and methods used are described, as well as evaluation and updating of recommendations.

Drug induced pancreatitis: A systematic review of case reports to determine potential drug associations

OBJECTIVE

A current assessment of case reports of possible drug-induced pancreatitis is needed. We systematically reviewed the case report literature to identify drugs with potential associations with acute pancreatitis and the burden of evidence supporting these associations.

METHODS

A protocol was developed a priori (PROSPERO CRD42017060473). We searched MEDLINE, Embase, the Cochrane Library, and additional sources to identify cases of drug-induced pancreatitis that met accepted diagnostic criteria of acute pancreatitis. Cases caused by multiple drugs or combination therapy were excluded. Established systematic review methods were used for screening and data extraction. A classification system for associated drugs was developed a priori based upon the number of cases, re-challenge, exclusion of non-drug causes of acute pancreatitis, and consistency of latency.

RESULTS

Seven-hundred and thirteen cases of potential drug-induced pancreatitis were identified, implicating 213 unique drugs. The evidence base was poor: exclusion of non-drug causes of acute pancreatitis was incomplete or poorly reported in all cases, 47% had at least one underlying condition predisposing to acute pancreatitis, and causality assessment was not conducted in 81%. Forty-five drugs (21%) were classified as having the highest level of evidence regarding their association with acute pancreatitis; causality was deemed to be probable or definite for 19 of these drugs (42%). Fifty-seven drugs (27%) had the lowest level of evidence regarding an association with acute pancreatitis, being implicated in single case reports, without exclusion of other causes of acute pancreatitis.

DISCUSSION

Much of the case report evidence upon which drug-induced pancreatitis associations are based is tenuous. A greater emphasis on exclusion of all non-drug causes of acute pancreatitis and on quality reporting would improve the evidence base. It should be recognized that reviews of case reports, are valuable scoping tools but have limited strength to establish drug-induced pancreatitis associations.

REGISTRATION

CRD42017060473.

Adverse Drug Events in Older Adults: Review of Adjudication Methods in Deprescribing Studies

OBJECTIVES

Polypharmacy is common in older adults and associated with adverse drug events (ADEs). Several methods have been described in studies to help correlate ADE causation. We performed a narrative review to identify methods for ADE adjudication. We compared their strengths and limitations to assess their applicability to deprescribing studies (of which clinical trials are a subset) and to encourage the use of a standardized method in future studies.

DESIGN

We performed a review of original articles (1946-2019) using the Medline (Ovid) and Cochrane databases. We also conducted a manual reference search of review articles. Abstracts were screened for relevance.

MEASUREMENTS

Adjudication methods were compared for advantages and limitations including validity, ease of use, and applicability to clinical trials with deprescribing as the primary intervention.

RESULTS

The search yielded 1881 articles of which 175 articles were included for full-text review. Following in-depth review, 135 were excluded: 79 had no ADE outcome data, 35 were not specific to older adults, 9 were not relevant, 6 were review articles, 5 contained duplicate data, and 1 was not written in French or English. Forty articles remained for analysis, from which we identified 10 unique ADE adjudication methods. No method was developed originally for use in a deprescribing setting.

CONCLUSION

A standard method to identify ADEs is important to capture the outcome reliably in deprescribing studies. All methods we identified had limitations in terms of capturing adverse events from the withdrawal of medications. Future work should focus on refining adjudication methods for capturing ADEs related not only to medication continuation and new drug starts but also to deprescribing and drug discontinuation. J Am Geriatr Soc 68:1594-1602, 2020.

Incidence, causes, and consequences of preventable adverse drug reactions occurring in inpatients: A systematic review of systematic reviews

BACKGROUND

Preventable adverse drug reactions (PADRs) in inpatients are associated with harm, including increased length of stay and potential loss of life, and result in elevated costs of care. We conducted an overview of reviews (i.e., a systematic review of systematic reviews) to determine the incidence of PADRs experienced by inpatients. Secondary review objectives were related to assessment of the effects of patient age, setting, and clinical specialty on PADR incidence.

METHODS

The protocol was registered in PROSPERO (CRD42016043220). We performed a search of Medline, Embase, and the Cochrane Library, limiting languages of publication to English and French. We included published systematic reviews that reported quantitative data on the incidence of PADRs in patients receiving acute or ambulatory care in a hospital setting. The full texts of all primary studies for which PADR data were reported in the included reviews were obtained and data relevant to review objectives were extracted. Quality of the included reviews was assessed using the AMSTAR-2 tool. Both narrative summaries of findings and meta-analyses of primary study data were undertaken.

RESULTS

Thirteen systematic reviews encompassing 37 unique primary studies were included. Across primary studies, the PADR incidence was highly varied, ranging from 0.006 to 13.3 PADRs per 100 patients, with a pooled incidence estimate of 0.59 PADRs per 100 patients. Substantial heterogeneity was present across both reviews and primary studies with respect to review/study objectives, patient age, hospital setting, medical discipline, definitions and assessment tools used, event detection methods, endpoints of interest, and units of measure. Thirteen primary studies used prospective event detection methods and had a pooled PADR incidence of 3.13 (2.87-3.38) PADRs per 100 patients; however, extreme statistical heterogeneity (I2 = 97%) indicated this finding should be considered with caution. Subgroup meta-analyses demonstrated that PADR incidence varied significantly with event detection method (prospective > retrospective > voluntary reporting methods), hospital setting (ICU > wards), and medical discipline (medical > surgical). High statistical heterogeneity (I2 > 80%) was present across all analyses, indicating results should be interpreted with caution. Effects of patient age could not be assessed due to poor reporting of age groups used in primary studies.

DISCUSSION

The method of event detection appeared to significantly influence PADR incidence, with prospective methods having the highest reported PADR rate. This finding is in agreement with the background literature. High methodological and statistical heterogeneity across primary studies evaluating adverse drug events reduces the validity of the overall PADR incidence derived from the meta-analyses of the pooled data. Data pooled from studies using only prospective methods of event detection should provide an overall estimate closest to the true PADR incidence; however, our estimate should be considered with caution due to the statistical heterogeneity found in this group of studies. Future studies should employ prospective methods of detection. This review demonstrates that the true overall incidence of PADRs is likely much greater than the overall pooled incidence estimate of 0.59 PADRs per 100 patients obtained when event detection method was not taken into consideration.

Comparison of different methods for causality assessment of adverse drug reactions

Background The causality assessment of adverse drug reactions (ADRs) remains a challenge, and none of the different available method of causality assessment used for assessing adverse reactions has been universally accepted as the gold standard. Objective To examine the agreement and correlation among three broad approaches for causality assessment of ADRs viz. World Health Organization-Uppsala Monitoring Centre (WHO-UMC) system, Naranjo algorithm, and updated Logistic method. Setting ADR monitoring centre (AMC) of a tertiary care teaching hospital in India. Method A total of 230 cases of ADR from April 2017 to August 2017 were retrospectively analyzed by each of these three methods. The agreement among the different methods was calculated by Cohen's kappa (κ), and Spearman's correlation was used to find the correlation among these methods. Main outcome measures Cohen's kappa value and Spearman's correlation coefficient for comparison among the different methods. Results The Cohen's κ used for analyzing the agreement between WHO-UMC system and Naranjo algorithm was 0.45, between WHO-UMC system and updated Logistic method was 0.405, and between Naranjo algorithm and updated Logistic method was 0.606. The Spearman's correlation coefficient was 0.793 for Naranjo algorithm vs. updated Logistic method, 0.735 for WHO-UMC system vs. Naranjo algorithm, and 0.696 for WHO-UMC system vs. updated Logistic method. Conclusion Causality assessment based on objective measurements (scores and probabilities) like updated Logistic method and Naranjo algorithm are less prone to subjective variations compared to the WHO-UMC system which is based on expert judgement.