Validation of the assessment of risk tool: patient prioritisation technology for clinical pharmacist interventions

A machine learning-based clinical predictive tool to identify patients at high risk of medication errors

A medication error is an inadvertent failure in the drug therapy process that can cause serious harm to patients by increasing morbidity and mortality and are associated with significant economic costs to the healthcare system. Medication reconciliation is the most cost-effective intervention and can result in a 66% reduction in medication errors. To improve patient safety, we developed a machine learning-based tool that prioritizes patients at risk of medication errors upon admission to the hospital to ensure that they undergo medication reconciliation by clinical pharmacists. The data were collected from the electronic health records of patients admitted to Reims University Hospital who underwent medication reconciliation between 2017 and 2023. The data from 7200 patients were used to train four machine learning-based models based on 52 variables in the development dataset. These models were used to prioritize admitted patients according to their likelihood of being exposed to a medication error. Our models, particularly the voting classifier model, demonstrated good performance (a recall of 0.75, precision of 0.65, F1 score of 0.70, AUROC of 0.74 and AUCPR of 0.75). In a retrospective evaluation simulating real-life use, the voting classifier model successfully identified 45% of the total patients selected who were found to have at least one unintended discrepancy compared to 21% when using the existing tool. The positive experimental results of this tool showed a superior improvement of 113% over the existing tool by targeting patients at risk of medication errors upon admission to Reims University Hospital.

Medication Profiles at Hospital Discharge Predict Poor Outcomes After Acute Ischemic Stroke

Objectives: To examine the relationship between medications prescribed during the first 6-months post-stroke and functional outcome. Materials and Methods: A retrospective analysis of ischemic stroke survivors enrolled in an observational stroke recovery study from June-2017 to July-2019 was performed. Survivors with favorable outcomes (modified rankin scale (mRS) score 0-2) were compared to those with unfavorable outcomes (mRS ≥3) 6-months after stroke on the following: discharge medication classes prescribed, achievement of recommended targets for blood pressure control, glycemic control, and LDL ≤70 mg/dL, medication changes, medication interactions, and medication list discrepancies. Results: Unfavorable 6-month outcomes occurred in 36/78 (46.2%) of survivors. Survivors with unfavorable outcomes were more likely to be prescribed a central nervous system-acting agent (97.2% vs 71.4%; P = .0022) and/or an anti-hyperglycemic agent (25.0% vs 9.5%; P = .009) at discharge. After adjustment of baseline covariates, total number of medications prescribed was associated with unfavorable 6-month outcomes (OR 1.13, 95% CI 1.0-1.28). Secondary stroke prevention measures were not achieved in a high proportion of survivors. Medication changes during 6-month follow up were common and survivors with unfavorable outcomes were more likely to have clinically significant drug-drug interactions. Discussion: At 6-months, survivors with unfavorable outcomes were found to be prescribed more medications, particularly central nervous system-acting and anti-hyperglycemic agents. There were also more drug-drug interactions in the medications prescribed compared to those with favorable outcomes. Together, these data suggest the need for enhanced screening of high-risk stroke survivors focused on close monitoring of polypharmacy, drug-drug interactions, and adverse events with pharmacotherapy.

Comparison of prioritisation algorithms for the selection of patients for medication reviews in the emergency department: a cross-sectional study

BACKGROUND

Risk prioritisation algorithms provide patients with a risk category that guides pharmacists to choose those needing medication reviews (MRs) the most. For this study the Medicine Risk Score (MERIS) and a modified Assessment of Risk Tool (ART) were used.

AIM

To examine how the selection of patients by the clinical pharmacists in an emergency department for MRs compared with the categorisation provided by MERIS and a modified version of ART (mART). Furthermore, examine the agreement between MERIS and mART.

METHOD

A cross-sectional study was conducted using data on all admitted patients during a two-month period. Data were entered into the prioritisation algorithms and independently ranked by the six pharmacists who were observed as they selected patients for MR. Risk scores and categorisations were compared between the algorithms and the pharmacists' ranking using t-test, Z-test, Chi square, Kruskal Wallis H-test, or Kappa statistics.

RESULTS

The study included 1133 patients. Significant differences were found between the pharmacists and the algorithms. The sensitivity and specificity of MERIS were 37.8% and 73.6%, for mART, 33.0% and 75.9%. Kappa was 0.58, showing moderate agreement. No significant differences were observed between the individual pharmacists' selection, but differences were significant between how pharmacists ranked the importance of the provided MRs.

CONCLUSION

Pharmacists disagreed with the risk categorisation provided by MERIS and mART. However, MERIS and mART had similar sensitivity, specificity, and moderate agreement. Further research should focus on how clinical algorithms affect the selection of patients and on the importance of the MRs carried out by pharmacists.

Development and validation of a pharmaceutical assessment screening tool to prioritise patient care in a tertiary care hospital

BACKGROUND

Clinical pharmacy plays an integral role in optimizing inpatient care. Nevertheless, prioritising patient care remains a critical challenge for pharmacists in a hectic medical ward. In Malaysia, clinical pharmacy practice has a paucity of standardized tools to prioritise patient care.

AIM

Our aim is to develop and validate a pharmaceutical assessment screening tool (PAST) to guide medical ward pharmacists in our local hospitals to effectively prioritise patient care.

METHOD

This study involved 2 major phases; (1) development of PAST through literature review and group discussion, (2) validation of PAST using a three-round Delphi survey. Twenty-four experts were invited by email to participate in the Delphi survey. In each round, experts were required to rate the relevance and completeness of PAST criteria and were given chance for open feedback. The 75% consensus benchmark was set and criteria with achieved consensus were retained in PAST. Experts' suggestions were considered and added into PAST for rating. After each round, experts were provided with anonymised feedback and results from the previous round.

RESULTS

Three Delphi rounds resulted in the final tool (rearranged as mnemonic 'STORIMAP'). STORIMAP consists of 8 main criteria with 29 subcomponents. Marks are allocated for each criteria in STORIMAP which can be combined to a total of 15 marks. Patient acuity level is determined based on the final score and clerking priority is assigned accordingly.

CONCLUSION

STORIMAP potentially serves as a useful tool to guide medical ward pharmacists to prioritise patients effectively, hence establishing acuity-based pharmaceutical care.

Systematic Review of Risk Factors Assessed in Predictive Scoring Tools for Drug-Related Problems in Inpatients

Drug-related problems (DRP, defined as adverse drug events/reactions and medication errors) are a common threat for patient safety. With the aim to aid improved allocation of specialist resources and to improve detection and prevention of DRP, numerous predictive scoring tools have been proposed. The external validation and evidence for the transferability of these tools still faces limitations. However, the proposed scoring tools include partly overlapping sets of similar factors, which may allow a new approach to estimate the external usability and validity of individual risk factors. Therefore, we conducted this systematic review and analysis. We identified 14 key studies that assessed 844 candidate risk factors for inclusion into predictive scoring tools. After consolidation to account for overlapping terminology and variable definitions, we assessed each risk factor in the number of studies it was assessed, and, if it was found to be a significant predictor of DRP, whether it was included in a final scoring tool. The latter included intake of ≥ 8 drugs, drugs of the Anatomical Therapeutic Chemical (ATC) class N, ≥1 comorbidity, an estimated glomerular filtration rate (eGFR) <30 mL/min and age ≥60 years. The methodological approach and the individual risk factors presented in this review may provide a new starting point for improved risk assessment.

Clinical usefulness of prediction tools to identify adult hospitalized patients at risk of drug-related problems: A systematic review of clinical prediction models and risk assessment tools

AIMS

This study aimed to review systematically all available prediction tools identifying adult hospitalized patients at risk of drug-related problems, and to synthesize the evidence on performance and clinical usefulness.

METHODS

PubMed, Scopus, Web of Science, Embase, and CINAHL databases were searched for relevant studies. Titles, abstracts and full-text studies were sequentially screened for inclusion by two independent reviewers. The Prediction Model Risk of Bias Assessment Tool (PROBAST) and the Revised Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) checklists were used to assess risk of bias and applicability of prediction tools. A narrative synthesis was performed.

RESULTS

A total of 21 studies were included, 14 of which described the development of new prediction tools (four risk assessment tools and ten clinical prediction models) and six studies were validation based and one an impact study. There were variations in tool development processes, outcome measures and included predictors. Overall, tool performance had limitations in reporting and consistency, with the discriminatory ability based on area under the curve receiver operating characteristics (AUROC) ranging from poor to good (0.62-0.81), sensitivity and specificity ranging from 57.0% to 89.9% and 30.2% to 88.0%, respectively. The Medicines Optimisation Assessment tool and Assessment of Risk tool were prediction tools with the lowest risk of bias and low concern for applicability. Studies reporting external validation and impact on patient outcomes were scarce.

CONCLUSION

Most prediction tools have limitations in development and validation processes, as well as scarce evidence of clinical usefulness. Future studies should attempt to either refine currently available tools or apply a rigorous process capturing evidence of acceptance, usefulness, performance and outcomes.

Development of measurable criteria to identify and prioritize patients for inclusion in comprehensive medication management programs within primary care settings

BACKGROUND: Pharmacists optimize medication use and ensure the safe and effective delivery of pharmacotherapy to patients using comprehensive medication management (CMM). Identifying and prioritizing individual patients who will most likely benefit from CMM can be challenging. Health systems have far more candidates for CMM than there are clinical pharmacists to provide this service. Furthermore, current evidence lacks widely accepted standards or automated mechanisms for identifying patients who would likely benefit from a pharmacist consultation. Existing tools to prioritize patients for pharmacist review often require manual chart review by a pharmacist or other clinicians or data collection by patient survey. OBJECTIVES: To (1) create new medication risk markers for identifying and prioritizing patients within a population and (2) identify patients who met these new markers, assess their clinical characteristics, and compare them with criteria that are widely used for medication therapy management (MTM). METHODS: Along with published literature, a panel of subject matter experts informed the development of 3 medication risk markers. To assess the prevalence of markers developed, we used Multum, a medication database, for medication-level characteristics, and for patient-level characteristics, we used QuintilesIMS, an administrative claims database derived from health plans across the United States, with data for 1,541,873 eligible individuals from 2014-2015. We compared the health care costs, utilization, and medication gap among patients identified through MTM criteria (both broad and narrow, as these are provided as ranges) and our new medication management score markers. RESULTS: We developed 3 claims-derivable markers: (1) instances when a patient filled a medication with high complexity that could affect adherence, (2) instances where a patient filled a medication defined as costly within a therapeutic category that could affect access, and (3) instances when a patient filled a medication defined as risky that could increase incidence of adverse drug events. In the QuintilesIMS database, individuals with 2 new medication risk markers plus at least 3 conditions and more than $3,017 in medication costs when compared with individuals meeting narrow MTM eligibility criteria (≥ 8 medications, ≥ 3 conditions, and > $3,017 medication costs) had increased costs ($36,000 vs $26,100 total; $24,800 vs 21,400 medical; $11,300 vs $4,800 pharmacy); acute care utilization (0.328 vs 0.256 inpatient admissions and 0.627 vs 0.579 emergency department visits); and 1 or more gaps in medication adherence(41.5% vs 34.7%). CONCLUSIONS: We identified novel markers of medication use risk that can be determined using insurance claims and can be useful to identify patients for CMM programs and prioritize patients who would benefit from clinical pharmacist intervention. These markers were associated with higher costs, acute care utilization, and gaps in medication use compared with the overall population and within certain subgroups. Providing CMM to these patients may improve health system performance in relevant quality measures. Evaluation of CMM services delivered by a pharmacist using these markers requires further investigation. DISCLOSURES: No outside funding supported this study. All authors are Johns Hopkins employees. The Johns Hopkins University receives royalties for nonacademic use of software based on the Johns Hopkins Adjusted Clinical Group (ACG) methodology. Chang, Kitchen, Weiner, and Kharrazi receive a portion of their salary support from this revenue. The authors have no conflicts of interests relevant to this study.

Prognostic prediction models and clinical tools based on consensus to support patient prioritization for clinical pharmacy services in hospitals: A scoping review

BACKGROUND

Identifying patients at high risk of adverse medication-related outcomes for targeted clinical pharmacy services is essential in hospital pharmacy. Models and predictive tools to prioritize patients are available to the clinical pharmacy services for hospital use.

OBJECTIVE

To describe and assess prognostic models and predictive tools used to identify inpatients at risk of adverse medication-related outcomes.

METHODS

We searched in Medline, Lilacs, Cochrane, CINAHL, Embase, Scopus and Web of Science, databases of theses and dissertations, and the references of the selected studies. The screening was carried out by two independent researchers. Cross-sectional studies, prospective or retrospective cohort studies, and case-control studies were eligible for inclusion. The studies addressed the development or validation of predictive models and clinical prioritization tools based on expert opinion to identify inpatients at risk of adverse medication-related outcomes.

RESULTS

25 studies were included, 13 of which were prognostic prediction models, seven were instrument development using the consensus method, and five were validation. The outcome events were drug-related problems (9), adverse drug reactions (8), adverse drug events (6), and medication errors (2). Most studies targeted adult patients (14), eight had older adult patients, one had obstetric patients, and others had pediatric patients. External validation was performed after the development study in three studies. The predictive model with a low risk of bias was the Medicines Optimisation Assessment Tool. Limited details on the method of expert involvement and the number of experts were identified in four studies.

CONCLUSION

The development of patient prioritization tools to optimize pharmacotherapy by clinical pharmacy services is a complex process. The predictive models and tools analyzed are limited in their development and validation process, hindering their effective use in prioritizing patients by the clinical pharmacy services. The development of additional prognostic prediction models for drug-related problems is a priority.

Development of a pharmacy 'patient prioritization tool' for use in a Tertiary Paediatric Hospital

WHAT IS KNOWN AND OBJECTIVE

Pharmacists play an integral role in paediatric patient care by ensuring the safe and optimal use of medications. There are increasing demands on pharmacists' time and challenges to meet them within allocated resources, and therefore, it is important to ensure that resources are used efficiently. Patient prioritization tools for clinical pharmacists have been proposed via many studies, but are generally adult-based and/or have not been validated to confirm their effectiveness. The aim of this study was to create, pilot and validate a patient prioritization tool to be used by pharmacists providing clinical pharmacy services to paediatric patients.

METHODS

A two-phase (retrospective and prospective) observational audit of pharmacists' interventions collected via notes made on their ward handover information sheets and patient case notes was conducted over a 2-year period in a tertiary paediatric hospital. A patient prioritization tool was created based on pharmacists' interventions in real time. This tool could be used at the start of the working day (without the need to review the patient or their case notes) to identify patients who would benefit most from a clinical pharmacist review. The tool was validated for effectiveness and selectivity.

RESULTS AND DISCUSSION

The tool was easy to use and effective in identifying that 43% of paediatric inpatients did not require a routine clinical pharmacist review. It had 98% specificity in identifying patients who require a pharmacist intervention. It could be easily used at the start of the day to select patients for pharmacist review.

WHAT IS NEW AND CONCLUSION

A new patient prioritization tool has been developed and validated for identifying paediatric inpatients requiring clinical pharmacist review.

Economic impact of clinical pharmaceutical activities in hospital wards: A systematic review

BACKGROUND

The positive impact of clinical pharmacy services (CPS) in improving clinical outcomes such as reduction of drug related problems is well demonstrated. Despite these results, the deployment of these activities is not systematically observed in the hospital setting.

OBJECTIVES

This systematic review first aimed to describe existing evidence regarding economic evaluation of ward-based CPS focusing on the entire treatment of a patient in a hospital setting. Secondly, the quality of economic evaluations of existing evidence was assessed.

METHODS

A comprehensive literature search was performed in PubMed/Medline, Science Direct and the NHS Economic Evaluation databases from January 2000 to March 2019. English or French language articles describing an economic evaluation of ward-based CPS on inpatients in hospital settings were included. Articles not describing a single study, dealing with a CPS not considering the entire medication regimen of the patient or presenting both inpatient and outpatient CPS were excluded. Selected articles were analyzed according to Drummond's check-list for assessing economic evaluations.

RESULTS

Forty-one studies were included. About one third were American publications. CPS implemented in ICU represented about half of the selected articles. Pharmacist-to-bed ratios varied according to countries and care unit type with the most favorable ratios in ICU and in American studies. Cost-avoidance was mostly used to express economic impact and ranged from €1579 to €3,089 328. Studies yielding the greater economic impact were conducted in the USA with implementation of full-time equivalents pharmacists or establishing of collaborative practice agreements. Only 6 articles dealt correctly with at least 7 of the 10 Drummond's checklist assessment criteria.

CONCLUSION

This review suggests that the existing evidence is not sufficient to conclude to a positive economic impact of CPS conducted according to clinical pharmacy guidelines. Funding resources, remuneration of clinical pharmacy activities and provision of standardized national clinical and economic databases appear to be essential evolutions to improve CPS development.

Screening Tools Used by Clinical Pharmacists to Identify Elderly Patients at Risk of Drug-Related Problems on Hospital Admission: A Systematic Review

None of the models met the four key stages required to create a quality risk prediction model. Further research is needed to either refine the tools developed to date or develop new ones that have good performance and have been externally validated before considering the potential impact and implementation of such tools.

Evaluation of drug-related problems and subsequent clinical pharmacists' interventions at a Swiss university hospital

WHAT IS KNOWN AND OBJECTIVE

The evaluation of clinical pharmacy services is essential for their further development and establishment. We analysed drug-related problems (DRPs) and subsequent clinical pharmacists' interventions (PIs) at a Swiss university hospital.

METHOD

We conducted a retrospective analysis of DRPs and subsequent PIs that were identified and implemented during interdisciplinary ward rounds in internal medicine at the University Hospital Basel, Switzerland, between 2015 and 2017. We estimated the potential clinical and economic impact of PIs using a validated evaluation tool (CLEO_de ).

RESULTS AND DISCUSSION

Based on medication reviews of 5441 patients, clinical pharmacists identified 5024 DRPs, of which 2892 DRPs (57.6%) were followed by a PI that was directly accepted and implemented by the physician in charge and included in the present analysis. The leading cause and type of PIs were inappropriate dose and dose adjustment, respectively. Overall, 97.8% of DRPs were followed by PIs with an expected clinical benefit for the patients (major: 11.1%; moderate: 27.6%; minor: 59.1%). The drugs most often involved in PIs of major clinical impact were antithrombotics, acid blockers and cardiovascular drugs. With regard to the economic impact, 40.7% of DRPs implied PIs resulting in an increase of immediate therapy costs, whereas 39.3% implied PIs resulting in a decrease of immediate therapy costs. The remaining PIs were cost-neutral.

WHAT IS NEW AND CONCLUSION

This study emphasizes that clinical pharmacists may help improve the effectiveness and safety of pharmacotherapy on acute care medical wards.

Development and performance evaluation of the Medicines Optimisation Assessment Tool (MOAT): a prognostic model to target hospital pharmacists' input to prevent medication-related problems

Background

Medicines optimisation is a key role for hospital pharmacists, but with ever-increasing demands on services, there is a need to increase efficiency while maintaining patient safety.

Objective

To develop a prediction tool, the Medicines Optimisation Assessment Tool (MOAT), to target patients most in need of pharmacists’ input in hospital.

Methods

Patients from adult medical wards at two UK hospitals were prospectively included into this cohort study. Data on medication-related problems (MRPs) were collected by pharmacists at the study sites as part of their routine daily clinical assessments. Data on potential risk factors, such as number of comorbidities and use of ‘high-risk’ medicines, were collected retrospectively. Multivariable logistic regression modelling was used to determine the relationship between risk factors and the study outcome: preventable MRPs that were at least moderate in severity. The model was internally validated and a simplified electronic scoring system developed.

Results

Among 1503 eligible admissions, 610 (40.6%) experienced the study outcome. Eighteen risk factors were preselected for MOAT development, with 11 variables retained in the final model. The MOAT demonstrated fair predictive performance (concordance index 0.66) and good calibration. Two clinically relevant decision thresholds (ie, the minimum predicted risk probabilities to justify pharmacists’ input) were selected, with sensitivities of 90% and 66% (specificity 30% and 61%); these equate to positive predictive values of 47% and 54%, respectively. Decision curve analysis suggests that the MOAT has potential value in clinical practice in guiding decision-making.

Conclusion

The MOAT has potential to predict those patients most at risk of moderate or severe preventable MRPs, experienced by 41% of admissions. External validation is now required to establish predictive accuracy in a new group of patients.

Patient prioritization for pharmaceutical care in hospital: A systematic review of assessment tools

BACKGROUND

Clinical pharmacy services improve patient safety, outcomes, and care quality; however, UK clinical pharmacy services face limited resources, insufficient capacity, and patients who present with increasingly complex medication regimes and morbidities. These indicate a need for the prioritization of pharmacy services. Several prioritization tools have been developed; however, there has been no comprehensive review of such tools to date.

OBJECTIVE

A systematic review was conducted to provide a structured overview and description of existing assessment tools with a focus on study quality, themes, tool validity, risk factors, and high-risk drug classes.

METHODS

Systematic searches for English-language publications (from 1990 to September 2017) were conducted in Embase, Medline, Scopus, International Pharmaceutical Abstracts, and Web of Science. Papers in the inpatient setting and in which the tool users were pharmacists or pharmacy technicians were included. Data on each study (e.g. aim and design) and the structure of tools (e.g. risk factors) from each included study were extracted by 2 independent reviewers. A descriptive analysis was conducted to summarize these tools along with a thematic analysis of study findings. The quality of each paper was assessed using the Hawker method.

RESULTS

Nineteen studies involving 17 risk assessment tools were included. Most tools were developed in Europe (76.5%) and published in the last 5 years (82%). Most tools (88%) were designed to identify patients at greatest risk of adverse drug reactions, adverse drug events, or medication errors and to guide appropriate pharmaceutical care. Ten out of 17 tools (59%) were validated. None showed a measurable impact on prescription errors or adverse drug events. Keys themes identified from the studies were the positive impact of risk assessment tools on both patient care and provision of pharmacy services as well as the limitations of risk assessment tools.

CONCLUSIONS

Current assessment tools are heterogeneous in their content, targeting diverse patient groups and clinical settings making generalization difficult. However, an underlying theme of all studies was that tools appear to achieve their aim in directing pharmaceutical care to where it is needed most which might provide reassurance and incentive for greater adoption and development of tools across clinical pharmacy services. However, further research is required to measure objectively the impact of tools on patient outcomes and on workforce efficiency so that comparisons can be made between tools.