Multicenter randomized comparative trial of Micromedex, Micromedex with Watson, or Google to answer drug information questions

A comparison of drug information question responses by a drug information center and by ChatGPT

PURPOSE

A study was conducted to assess the accuracy and ability of Chat Generative Pre-trained Transformer (ChatGPT) to systematically respond to drug information inquiries relative to responses of a drug information center (DIC).

METHODS

Ten drug information questions answered by the DIC in 2022 or 2023 were selected for analysis. Three pharmacists created new ChatGPT accounts and submitted each question to ChatGPT at the same time. Each question was submitted twice to identify consistency in responses. Two days later, the same process was conducted by a fourth pharmacist. Phase 1 of data analysis consisted of a drug information pharmacist assessing all 84 ChatGPT responses for accuracy relative to the DIC responses. In phase 2, 10 ChatGPT responses were selected to be assessed by 3 blinded reviewers. Reviewers utilized an 8-question predetermined rubric to evaluate the ChatGPT and DIC responses.

RESULTS

When comparing the ChatGPT responses (n = 84) to the DIC responses, ChatGPT had an overall accuracy rate of 50%. Accuracy across the different question types varied. In regards to the overall blinded score, ChatGPT responses scored higher than the responses by the DIC according to the rubric (overall scores of 67.5% and 55.0%, respectively). The DIC responses scored higher in the categories of references mentioned and references identified.

CONCLUSION

Responses generated by ChatGPT have been found to be better than those created by a DIC in clarity and readability; however, the accuracy of ChatGPT responses was lacking. ChatGPT responses to drug information questions would need to be carefully reviewed for accuracy and completeness.

Analysis of the Integrated Management Model of Medical Care and Medication in Intravenous Treatment for Critically Ill Patients

Objective

To explore the effect of the Integrated Management Model of Doctor-Nurse-Pharmacist Collaboration on the Safety of Intravenous Therapy in Critically Ill Patients.

Methods

1587 patients who were hospitalized in the Intensive Care Department of the Fourth Hospital of Hebei Medical University in China from January 2021 to December 2022 were selected. 768 patients before the implementation of the integrated medical, nursing, and drug management model were selected as the control group, and 819 patients who implemented the integrated medical, nursing, and drug management model were selected as the observation group.

Results

Compared with the control group, the incidence of drug compatibility contraindications in the observation group decreased from 3.5% to 1.5% (χ2=6.957 P=0.008), the central venous catheter (CVC) blockage rate decreased from 2.5% to 1.0% (χ2=5.249 P=0.022), the daily incidence of catheter related bloodstream infections decreased from (1.84 ± 2.17) to (0.91 ± 1.19)(t=6.988 P=0.015), and the incidence of peripheral venous treatment related complications decreased from 10.3% to 2.9% (χ2=16.663 P=0.000). Among them, the incidence of phlebitis decreased from 5% to 1.6% (χ2=4.817 P=0.028). The incidence of drug exudation decreased from 3.4% to 0.8% (χ2=0.031 P=0.019). The incidence of extravasation has decreased from 2.5% to 0.4% (χ2=0.044 P=0.027). The differences were statistically significant (P<0.05).

Conclusion

The Integrated Management Model of Doctor-Nurse-Pharmacist Collaboration significantly reduced the incidence of catheter-related bloodstream infections (CRBSI), drug incompatibility, and other intravenous therapy-related complications, thereby enhancing the safety of intravenous therapy in critically ill patients.

Development of a novel drug information provision system for Kampo medicine using natural language processing technology

BACKGROUND

Kampo medicine is widely used in Japan; however, most physicians and pharmacists have insufficient knowledge and experience in it. Although a chatbot-style system using machine learning and natural language processing has been used in some clinical settings and proven useful, the system developed specifically for the Japanese language using this method has not been validated by research. The purpose of this study is to develop a novel drug information provision system for Kampo medicines using a natural language classifier® (NLC®) based on IBM Watson.

METHODS

The target Kampo formulas were 33 formulas listed in the 17th revision of the Japanese Pharmacopoeia. The information included in the system comes from the package inserts of Kampo medicines, Manuals for Management of Individual Serious Adverse Drug Reactions, and data on off-label usage. The system developed in this study classifies questions about the drug information of Kampo formulas input by natural language into preset questions and outputs preset answers for the questions. The system uses morphological analysis, synonym conversion by thesaurus, and NLC®. We fine-tuned the information registered into NLC® and increased the thesaurus. To validate the system, 900 validation questions were provided by six pharmacists who were classified into high or low levels of knowledge and experience of Kampo medicines and three pharmacy students.

RESULTS

The precision, recall, and F-measure of the system performance were 0.986, 0.915, and 0.949, respectively. The results were stable even with differences in the amount of expertise of the question authors.

CONCLUSIONS

We developed a system using natural language classification that can give appropriate answers to most of the validation questions.

Googling for a veterinary diagnosis: A replication study using Google as a diagnostic aid

BACKGROUND

The purpose of this study was to replicate in the veterinary context a BMJ study using Google to assist in diagnosis of complex cases.

HYPOTHESIS/OBJECTIVES

To assess percentage of diagnoses identified using Google as a diagnostic aid in veterinary medicine.

ANIMALS

None; 13 cases in cats and 17 in dogs published in JAVMA.

METHODS

Cross-sectional survey of Google results from searches using keywords generated independently by a generalist and a specialist veterinarian who reviewed the published case history and diagnostic components while blind to the diagnosis. They offered diagnoses and generated up to 5 search strategies for each case. The top 30 Google results for each search were reviewed by the generalist to inform a final Google-aided diagnosis. Both veterinarians' initial diagnoses and the Google-aided diagnoses were compared with the published diagnoses.

RESULTS

Google searching led to 52 diagnoses out of 60 possible. Twenty-two (42%, 95% confidence interval [95% CI] 29%-55%) Google-aided diagnoses matched the JAVMA diagnosis. This accuracy rate does not differ significantly from 58% (n = 15/26, 95% CI 38%-77%) identified in the BMJ study. Google-aided results were not statistically different from those achieved unaided by each veterinarian (33%, 95% CI 16%-50%).

CONCLUSIONS AND CLINICAL IMPORTANCE

Published information found searching Google using keywords related to complicated or unusual cases could assist veterinarians to reinforce their initial diagnosis or consider other differential diagnoses. Search strategies using words representing either signs or the preliminary diagnoses can yield results useful to confirming a correct diagnosis.