Accuracy and Reliability of Chatbot Responses to Physician Questions

Importance

Natural language processing tools, such as ChatGPT (generative pretrained transformer, hereafter referred to as chatbot), have the potential to radically enhance the accessibility of medical information for health professionals and patients. Assessing the safety and efficacy of these tools in answering physician-generated questions is critical to determining their suitability in clinical settings, facilitating complex decision-making, and optimizing health care efficiency.

Objective

To assess the accuracy and comprehensiveness of chatbot-generated responses to physician-developed medical queries, highlighting the reliability and limitations of artificial intelligence-generated medical information.

Design, Setting, and Participants

Thirty-three physicians across 17 specialties generated 284 medical questions that they subjectively classified as easy, medium, or hard with either binary (yes or no) or descriptive answers. The physicians then graded the chatbot-generated answers to these questions for accuracy (6-point Likert scale with 1 being completely incorrect and 6 being completely correct) and completeness (3-point Likert scale, with 1 being incomplete and 3 being complete plus additional context). Scores were summarized with descriptive statistics and compared using the Mann-Whitney U test or the Kruskal-Wallis test. The study (including data analysis) was conducted from January to May 2023.

Main Outcomes and Measures

Accuracy, completeness, and consistency over time and between 2 different versions (GPT-3.5 and GPT-4) of chatbot-generated medical responses.

Results

Across all questions (n = 284) generated by 33 physicians (31 faculty members and 2 recent graduates from residency or fellowship programs) across 17 specialties, the median accuracy score was 5.5 (IQR, 4.0-6.0) (between almost completely and complete correct) with a mean (SD) score of 4.8 (1.6) (between mostly and almost completely correct). The median completeness score was 3.0 (IQR, 2.0-3.0) (complete and comprehensive) with a mean (SD) score of 2.5 (0.7). For questions rated easy, medium, and hard, the median accuracy scores were 6.0 (IQR, 5.0-6.0), 5.5 (IQR, 5.0-6.0), and 5.0 (IQR, 4.0-6.0), respectively (mean [SD] scores were 5.0 [1.5], 4.7 [1.7], and 4.6 [1.6], respectively; P = .05). Accuracy scores for binary and descriptive questions were similar (median score, 6.0 [IQR, 4.0-6.0] vs 5.0 [IQR, 3.4-6.0]; mean [SD] score, 4.9 [1.6] vs 4.7 [1.6]; P = .07). Of 36 questions with scores of 1.0 to 2.0, 34 were requeried or regraded 8 to 17 days later with substantial improvement (median score 2.0 [IQR, 1.0-3.0] vs 4.0 [IQR, 2.0-5.3]; P < .01). A subset of questions, regardless of initial scores (version 3.5), were regenerated and rescored using version 4 with improvement (mean accuracy [SD] score, 5.2 [1.5] vs 5.7 [0.8]; median score, 6.0 [IQR, 5.0-6.0] for original and 6.0 [IQR, 6.0-6.0] for rescored; P = .002).

Conclusions and Relevance

In this cross-sectional study, chatbot generated largely accurate information to diverse medical queries as judged by academic physician specialists with improvement over time, although it had important limitations. Further research and model development are needed to correct inaccuracies and for validation.

Mycobacterium tuberculosis Gene Expression Associated With Fluoroquinolone Resistance and Efflux Pump Inhibition

BACKGROUND

We evaluated the relationship between response to efflux pump inhibition in fluoroquinolone-resistant Mycobacterium tuberculosis (Mtb) isolates and differences in gene expression and expression quantitative trait loci (eQTL).

METHODS

We determined ofloxacin minimum inhibitory concentration (MIC) for ofloxacin-resistant and -susceptible Mtb isolates without and with the efflux pump inhibitor verapamil. We performed RNA sequencing (RNA-seq), whole genome sequencing (WGS), and eQTL analysis, focusing on efflux pump, transport, and secretion-associated genes.

RESULTS

Of 42 ofloxacin-resistant Mtb isolates, 27 had adequate WGS coverage and acceptable RNA-seq quality. Of these 27, 7 had >2-fold reduction in ofloxacin MIC with verapamil; 6 had 2-fold reduction, and 14 had <2-fold reduction. Five genes (including Rv0191) had significantly increased expression in the MIC fold change >2 compared to <2 groups. Among regulated genes, 31 eQTLs (without ofloxacin) and 35 eQTLs (with ofloxacin) had significant allele frequency differences between MIC fold change >2 and <2 groups. Of these, Rv1410c, Rv2459, and Rv3756c (without ofloxacin) and Rv0191 and Rv3756c (with ofloxacin) have previously been associated with antituberculosis drug resistance.

CONCLUSIONS

In this first reported eQTL analysis in Mtb, Rv0191 had increased gene expression and significance in eQTL analysis, making it a candidate for functional evaluation of efflux-mediated fluoroquinolone resistance in Mtb.

Failure to decrease HbA1c levels following TB treatment is associated with elevated Th1/Th17 CD4+ responses

Introduction

The rising global burden of metabolic disease impacts the control of endemic tuberculosis (TB) in many regions, as persons with diabetes mellitus (DM) are up to three times more likely to develop active TB than those without DM. Active TB can also promote glucose intolerance during both acute infection and over a longer term, potentially driven by aspects of the immune response. Identifying patients likely to have persistent hyperglycemia following TB treatment would enable closer monitoring and care, and an improved understanding of underlying immunometabolic dysregulation.

Methods

We measured the relationship of plasma cytokine levels, T cell phenotypes and functional responses with the change in hemoglobin A1c (HbA1c) before and after treatment of pulmonary TB in a prospective observational cohort in Durban, South Africa. Participants were stratified based on stable/increased HbA1c (n = 16) versus decreased HbA1c (n = 46) levels from treatment initiation to 12 month follow-up.

Results

CD62 P-selectin was up- (1.5-fold) and IL-10 downregulated (0.85-fold) in plasma among individuals whose HbA1c remained stable/increased during TB treatment. This was accompanied by increased pro-inflammatory TB-specific IL-17 production (Th17). In addition, Th1 responses were upregulated in this group, including TNF-α production and CX3CR1 expression, with decreased IL-4 and IL-13 production. Finally, the TNF-α+ IFNγ+ CD8+ T cells were associated with stable/increased HbA1c. These changes were all significantly different in the stable/increased HbA1c relative to the decreased HbA1c group.

Discussion

Overall, these data suggest that patients with stable/increased HbA1c had an increased pro-inflammatory state. Persistent inflammation and elevated T cell activity in individuals with unresolved dysglycemia following TB treatment may indicate failure to fully resolve infection or may promote persistent dysglycemia in these individuals, and further studies are needed to explore potential mechanisms.

Variation in missed doses and reasons for discontinuation of anti-tuberculosis drugs during hospital treatment for drug-resistant tuberculosis in South Africa

BACKGROUND

Updated World Health Organization (WHO) treatment guidelines prioritize all-oral drug-resistant tuberculosis (DR-TB) regimens. Several poorly tolerated drugs, such as amikacin and para-aminosalicylic acid (PAS), remain treatment options for DR-TB in WHO-recommended longer regimens as Group C drugs. Incomplete treatment with anti-TB drugs increases the risk of treatment failure, relapse, and death. We determined whether missed doses of individual anti-TB drugs, and reasons for their discontinuation, varied in closely monitored hospital settings prior to the 2020 WHO DR-TB treatment guideline updates.

METHODS

We collected retrospective data on adult patients with microbiologically confirmed DR-TB between 2008 and 2015 who were selected for a study of acquired drug resistance in the Western Cape Province of South Africa. Medical records through mid-2017 were reviewed. Patients received directly observed treatment during hospitalization at specialized DR-TB hospitals. Incomplete treatment with individual anti-TB drugs, defined as the failure to take medication as prescribed, regardless of reason, was determined by comparing percent missed doses, stratified by HIV status and DR-TB regimen. We applied a generalized mixed effects model.

RESULTS

Among 242 patients, 131 (54%) were male, 97 (40%) were living with HIV, 175 (72%) received second-line treatment prior to first hospitalization, and 191 (79%) died during the study period. At initial hospitalization, 134 (55%) patients had Mycobacterium tuberculosis with resistance to rifampicin and isoniazid (multidrug-resistant TB [MDR-TB]) without resistance to ofloxacin or amikacin, and 102 (42%) had resistance to ofloxacin and/or amikacin. Most patients (129 [53%]) had multiple hospitalizations and DST changes occurred in 146 (60%) by the end of their last hospital discharge. Incomplete treatment was significantly higher for amikacin (18%), capreomycin (18%), PAS (17%) and kanamycin (16%) than other DR-TB drugs (P<0.001), including ethionamide (8%), moxifloxacin (7%), terizidone (7%), ethambutol (7%), and pyrazinamide (6%). Among the most frequently prescribed drugs, second-line injectables had the highest rates of discontinuation for adverse events (range 0.56-1.02 events per year follow-up), while amikacin, PAS and ethionamide had the highest rates of discontinuation for patient refusal (range 0.51-0.68 events per year follow-up). Missed doses did not differ according to HIV status or anti-TB drug combinations.

CONCLUSION

We found that incomplete treatment for second-line injectables and PAS during hospitalization was higher than for other anti-TB drugs. To maximize treatment success, interventions to improve person-centered care and mitigate adverse events may be necessary in cases when PAS or amikacin (2020 WHO recommended Group C drugs) are needed.

High mortality among patients hospitalized for drug-resistant tuberculosis with acquired second-line drug resistance and high HIV prevalence

OBJECTIVES

We compared mortality between HIV-positive and HIV-negative South African adults with drug-resistant tuberculosis (DR-TB) and high incidence of acquired second-line drug resistance.

METHODS

We performed a retrospective review of DR-TB patients with serial second-line TB drug susceptibility tests (2008-2015) who were hospitalized at a specialized TB hospital. We used Kaplan-Meier analysis and Cox models to examine associations with mortality.

RESULTS

Of 245 patients, the median age was 33 years, 54% were male and 40% were HIV-positive, 96% of whom had ever received antiretroviral therapy (ART). At initial drug resistance detection, 99% of patients had resistance to at least rifampicin and isoniazid, and 18% had second-line drug resistance (fluoroquinolones and/or injectable drugs). At later testing, 88% of patients had acquired additional second-line drug resistance. Patient-initiated treatment interruptions (> 2 months) occurred in 47%. Mortality was 79%. Those with HIV had a shorter time to death (p = 0.02; log-rank): median survival time from DR-TB treatment initiation was 2.44 years [95% confidence interval (CI): 2.09-3.15] versus 3.99 years (95% CI: 3.12-4.75) for HIV-negative patients. HIV-positive patients who received ART within 6 months before DR-TB treatment had a higher mortality hazard than HIV-negative patients [adjusted hazard ratio (aHR) ratio = 1.82, 95% CI: 1.21-2.74]. By contrast, HIV-positive patients who did not receive ART within 6 months before DR-TB treatment did not have a significantly higher mortality hazard than HIV-negative patients (aHR = 1.09; 95% CI: 0.72-1.65), although those on ART had lower median CD4 counts than those not on ART (157 vs. 281 cells/μL, respectively; p = 0.02).

CONCLUSIONS

A very high incidence of acquired second-line drug resistance and high overall mortality were observed, reinforcing the need to reduce the risk of acquired resistance and for more effective treatment.

Public health and hospital-based nursing intersection: Case study of drug-resistant tuberculosis patients

OBJECTIVE

Public health nurses (PHN) are key partners in continuity of care for drug-resistant tuberculosis (DR-TB) patients. We examined complexities in DR-TB care transition between community- and hospital-based care.

DESIGN

We conducted a case study using medical record data. Four patients were purposively selected to illustrate intersectional complexities in DR-TB care transition involving PHN.

RESULTS

Case A (HIV negative male) received PHN care at a community-based facility 124 km from Cape Town. Cases B, C, and D (males living with HIV) received PHN community-based care, averaging 25 km from the hospital. Treatment failed in cases A, B, and C; they subsequently died. Case D was cured. All cases were granted leave of absence at least once while hospitalized. None returned when expected mainly due to lack of transport funds. PHN played critical roles regarding patients' return by conducting home visits, interacting with relatives, and assisting emergency officers to transport patients back to the hospital. PHN supported relatives to endure protracted patient hospitalizations.

CONCLUSION

The role of PHN in continuity of DR-TB care in low-middle income countries is unambiguous. PHN are key partners in the DR-TB care cascade, namely facilitating retention in care between hospital and community-based care. Effective DR-TB control relies on effective partnerships among healthcare personnel, patients, and their families.

Cytomegalovirus Infection Is Associated With Increased Prevalence of Latent Tuberculosis Infection

Cytomegalovirus (CMV) increases tuberculosis (TB) risk, but its relationship with latent TB infection (LTBI) is unknown. Using US nationally representative data, we report that CMV was independently associated with LTBI (odds ratio, 2.94; 95% CI, 1.19–7.28; P=.02). CMV and LTBI were associated with higher C-reactive protein, suggesting chronic inflammation.

Resistance-Conferring Mutations on Whole-Genome Sequencing of Fluoroquinolone-resistant and -Susceptible Mycobacterium tuberculosis Isolates: A Proposed Threshold for Identifying Resistance

BACKGROUND

Fluoroquinolone resistance in Mycobacterium tuberculosis (Mtb) is conferred by DNA gyrase mutations, but not all fluoroquinolone-resistant Mtb isolates have mutations detected. The optimal allele frequency threshold to identify resistance-conferring mutations by whole-genome sequencing is unknown.

METHODS

Phenotypically ofloxacin-resistant and lineage-matched ofloxacin-susceptible Mtb isolates underwent whole-genome sequencing at an average coverage depth of 868 reads. Polymorphisms within the quinolone-resistance-determining region (QRDR) of gyrA and gyrB were identified. The allele frequency threshold using the Genome Analysis Toolkit pipeline was ~8%; allele-level data identified the predominant variant allele frequency and mutational burden (ie, sum of all variant allele frequencies in the QRDR) in gyrA, gyrB, and gyrA + gyrB for each isolate. Receiver operating characteristic (ROC) curves assessed the optimal measure of allele frequency and potential thresholds for identifying phenotypically resistant isolates.

RESULTS

Of 42 ofloxacin-resistant Mtb isolates, area under the ROC curve (AUC) was highest for predominant variant allele frequency, so that measure was used to evaluate optimal mutation detection thresholds. AUCs for 8%, 2.5%, and 0.8% thresholds were 0.8452, 0.9286, and 0.9069, respectively. Sensitivity and specificity were 69% and 100% for 8%, 86% and 100% for 2.5%, 91% and 91% for 0.8%. The sensitivity of the 2.5% and 0.8% thresholds were significantly higher than the 8% threshold (P = .016 and .004, respectively) but not significantly different between one another (P = .5).

CONCLUSIONS

A predominant mutation allele frequency threshold of 2.5% had the highest AUC for detecting DNA gyrase mutations that confer ofloxacin resistance, and was therefore the optimal threshold.

Increased Neutrophil Count and Decreased Neutrophil CD15 Expression Correlate With TB Disease Severity and Treatment Response Irrespective of HIV Co-infection

Tuberculosis remains a leading cause of death globally despite curative treatment, partly due to the difficulty of identifying patients who will not respond to therapy. Simple host biomarkers that correlate with response to drug treatment would facilitate improvement in outcomes and the evaluation of novel therapies. In a prospective longitudinal cohort study, we evaluated neutrophil count and phenotype at baseline, as well as during TB treatment in 79 patients [50 (63%) HIV-positive] with microbiologically confirmed drug susceptible TB undergoing standard treatment. At time of diagnosis, blood neutrophils were highly expanded and surface expression of the neutrophil marker CD15 greatly reduced compared to controls. Both measures changed rapidly with the commencement of drug treatment and returned to levels seen in healthy control by treatment completion. Additionally, at the time of diagnosis, high neutrophil count, and low CD15 expression was associated with higher sputum bacterial load and more severe lung damage on chest x-ray, two clinically relevant markers of disease severity. Furthermore, CD15 expression level at diagnosis was associated with TB culture conversion after 2 months of therapy (OR: 0.14, 95% CI: 0.02, 0.89), a standard measure of early TB treatment success. Importantly, our data was not significantly impacted by HIV co-infection. These data suggest that blood neutrophil metrics could potentially be exploited to develop a simple and rapid test to help determine TB disease severity, monitor drug treatment response, and identify subjects at diagnosis who may respond poorly to treatment.

Gaps in the care cascade for screening and treatment of refugees with tuberculosis infection in Middle Tennessee: a retrospective cohort study

BACKGROUND

Treatment of tuberculosis infection (TBI) in individuals at high risk for tuberculosis (TB) disease is a priority for TB elimination in the US. Newly arrived refugees in Middle Tennessee are screened for TBI, but factors associated with gaps in the TBI care cascade are not well characterized.

METHODS

We assessed the TBI care cascade from US entry to completion of treatment for refugees who resettled in Middle Tennessee from 2012 through 2016. We assessed factors associated with treatment initiation and completion using logistic regression models.

RESULTS

Of 6776 refugees who completed initial health screening, 1681 (25%) screened positive for TBI, 1208 were eligible for treatment, 690 started treatment, and 432 completed treatment. Male sex (Odds Ratio [OR]: 1.42; 95% Confidence Interval [CI]: 1.06, 1.89) and screening with interferon gamma release assay compared to tuberculin skin test (OR: 2.89; 95% CI: 1.59, 5.27) were associated with increased treatment initiation; living farther away from TB clinic was associated with decreased treatment initiation (OR: 0.91; 95% CI: 0.83, 0.99). Existing diabetes (OR: 7.27; 95% CI: 1.93, 27.30), receipt of influenza vaccination (OR: 1.65; 95% CI: 1.14, 2.40) and region of origin from South-Eastern or Southern Asia (ORSEAsia: 2.30; 95% CI: 1.43, 3.70; ORSAsia: 1.64; 95% CI: 1.02, 2.64) were associated with increased treatment completion. Six refugees developed TB disease after declining (n = 4) or partially completing (n = 2) TBI treatment; none who completed treatment developed TB disease.

CONCLUSIONS

We determined gaps in the TBI care cascade among refugees in Middle Tennessee. Further assessment of barriers to treatment initiation and completion and interventions to assist refugees are warranted to improve these gaps and prevent TB disease.

Building capacity for advances in tuberculosis research; proceedings of the third RePORT international meeting

RePORT International is a global network of research sites in India, Brazil, Indonesia, South Africa, China, and the Philippines dedicated to collaborative tuberculosis research in the context of HIV. A standardized research protocol (the Common Protocol) guides the enrollment of participants with active pulmonary tuberculosis and contacts into observational cohorts. The establishment of harmonized clinical data and bio-repositories will allow cutting-edge, large-scale advances in the understanding of tuberculosis, including identification of novel biomarkers for progression to active tuberculosis and relapse after treatment. The RePORT International infrastructure aims to support research capacity development through enabling globally-diverse collaborations. To that end, representatives from the RePORT International network sites, funding agencies, and other stakeholders gathered together in Brazil in September 2017 to present updates on relevant research findings and discuss ideas for collaboration. Presenters emphasized research involving biomarker identification for incipient tuberculosis, host immunity and pharmacogenomics, co-morbidities such as HIV and type 2 diabetes mellitus, and tuberculosis transmission in vulnerable and high-risk populations. Currently, 962 active TB participants and 670 household contacts have contributed blood, sputum, urine and microbes to in-country biorepositories. Cross-consortium collaborations have begun sharing data and specimens to analyze molecular and cytokine predictive patterns.

Missed opportunities for tuberculosis screening in primary care

OBJECTIVE

To assess how frequently pediatric practitioners perform latent tuberculosis infection (LTBI) screening according to guidelines. We hypothesized that screening occurs less frequently among children whose parents do not speak English as the primary language.

STUDY DESIGN

We conducted a retrospective cohort study of patients attending well-child visits in an urban academic pediatric primary care clinic between April 1, 2012, and March 31, 2013. We assessed documentation of 3 LTBI screening components and tested the association between parent primary language and tuberculin skin test (TST) placement and documentation of results.

RESULTS

During the study period, 387 of 9143 children (4%) had no documentation of screening question responses. Among the other 8756 children, 831 (10%) were identified as at high risk for LTBI. Of these, 514 (62%) did not have documented TST placement in the appropriate time frame. Thirty-nine of 213 children (18%) who had a TST placed did not have documented results. Multivariable regression showed that parent language was not associated with TST placement or documentation of results, but non-Hispanic Black children were more likely to not have a documented test result (aOR, 2.12; 95% CI, 1.07-4.19; P=.03) when adjusting for age, sex, parent primary language, insurance status, day of the week, and study year of TST placement.

CONCLUSION

Parent primary language was not associated with LTBI testing. However, we found substantial gaps in TST placement and documentation of TST results among high-risk children, the latter of which was associated with race/ethnicity. Targeted quality improvement efforts should focus on developing processes to ensure complete screening in high-risk children.

Clinical Impact of Blood Cultures Contaminated with Coagulase-Negative Staphylococci at an Academic Medical Center

Of all blood cultures positive for coagulase-negative staphylococci collected in 1 year at an academic hospital, 100 were selected randomly for review and designated true positives or contaminated. For the 85 patients whose cultures were determined to be contaminated, chart abstractions revealed substantial unnecessary antibiotic administration, additional laboratory tests and procedures, and hospital readmissions.

Fluoroquinolone susceptibility in Mycobacterium tuberculosis after pre-diagnosis exposure to older- versus newer-generation fluoroquinolones

Fluoroquinolone exposure before tuberculosis (TB) diagnosis is common. We anticipated that exposure to older-generation fluoroquinolones is associated with greater fluoroquinolone MICs in Mycobacterium tuberculosis than exposure to newer agents. A nested case-control study was performed among newly diagnosed TB patients reported to the Tennessee Department of Health (January 2002-December 2009). Each fluoroquinolone-resistant case (n=25) was matched to two fluoroquinolone-susceptible controls (n=50). Ciprofloxacin and ofloxacin were classified as older-generation fluoroquinolones; levofloxacin, moxifloxacin and gatifloxacin were considered newer agents. There was no difference between median ofloxacin MIC for isolates from 9 patients exposed only to older fluoroquinolones, 25 exposed only to newer fluoroquinolones, 6 exposed to both and 35 fluoroquinolone-unexposed patients (Kruskal-Wallis, P=0.35). Using multivariate proportional odds logistic regression adjusting for age and sex, duration of exposure to newer fluoroquinolones was independently associated with higher MIC (OR=1.79, 95% CI 1.22-2.64), but duration of exposure to older fluoroquinolones was not (OR=0.94, 95% CI 0.50-1.78). Isolates from patients exposed only to newer fluoroquinolones tended to have mutations at gyrA codons 90, 91 or 94 more frequently than those exposed only to older fluoroquinolones (44% vs. 11%). We were surprised to find that duration of exposure to newer fluoroquinolones, but not older ones, was independently associated with higher ofloxacin MIC. This suggests that the mutant selection window lower boundary is likely to have clinical relevance; caution is warranted when newer fluoroquinolones are prescribed to patients with TB risk factors.

A systematic review of gyrase mutations associated with fluoroquinolone-resistant Mycobacterium tuberculosis and a proposed gyrase numbering system

Fluoroquinolone resistance in Mycobacterium tuberculosis has become increasingly important. A review of mutations in DNA gyrase, the fluoroquinolone target, is needed to improve the molecular detection of resistance. We performed a systematic review of studies reporting mutations in DNA gyrase genes in clinical M. tuberculosis isolates. From 42 studies that met inclusion criteria, 1220 fluoroquinolone-resistant M. tuberculosis isolates underwent sequencing of the quinolone resistance-determining region (QRDR) of gyrA; 780 (64%) had mutations. The QRDR of gyrB was sequenced in 534 resistant isolates; 17 (3%) had mutations. Mutations at gyrA codons 90, 91 or 94 were present in 654/1220 (54%) resistant isolates. Four different GyrB numbering systems were reported, resulting in mutation location discrepancies. We propose a consensus numbering system. Most fluoroquinolone-resistant M. tuberculosis isolates had mutations in DNA gyrase, but a substantial proportion did not. The proposed consensus numbering system can improve molecular detection of resistance and identification of novel mutations.