Investigating the clinical reasoning abilities of large language model GPT-4: an analysis of postoperative complications from renal surgeries

PURPOSE

Large language models, a subset of artificial intelligence, have immense potential to support human tasks. The role of these models in science and medicine is unclear, requiring strong critical thinking and analysis skills. The objective of our study was to evaluate GPT-4's abilities to assess postoperative complications after renal surgeries.

MATERIALS AND METHODS

Discharge summaries were compiled, and patient information was deidentified in a Python-based program. Prompts were engineered in GPT-4 to assess for the presence of postoperative complications. GPT-4 was further asked to interpret each complication's Clavien-Dindo classification and institutional-specific category. GPT-4's database was compared to a human-curated database. Discrepancies were manually reviewed to calculate match and accuracy rates.

RESULTS

Approximately 944 renal surgeries were conducted from August 2005 to March 2022. There was a 79.6% match rate between GPT-4 and human-curated data in detecting postoperative complications. Accuracy rates were 86.7% for GPT-4 and 92.9% for human-curated. A subgroup of 139 patients had a complication detected by both GPT-4 and human with available Clavien-Dindo classification and category information. There was a 37.4% overall match rate for Clavien-Dindo grade and 55.4% match rate for category.

CONCLUSIONS

GPT-4 was able to accurately detect if there were any postoperative complications. It struggled with the complex task of further analyzing complications, especially with Clavien-Dindo classification, which requires more critical thinking and interpretation. While GPT-4 is not yet ready for advanced postoperative complication analysis, it can still be used to support clinicians in this endeavor.

Evaluating Diagnostic Accuracy and Inter-reader Agreement of the Prostate Imaging After Focal Ablation Scoring System

Background and objective

Focal therapy (FT) is increasingly recognized as a promising approach for managing localized prostate cancer (PCa), notably reducing treatment-related morbidities. However, post-treatment anatomical changes present significant challenges for surveillance using current imaging techniques. This study aimed to evaluate the inter-reader agreement and efficacy of the Prostate Imaging after Focal Ablation (PI-FAB) scoring system in detecting clinically significant prostate cancer (csPCa) on post-FT multiparametric magnetic resonance imaging (mpMRI).

Methods

A retrospective cohort study was conducted involving patients who underwent primary FT for localized csPCa between 2013 and 2023, followed by post-FT mpMRI and a prostate biopsy. Two expert genitourinary radiologists retrospectively evaluated post-FT mpMRI using PI-FAB. The key measures included inter-reader agreement of PI-FAB scores, assessed by quadratic weighted Cohen's kappa (κ), and the system's efficacy in predicting in-field recurrence of csPCa, with a PI-FAB score cutoff of 3. Additional diagnostic metrics including sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and overall accuracy were also evaluated.

Key findings and limitations

Scans from 38 patients were analyzed, revealing a moderate level of agreement in PI-FAB scoring (κ = 0.56). Both radiologists achieved sensitivity of 93% in detecting csPCa, although specificity, PPVs, NPVs, and accuracy varied.

Conclusions and clinical implications

The PI-FAB scoring system exhibited high sensitivity with moderate inter-reader agreement in detecting in-field recurrence of csPCa. Despite promising results, its low specificity and PPV necessitate further refinement. These findings underscore the need for larger studies to validate the clinical utility of PI-FAB, potentially aiding in standardizing post-treatment surveillance.

Patient summary

Focal therapy has emerged as a promising approach for managing localized prostate cancer, but limitations in current imaging techniques present significant challenges for post-treatment surveillance. The Prostate Imaging after Focal Ablation (PI-FAB) scoring system showed high sensitivity for detecting in-field recurrence of clinically significant prostate cancer. However, its low specificity and positive predictive value necessitate further refinement. Larger, more comprehensive studies are needed to fully validate its clinical utility.

Evaluating the performance of Generative Pre-trained Transformer-4 (GPT-4) in standardizing radiology reports

OBJECTIVE

Radiology reporting is an essential component of clinical diagnosis and decision-making. With the advent of advanced artificial intelligence (AI) models like GPT-4 (Generative Pre-trained Transformer 4), there is growing interest in evaluating their potential for optimizing or generating radiology reports. This study aimed to compare the quality and content of radiologist-generated and GPT-4 AI-generated radiology reports.

METHODS

A comparative study design was employed in the study, where a total of 100 anonymized radiology reports were randomly selected and analyzed. Each report was processed by GPT-4, resulting in the generation of a corresponding AI-generated report. Quantitative and qualitative analysis techniques were utilized to assess similarities and differences between the two sets of reports.

RESULTS

The AI-generated reports showed comparable quality to radiologist-generated reports in most categories. Significant differences were observed in clarity (p = 0.027), ease of understanding (p = 0.023), and structure (p = 0.050), favoring the AI-generated reports. AI-generated reports were more concise, with 34.53 fewer words and 174.22 fewer characters on average, but had greater variability in sentence length. Content similarity was high, with an average Cosine Similarity of 0.85, Sequence Matcher Similarity of 0.52, BLEU Score of 0.5008, and BERTScore F1 of 0.8775.

CONCLUSION

The results of this proof-of-concept study suggest that GPT-4 can be a reliable tool for generating standardized radiology reports, offering potential benefits such as improved efficiency, better communication, and simplified data extraction and analysis. However, limitations and ethical implications must be addressed to ensure the safe and effective implementation of this technology in clinical practice.

CLINICAL RELEVANCE STATEMENT

The findings of this study suggest that GPT-4 (Generative Pre-trained Transformer 4), an advanced AI model, has the potential to significantly contribute to the standardization and optimization of radiology reporting, offering improved efficiency and communication in clinical practice.

KEY POINTS

• Large language model-generated radiology reports exhibited high content similarity and moderate structural resemblance to radiologist-generated reports. • Performance metrics highlighted the strong matching of word selection and order, as well as high semantic similarity between AI and radiologist-generated reports. • Large language model demonstrated potential for generating standardized radiology reports, improving efficiency and communication in clinical settings.

Initial experience and cancer detection rates of office-based transperineal magnetic resonance imaging-ultrasound fusion prostate biopsy under local anesthesia

INTRODUCTION

We aimed to demonstrate feasibility and cancer detection rates of office-based ultrasound-guided transperineal magnetic resonance imaging-ultrasound (MRI-US) fusion (TFB) prostate biopsy under local anesthesia.

METHODS

With institutional review board approval, records of men undergoing TFB in the office setting under local anesthesia were reviewed. Baseline patient characteristics, MRI findings, cancer detection rates, and complications were recorded. The PrecisionPoint Transperineal Access System (Perineologic, Cumberland, MD, U.S.) along with UroNav 3.0 image-fusion system (Invivo International, Best, The Netherlands) were used for all procedures. Following biopsy, men were surveyed to assess patient experience.

RESULTS

Between January 2019 and February 2020, 200 TFBs were performed, of which 141 (71%) were positive for prostate cancer, with 117 (83%) Gleason grade group 2 or higher. A total of 259 of 265 MRI lesions were biopsied, with 127 (49%) positive overall. Prostate Imaging-Reporting and Data System (PI-RADS) 4-5 lesions were positive for prostate cancer in 59% of cases. The mean procedural time was 20 minutes, with a patient enter-to-exit room time of 54 minutes. There were no septic complications, no patients required post-procedure hospital admission, and all procedures were successfully completed. Seventy-five percent of patients surveyed reported complete resolution of pain at three days following the procedure.

CONCLUSIONS

Office-based transperineal fusion biopsy represents a viable approach to prostate cancer detection following prostate MRI. Larger-scale assessment is needed to categorize cancer detection rates more accurately by PI-RADs subset, patient selection factors, complication rate, and cost relative to TFB under anesthesia.

Prostate Cancer Screening

This chapter introduces the controversy of prostate cancer screening that has emerged over the past two decades, starting with the widespread use of the prostate-specific antigen (PSA) test in 1994. The PSA test revolutionized prostate cancer detection as it is the first biomarker available to physicians to identify prostate cancers that could not be detected by a digital rectal exam. However, unrestricted utilization of the PSA test in asymptomatic men led to an increase in overdiagnosis and treatment-related morbidity. When reviewing evidence relating prostate cancer screening to prostate cancer–related morbidity and mortality, the US Preventative Services Task Force (USPSTF) issued a recommendation in 2012 that screening among asymptomatic men for prostate cancer produces more harm than benefit. These recommendations were updated in 2018 to offer screening selectively for men between the ages of 55 and 69 years. Both recommendations called for physicians to engage patients in shared decision making regarding screening. Consequences of these recommendations on shared decision making, health communication, and efforts to address disparities in screening among underserved and at-risk populations are discussed.

Management and treatment options when facing malignant ureteral obstruction

Malignant ureteral obstruction is an unfortunate finding that can be caused by a wide-ranging number of malignancies with a prognosis of limited survival. Given its presentation and progression, it can be refractory to treatment by traditional single polymeric ureteral stents. With a higher failure rate than causes of benign ureteral obstruction, a number of other options are available for initial management, as well as in cases of first-line therapy failure, including tandem stents, metallic stents, percutaneous nephrostomies and extra-anatomic stents. We reviewed the literature and carried out a PubMed search including the following keywords and phrases: "malignant ureteral obstruction," "tandem ureteral stents," "metallic ureteral stents," "resonance stent," "metal mesh ureteral stents" and "extra-anatomic stents." The vast majority of studies were small and retrospective, with a large number of studies related to metallic stents. Given the heterogenous patient population and diversity of practice, it is difficult to truly assess the efficacy of each method. As there are no guidelines or major head-to-head prospective trials involving these techniques, it makes practicing up to the specific provider. However, this article attempts to provide a framework with which the urologist who is presented with malignant ureteral obstruction can plan in order to provide the individualized care on a case-by-case basis. What is clear is that prospective, randomized clinical trials are necessary to help bring evidence-based medicine and guidelines for patients with malignant ureteral obstruction.

Triple Antiplatelet Therapy and Combinations with Oral Anticoagulants After Stent Implantation

Triple oral anticoagulation or triple antiplatelet therapies may be administered for various reasons. They reduce cardiac complications following percutaneous coronary intervention and stroke or other thromboembolic phenomenon in conditions such as atrial fibrillation. There is an elevated risk of severe bleeding, so it is necessary to balance risk and benefits. Newer oral anticoagulants and antiplatelet drugs may be considered; the number of options is increasing. This article examines triple therapies and the efficacy and safety of combinations of traditional anticoagulant and antiplatelet drugs, and reviews clinical trial data on novel agents. Guidelines to inform clinical decision-making are presented.