Concordance between treatment recommendations provided by IBM Watson for Oncology and a multidisciplinary tumor board for breast cancer in China

ChatGPT's Gastrointestinal Tumor Board Tango: A limping dance partner?

OBJECTIVES

This study aimed to assess the consistency and replicability of treatment recommendations provided by ChatGPT 3.5 compared to gastrointestinal tumor cases presented at multidisciplinary tumor boards (MTBs). It also aimed to distinguish between general and case-specific responses and investigated the precision of ChatGPT's recommendations in replicating exact treatment plans, particularly regarding chemotherapy regimens and follow-up protocols.

MATERIAL AND METHODS

A retrospective study was carried out on 115 cases of gastrointestinal malignancies, selected from 448 patients reviewed in MTB meetings. A senior resident fed patient data into ChatGPT 3.5 to produce treatment recommendations, which were then evaluated against the tumor board's decisions by senior oncology fellows.

RESULTS

Among the examined cases, ChatGPT 3.5 provided general information about the malignancy without considering individual patient characteristics in 19% of cases. However, only in 81% of cases, ChatGPT generated responses that were specific to the individual clinical scenarios. In the subset of case-specific responses, 83% of recommendations exhibited overall treatment strategy concordance between ChatGPT and MTB. However, the exact treatment concordance dropped to 65%, notably lower in recommending specific chemotherapy regimens. Cases recommended for surgery showed the highest concordance rates, while those involving chemotherapy recommendations faced challenges in precision.

CONCLUSIONS

ChatGPT 3.5 demonstrates potential in aligning conceptual approaches to treatment strategies with MTB guidelines. However, it falls short in accurately duplicating specific treatment plans, especially concerning chemotherapy regimens and follow-up procedures. Ethical concerns and challenges in achieving exact replication necessitate prudence when considering ChatGPT 3.5 for direct clinical decision-making in MTBs.

Clinical decision support systems for multidisciplinary team decision-making in patients with solid cancer: Composition of an implementation model based on a scoping review

Generating guideline-based recommendations during multidisciplinary team (MDT) meetings in solid cancers is getting more complex due to increasing amount of information needed to follow the guidelines. Usage of clinical decision support systems (CDSSs) can simplify and optimize decision-making. However, CDSS implementation is lagging behind. Therefore, we aim to compose a CDSS implementation model. By performing a scoping review of the currently reported CDSSs for MDT decision-making we determined 102 barriers and 86 facilitators for CDSS implementation out of 44 papers describing 20 different CDSSs. The most frequently reported barriers and facilitators for CDSS implementation supporting MDT decision-making concerned CDSS maintenance (e.g. incorporating guideline updates), validity of recommendations and interoperability with electronic health records. Based on the identified barriers and facilitators, we composed a CDSS implementation model describing clinical utility, analytic validity and clinical validity to guide CDSS integration more successfully in the clinical workflow to support MDTs in the future.

Artificial intelligence in urologic oncology: the actual clinical practice results of IBM Watson for Oncology in South Korea

Background

Artificial intelligence (AI) is changing our life, including the medical field. Repeated machine learning using big data made various fields more predictable and accurate. In medicine, IBM Watson for Oncology (WFO), trained by Memorial Slone Kettering Cancer Center (MSKCC), was first introduced and applied in 14 countries worldwide.Our study was designed to assess the feasibility of WFO in actual clinical practice. We aimed to investigate the concordance rate between WFO and multidisciplinary tumor board (MTB) in Urologic cancer patients.

Materials and methods

We reviewed retrospectively collected data for consecutive patients who underwent WFO and MTB simultaneously in the diagnosis of urologic malignancy before determining further treatment between August 2017 and September 2020. We compared the recommendation of the AI system, WFO (IBM Watson Health, Cambridge, MA), with the opinion of MTB for further managing all patients diagnosed with urologic malignancies such as prostate, bladder, and kidney cancer.

Results

A total of 55 patients were enrolled in our study. The number of patients with prostate cancer was 48. The number of bladder and kidney cancer patients was 5 and 2, respectively. The overall concordance rate between WFO and MTB was 92.7%. Three patients could not suggest proper treatment options using WFO, and the recommended choice of WFO was not feasible in the Korean Health Insurance Review and Assessment Service.

Conclusions

The decision of WFO showed a high concordance rate with a multidisciplinary tumor board for urologic oncology. However, some recommendations of WFO were not feasible in actual practice, and WFO still has some points to improve and modify. Interestingly, applying WFO is likely to facilitate a multidisciplinary team approach.

Challenging ChatGPT 3.5 in Senology-An Assessment of Concordance with Breast Cancer Tumor Board Decision Making

With the recent diffusion of access to publicly available large language models (LLMs), common interest in generative artificial-intelligence-based applications for medical purposes has skyrocketed. The increased use of these models by tech-savvy patients for personal health issues calls for a scientific evaluation of whether LLMs provide a satisfactory level of accuracy for treatment decisions. This observational study compares the concordance of treatment recommendations from the popular LLM ChatGPT 3.5 with those of a multidisciplinary tumor board for breast cancer (MTB). The study design builds on previous findings by combining an extended input model with patient profiles reflecting patho- and immunomorphological diversity of primary breast cancer, including primary metastasis and precancerous tumor stages. Overall concordance between the LLM and MTB is reached for half of the patient profiles, including precancerous lesions. In the assessment of invasive breast cancer profiles, the concordance amounts to 58.8%. Nevertheless, as the LLM makes considerably fraudulent decisions at times, we do not identify the current development status of publicly available LLMs to be adequate as a support tool for tumor boards. Gynecological oncologists should familiarize themselves with the capabilities of LLMs in order to understand and utilize their potential while keeping in mind potential risks and limitations.

Use and accuracy of decision support systems using artificial intelligence for tumor diseases: a systematic review and meta-analysis

Background

For therapy planning in cancer patients multidisciplinary team meetings (MDM) are mandatory. Due to the high number of cases being discussed and significant workload of clinicians, Clinical Decision Support System (CDSS) may improve the clinical workflow.

Methods

This review and meta-analysis aims to provide an overview of the systems utilized and evaluate the correlation between a CDSS and MDM.

Results

A total of 31 studies were identified for final analysis. Analysis of different cancers shows a concordance rate (CR) of 72.7% for stage I-II and 73.4% for III-IV. For breast carcinoma, CR for stage I-II was 72.8% and for III-IV 84.1%, P≤ 0.00001. CR for colorectal carcinoma is 63% for stage I-II and 67% for III-IV, for gastric carcinoma 55% and 45%, and for lung carcinoma 85% and 83% respectively, all P>0.05. Analysis of SCLC and NSCLC yields a CR of 94,3% and 82,7%, P=0.004 and for adenocarcinoma and squamous cell carcinoma in lung cancer a CR of 90% and 86%, P=0.02.

Conclusion

CDSS has already been implemented in clinical practice, and while the findings suggest that its use is feasible for some cancers, further research is needed to fully evaluate its effectiveness.

Comparison of case-based learning using Watson for oncology and traditional method in teaching undergraduate medical students

BACKGROUND

Watson for Oncology (WFO) is a decision-making system generated by artificial intelligence (AI) and has been widely used in treatment recommendations of cancer patients. However, the application of WFO in clinical teaching among medical students has not been reported.

OBJECTIVE

To establish a novel teaching and learning method with WFO in undergraduate medical students and evaluate its efficiency and students' satisfaction compared with traditional case-based learning model.

METHODS

72 undergraduates majoring in clinical medicine in Wuhan University were enrolled and were randomly divided into the WFO-based group and the control group. 36 students in the WFO-based group learned clinical oncology cases via WFO platform while 36 students in the control group using traditional teaching methods. After the course, final examination and questionnaire survey of teaching assessment were conducted on the two groups of students.

RESULTS

According to the questionnaire survey of teaching assessment, WFO-based group showed significant higher score in the aspect of cultivating ability of independent learning (17.67 ± 1.39 vs. 15.17 ± 2.02, P = 0.018), increasing knowledge mastery (17.75 ± 1.10 vs. 16.25 ± 1.18, P = 0.001), enhancing learning interest (18.41 ± 1.42 vs. 17.00 ± 1.37, P = 0.002), increasing course participation (18.33 ± 1.67 vs. 15.75 ± 1.67, P = 0.001) and the overall course satisfaction (89.25 ± 5.92 vs. 80.75 ± 3.42, P = 0.001) than those of the control group students.

CONCLUSION

Our practice has established a novel clinical case-based teaching pattern with WFO, providing undergraduate students with convenient and scientific training and guidance. It empowers students with improved learning experiences and equips them with essential tools for clinical practices.

Identifying cancer-associated leukocyte profiles using high-resolution flow cytometry screening and machine learning

Background

Machine learning (ML) is a valuable tool with the potential to aid clinical decision making. Adoption of ML to this end requires data that reliably correlates with the clinical outcome of interest; the advantage of ML is that it can model these correlations from complex multiparameter data sets that can be difficult to interpret conventionally. While currently available clinical data can be used in ML for this purpose, there exists the potential to discover new "biomarkers" that will enhance the effectiveness of ML in clinical decision making. Since the interaction of the immune system and cancer is a hallmark of tumor establishment and progression, one potential area for cancer biomarker discovery is through the investigation of cancer-related immune cell signatures. Hence, we hypothesize that blood immune cell signatures can act as a biomarker for cancer progression.

Methods

To probe this, we have developed and tested a multiparameter cell-surface marker screening pipeline, using flow cytometry to obtain high-resolution systemic leukocyte population profiles that correlate with detection and characterization of several cancers in murine syngeneic tumor models.

Results

We discovered a signature of several blood leukocyte subsets, the most notable of which were monocyte subsets, that could be used to train CATboost ML models to predict the presence and type of cancer present in the animals.

Conclusions

Our findings highlight the potential utility of a screening approach to identify robust leukocyte biomarkers for cancer detection and characterization. This pipeline can easily be adapted to screen for cancer specific leukocyte markers from the blood of cancer patient.

Assessing the decision quality of artificial intelligence and oncologists of different experience in different regions in breast cancer treatment

Background

AI-based clinical decision support system (CDSS) has important prospects in overcoming the current informational challenges that cancer diseases faced, promoting the homogeneous development of standardized treatment among different geographical regions, and reforming the medical model. However, there are still a lack of relevant indicators to comprehensively assess its decision-making quality and clinical impact, which greatly limits the development of its clinical research and clinical application. This study aims to develop and application an assessment system that can comprehensively assess the decision-making quality and clinical impacts of physicians and CDSS.

Methods

Enrolled adjuvant treatment decision stage early breast cancer cases were randomly assigned to different decision-making physician panels (each panel consisted of three different seniority physicians in different grades hospitals), each physician made an independent "Initial Decision" and then reviewed the CDSS report online and made a "Final Decision". In addition, the CDSS and guideline expert groups independently review all cases and generate "CDSS Recommendations" and "Guideline Recommendations" respectively. Based on the design framework, a multi-level multi-indicator system including "Decision Concordance", "Calibrated Concordance", " Decision Concordance with High-level Physician", "Consensus Rate", "Decision Stability", "Guideline Conformity", and "Calibrated Conformity" were constructed.

Results

531 cases containing 2124 decision points were enrolled; 27 different seniority physicians from 10 different grades hospitals have generated 6372 decision opinions before and after referring to the "CDSS Recommendations" report respectively. Overall, the calibrated decision concordance was significantly higher for CDSS and provincial-senior physicians (80.9%) than other physicians. At the same time, CDSS has a higher " decision concordance with high-level physician" (76.3%-91.5%) than all physicians. The CDSS had significantly higher guideline conformity than all decision-making physicians and less internal variation, with an overall guideline conformity variance of 17.5% (97.5% vs. 80.0%), a standard deviation variance of 6.6% (1.3% vs. 7.9%), and a mean difference variance of 7.8% (1.5% vs. 9.3%). In addition, provincial-middle seniority physicians had the highest decision stability (54.5%). The overall consensus rate among physicians was 64.2%.

Conclusions

There are significant internal variation in the standardization treatment level of different seniority physicians in different geographical regions in the adjuvant treatment of early breast cancer. CDSS has a higher standardization treatment level than all physicians and has the potential to provide immediate decision support to physicians and have a positive impact on standardizing physicians' treatment behaviors.

Data-driven support to decision-making in molecular tumour boards for lymphoma: A design science approach

Background

The increasing amount of molecular data and knowledge about genomic alterations from next-generation sequencing processes together allow for a greater understanding of individual patients, thereby advancing precision medicine. Molecular tumour boards feature multidisciplinary teams of clinical experts who meet to discuss complex individual cancer cases. Preparing the meetings is a manual and time-consuming process.

Purpose

To design a clinical decision support system to improve the multimodal data interpretation in molecular tumour board meetings for lymphoma patients at Karolinska University Hospital, Stockholm, Sweden. We investigated user needs and system requirements, explored the employment of artificial intelligence, and evaluated the proposed design with primary stakeholders.

Methods

Design science methodology was used to form and evaluate the proposed artefact. Requirements elicitation was done through a scoping review followed by five semi-structured interviews. We used UML Use Case diagrams to model user interaction and UML Activity diagrams to inform the proposed flow of control in the system. Additionally, we modelled the current and future workflow for MTB meetings and its proposed machine learning pipeline. Interactive sessions with end-users validated the initial requirements based on a fictive patient scenario which helped further refine the system.

Results

The analysis showed that an interactive secure Web-based information system supporting the preparation of the meeting, multidisciplinary discussions, and clinical decision-making could address the identified requirements. Integrating artificial intelligence via continual learning and multimodal data fusion were identified as crucial elements that could provide accurate diagnosis and treatment recommendations.

Impact

Our work is of methodological importance in that using artificial intelligence for molecular tumour boards is novel. We provide a consolidated proof-of-concept system that could support the end-to-end clinical decision-making process and positively and immediately impact patients.

Conclusion

Augmenting a digital decision support system for molecular tumour boards with retrospective patient material is promising. This generates realistic and constructive material for human learning, and also digital data for continual learning by data-driven artificial intelligence approaches. The latter makes the future system adaptable to human bias, improving adequacy and decision quality over time and over tasks, while building and maintaining a digital log.

Future Medical Artificial Intelligence Application Requirements and Expectations of Physicians in German University Hospitals: Web-Based Survey

BACKGROUND

The increasing development of artificial intelligence (AI) systems in medicine driven by researchers and entrepreneurs goes along with enormous expectations for medical care advancement. AI might change the clinical practice of physicians from almost all medical disciplines and in most areas of health care. While expectations for AI in medicine are high, practical implementations of AI for clinical practice are still scarce in Germany. Moreover, physicians' requirements and expectations of AI in medicine and their opinion on the usage of anonymized patient data for clinical and biomedical research have not been investigated widely in German university hospitals.

OBJECTIVE

This study aimed to evaluate physicians' requirements and expectations of AI in medicine and their opinion on the secondary usage of patient data for (bio)medical research (eg, for the development of machine learning algorithms) in university hospitals in Germany.

METHODS

A web-based survey was conducted addressing physicians of all medical disciplines in 8 German university hospitals. Answers were given using Likert scales and general demographic responses. Physicians were asked to participate locally via email in the respective hospitals.

RESULTS

The online survey was completed by 303 physicians (female: 121/303, 39.9%; male: 173/303, 57.1%; no response: 9/303, 3.0%) from a wide range of medical disciplines and work experience levels. Most respondents either had a positive (130/303, 42.9%) or a very positive attitude (82/303, 27.1%) towards AI in medicine. There was a significant association between the personal rating of AI in medicine and the self-reported technical affinity level (H4=48.3, P<.001). A vast majority of physicians expected the future of medicine to be a mix of human and artificial intelligence (273/303, 90.1%) but also requested a scientific evaluation before the routine implementation of AI-based systems (276/303, 91.1%). Physicians were most optimistic that AI applications would identify drug interactions (280/303, 92.4%) to improve patient care substantially but were quite reserved regarding AI-supported diagnosis of psychiatric diseases (62/303, 20.5%). Of the respondents, 82.5% (250/303) agreed that there should be open access to anonymized patient databases for medical and biomedical research.

CONCLUSIONS

Physicians in stationary patient care in German university hospitals show a generally positive attitude towards using most AI applications in medicine. Along with this optimism comes several expectations and hopes that AI will assist physicians in clinical decision making. Especially in fields of medicine where huge amounts of data are processed (eg, imaging procedures in radiology and pathology) or data are collected continuously (eg, cardiology and intensive care medicine), physicians' expectations of AI to substantially improve future patient care are high. In the study, the greatest potential was seen in the application of AI for the identification of drug interactions, assumedly due to the rising complexity of drug administration to polymorbid, polypharmacy patients. However, for the practical usage of AI in health care, regulatory and organizational challenges still have to be mastered.

[Artificial intelligence in oncological radiology : A (p)review]

BACKGROUND

Artificial intelligence (AI) has the potential to fundamentally change medicine within the coming decades. Radiological imaging is one of the primary fields of its clinical application.

OBJECTIVES

In this article, we summarize previous AI developments with a focus on oncological radiology. Based on selected examples, we derive scenarios for developments in the next 10 years.

MATERIALS AND METHODS

This work is based on a review of various literature and product databases, publications by regulatory authorities, reports, and press releases.

CONCLUSIONS

The clinical use of AI applications is still in an early stage of development. The large number of research publications shows the potential of the field. Several certified products have already become available to users. However, for a widespread adoption of AI applications in clinical routine, several fundamental prerequisites are still awaited. These include the generation of evidence justifying the use of algorithms through representative clinical studies, adjustments to the framework for approval processes and dedicated education and teaching resources for its users. It is expected that use of AI methods will increase, thus, creating new opportunities for improved diagnostics, therapy, and more efficient workflows.