Disparity dashboards: an evaluation of the literature and framework for health equity improvement

The growing recognition of differences in health outcomes across populations has led to a slow but increasing shift towards transparent reporting of patient outcomes. In addition, pay-for-equity initiatives, such as those proposed by the Centers for Medicare and Medicaid, will require the reporting of health outcomes across subgroups over time. Dashboards offer one means of visualising data in the health-care context that can highlight essential disparities in clinical outcomes, guide targeted quality-improvement efforts, and ultimately improve health equity. In this Viewpoint, we evaluate all studies that have reported the successful development of a disparity dashboard and share the data collected and unintended consequences reported. We propose a framework for systematic equality improvement through incentivisation of the collecting and reporting of health data and through implementation of reward systems to reduce health disparities.

Evaluating equitable care in the ICU: Creating a causal inference framework to assess the impact of life-sustaining interventions across racial and ethnic groups

Background

Variability in the provision of intensive care unit (ICU)-interventions may lead to disparities between socially defined racial-ethnic groups.

Research Question

We used causal inference to examine the use of invasive mechanical ventilation (IMV), renal replacement therapy (RRT), and vasopressor agents (VP) to identify disparities in outcomes across race-ethnicity in patients with sepsis.

Study Design and Methods

Single-center, academic referral hospital in Boston, Massachusetts, USA. Retrospective analysis of treatment effect with a targeted trial design categorized by treatment assignment within the first 24 hours in the MIMIC-IV dataset (2008- 2019) using targeted maximum likelihood estimation. Of 76,943 ICU stays in MIMIC-IV, 32,971 adult stays fulfilling sepsis-3 criteria were included. The primary outcome was in-hospital mortality. Secondary outcomes were hospital-free days, and occurrence of nosocomial infection stratified by predicted mortality probability ranges and self-reported race-ethnicity. Average treatment effects by treatment type and race-ethnicity, Racial-ethnic group (REG) or White group (WG), were estimated.

Results

Of 19,419 admissions that met inclusion criteria, median age was 68 years, 57.4% were women, 82% were White, and mortality was 18.2%. There was no difference in mortality benefit associated with the administration of IMV, RRT, or VP between the REG and the WG. There was also no difference in hospital-free days or nosocomial infections. These findings are unchanged with different eligibility periods.

Interpretation

There were no differences in the treatment outcomes from three life-sustaining interventions in the ICU according to race-ethnicity. While there was no discernable harm from the treatments across mortality risk, there was also no measurable benefit. These findings highlight the need for research to understand better the risk-benefit of life-sustaining interventions in the ICU.

Strategies and solutions to address Digital Determinants of Health (DDOH) across underinvested communities

Healthcare has long struggled to improve services through technology without further widening health disparities. With the significant expansion of digital health, a group of healthcare professionals and scholars from across the globe are proposing the official usage of the term "Digital Determinants of Health" (DDOH) to explicitly call out the relationship between technology, healthcare, and equity. This is the final paper in a series published in PLOS Digital Health that seeks to understand and summarize current knowledge of the strategies and solutions that help to mitigate the negative effects of DDOH for underinvested communities. Through a search of English-language Medline, Scopus, and Google Scholar articles published since 2010, 345 articles were identified that discussed the application of digital health technology among underinvested communities. A group of 8 reviewers assessed 132 articles selected at random for the mention of solutions that minimize differences in DDOH. Solutions were then organized by categories of policy; design and development; implementation and adoption; and evaluation and ongoing monitoring. The data were then assessed by category and the findings summarized. The reviewers also looked for common themes across the solutions and evidence of effectiveness. From this limited scoping review, the authors found numerous solutions mentioned across the papers for addressing DDOH and many common themes emerged regardless of the specific community or digital health technology under review. There was notably less information on solutions regarding ongoing evaluation and monitoring which corresponded with a lack of research evidence regarding effectiveness. The findings directionally suggest that universal strategies and solutions can be developed to address DDOH independent of the specific community under focus. With the need for the further development of DDOH measures, we also provide a framework for DDOH assessment.

A new tool for evaluating health equity in academic journals; the Diversity Factor

Current methods to evaluate a journal's impact rely on the downstream citation mapping used to generate the Impact Factor. This approach is a fragile metric prone to being skewed by outlier values and does not speak to a researcher's contribution to furthering health outcomes for all populations. Therefore, we propose the implementation of a Diversity Factor to fulfill this need and supplement the current metrics. It is composed of four key elements: dataset properties, author country, author gender and departmental affiliation. Due to the significance of each individual element, they should be assessed independently of each other as opposed to being combined into a simplified score to be optimized. Herein, we discuss the necessity of such metrics, provide a framework to build upon, evaluate the current landscape through the lens of each key element and publish the findings on a freely available website that enables further evaluation. The OpenAlex database was used to extract the metadata of all papers published from 2000 until August 2022, and Natural language processing was used to identify individual elements. Features were then displayed individually on a static dashboard developed using TableauPublic, which is available at www.equitablescience.com. In total, 130,721 papers were identified from 7,462 journals where significant underrepresentation of LMIC and Female authors was demonstrated. These findings are pervasive and show no positive correlation with the Journal's Impact Factor. The systematic collection of the Diversity Factor concept would allow for more detailed analysis, highlight gaps in knowledge, and reflect confidence in the translation of related research. Conversion of this metric to an active pipeline would account for the fact that how we define those most at risk will change over time and quantify responses to particular initiatives. Therefore, continuous measurement of outcomes across groups and those investigating those outcomes will never lose importance. Moving forward, we encourage further revision and improvement by diverse author groups in order to better refine this concept.

Screening Health-Related Social Needs in Hospitals: A Systematic Review of Health Care Professional and Patient Perspectives

Health outcomes are markedly influenced by health-related social needs (HRSN) such as food insecurity and housing instability. Under new Joint Commission requirements, hospitals have recently increased attention to HRSN to reduce health disparities. To evaluate prevailing attitudes and guide hospital efforts, the authors conducted a systematic review to describe patients' and health care providers' perceptions related to screening for and addressing patients' HRSN in US hospitals. Articles were identified through PubMed and by expert recommendations, and synthesized by relevance of findings and basic study characteristics. The review included 22 articles, which showed that most health care providers believed that unmet social needs impact health and that screening for HRSN should be a standard part of hospital care. Notable differences existed between perceived importance of HRSN and actual screening rates, however. Patients reported high receptiveness to screening in hospital encounters, but cautioned to avoid stigmatization and protect privacy when screening. Limited knowledge of resources available, lack of time, and lack of actual resources were the most frequently reported barriers to screening for HRSN. Hospital efforts to screen and address HRSN will likely be facilitated by stakeholders' positive perceptions, but common barriers to screening and referral will need to be addressed to effectively scale up efforts and impact health disparities.

Artificial Intelligence and Machine Learning Technologies in Cancer Care: Addressing Disparities, Bias, and Data Diversity

SUMMARY

Artificial intelligence (AI) and machine learning (ML) technologies have not only tremendous potential to augment clinical decision-making and enhance quality care and precision medicine efforts, but also the potential to worsen existing health disparities without a thoughtful, transparent, and inclusive approach that includes addressing bias in their design and implementation along the cancer discovery and care continuum. We discuss applications of AI/ML tools in cancer and provide recommendations for addressing and mitigating potential bias with AI and ML technologies while promoting cancer health equity.

For the Commercially Insured, Equitable Health Benefits Begin With Equitable Health Insurance Design

Equitable health benefit design is central to addressing the health inequities of individuals with commercial health insurance in the United States. To do so, employers and other plan sponsors must take action to identify and address unmet health and well-being priorities among racialized groups and low-income workers. These historically underrepresented subpopulations will also benefit from more equitable approaches to healthcare benefits design that recognize and meaningfully address access and affordability concerns. Targeted appropriately, these actions have the potential to foster greater employee engagement and productivity, leading to enhanced business performance.

Abstract IA-04: Leveraging the potential of artificial and machine learning technologies to address cancer health disparities

How can we realize the potential of artificial intelligence and machine learning for addressing cancer inequities and advancing cancer health disparities research? This presentation will discuss areas where bias can be introduced into the AI and machine learning design, development, implementation, use and standardization for cancer care. What are the factors that contribute to bias in the research continuum and how can we mitigate them? The presentation will provide some promising and actionable examples of how we can leverage the transformative potential of AI and machine learning to address cancer health disparities.

Citation Format: Irene Dankwa-Mullan. Leveraging the potential of artificial and machine learning technologies to address cancer health disparities [abstract]. In: Proceedings of the AACR Virtual Conference: 14th AACR Conference on the Science of Cancer Health Disparities in Racial/Ethnic Minorities and the Medically Underserved; 2021 Oct 6-8. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2022;31(1 Suppl):Abstract nr IA-04.

Artificial intelligence applications in social media for depression screening: A systematic review protocol for content validity processes

BACKGROUND

The popularization of social media has led to the coalescing of user groups around mental health conditions; in particular, depression. Social media offers a rich environment for contextualizing and predicting users' self-reported burden of depression. Modern artificial intelligence (AI) methods are commonly employed in analyzing user-generated sentiment on social media. In the forthcoming systematic review, we will examine the content validity of these computer-based health surveillance models with respect to standard diagnostic frameworks. Drawing from a clinical perspective, we will attempt to establish a normative judgment about the strengths of these modern AI applications in the detection of depression.

METHODS

We will perform a systematic review of English and German language publications from 2010 to 2020 in PubMed, APA PsychInfo, Science Direct, EMBASE Psych, Google Scholar, and Web of Science. The inclusion criteria span cohort, case-control, cross-sectional studies, randomized controlled studies, in addition to reports on conference proceedings. The systematic review will exclude some gray source materials, specifically editorials, newspaper articles, and blog posts. Our primary outcome is self-reported depression, as expressed on social media. Secondary outcomes will be the types of AI methods used for social media depression screen, and the clinical validation procedures accompanying these methods. In a second step, we will utilize the evidence-strengthening Population, Intervention, Comparison, Outcomes, Study type (PICOS) tool to refine our inclusion and exclusion criteria. Following the independent assessment of the evidence sources by two authors for the risk of bias, the data extraction process will culminate in a thematic synthesis of reviewed studies.

DISCUSSION

We present the protocol for a systematic review which will consider all existing literature from peer reviewed publication sources relevant to the primary and secondary outcomes. The completed review will discuss depression as a self-reported health outcome in social media material. We will examine the computational methods, including AI and machine learning techniques which are commonly used for online depression surveillance. Furthermore, we will focus on standard clinical assessments, as indicating content validity, in the design of the algorithms. The methodological quality of the clinical construct of the algorithms will be evaluated with the COnsensus-based Standards for the selection of health status Measurement Instruments (COSMIN) framework. We conclude the study with a normative judgment about the current application of AI to screen for depression on social media.

SYSTEMATIC REVIEW REGISTRATION

International Prospective Register of Systematic Reviews PROSPERO (registration number CRD42020187874).

Health Care Equity in the Use of Advanced Analytics and Artificial Intelligence Technologies in Primary Care

The integration of advanced analytics and artificial intelligence (AI) technologies into the practice of medicine holds much promise. Yet, the opportunity to leverage these tools carries with it an equal responsibility to ensure that principles of equity are incorporated into their implementation and use. Without such efforts, tools will potentially reflect the myriad of ways in which data, algorithmic, and analytic biases can be produced, with the potential to widen inequities by race, ethnicity, gender, and other sociodemographic factors implicated in disparate health outcomes. We propose a set of strategic assertions to examine before, during, and after adoption of these technologies in order to facilitate healthcare equity across all patient population groups. The purpose is to enable generalists to promote engagement with technology companies and co-create, promote, or support innovation and insights that can potentially inform decision-making and health care equity.

Comparison of an oncology clinical decision-support system's recommendations with actual treatment decisions

Objective

IBM^(R) Watson for Oncology (WfO) is a clinical decision-support system (CDSS) that provides evidence-informed therapeutic options to cancer-treating clinicians. A panel of experienced oncologists compared CDSS treatment options to treatment decisions made by clinicians to characterize the quality of CDSS therapeutic options and decisions made in practice.

Methods

This study included patients treated between 1/2017 and 7/2018 for breast, colon, lung, and rectal cancers at Bumrungrad International Hospital (BIH), Thailand. Treatments selected by clinicians were paired with therapeutic options presented by the CDSS and coded to mask the origin of options presented. The panel rated the acceptability of each treatment in the pair by consensus, with acceptability defined as compliant with BIH’s institutional practices. Descriptive statistics characterized the study population and treatment-decision evaluations by cancer type and stage.

Results

Nearly 60% (187) of 313 treatment pairs for breast, lung, colon, and rectal cancers were identical or equally acceptable, with 70% (219) of WfO therapeutic options identical to, or acceptable alternatives to, BIH therapy. In 30% of cases (94), 1 or both treatment options were rated as unacceptable. Of 32 cases where both WfO and BIH options were acceptable, WfO was preferred in 18 cases and BIH in 14 cases. Colorectal cancers exhibited the highest proportion of identical or equally acceptable treatments; stage IV cancers demonstrated the lowest.

Conclusion

This study demonstrates that a system designed in the US to support, rather than replace, cancer-treating clinicians provides therapeutic options which are generally consistent with recommendations from oncologists outside the US.

TechQuity is an imperative for health and technology business: Let's work together to achieve it

Open discussions of social justice and health inequities may be an uncommon focus within information technology science, business, and health care delivery partnerships. However, the COVID-19 pandemic—which disproportionately affected Black, indigenous, and people of color—has reinforced the need to examine and define roles that technology partners should play to lead anti-racism efforts through our work. In our perspective piece, we describe the imperative to prioritize TechQuity—equity and social justice as a technology business strategy—through collaborating in partnerships that focus on eliminating racial and social inequities.

Comparison of Methods to Reduce Bias From Clinical Prediction Models of Postpartum Depression

IMPORTANCE

The lack of standards in methods to reduce bias for clinical algorithms presents various challenges in providing reliable predictions and in addressing health disparities.

OBJECTIVE

To evaluate approaches for reducing bias in machine learning models using a real-world clinical scenario.

DESIGN, SETTING, AND PARTICIPANTS

Health data for this cohort study were obtained from the IBM MarketScan Medicaid Database. Eligibility criteria were as follows: (1) Female individuals aged 12 to 55 years with a live birth record identified by delivery-related codes from January 1, 2014, through December 31, 2018; (2) greater than 80% enrollment through pregnancy to 60 days post partum; and (3) evidence of coverage for depression screening and mental health services. Statistical analysis was performed in 2020.

EXPOSURES

Binarized race (Black individuals and White individuals).

MAIN OUTCOMES AND MEASURES

Machine learning models (logistic regression [LR], random forest, and extreme gradient boosting) were trained for 2 binary outcomes: postpartum depression (PPD) and postpartum mental health service utilization. Risk-adjusted generalized linear models were used for each outcome to assess potential disparity in the cohort associated with binarized race (Black or White). Methods for reducing bias, including reweighing, Prejudice Remover, and removing race from the models, were examined by analyzing changes in fairness metrics compared with the base models. Baseline characteristics of female individuals at the top-predicted risk decile were compared for systematic differences. Fairness metrics of disparate impact (DI, 1 indicates fairness) and equal opportunity difference (EOD, 0 indicates fairness).

RESULTS

Among 573 634 female individuals initially examined for this study, 314 903 were White (54.9%), 217 899 were Black (38.0%), and the mean (SD) age was 26.1 (5.5) years. The risk-adjusted odds ratio comparing White participants with Black participants was 2.06 (95% CI, 2.02-2.10) for clinically recognized PPD and 1.37 (95% CI, 1.33-1.40) for postpartum mental health service utilization. Taking the LR model for PPD prediction as an example, reweighing reduced bias as measured by improved DI and EOD metrics from 0.31 and -0.19 to 0.79 and 0.02, respectively. Removing race from the models had inferior performance for reducing bias compared with the other methods (PPD: DI = 0.61; EOD = -0.05; mental health service utilization: DI = 0.63; EOD = -0.04).

CONCLUSIONS AND RELEVANCE

Clinical prediction models trained on potentially biased data may produce unfair outcomes on the basis of the chosen metrics. This study's results suggest that the performance varied depending on the model, outcome label, and method for reducing bias. This approach toward evaluating algorithmic bias can be used as an example for the growing number of researchers who wish to examine and address bias in their data and models.

Accuracy of an Artificial Intelligence System for Cancer Clinical Trial Eligibility Screening: Retrospective Pilot Study

Background

Screening patients for eligibility for clinical trials is labor intensive. It requires abstraction of data elements from multiple components of the longitudinal health record and matching them to inclusion and exclusion criteria for each trial. Artificial intelligence (AI) systems have been developed to improve the efficiency and accuracy of this process.

Objective

This study aims to evaluate the ability of an AI clinical decision support system (CDSS) to identify eligible patients for a set of clinical trials.

Methods

This study included the deidentified data from a cohort of patients with breast cancer seen at the medical oncology clinic of an academic medical center between May and July 2017 and assessed patient eligibility for 4 breast cancer clinical trials. CDSS eligibility screening performance was validated against manual screening. Accuracy, sensitivity, specificity, positive predictive value, and negative predictive value for eligibility determinations were calculated. Disagreements between manual screeners and the CDSS were examined to identify sources of discrepancies. Interrater reliability between manual reviewers was analyzed using Cohen (pairwise) and Fleiss (three-way) κ, and the significance of differences was determined by Wilcoxon signed-rank test.

Results

In total, 318 patients with breast cancer were included. Interrater reliability for manual screening ranged from 0.60-0.77, indicating substantial agreement. The overall accuracy of breast cancer trial eligibility determinations by the CDSS was 87.6%. CDSS sensitivity was 81.1% and specificity was 89%.

Conclusions

The AI CDSS in this study demonstrated accuracy, sensitivity, and specificity of greater than 80% in determining the eligibility of patients for breast cancer clinical trials. CDSSs can accurately exclude ineligible patients for clinical trials and offer the potential to increase screening efficiency and accuracy. Additional research is needed to explore whether increased efficiency in screening and trial matching translates to improvements in trial enrollment, accruals, feasibility assessments, and cost.

Abstract PS8-22: Augmentation of a minimal multidisciplinary tumor board with clinical decision support to triage breast cancer patients in the UK

BackgroundAll UK cancer patients undergo required assessments by a full Multidisciplinary Tumor Board (fMTB) at key treatment decision points, placing a resource burden on the healthcare system. Watson for Oncology (WfO) is a decision-support system that presents therapeutic options to cancer-treating clinicians. This study is an initial phase of an evaluation at Guys and St. Thomas’ NHS Hospital (GSTT), designed to explore the extent to which WfO can be used by the fMTB to triage less complex patient cases and ultimately reduce workload and time pressures currently experienced by fMTBs. We conducted a concordance study with two minimal MTB teams (mMTB) for Stage I-III breast cancer patients.

MethodsBreast cancer cases (N=63) treated from 2017-2018 at GSTT were evaluated by 2 independent mMTBs, blinded to each other and previous fMTB decisions rendered prior to this study. Each mMTB consisted of a senior medical oncologist and surgeon; GSTT’s 12+ member fMTB is comprised of oncologists, surgeons, radiologists, pathologists and others. mMTBs were shown options that were either listed as ‘recommended’ or ‘for consideration’ by WfO and given the opportunity to revise prior decisions. The combined 4-person minimal MTB (cmMTB) consisting of both 2-person mMTBs provided a current consensus best-practice plan and systemic therapy recommendations for discordant cases. We evaluated the concordance of WfO’s systemic therapeutic recommendations and mMTBs, as well as concordance with the cmMTB. Previous decisions by the fMDTB were also compared to decisions by the cmMTB. Univariate logistic regression explored characteristics predictive of concordance with the cmMTB.

ResultsFor treatment plans, WFO’s therapeutic options had higher concordance with cmMTB decisions than either mMTB alone (concordance 93.7% vs. 92.1%) or the previous decisions by the fMTB (87.3). For systemic therapy decisions, the WfO-cmMDTB concordance was 70.2%; however, adjusting for non-NICE approved drugs and the common practice of Carboplatin use in the UK, concordance increased to 91.5%. Previous decisions by the fMTB had the lowest concordance with the cmMTB (87.3%). Adjusting for the UK-practice related use of Carboplatin, WfO had slightly higher concordance with cmMTB systemic therapy decisions than either mMTB alone (89.4% and 87.2%). Univariate analysis with this limited sample revealed non-significant trends in association between mMTB’s concordance with WfO and stage of cN at diagnosis, HER2 status, tumor location and grade. For example, mMTBs concordance with WfO tended to improve when tumor grade was high. Non-significant trends were also identified in the association between WfO-treatment concordance and tumor location, where treatment concordance increased with medial tumor location.

ConclusionIn this small cohort study, a clinical decision-support tool demonstrated better agreement with UK best practice treatment than a 2-person mMTB and may have a role in triaging breast cancer cases in the UK.

Citation Format: Hartmut Kristeleit, Martha Martin, Christina Karampera, Rezzan Hekmat, Bertha IntHout, Ashutosh Kothari, Majid Kazmi, Amanda Clery, Yanzhong Wang, Bolaji Coker, Winnie Felix, Anita Preininger, Suwei Wang, Roy Vergis, Tom Eggebraaten, Christopher Gloe, Irene Dankwa-Mullan, Gretchen Jackson, Anna Rigg, Danny Ruta. Augmentation of a minimal multidisciplinary tumor board with clinical decision support to triage breast cancer patients in the UK [abstract]. In: Proceedings of the 2020 San Antonio Breast Cancer Virtual Symposium; 2020 Dec 8-11; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2021;81(4 Suppl):Abstract nr PS8-22.

Early-onset colorectal cancer in younger patients: A population-based study

124

Background: In the US, the incidence of colorectal cancer (CRC) is increasing in patients younger than 50 years who may present with advanced stage, high grade, left-sided colon or rectal cancers with signet ring cell histopathology, aggressive clinical course, and reduced overall survival. Understanding the characteristics of this population could inform screening, early detection, and optimal treatment. In this study, we describe the attributes of adults who are 50 years and younger with a first diagnosis of CRC and ascertain molecular testing rates and time to surgery by using data from a commercially insured cohort in the U.S. Methods: This retrospective study of patients ages 50 and younger with a first diagnosis of CRC utilizes the IBM MarketScan database, and focuses on claims from January 2013 to December 2018. Included patients had continuous insurance enrollment of 12 months before and 6 months after diagnosis. We determined rates of tumor testing for microsatellite instability (MSI) or immunohistochemistry (IHC) for mismatch repair (MMR) proteins and referral to genetic services in all patients, as well as mutational analysis of KRAS, NRAS, and BRAF in metastatic CRC patients. Time to surgical resection of primary tumor (TTS) in non-metastatic colon cancer patients was measured. Results: During the 5-year period, 10,577 patients ages 18 to 50 years had a first diagnosis of CRC, which was 15.6% of the 67,921 adults of all ages with CRC. Claims for MSI or IHC for MMR proteins within 120 days of initial diagnosis were done in 4,429 (41.9%) patients and referral to genetics services/counseling within 1 year of initial diagnosis were done in 443 (4.1%) patients. Among metastatic CRC patients, KRAS, NRAS, or BRAF tumor mutational analyses within 120 days of initial diagnosis were documented in 323 (31.5%). The median TTS ranged from 7 to 15 days with no statistically significant differences based on geographic region or health insurance plan type. Conclusions: Younger patients with early onset CRC had low rates of referral to genetics services, tumor MSI or IHC for MMR proteins testing, and KRAS, NRAS, and BRAF mutational analysis. There were no geographic or insurance type trends in TTS in non-metastatic colon cancer patients. Although underreporting is possible in our study, the findings of low utilization of genetic services and tumor genomic testing in these younger patients with early onset CRC should alert the oncology community to critical management gaps in the care of this population.

Factors associated with time to targeted therapy for patients with metastatic lung cancer with driver mutations

120

Background: Targeted therapies are superior to chemotherapy in metastatic lung cancer with driver gene mutations. Delays in initiation of targeted therapies may result in faster symptom progression, decline in quality of life, and shortened survival. We examined factors associated with time to initiation of targeted therapy (TTT) in patients with metastatic lung cancer with selected driver mutations. Methods: In this retrospective cohort study, IBM MarketScan claims data was used to identify patients who had an initial diagnosis of metastatic lung cancer, defined as continuous insurance enrollment 12 months pre- to 6 months post-diagnosis, with tumor biomarker (i.e., EGFR, ALK, ROS1, BRAF V600E, NTRK)-directed targeted therapy performed within 6 months of the initial diagnosis, during the timeframe of 1/1/2013 to 12/31/2018. Trends in TTT were evaluated with Wilcoxon–Mann–Whitney. Quantile regression, a robust model that analyzed factors on different outcome-related quantiles, was used to identify associations among TTT and covariates including age, sex, comorbidity, insurance type, and US region. Results: Among 8977 patients identified with an initial diagnosis of metastatic lung cancer, 710 (7.9%) received targeted therapies within the 6-month timeframe, and 1040 (12%) had tumor biomarker testing performed. The overall median TTT was 21 days (IQR = 36 days). Median TTT decreased from 25 days in 2013 to 18 days in 2018 (p = 0.03). Factors associated with longer TTT (median, 50% quantile) were increasing age (p = 0.04), cardiovascular disease (“CVD”, p = 0.03), HIV (p = 0.04), and mild liver disease (p = 0.05). For the lower quantile ( < = 1 day, 5% quantile), female sex (p = 0.01), HIV (p = 0.04), and mild liver disease (p = 0.002) were associated with longer TTT. Having a PPO health plan extended TTT (p = 0.05) at the upper quantile (79 days, 90% quantile). Conclusions: Our study showed an encouraging 5-year trend of the median TTT decreasing by 28%. Numerous factors associated with longer TTT included increasing age, CVD, HIV, mild liver disease, female sex, and PPO plan. This study provides insights into patient-related factors associated with longer time to initiation of targeted therapies for patients with metastatic lung cancer with driver mutations. Additional research is needed to identify the reasons for longer TTT and to develop strategies to expedite delivery of optimal therapies.

Artificial Intelligence Tool for Optimizing Eligibility Screening for Clinical Trials in a Large Community Cancer Center

PURPOSE

Less than 5% of patients with cancer enroll in clinical trials, and 1 in 5 trials are stopped for poor accrual. We evaluated an automated clinical trial matching system that uses natural language processing to extract patient and trial characteristics from unstructured sources and machine learning to match patients to clinical trials.

PATIENTS AND METHODS

Medical records from 997 patients with breast cancer were assessed for trial eligibility at Highlands Oncology Group between May and August 2016. System and manual attribute extraction and eligibility determinations were compared using the percentage of agreement for 239 patients and 4 trials. Sensitivity and specificity of system-generated eligibility determinations were measured, and the time required for manual review and system-assisted eligibility determinations were compared.

RESULTS

Agreement between system and manual attribute extraction ranged from 64.3% to 94.0%. Agreement between system and manual eligibility determinations was 81%-96%. System eligibility determinations demonstrated specificities between 76% and 99%, with sensitivities between 91% and 95% for 3 trials and 46.7% for the 4th. Manual eligibility screening of 90 patients for 3 trials took 110 minutes; system-assisted eligibility determinations of the same patients for the same trials required 24 minutes.

CONCLUSION

In this study, the clinical trial matching system displayed a promising performance in screening patients with breast cancer for trial eligibility. System-assisted trial eligibility determinations were substantially faster than manual review, and the system reliably excluded ineligible patients for all trials and identified eligible patients for most trials.

Abstract 4341: Regional disparities in time to treatment for breast conserving surgery and mastectomy in women with early-stage breast cancer

BACKGROUND:

Although several studies have reported regional differences in type of surgery performed for early-stage breast cancer (ESBCa), no research has examined geographic trends in time from diagnosis to surgical treatment (TtS), a quality measure for early-stage breast cancer care. Given evidence for improved survival outcomes of expedited TtS, we assessed how TtS for both breast conserving surgery (BCS) and mastectomy (MAST) has changed over time within region.

METHODS:

IBM® MarketScan® claims data were used to select women diagnosed with non-metastatic invasive ESBCa from January 2012 to March 2018. Eligibility criteria included: 1) absence of other cancers 2) ≥ 6 months of continuous insurance enrollment pre- and post- diagnosis 3) treatment with BCS or MAST within 6 months. Days elapsed between diagnosis and first surgery was the dependent variable. Region-specific quantile regression models of median TtS were estimated, which included a vector of patient- and community-level covariates.

RESULTS:

A total of 57,299 women met the inclusion criteria. Among those receiving MAST (n=18,825), 11% had neoadjuvant chemotherapy and 40% had adjuvant chemotherapy. In the BCS cohort (n=38,474), 4% had neoadjuvant chemotherapy, 28% had adjuvant chemotherapy, and 25% had adjuvant radiation therapy. As expected, receipt of neoadjuvant chemotherapy prolonged TtS by 116-128 days (p&lt;0.01). From 2012 to 2017, TtS for MAST significantly increased in the South (3.8 days; p&lt;0.01) and West (8.0 days; p&lt;0.01). Women residing in more urban areas waited longer by 21.9 days in the NE (p=0.01) and 14.2 days in the South (p&lt;0.01). For patients in Black communities, TtS for MAST was greater by 20.7 days (p=0.02) in the Midwest (MW) and 57.8 days in the West (p=0.04). Among women living in more Hispanic areas, median TtS for MAST was lower by 43.4 days (p=0.02) in the Northeast (NE), but was higher by 23.0 days (p&lt;0.01) in the West. In all regions except the NE, TtS for BCS significantly (p&lt;0.01) increased from 2012 to 2017 by between 3.0 (MW) to 6.5 days (West). In more urban areas, women in the NE (14.4 days; p&lt;0.01) and West (20.2 days; p=0.03) had longer TtS. In the NE, women in communities with higher densities of Asians (-19.2 days; p=0.01), Blacks (-20.0 days; p=0.04) and Hispanics (-24.3 days; p=0.05) experienced shorter wait times. However, women from more Hispanic communities in the South had longer TtS for BCS by 6.7 days (p=0.02).

CONCLUSIONS:

Understanding these geographical variations is important to identify potential region-specific and community-level factors influencing time to primary surgery in order to address healthcare inequalities in breast cancer treatment quality. These findings call for policy attention in areas where surgical delays that potentially impact outcomes could be addressed.

Citation Format: Irene Dankwa-Mullan, M Christopher Roebuck, Joseph Tkacz, Judy George, Fredy Reyes, Yull Edwin Arriaga. Regional disparities in time to treatment for breast conserving surgery and mastectomy in women with early-stage breast cancer [abstract]. In: Proceedings of the Annual Meeting of the American Association for Cancer Research 2020; 2020 Apr 27-28 and Jun 22-24. Philadelphia (PA): AACR; Cancer Res 2020;80(16 Suppl):Abstract nr 4341.

A retrospective evaluation of treatment decision making for thyroid cancer using clinical decision support in Brazil

e19193

Background: Artificial intelligence-driven clinical decision-support systems such as Watson for Oncology (WfO) may aid cancer care in economically challenged health systems. Evidence of the applicability of such tools in resource-constrained settings is limited. The study objective was to evaluate treatment agreement between physician-prescribed therapy and WfO recommended treatment options in thyroid cancer in Brazil. An in-depth evaluation of discordant cases by a blinded expert panel of medical oncologists and cancer surgeons was performed to identify preferred therapies and predictors of discordance. Methods: Thyroid cancer patients treated at the Instituto do Câncer do Ceará, Brazil from July 2018 to June 2019, but not processed in WfO, were selected for entry into WfO in January 2020. Blinded to treatment-plan source (i.e., WfO or historical), the expert panel reviewed all WfO therapeutic options and historical physician-prescribed treatment plans for discordant cases and selected their preferred treatment options. Clinical and demographic characteristics were analyzed using logistic regression. Results: Thyroid cancer patients (n = 83) evaluated for concordance between WfO therapeutic options and historical treatments were mostly female (91%) and between the ages of 18 - 78 years (mean 47.7). Concordance between historical physician-prescribed treatment decisions and WfO was 73.5% (61/83). Demographics and clinical characteristics associated with discordance are shown in Table. For all discordant cases (n = 22), preferred treatment decisions, as determined by the expert panel, were in agreement with WfO. Conclusions: High concordance between WfO recommended treatment options and historical treatment decisions for thyroid cancer was observed at Instituto do Câncer do Ceará. For discordant cases, a blinded expert panel agreed with WfO recommended treatment options in all cases, demonstrating there may be a role for decision support in aiding individual oncologists to make best-practice and evidence-informed treatment decisions. [Table: see text]

Disparities in receipt of and time to adjuvant therapy after lumpectomy

534

Background: Adjuvant treatment after breast conserving surgery (BCS) has been shown to improve outcomes, but the degree of uptake varies considerably. We sought to examine factors associated with post-BCS receipt of and time to treatment (TTT) for adjuvant radiation therapy (ART), cytotoxic chemotherapy (ACT) and endocrine therapy (AET) among women with breast cancer. Methods: IBM MarketScan claims data were used to select women diagnosed with non-metastatic invasive breast cancer from 01/01/2012 to 03/31/2018, who received primary BCS without any neoadjuvant therapy, and who had continuous insurance eligibility 60 days post-BCS. Logistic and quantile regressions were used to identify factors associated with receipt of adjuvant therapy (ART, ACT, AET) and median TTT in days for ART (rTTT), ACT (cTTT), and AET (eTTT), respectively, after adjustment for covariates including age, year, region, insurance plan type, comorbidities, and a vector of ZIP3-level measures (e.g., community race/ethnicity-density, education level) from the 2019 Area Health Resource Files. Results: 36,270 patients were identified: 11,996 (33%) received ART only, 4,837 (13%) received ACT only, 3,458 (10 %) received AET only, 5,752 (16%) received both ART and AET, and 9,909 (27%) received no adjuvant therapy within 6 months of BCS. (318) 1% of patients received combinations of either ART, AET or ACT. Relative to having no adjuvant therapy, patients > 80 years were significantly less likely to receive ART only (relative risk ratio [RRR] 0.65), ACT only (RRR 0.05), or combination ART/AET (RRR 0.66) but more likely to receive AET alone (RRR 3.61) (all p < .001). Patients from communities with high proportions of Black (RRR 0.14), Asian (RRR 0.13), or Hispanic (RRR 0.45) residents were significantly less likely to receive combination ART and AET (all p < .001). Having HIV/AIDS (+11 days; p = .01) and residing in highly concentrated Black (+8.5 days; p = .01) and Asian (+12.2 days; p = .04) communities were associated with longer rTTT. Longer cTTT was associated with having comorbidities of cerebrovascular disease (+6.0 days; p < .001), moderate to severe liver disease (+12.3 days; p < .001) and residing in high-density Asian communities (+18.0 days; p < .001). Shorter eTTT (-11.4 days; p = .06) and cTTT (-14.8 days; p < .001) was observed in patients with comorbidities of dementia. Conclusions: Results from this cohort of privately insured patients demonstrate disparities in receipt of post-BCS adjuvant radiation and systemic therapy along multiple demographic dimensions and expose opportunities to promote timely receipt of care.

Associations between concordance with oncology clinical decision support and clinical outcomes in lung cancer patients

e14114

Background: Watson for Oncology (WfO) is an artificial intelligence-based clinical decision-support system which provides therapeutic options and associated scientific evidence to cancer-treating physicians. Oncologists at Bumrungrad International Hospital (BIH) have used WfO since 2015. We examined the association between concordance of WfO therapeutic options and BIH treatment decisions with short-term clinical outcomes for lung cancer patients. Methods: This study included lung cancer patients seen at BIH for treatment and follow-up care and for whom WfO was used from 2015 to 2018. Charts were reviewed for concordance with WfO, documentation of disease progression, response to treatment, and survival. We evaluated concordance between oncologists’ treatments and therapeutic options listed as “recommended” by WfO. We evaluated association between WfO concordance and partial or complete response rates over a 24-month period by comparison of proportions with odds ratio. Progression-free survival (PFS, time from diagnosis until progression or death) was evaluated by Kaplan-Meier log-rank test. Results: Seventy-nine lung cancer patients were included. We identified a trend towards higher response rates in concordant cases (59.2%, N = 32), as compared to discordant (48.0%, N = 12), with an odds ratio of 1.56 (see table). There was not a significant difference in PFS between concordant and discordant cohorts. Conclusions: In this small-cohort, retrospective study, lung cancer patients receiving treatments that are concordant with WfO recommended therapeutic options trended towards higher response rates than patients with discordant treatments. Use of a clinical decision-support system may help support cancer-treating physicians in delivering best practice and evidence-based care that may improve short-term outcomes. Prospective studies with larger samples and other cancer types are underway. [Table: see text]

Biomarker testing patterns and trends among patients with metastatic lung cancer

e13667

Background: Guidelines for biomarker testing of metastatic lung cancer patients aid oncologists in making targeted treatment decisions. Despite evidence demonstrating the benefits of genomic and immune biomarker identification in these patients, variations in testing exist. This population-based, retrospective, observational study examined trends in testing rates and timing, assessing associations between testing and patient characteristics, sociodemographic factors, and regional patterns using insurance claims data. Methods: We evaluated patterns of biomarker testing in the IBM MarketScan database between 1/1/2013-12/31/2018. Inclusion criteria consisted of lung cancer patients with an initial diagnosis of metastasis within the study period, continuous insurance coverage from 12 months before to 4 months post-diagnosis, and biomarker testing (EGFR, ALK, ROS1, BRAF V600E, NTRK, PD-L1) within 4 months of diagnosis. Temporal trends were evaluated by the Cochran-Armitage method. Multivariate logistic regression evaluated associations between testing rates and patient-specific factors (i.e., age, gender, comorbid conditions), insurance type, and region (i.e., Northeastern, North central, Southern, and Western) in the United States (US). Results: Of the 8977 patients with metastatic lung cancer, 1040 (12%) had claims for biomarker testing. During the study period, testing rates increased significantly, from 8.4% in 2013 to 20.6% in 2018 (P <.0001); the likelihood of testing increased by year (2014, OR 1.20, 95% CI 0.97 - 1.48 vs. 2018, OR 2.83, 95% CI 2.26 - 3.54). Of patients tested, 25.8% (N = 268) were tested on the day of diagnosis, 70.7 % (N = 735) within 30 days, and 85.6% (N = 890) within 60 days. A lower likelihood of testing was associated with increasing age (OR = 0.97, 95% CI 0.96 - 0.98), enrollment in preferred provider health plans (OR 0.69, 95% CI 0.53 – 0.93), or pre-existing comorbidities of congestive heart failure (OR 0.76, 95% CI 0.59 – 0.98) or diabetes (OR 0.82, 95% CI 0.68 – 0.99). Testing was more likely to occur in females (OR 1.24, 95% CI 1.09 – 1.42), age < 55 years (OR 1.67, 95% CI 1.32 – 2.12) or residence in Northeastern US (OR 1.26, 95% CI 1.05 -1.51). Conclusions: Biomarker testing rates for an insured cohort of metastatic lung cancer patients increased significantly over time, but the likelihood of testing varied based on age, sex, insurance type, comorbidities, and region. Results of this study may inform policy or outreach strategies by highlighting population-based factors influencing biomarker testing rates.

A review of gynecological cancers studies of concordance with individual clinicians or multidisciplinary tumor boards for an artificial intelligence-based clinical decision-support system

e14070

Background: Watson for Oncology (WfO) is an artificial intelligence-based clinical decision-support system that offers potential therapeutic options to cancer-treating physicians. We reviewed studies of concordance between therapeutic options offered by WfO and treatment decisions made by individual clinicians (IC) and multidisciplinary tumor boards (MTB) in practice in gynecological cancers. Methods: We searched PubMed and an internal database to identify peer-reviewed WfO concordance studies of gynecological cancers published between 01/01/2015 and 06/30/2019. Concordance was defined as agreement between therapeutic options recommended or offered for consideration by WfO and treatment decisions made by IC or MTB. Mean concordance was calculated as a weighted average based on the number of patients per study. Statistical significance was evaluated by z-test of two proportions. Results: Our search identified 5 retrospective studies with 635 patients with cervical and ovarian cancers in China and Thailand; 4 compared WfO to MTB and 1 to IC. Overall WfO concordance with MTB and IC for both cancers was 77.2% (SD 11.6%). The concordance between MTB and WfO in cervical and ovarian cancers was 80.5% and 86.2%, respectively ( P = .21); IC concordance with WfO in cervical and ovarian cancers was 65.2% and 73.2%, respectively ( P = .18). MTB concordance with WfO for both cancers combined was 81.5%, significantly higher than the 67.9% IC concordance with WfO for both cancers ( P = .01). Conclusions: Studies of cervical and ovarian cancers demonstrated a statistically significantly higher concordance of MTB and WfO than IC and WFO, suggesting a role for WfO in supporting treatment-decision making in gynecological cancers that aligns with decisions made by MTB. Larger prospective studies are needed to evaluate the technical performance, usability, workflow integration, and clinical impact of WfO in gynecological cancers.[Table: see text]

Using implementation science to examine impact of a social responsibility agenda on addressing cancer health disparities in Ceará, Brazil

e19071

Background: Programs to address disparities in cancer care outcomes in resource-limited settings require attention to social determinants of health (SDoH) to achieve successful clinical care implementation. The Instituto de Câncer do Ceará, the largest cancer center in northeastern Brazil, has implemented a Social Responsibility Agenda (SRA) to guide equitable cancer care delivery. This goal of this study was to develop a framework for an implementation science (IS) study evaluating the longitudinal impact of the SRA on cancer outcomes. Methods: We outlined a mixed-methods and participatory study incorporating a process model, the Consolidated Framework for Implementation Research (CFIR) and the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) evaluation framework. A list of constructs and links to measurement tools associated with IS models were identified to guide the study phases. Results: We established a logic model to guide in evaluating the health and economic impact of the SRA. We identified >30 constructs and measures across domains of IS models. The table shows a driver diagram to inform the framework. Conclusions: Understanding determinants, key drivers and change concepts are important initial steps in an ongoing evaluation of the impact of evidence based SDoH interventions to address cancer disparities. [Table: see text]

Evaluation of an artificial intelligence clinical trial matching system in Australian lung cancer patients

Objective

The objective of this technical study was to evaluate the performance of an artificial intelligence (AI)-based system for clinical trials matching for a cohort of lung cancer patients in an Australian cancer hospital.

Methods

A lung cancer cohort was derived from clinical data from patients attending an Australian cancer hospital. Ten phases I–III clinical trials registered on clinicaltrials.gov and open to lung cancer patients at this institution were utilized for assessments. The trial matching system performance was compared to a gold standard established by clinician consensus for trial eligibility.

Results

The study included 102 lung cancer patients. The trial matching system evaluated 7252 patient attributes (per patient median 74, range 53–100) against 11 467 individual trial eligibility criteria (per trial median 597, range 243–4132). Median time for the system to run a query and return results was 15.5 s (range 7.2–37.8). In establishing the gold standard, clinician interrater agreement was high (Cohen’s kappa 0.70–1.00). On a per-patient basis, the performance of the trial matching system for eligibility was as follows: accuracy, 91.6%; recall (sensitivity), 83.3%; precision (positive predictive value), 76.5%; negative predictive value, 95.7%; and specificity, 93.8%.

Discussion and Conclusion

The AI-based clinical trial matching system allows efficient and reliable screening of cancer patients for clinical trials with 95.7% accuracy for exclusion and 91.6% accuracy for overall eligibility assessment; however, clinician input and oversight are still required. The automated system demonstrates promise as a clinical decision support tool to prescreen a large patient cohort to identify subjects suitable for further assessment.

Systematic review of gastrointestinal cancer studies of concordance with expert opinion for a clinical decision support system (CDSS)

250

Background: Watson for Oncology (WfO), a cognitive CDSS, provides therapeutic options to cancer-treating physicians. We reviewed the concordance of WfO therapeutic options in gastrointestinal cancers with experts’ treatment decisions. Methods: Systematic review to identify WfO concordance studies in gastrointestinal cancers, published from June 2015 to June 2019. Concordance was defined as agreement between WfO “Recommended” and “For Consideration” treatment options and decisions made by experts. Mean concordance rates were calculated as an average, weighted by the number of patients in each study. Results: 2,407 patients were identified (Table). Overall treatment decision concordance was 67.2% (SD 25.7%). Concordance for rectal, colon, hepatocellular, and gastric cancers were 90.5% (SD 9.4%), 80.9% (SD 24.3%), 58.5%, and 47.5% (SD 33.9%), respectively. Concordance with WfO were significantly higher for rectal versus colon cancer ( p = .001), rectal versus gastric cancer ( p < .0001) and for colon versus gastric cancer ( p <.0001). Conclusions: Concordance between WfO and treatment decisions by experts for rectal and colon cancers were high. Concordance for HCC and gastric cancer were the lowest. A higher discordance in gastric cancer is likely related to disease-specific and management differences compared to United States practice. Variable concordance between expert clinical decisions and CDSS suggestions and can be minimized by localization efforts.

[Table: see text]

Concordance between a clinical decision-support system and treatments selected by clinicians as a function of cancer type or stage

95

Background: Watson for Oncology (WFO) is an artificial intelligence (AI) based clinical decision-support tool trained by Memorial Sloan Kettering. This retrospective observational study of breast, lung, colon and rectal cancer examined the concordance of treatment options provided by WFO to treatments selected by clinicians at Bumrungrad International Hospital (BIH) as a function of stage or cancer type. Methods: Concordance between WFO treatment options and treatments selected by BIH clinicians (WFO-BIH concordance) was defined as identical or equally acceptable treatments, as determined by a panel of experts blinded to the source of treatment. Relationships between stage or type of cancer and WFO-BIH concordant treatments were evaluated by Chi-squared analysis. Results: Analysis revealed a statistically significant association ( P = 0.02) between cancer stage and concordance. For all 4 cancer types combined, stages I-III demonstrated higher concordance than stage IV. A highly significant association ( P < 0.001) between concordance and cancer type was identified. Colon cancer demonstrated the highest concordance, followed by rectal, lung and breast cancer. Reasons for discordance, when given, related to oncologist or patient preferences, and treatment availability. Conclusions: BIH clinicians tended to agree more with WFO therapeutic options for stage I-III cancers and colon cancer in general, as compared to relatively less agreement for stage IV cancers and breast cancer in general, suggesting the need to understand reasons for discordance among all cancer types and stages. An AI tool, trained by experts in the U.S., provides treatment options consistent with some therapies selected in international settings, but preferences and treatment availability may affect choices made in practice. [Table: see text]

Reasons for discordance in treatment approaches between oncology practice and clinical decision support in China

6555

Background: Therapeutic clinical decision-support systems (CDSS) are often evaluated by comparisons between CDSS options and actual practice decisions or expert opinions. Few such studies have carefully examined reasons for discordance. Methods: We reviewed 11 concordance studies from different hospitals across 8 provinces in China, published between 2017 and 2018. The studies compared IBM Watson for Oncology (WfO) therapeutic options to treatments selected by oncologists or a tumor board involved in review of cases for lung, colon, rectal, breast, gastric, and gynecological cancers. We identified given reasons for discordance and summarized themes across studies. Results: Of the 11 studies, 9 provided 1 or more reasons for discordance which could be analyzed. We found three major themes related to discordance: formulary restrictions, treatment-protocol differences, and physician or patient preferences (Table). Formulary differences between WfO and regional practices included off-label drug uses or unavailable therapies. Treatment-protocol differences included variations in regimens, such as simultaneous versus sequential treatments. Physician or patient preferences included factors such as the cost of treatment and logistics associated with various treatments. Conclusions: This study identified multiple reasons for discordance between an oncology CDSS option and oncologists’ treatment choices in China. Treatment differences arose from local formulary or protocol differences as well as provider and patient preferences. Future studies of CDSS should include reasons for discordance when assessing system performance in this manner. [Table: see text]

A blinded evaluation of a clinical decision-support system at a regional cancer care center

6553

Background: Clinical decision-support systems (CDSS) such as Watson for Oncology (WFO) may reduce treatment variation in oncology, provided options offered by the system are at least as acceptable as expert, evidence-based options. Deviation from expert consensus in practice is not well documented. In this blinded study, WFO therapeutic options and treatment decisions made by individual oncologists at Bumrungrad International Hospital (BIH) were evaluated by expert panel. Methods: Treatments selected by BIH that were labeled as either “for consideration” or “not recommended” by WFO were evaluated by a panel of 3 oncologists in 2018. The panel evaluated WFO options and previous BIH treatments for prospective cases from 2016-2018, blinded to the source of treatment option. Consensus of panel rated treatment pairs as: identical; both acceptable and roughly equivalent; both acceptable, but one preferred; one is acceptable and the other, unacceptable; neither is acceptable. The results of 321 treatment choices for breast, lung, colon and rectal cancers were analyzed, and McNemar’s test, a modified pairwise chi-square, was applied to identify differences between BIH and WFO. Results: 71% of both BIH and WFO treatments across all 4 cancer types were considered acceptable or identical by the panel. In 18 cases (5.6%), WFO treatments were preferred; in 14 cases (4.4%), BIH cases were preferred. Unacceptable treatments by either BIH or WFO were identified in 15% and 23% of treatments, respectively. Statistical analysis of treatment pairs revealed no significant difference between BIH and WFO treatments for breast, colon and rectal cancer. Treatment for lung cancer differed significantly ( p = 0.004); in 6% of cases, WFO was unacceptable and BIH acceptable; in 1% of cases, BIH was unacceptable and WfO was acceptable. Conclusions: This study is one of the first to compare therapeutic options from CDSS to treatment decisions made in practice, evaluated in a blinded fashion by an expert panel. 71% of treatments suggested by WFO CDSS were as acceptable as those selected by clinicians at the point of care, and some were considered superior. Decisions made in practice were unacceptable to the panel in 15% of cases, suggesting a role for CDSS.

Shared-decision making in prostate cancer with clinical decision-support

e16576

Background: Shared decision-making is the process of deliberately interacting with patients who wish to make informed value-based choices, when there are no indicated best treatment options. Given the wide variation in prostate cancer treatment options, clinical decision-support systems (CDSS) may effectively support treatment decisions for patients with challenging risk-benefit profiles. However, limited data are available regarding CDSS in shared decision making. This study aimed to assess the alignment of CDSS therapeutic options with treatment received through a shared decision process. Methods: We identified patients with prostate cancer (Gleason Groups 1-5) who were engaged in shared treatment decision making, (from August–September 2018) at the Instituto do Câncer do Ceará, Brazil. IBM Watson for Oncology (WfO), a CDSS was used for the study. Treatment decisions were compared with WfO options (active surveillance, clinical trial, chemotherapy [CT], hormone therapy [HT], radiation [RT], brachytherapy [brachy], surgery and systemic therapy with GnRH suppression) and categorized as concordant (equivalent), partially concordant (a partial match), or discordant. Results: Concordance between WfO and shared treatment decisions was observed in 54% (26/48) of patients, partial concordance in 15% (7/48) and discordance in 31% (15/48). Most frequent treatments were RT+HT combination therapy (25%) and prostatectomy (21%). 8/15 (53%) discordant cases were due to patient preference for treatment over active surveillance. Patient preference for treatment over active surveillance was the most common reason (53%) for discordance. Conclusions: Variation in prostate cancer treatment exists. CDSS therapy options may be useful in quantifying and modifying unwarranted variations in prostate cancer treatment. Future studies are important for understanding reasons for variations. [Table: see text]

Employing an oncology decision-support system to quantify treatment variation

e18067

Background: Factors affecting cancer treatment may include evidence for effectiveness, cost, and preference. These influences can lead to treatment variation across institutions and populations. Decision-support systems have been proposed as tools to reduce variation. This study quantified concordance between treatment provided by oncologists in China and therapeutic options presented by a decision-support tool. Methods: We identified and analyzed concordance studies in nine unique institutions located in seven provinces in China, published in 2017-2018 using Watson for Oncology (WFO), a clinical decision-support tool. Published rates of concordance were compared by cancer type and institution. Results: Concordance of all combined cases was 59% (2012/3388). Concordance rates varied by cancer type and institution (Table). Concordance rates were highest for ovarian (96%), rectal (94%) and breast (89%) cancers but lowest in gastric (12%), ovarian (43%) and breast (55%) cancers. Conclusions: Concordance between treatments and therapeutic options from an oncology decision-support tool varied significantly across cancer types and institutions in China, suggesting significant practice variation. Without established guidelines for treatment, clinical decisions may be influenced by preferences and local factors. Future studies are needed to identify reasons for variation and improve adherence to regional evidence-based guidelines. [Table: see text]

Clinical insights for hematological malignancies from an artificial intelligence decision-support tool

e13023

Background: Next generation sequencing (NGS) in hematological tumors is increasingly shaping clinical treatment decisions at the point of care. While the impact of NGS panels in solid tumors is largely therapeutic, targeted sequencing in hematological tumors can additionally provide diagnostic and prognostic insights. Additional data generated in hematological tumor sequencing makes manual interpretation and annotation of variants tedious and non-scalable. In this study we compared hematological tumor variant interpretation using an artificial intelligence decision-support system, Watsonä for Genomics (WfG), with expert guided manual curation. Methods: Patients with hematological tumors at Hallym University, College of Medicine between December 2017 and December 2018, were sequenced using the 54 gene Illumina TruSight Myeloid Panel. WfG interpreted and annotated all patients’ sequencing results, a subset of which were assessed manually to ascertain concordance. Results: 54 South Korean patients with hematological malignancies were analyzed (23 Acute Myeloid Leukemia, 12 myeloproliferative neoplasm, 5 myelodysplastic syndrome, 5 multiple myeloma and 9 others). Comparison of manual and WfG interpretation of 10 randomly selected cases yielded 90% (9/10) concordance and identification of 9 clinically actionable variants (33%) not found in manual interpretation. In total, WfG identified that 71% (38/54) of all cases had at least one clinically actionable therapeutic alteration (a variant targeted by a US FDA approved drug, off-label drug, or clinical trial). 33% (18/54) of cases had genes that were targeted by a US FDA approved therapy including JAK2, IDH1, IDH2, and FLT3. In cases without therapeutic alterations, WfG identified diagnostic or prognostic insights in an additional 20% (11/54) of patients. 9% (5/54) had no clinically actionable information. Conclusions: WfG variant interpretation correlated well with manually curated expert opinion and identified clinically actionable insights missed by manual interpretation. WfG has obviated the need for labor-intensive manual curation of clinical trials and therapy, enabling our center to exponentially scale our NGS operations.

Deployment and integration of a cognitive technology in China: Experiences and lessons learned

6538

Background: Cognitive technologies are rapidly being introduced in oncology for decision-support, prescribing therapy, predicting risk, reducing medical errors and for care management. Few studies have reported on successful approaches for clinical adoption. We report the early adopter experience of BahealIntelligenceTechnologyCo., Ltd. (Baheal), across China within a 2-year period (April 2017-January 2019). We also describe lessons and experiences of oncology users. Methods: Baheal developed collaborative agreements for use of IBM Watson for Oncology (WFO) in 96 hospitals across 8 provinces. Key opinion leaders who saw the potential for AI were recruited as champion advocates. A 29-item survey conducted included usability and integration within clinical workflow. 85 questionnaires were distributed to oncologists who were major WfO users; 51 were completed. All questionnaires were completed anonymously and de-identified prior to analysis. Results: As of January 31, 2019, 866 physicians have entered a total of 52,537 cancer cases into WfO. Most users approved of both the quality (44/51, 86.3%) and comprehensibility (45/51, 88.2%) of treatment options, rationales, and literature references. WfO was most frequently applied in the context of inpatient cases reviewed by a multidisciplinary tumor board (MDT) (44/51, 86.3%). A lack of locally available treatments in WfO was cited as an area for improvement by two-thirds of users (34/51, 66.7%). CDS was rated from 0 (lowest) to 10 (highest) for each of the following uses: EBM medical education (8.1); assistance with literature (7.7); medical care quality control (7.3); second opinion consultations (7.0); case review with tumor board (6.9); and decision support (6.4). Overall, users were willing to recommend CDS to patients and other clinicians (7.3). Conclusions: WfO CDS, employed in a variety of settings, was viewed positively by more than 86% of users, with perceived benefits differing by context. Future incorporation of locally available treatments and understanding reasons for their omission in CDS may increase perceived value, improve standardization, quality of cancer care and equity of care in China.

An evaluation of artificial intelligence-based clinical decision supports use in Brazil

e18081

Background: The Instituto do Câncer do Ceará (ICC), a 160-bed oncology hospital located in Brazil, serves approximately 23,000 patients monthly. In December of 2017, ICC implemented Watson for Oncology (WFO), an artificial intelligence (AI)-based clinical decision-support (CDS) tool to help enhance personalized cancer care. As of December 2018, 903 cases involving mainly breast, prostate and gastric cancers were entered in WFO. The purpose of this study was to investigate how implementation of WFO and use by oncologists affects clinical decision-making and workflow. Methods: 7 oncologists who employed WfO during and after the patients’ first visit were recruited to complete a survey regarding usability, decision-making and workflow. The group consisted of 1 urologist, 3 gastric surgeons, 1 gynecologist, 1 breast surgeon, 1 head-neck surgeon. Survey questions integrated the CDS Five Rights framework. Results: Most oncologists agreed that WFO is easy to understand and provides complete, relevant and actionable information at an appropriate time (Table). Opinions on the impact on treatment decisions varied. 71.4% expressed positive statements (agree or strongly agree) pertaining to the use of WFO. Conclusions: In this study, oncologists felt WFO met 5 Rights expectations for CDS; 57% felt that WFO exceed expectations. Further research is needed to understand how variation in experience affects decision impact. [Table: see text]

Impact of decision-support system and guideline treatment concordance on response rate in advanced lung cancer

e20006

Background: Clinical decision-support systems (CDSS) provide up-to-date evidence to practitioners overwhelmed by the deluge of clinical findings. Few studies, however, have evaluated their impact on patient outcomes. This cross-sectional retrospective study was conducted to: 1) measure concordance of real-world clinical decisions with therapeutic options from the IBM Watson for Oncology (WfO) CDSS and the Chinese Society of Clinical Oncology (CSCO) guidelines, 2) the effect of concordance on objective response rate (ORR). Methods: Health records from patients receiving 1stline treatment at Peking University International Hospital Oncology Center in China between January 2016 and December 2018 (69 stage IV NSCLC, 30 with SCLC) with documented tumor progression after 2+ cycles of treatment, were reviewed to determine concordance of actual treatment with WfO therapeutic options and CSCO guidelines. Patients’ treatments were grouped as concordant with: WfO+CSCO, WFO only, CSCO only, or neither. ORR, defined as partial or complete response after 2+ treatment cycles (RECIST criteria)was determined for each group. Results: For NSCLC, ORR ranged from 21.4% for discordance with both WfO and CSCO to 100.0% for WfO only concordance. For SCLC, ORR ranged from 37.5% for CSCO only concordance to 73.3% for WfO+CSCO concordance (Table). The main reasons for discordance were: 1stgeneration TKIs like Gefitinib and Erlotinib (vs. Osimertinib) are standard of care for EGFR mutant patients in China, (2) Local CSCO guideline drugs like Lobaplatin and Icotinib are not included in WfO. Conclusions: This study provides preliminary evidence to suggest that treatment concordance with WfO may be associated with improved ORR in some cases of NSCLC (WfO only) and SCLC (WfO + CSCO). ORR in NSCLC patients who were discordant with both WFO and CSCO guidelines was the lowest at 21.4%. Larger studies are needed to understand the effect of guideline and WfO concordance on ORR. [Table: see text]

Enhancing evidence-based medicine skills in oncology training with cognitive technology

10532

Background: Evidence-based medicine (EBM) requires applying literature evidence to inform practice. Students from Taipei Medical University Hospital, trained in EBM concepts, participated in a preliminary study using Watson for Oncology (WfO), an evidence-based decision-support system to enhance the EBM skills of medical students. Methods: A class of 50 medical students compared traditional search methods (TSM) and WfO in a workshop divided into 2 sequential sessions on colon and lung cancer, respectively. All students were trained on WfO, and 2 groups of 25 students each were randomly assigned to either TSM or WfO in the first session. Those groups were then assigned to the alternate approach in the second session. Students completed a profile that included their clinical experience with each cancer type. Students used either WfO or TSM to help answer a series of questions related to colon or lung cancer. Students then completed a survey of attitudes towards the technology, followed by a constructed-response learning assessment without the aid of TSM or WfO. Assessments were scored and results compared using a Mann-Whitney U Test; outcomes at two different experience levels, based on student profiles, were compared using a Kruskal-Wallis test. Results: In this preliminary study, more than 70% of students reported limited clinical experience with either cancer. On the colon cancer assessment, students in the WfO group performed significantly better than the TSM group ( p = 0.0001); there was no significant difference detected for lung cancer. Students with more clinical experience felt that TSM was easier to learn than WfO ( p= 0.005); students with less experience felt that WfO was clearer and more understandable than TSM ( p= 0.002). Conclusions: These preliminary results are consistent with better learning outcomes for students using WfO in the colon cancer module. Students with more clinic experience reported that TSM was easier to learn than WfO, however it is unknown if this might be due to a potentially greater familiarity with TSM in this more experienced group. More studies are needed to determine what features, if any, of WfO can facilitate EBM approaches in oncology education.

Harnessing AI for health equity in oncology research and practice

67

Background: Recent advances in artificial intelligence (AI) carry underexplored practical and ethical implications for the practice of clinical oncology. As oncologic applications of AI proliferate, a framework for guiding their ethical implementations and equitable distribution will be crucial. Methods: We reviewed the current landscape of AI applications in oncology research and clinical practice by reviewing the current body of evidence in PubMed and Medline. Key ethical challenges and opportunities to address health equity are critically evaluated and highlighted. Ethical implications for patients, clinicians and society at large are delineated, with particular focus on the impact and ramifications of AI with respect to healthcare disparities and equity of oncology care delivery. Results: Growing concerns that AI may widen disparities in oncologic care by virtue of lack of affordability, inconsistent accessibility and biased machine-learning models are addressed. Although there is potential for AI to widen disparities in oncology care, using foresight in application, AI has the potential to (1) democratize access to specialized clinical knowledge, (2) improve the accuracy of predicting cancer susceptibility, recurrence and mortality, (3) prevent diagnostic errors in under-resourced settings, (4) minimize unintended bias and (5) enable access to tailored therapeutic options including clinical trials if appropriately deployed. Separately, AI can be harnessed to identify areas of underserved needs and optimize systems of health-information sharing and reimbursements as blockchain technology converges with AI. As AI advances it will have a larger presence in oncology research and clinical practice. Conclusions: A strategic framework integrating ethical standards and emphasizing equitable implementation can help ensure that the potential of AI to address disparities in oncology are maximally captured and its perils averted. Further work is being done on exploring these challenges and will be submitted as a manuscript.

Transforming Diabetes Care Through Artificial Intelligence: The Future Is Here

An estimated 425 million people globally have diabetes, accounting for 12% of the world's health expenditures, and yet 1 in 2 persons remain undiagnosed and untreated. Applications of artificial intelligence (AI) and cognitive computing offer promise in diabetes care. The purpose of this article is to better understand what AI advances may be relevant today to persons with diabetes (PWDs), their clinicians, family, and caregivers. The authors conducted a predefined, online PubMed search of publicly available sources of information from 2009 onward using the search terms "diabetes" and "artificial intelligence." The study included clinically-relevant, high-impact articles, and excluded articles whose purpose was technical in nature. A total of 450 published diabetes and AI articles met the inclusion criteria. The studies represent a diverse and complex set of innovative approaches that aim to transform diabetes care in 4 main areas: automated retinal screening, clinical decision support, predictive population risk stratification, and patient self-management tools. Many of these new AI-powered retinal imaging systems, predictive modeling programs, glucose sensors, insulin pumps, smartphone applications, and other decision-support aids are on the market today with more on the way. AI applications have the potential to transform diabetes care and help millions of PWDs to achieve better blood glucose control, reduce hypoglycemic episodes, and reduce diabetes comorbidities and complications. AI applications offer greater accuracy, efficiency, ease of use, and satisfaction for PWDs, their clinicians, family, and caregivers.

Concordance assessment of a cognitive computing system in Thailand

6589

Background: IBM Watson for Oncology (WFO) was trained by Memorial Sloan Kettering and is a cognitive computing system that uses natural language processing to ingest patient data in structured and unstructured formats. The system provides physicians with treatment options that are derived from established guidelines, the medical literature, and training from patient cases. In this study, we assessed the degree of concordance between treatment recommendations proposed by WFO and oncologists at Bumrungrad International Hospital (BIH). BIH is a 580-bed multispecialty hospital in Bangkok, Thailand. Methods: Data from breast, colorectal, gastric, and lung cancer patients treated at BIH were entered into WFO in 2015 and 2016. Retrospective cases were entered after a treatment plan had been determined, and prospective cases were entered during patients’ treatment planning sessions. WFO recommendations were provided in 3 categories: “Recommended”, “For Consideration”, and “Not Recommended.” Concordance was analyzed by comparing the decisions made by the oncologists to those proposed by WFO. Concordance was achieved when the oncologist’s treatment suggestion was in the “Recommended” or “For Consideration” categories given by WFO. Results: A total of 211 cases were assessed, 92 were retrospective and 119 were prospective. The overall concordance rate was 83%; 89% for colorectal, 91% for lung, 76% for breast, and 78% for gastric cancer. Similar concordance rates were observed when retrospective and prospective cases were analyzed separately. Discordance was attributable in part to local oncologists’ preferences for non-U.S. guidelines for certain cancers, especially gastric cancer. Conclusions: There was a high degree of concordance between WFO treatment options and the decisions made by local oncologists. Similar results were recently reported in a breast cancer concordance study conducted using WFO in India (San Antonio Breast Cancer Symposium 2016, Somashekhar et al). WFO’s capabilities as a cognitive decision support tool can be further improved by incorporating regional guidelines. Future work will analyze reasons for discordance such as cost, insurance requirements, and patient and physician preference.

Use of a cognitive computing system for treatment of colon and gastric cancer in South Korea

e18204

Background: IBM Watson for Oncology (WFO) is a Memorial Sloan Kettering-trained cognitive computing system that provides oncologists with evidence-based treatment options for cancer. Treatments are presented in three categories: “Recommended”, “For Consideration” and “Not Recommended”. We examined the concordance of treatment options between WFO and the tumor board from Gachon University Gil Medical Centre (GMC), Incheon, South Korea. GMC is an urban center that cares for 50,000 cancer patients annually. Methods: We enrolled 340 patients with stage II, III and IV colon cancer and 185 with chemotherapy-naïve advanced gastric cancer, all treated between 2012 and 2016. Cases were processed using WFO, and the output was compared to blinded tumor board recommendations. Treatment options were considered concordant when the GMC recommendation was included in the “Recommended” or “For Consideration” categories. Results: Treatment recommendations were concordant in 248 (73%) of the 340 evaluated colon cancer cases. Of 250 patients treated in the adjuvant setting, 212 (85%) were concordant. Of 90 patients with metastatic disease, 36 (40%) were concordant. Treatment recommendations were concordant in 90 (49%) of 185 chemotherapy-naïve gastric cancer patients. Low concordance rates in gastric cancer were explained by two observations: (1) The trastzumab/FOLFOX regimen is not covered by the Korean National Health Insurance System, and (2) A regimen known as S-1 (tegafur, gimeracil, and oteracil) plus cisplatin is routinely used in Korea and is not used in the U.S. Conclusions: Treatment options suggested by WFO were concordant with the therapeutic decisions of GMC in the large majority of colon cancer patients treated in the adjuvant setting. Lower degrees of concordance were seen in patients with metastatic colon and gastric cancer, reflecting differences in practice patterns between the United States, where WFO was trained, and GMC, in Korea. Geography-specific customization is available in WFO and should enable physicians and patients to benefit from WFO worldwide. WFO's ability to learn from gastric cancer cases in a part of the world with increased incidence may reveal insights that are applicable elsewhere.