Efficient and Accurate Extracting of Unstructured EHRs on Cancer Therapy Responses for the Development of RECIST Natural Language Processing Tools: Part I, the Corpus

Framework for the Use of External Controls to Evaluate Treatment Outcomes in Precision Oncology Trials

Advances in genomics have enabled anticancer therapies to be tailored to target specific genomic alterations. Single-arm trials (SATs), including those incorporated within umbrella, basket, and platform trials, are widely adopted when it is not feasible to conduct randomized controlled trials in rare biomarker-defined subpopulations. External controls (ECs), defined as control arm data derived outside the clinical trial, have gained renewed interest as a strategy to supplement evidence generated from SATs to allow comparative analysis. There are increasing examples demonstrating the application of EC in precision oncology trials. The prospective application of EC in conducting comparative studies is associated with distinct methodological challenges, the specific considerations for EC use in biomarker-defined subpopulations have not been adequately discussed, and a formal framework is yet to be established. In this review, we present a framework for conducting a prospective comparative analysis using EC. Key steps are (1) defining the purpose of using EC to address the study question, (2) determining if the external data are fit for purpose, (3) developing a transparent study protocol and a statistical analysis plan, and (iv) interpreting results and drawing conclusions on the basis of a prespecified hypothesis. We specify the considerations required for the biomarker-defined subpopulations, which include (1) specifying the comparator and biomarker status of the comparator group, (2) defining lines of treatment, (3) assessment of the biomarker testing panels used, and (4) assessment of cohort stratification in tumor-agnostic studies. We further discuss novel clinical trial designs and statistical techniques leveraging EC to propose future directions to advance evidence generation and facilitate drug development in precision oncology.

Developing Artificial Intelligence Models for Extracting Oncologic Outcomes from Japanese Electronic Health Records

INTRODUCTION

A framework that extracts oncological outcomes from large-scale databases using artificial intelligence (AI) is not well established. Thus, we aimed to develop AI models to extract outcomes in patients with lung cancer using unstructured text data from electronic health records of multiple hospitals.

METHODS

We constructed AI models (Bidirectional Encoder Representations from Transformers [BERT], Naïve Bayes, and Longformer) for tumor evaluation using the University of Miyazaki Hospital (UMH) database. This data included both structured and unstructured data from progress notes, radiology reports, and discharge summaries. The BERT model was applied to the Life Data Initiative (LDI) data set of six hospitals. Study outcomes included the performance of AI models and time to progression of disease (TTP) for each line of treatment based on the treatment response extracted by AI models.

RESULTS

For the UMH data set, the BERT model exhibited higher precision accuracy compared to the Naïve Bayes or the Longformer models, respectively (precision [0.42 vs. 0.47 or 0.22], recall [0.63 vs. 0.46 or 0.33] and F1 scores [0.50 vs. 0.46 or 0.27]). When this BERT model was applied to LDI data, prediction accuracy remained quite similar. The Kaplan-Meier plots of TTP (months) showed similar trends for the first (median 14.9 [95% confidence interval 11.5, 21.1] and 16.8 [12.6, 21.8]), the second (7.8 [6.7, 10.7] and 7.8 [6.7, 10.7]), and the later lines of treatment for the predicted data by the BERT model and the manually curated data.

CONCLUSION

We developed AI models to extract treatment responses in patients with lung cancer using a large EHR database; however, the model requires further improvement.

Real-world treatment response in Japanese patients with cancer using unstructured data from electronic health records

Purpose

We generated methods for evaluating clinical outcomes including treatment response in oncology using the unstructured data from electronic health records (EHR) in Japanese language.

Methods

This retrospective analysis used medical record database and administrative data of University of Miyazaki Hospital in Japan of patients with lung/breast cancer. Treatment response (objective response [OR], stable disease [SD] or progressive disease [PD]) was adjudicated by two evaluators using clinicians’ progress notes, radiology reports and pathological reports of 15 patients with lung cancer (training data set). For assessing key terms to describe treatment response, natural language processing (NLP) rules were created from the texts identified by the evaluators and broken down by morphological analysis. The NLP rules were applied for assessing data of other 70 lung cancer and 30 breast cancer patients, who were not adjudicated, to examine if any difference in using key terms exist between these patients.

Results

A total of 2,039 records in progress notes, 131 in radiology reports and 60 in pathological reports of 15 patients, were adjudicated. Progress notes were the most common primary source data for treatment assessment (60.7%), wherein, the most common key terms with high sensitivity and specificity to describe OR were “reduction/shrink”, for SD were “(no) remarkable change/(no) aggravation)” and for PD were “(limited) effect” and “enlargement/grow”. These key terms were also found in other larger cohorts of 70 patients with lung cancer and 30 patients with breast cancer.

Conclusion

This study demonstrated that assessing response to anticancer therapy using Japanese EHRs is feasible by interpreting progress notes, radiology reports and Japanese key terms using NLP.

Correlation Between Surrogate End Points and Overall Survival in a Multi-institutional Clinicogenomic Cohort of Patients With Non-Small Cell Lung or Colorectal Cancer

IMPORTANCE

Contemporary observational cancer research requires associating genomic biomarkers with reproducible end points; overall survival (OS) is a key end point, but interpretation can be challenging when multiple lines of therapy and prolonged survival are common. Progression-free survival (PFS), time to treatment discontinuation (TTD), and time to next treatment (TTNT) are alternative end points, but their utility as surrogates for OS in real-world clinicogenomic data sets has not been well characterized.

OBJECTIVE

To measure correlations between candidate surrogate end points and OS in a multi-institutional clinicogenomic data set.

DESIGN, SETTING, AND PARTICIPANTS

A retrospective cohort study was conducted of patients with non-small cell lung cancer (NSCLC) or colorectal cancer (CRC) whose tumors were genotyped at 4 academic centers from January 1, 2014, to December 31, 2017, and who initiated systemic therapy for advanced disease. Patients were followed up through August 31, 2020 (NSCLC), and October 31, 2020 (CRC). Statistical analyses were conducted on January 5, 2021.

EXPOSURES

Candidate surrogate end points included TTD; TTNT; PFS based on imaging reports only; PFS based on medical oncologist ascertainment only; PFS based on either imaging or medical oncologist ascertainment, whichever came first; and PFS defined by a requirement that both imaging and medical oncologist ascertainment have indicated progression.

MAIN OUTCOMES AND MEASURES

The primary outcome was the correlation between candidate surrogate end points and OS.

RESULTS

There were 1161 patients with NSCLC (648 women [55.8%]; mean [SD] age, 63 [11] years) and 1150 with CRC (647 men [56.3%]; mean [SD] age, 54 [12] years) identified for analysis. Progression-free survival based on both imaging and medical oncologist documentation was most correlated with OS (NSCLC: ρ = 0.76; 95% CI, 0.73-0.79; CRC: ρ = 0.73; 95% CI, 0.69-0.75). Time to treatment discontinuation was least associated with OS (NSCLC: ρ = 0.45; 95% CI, 0.40-0.50; CRC: ρ = 0.13; 95% CI, 0.06-0.19). Time to next treatment was modestly associated with OS (NSCLC: ρ = 0.60; 0.55-0.64; CRC: ρ = 0.39; 95% CI, 0.32-0.46).

CONCLUSIONS AND RELEVANCE

This cohort study suggests that PFS based on both a radiologist and a treating oncologist determining that a progression event has occurred was the surrogate end point most highly correlated with OS for analysis of observational clinicogenomic data.