Automated Electronic Health Record-Based Tool for Identification of Patients With Metastatic Disease to Facilitate Clinical Trial Patient Ascertainment

PURPOSE

To facilitate identification of clinical trial participation candidates, we developed a machine learning tool that automates the determination of a patient's metastatic status, on the basis of unstructured electronic health record (EHR) data.

METHODS

This tool scans EHR documents, extracting text snippet features surrounding key words (such as metastatic, progression, and local). A regularized logistic regression model was trained and used to classify patients across five metastatic categories: highly likely and likely positive, highly likely and likely negative, and unknown. Using a real-world oncology database of patients with solid tumors with manually abstracted information as reference, we calculated sensitivity, specificity, negative predictive value (NPV), and positive predictive value (PPV). We validated the performance in a real-world data set, evaluating accuracy gains upon additional user review of tool's outputs after integration into clinic workflows.

RESULTS

In the training data set (N = 66,532), the model sensitivity and specificity (% [95% CI]) were 82.4 [81.9 to 83.0] and 95.5 [95.3 to 96.7], respectively; the PPV was 89.3 [88.8 to 90.0], and the NPV was 94.0 [93.8 to 94.2]. In the validation sample (n = 200 from five distinct care sites), after user review of model outputs, values increased to 97.1 [85.1 to 99.9] for sensitivity, 98.2 [94.8 to 99.6] for specificity, 91.9 [78.1 to 98.3] for PPV, and 99.4 [96.6 to 100.0] for NPV. The model assigned 163 of 200 patients to the highly likely categories. The error prevalence was 4% before and 2% after user review.

CONCLUSION

This tool infers metastatic status from unstructured EHR data with high accuracy and high confidence in more than 75% of cases, without requiring additional manual review. By enabling efficient characterization of metastatic status, this tool could mitigate a key barrier for patient ascertainment and clinical trial participation in community clinics.

An automated EHR-based tool to facilitate patient identification for biomarker-driven trials

1539

Background: Clinical trial eligibility increasingly requires information found in NGS tests; lack of structured NGS results hinders the automation of trial matching for this criterion, which may be a deterrent to open biomarker-driven trials in certain sites. We developed a machine learning tool that infers the presence of NGS results in the EHR, facilitating clinical trial matching. Methods: The Flatiron Health EHR-derived database contains patient-level pathology and genetic counseling reports from community oncology practices. An internal team of clinical experts reviewed a random sample of patients across this network to generate labels of whether each patient had been NGS tested. A supervised ML model was trained by scanning documents in the EHR and extracting n-gram features from text snippets surrounding relevant keywords (i.e. 'Lung biomarker', 'Biomarker negative'). Through k-fold cross-validation and l2-regularization, we found that a logistic regression was able to classify patients' NGS testing status. The model's offline performance on a 20% hold-out test set was measured with standard classification metrics: sensitivity, specificity, positive predictive value (PPV) and NPV. In an online setting, we integrated the tool into Flatiron's clinical trial matching software OncoTrials by including in each patient's profile an indicator of "likely NGS tested" or "unlikely NGS tested" based on the classifier's prediction. For patients inferred as tested, the model linked users to a test report view in the EHR. In this online setting, we measured sensitivity and specificity of the model after user review in two community oncology practices. Results: This NGS testing status inference model was characterized using a test sample of 15,175 patients. The model sensitivity and specificity (95%CI) were 91.3% (90.2, 92.3) and 96.2% (95.8, 96.5), respectively; PPV was 84.5% (83.2, 85.8) and NPV was 98.0% (97.7, 98.2). In the validation sample (N = 200 originated from 2 distinct care sites), users identified NGS testing status with a sensitivity of 95.2% (88.3%, 98.7%). Conclusions: This machine learning model facilitates the screening for potential patient enrollment in biomarker-driven trials by automatically surfacing patients with NGS test results at high sensitivity and specificity into a trial matching application to identify candidates. This tool could mitigate a key barrier for participation in biomarker-driven trials for community clinics.

An automated EHR-based tool for identification of patients (pts) with metastatic disease to facilitate clinical trial pt ascertainment

2051

Background: Efforts to facilitate patient identification for clinical trials in routine practice, such as automating electronic health record (EHR) data reviews, are hindered by the lack of information on metastatic status in structured format. We developed a machine learning tool that infers metastatic status from unstructured EHR data, and we describe its real-world implementation. Methods: This machine learning model scans EHR documents, extracting features from text snippets surrounding key words (ie, ‘Metastatic’ ‘Progression’ ‘Local’). A regularized logistic regression model was trained, and used to classify patients across 5 metastatic status inference categories: highly-likely and likely positive, highly-likely and likely negative, and unknown. The model accuracy was characterized using the Flatiron Health EHR-derived de-identified database of patients with solid tumors, where manually abstracted information served as standard accurate reference. We assessed model accuracy using sensitivity and specificity (patients in the ‘unknown’ category omitted from numerator), negative and positive predictive values (NPV, PPV; patients ‘unknown’ included in denominator), and its performance in a real-world dataset. In a separate validation, we evaluated the accuracy gained upon additional user review of the model outputs after integration of this tool into workflows. Results: This metastatic status inference model was characterized using a sample of 66,532 patients. The model sensitivity and specificity (95%CI) were 82.% (82, 83) and 95% (95, 96), respectively; PPV was 89% (89, 90) and NPV was 94% (94, 94). In the validation sample (N = 200 originated from 5 distinct care sites), and after user review of model outputs, values increased to 97% (85, 100) for sensitivity, 98% (95, 100) for specificity, 92 (78, 98) for PPV and 99% (97, 100) for NPV. The model assigned 163/200 patients to the highly-likely categories, which were deemed not to require further EHR review by users. The prevalence of errors was 4% without user review, and 2% after user review. Conclusions: This machine learning model infers metastatic status from unstructured EHR data with high accuracy. The tool assigns metastatic status with high confidence in more than 75% of cases without requiring additional manual review, allowing more efficient identification of clinical trial candidates and clinical trial matching, thus mitigating a key barrier for clinical trial participation in community clinics.

Recapitulation of treatment response patterns in a novel humanized mouse model for chronic hepatitis B virus infection

There are ~350 million chronic carriers of hepatitis B (HBV). While a prophylactic vaccine and drug regimens to suppress viremia are available, chronic HBV infection is rarely cured. HBV's limited host tropism leads to a scarcity of susceptible small animal models and is a hurdle to developing curative therapies. Mice that support engraftment with human hepatoctyes have traditionally been generated through crosses of murine liver injury models to immunodeficient backgrounds. Here, we describe the disruption of fumarylacetoacetate hydrolase directly in the NOD Rag1^-/- IL2RγNULL (NRG) background using zinc finger nucleases. The resultant human liver chimeric mice sustain persistent HBV viremia for >90 days. When treated with standard of care therapy, HBV DNA levels decrease below detection but rebound when drug suppression is released, mimicking treatment response observed in patients. Our study highlights the utility of directed gene targeting approaches in zygotes to create new humanized mouse models for human diseases.

A bibliometric analysis of dengue-related publications in the Science Citation Index Expanded

Dengue is a re-emerging infection that poses significant threat to about half of the world's population, but there was little information on the bibliometric trend and patterns of dengue research globally. The purpose of this research is to provide a bibliometric analysis of dengue research from 1991 to 2014, and to assess the relationship between the burden of dengue and scientific publications. Most papers were classified under the topics of tropical medicine, virology, infectious diseases, parasitology or immunology. India may soon surpass Thailand in the ranking. Overall, international collaboration appeared to play a significant role in dengue research. Regional specificity of dengue may also influence the bibiliometric profile of dengue research. Nevertheless, dengue research output appeared to be positively and significantly related to the level of dengue cases, an indication that the scientific community was responding well to the needs of the population.

Modeling malaria in humanized mice: opportunities and challenges

About half of the world is at risk for developing malaria, which killed over 600,000 people in 2012 alone. The limited host range of the Plasmodium species that cause malaria in humans poses challenges for studying these parasites in experimentally tractable in vivo systems. To address this challenge, humanized mice have been constructed in which the tissue compartments relevant to the mammalian stages of plasmodial parasites are humanized. The development of human liver chimeric mice, human erythroid chimeric mice, and dually engrafted mice now allows for faithful replication of the entire Plasmodium falciparum life cycle. Although refinements to these models are still needed, humanized mice provide a promising small-animal model to study development of P. falciparum and, conceivably, other human malarial parasite species, and may also aid in drug and vaccine development.

Publication patterns of cancer cost-effectiveness studies presented at major conferences

OBJECTIVE

To be useful to policymakers and stakeholders, cost-effectiveness analyses (ceas) should be published in a timely manner and without bias. The aims of the present study were to examine the time between conference abstract presentation and subsequent publication, to determine the factors associated with time to publication, to evaluate potential publication bias, and to examine discrepancies in the results between abstract and publication.

METHODS

Abstracts of ceas presented at the annual meetings of the American Society of Clinical Oncology (asco), the American Society of Hematology (ash), and the International Society for Pharmacoeconomics and Outcomes Research (ispor) between 1997 and 2007 were reviewed. Time-to-event analysis was performed to assess the timeliness of publication and to examine factors associated with time to publication. Summary statistics were used to assess discrepancies in incremental cost-effectiveness ratios (icers) between abstract and publication.

RESULTS

Of 164 abstracts identified, 65 (39.6%) were subsequently published. The 1-, 2-, 3-, and 5-year publication rates were 12.8%, 25%, 34.2%, and 40.5% respectively. Abstracts were more likely to be published if presented at asco than at ispor (hazard ratio: 1.94; p = 0.038). There was no direct evidence of publication bias for abstracts with favourable icers. Comparing icers between abstracts and publications, the mean absolute difference was 23.8%; 50% of studies had a change in icer exceeding 10%.

CONCLUSIONS

Publication rates for ceas were low, and publication was not timely with respect to informing the decision-making process for funding. Abstract results often differed from publication results and cannot reliably be used in the decision-making process for funding.