Natural history of rare diseases using natural language processing of narrative unstructured electronic health records: The example of Dravet syndrome

Ontology accelerates few-shot learning capability of large language model: A study in extraction of drug efficacy in a rare pediatric epilepsy

OBJECTIVE

Dravet Syndrome (DS) is a developmental and epileptic encephalopathy that is characterized by severe, prolonged motor seizures and high resistance to multiple antiseizure medications (ASMs) with multiple comorbidities. Evaluating the efficacy of new drugs in DS preclinical models and mapping them to human phenotypes of DS through analysis of published literature is an important goal for improving outcomes in this rare pediatric epilepsy.

MATERIALS AND METHODS

Large language models (LLM) have demonstrated great promise in parsing published literature; however, the performance of LLMs falls short in medical applications. In this study, we investigate the effectiveness of domain ontology developed by human experts to optimize LLMs for medical text processing in a rare disease. Utilizing a benchmark dataset that describes the efficacy of 17 ASMs tested in preclinical models and DS patients, we define a new ontology-augmented phased in-context learning (PCL) approach to process 4935 full-text DS articles. We expand this analysis to 7 new drugs that demonstrate efficacy in reducing seizures to identify gaps in current knowledge for designing new experimental studies for drug discovery in DS.

RESULTS

Few-shot or in-context learning is a foundational capability of LLMs and the few-shot learning capability of the Gemini 1.0 Pro version LLM dramatically increases when we augment prompts with the DS epilepsy ontology. The DS epilepsy ontology is the largest epilepsy and seizure ontology in clinical use that was developed by DS basic scientists and clinical neurologists. The ontology-augmented PCL prompt achieves 100% accuracy in reproducing the benchmark drug efficacy dataset for 17 ASMs with only two examples for in-context learning.

CONCLUSION

The new ontology-augmented PCL approach significantly accelerates the few-shot learning capabilities of the Gemini LLM, thereby reducing the number of required examples and time needed to optimize LLMs for medical applications.

Neighborhood-level social determinants of suicidality in youth with schizophrenia: An EHR-based study

BACKGROUND

Suicidal thoughts and behaviors (STB) among youth with schizophrenia represent a significant public health concern. It is well-established that neighborhood-level social determinants of health (SDoHs) can impact health outcomes in individuals with schizophrenia. We aimed to investigate the effects of neighborhood-level social determinants on developing future STB in youth with schizophrenia.

METHODS

We conducted a retrospective cohort study using electronic health records from the INSIGHT Clinical Research Network, which contains >22 million unique patients across five healthcare systems in New York City. Patients' neighborhood-level SDoHs were measured at their residential ZIP Code Tabulation Area using a composite measure, Social Deprivation Index (SDI), as well as specific components derived from the American Community Survey. Survival analysis was used to study the association between neighborhood-level SDoHs and time to STB since the first schizophrenia diagnosis.

RESULTS

Between 10/1/2015 and 10/1/2022, we identified 1209 youth aged between 10 and 25 years with a schizophrenia diagnosis and no prior STB, among whom 176 developed STB during follow-up. SDI quintiles were not associated with the risk of future STB, whereas two specific neighborhood characteristics, Gini index and percentage of residents commuting by car/truck/van, were associated with a decreased risk of STB, after controlling for patients' demographic characteristics.

CONCLUSIONS

Although the overall neighborhood deprivation level was not associated with the risk of STB among youth with schizophrenia, specific neighborhood characteristics were. These findings underscore the need for more targeted community-based suicide prevention strategies. Further research is essential to better understand the underlying mechanism of these associations.

Objectivizing issues in the diagnosis of complex rare diseases: lessons learned from testing existing diagnosis support systems on ciliopathies

BACKGROUND

There are approximately 8,000 different rare diseases that affect roughly 400 million people worldwide. Many of them suffer from delayed diagnosis. Ciliopathies are rare monogenic disorders characterized by a significant phenotypic and genetic heterogeneity that raises an important challenge for clinical diagnosis. Diagnosis support systems (DSS) applied to electronic health record (EHR) data may help identify undiagnosed patients, which is of paramount importance to improve patients' care. Our objective was to evaluate three online-accessible rare disease DSSs using phenotypes derived from EHRs for the diagnosis of ciliopathies.

METHODS

Two datasets of ciliopathy cases, either proven or suspected, and two datasets of controls were used to evaluate the DSSs. Patient phenotypes were automatically extracted from their EHRs and converted to Human Phenotype Ontology terms. We tested the ability of the DSSs to diagnose cases in contrast to controls based on Orphanet ontology.

RESULTS

A total of 79 cases and 38 controls were selected. Performances of the DSSs on ciliopathy real world data (best DSS with area under the ROC curve = 0.72) were not as good as published performances on the test set used in the DSS development phase. None of these systems obtained results which could be described as "expert-level". Patients with multisystemic symptoms were generally easier to diagnose than patients with isolated symptoms. Diseases easily confused with ciliopathy generally affected multiple organs and had overlapping phenotypes. Four challenges need to be considered to improve the performances: to make the DSSs interoperable with EHR systems, to validate the performances in real-life settings, to deal with data quality, and to leverage methods and resources for rare and complex diseases.

CONCLUSION

Our study provides insights into the complexities of diagnosing highly heterogenous rare diseases and offers lessons derived from evaluation existing DSSs in real-world settings. These insights are not only beneficial for ciliopathy diagnosis but also hold relevance for the enhancement of DSS for various complex rare disorders, by guiding the development of more clinically relevant rare disease DSSs, that could support early diagnosis and finally make more patients eligible for treatment.

Performance and clinical utility of a new supervised machine-learning pipeline in detecting rare ciliopathy patients based on deep phenotyping from electronic health records and semantic similarity

BACKGROUND

Rare diseases affect approximately 400 million people worldwide. Many of them suffer from delayed diagnosis. Among them, NPHP1-related renal ciliopathies need to be diagnosed as early as possible as potential treatments have been recently investigated with promising results. Our objective was to develop a supervised machine learning pipeline for the detection of NPHP1 ciliopathy patients from a large number of nephrology patients using electronic health records (EHRs).

METHODS AND RESULTS

We designed a pipeline combining a phenotyping module re-using unstructured EHR data, a semantic similarity module to address the phenotype dependence, a feature selection step to deal with high dimensionality, an undersampling step to address the class imbalance, and a classification step with multiple train-test split for the small number of rare cases. The pipeline was applied to thirty NPHP1 patients and 7231 controls and achieved good performances (sensitivity 86% with specificity 90%). A qualitative review of the EHRs of 40 misclassified controls showed that 25% had phenotypes belonging to the ciliopathy spectrum, which demonstrates the ability of our system to detect patients with similar conditions.

CONCLUSIONS

Our pipeline reached very encouraging performance scores for pre-diagnosing ciliopathy patients. The identified patients could then undergo genetic testing. The same data-driven approach can be adapted to other rare diseases facing underdiagnosis challenges.