Enhancing Early Detection of Cognitive Decline in the Elderly: A Comparative Study Utilizing Large Language Models in Clinical Notes

Background

Large language models (LLMs) have shown promising performance in various healthcare domains, but their effectiveness in identifying specific clinical conditions in real medical records is less explored. This study evaluates LLMs for detecting signs of cognitive decline in real electronic health record (EHR) clinical notes, comparing their error profiles with traditional models. The insights gained will inform strategies for performance enhancement.

Methods

This study, conducted at Mass General Brigham in Boston, MA, analyzed clinical notes from the four years prior to a 2019 diagnosis of mild cognitive impairment in patients aged 50 and older. We used a randomly annotated sample of 4,949 note sections, filtered with keywords related to cognitive functions, for model development. For testing, a random annotated sample of 1,996 note sections without keyword filtering was utilized. We developed prompts for two LLMs, Llama 2 and GPT-4, on HIPAA-compliant cloud-computing platforms using multiple approaches (e.g., both hard and soft prompting and error analysis-based instructions) to select the optimal LLM-based method. Baseline models included a hierarchical attention-based neural network and XGBoost. Subsequently, we constructed an ensemble of the three models using a majority vote approach.

Results

GPT-4 demonstrated superior accuracy and efficiency compared to Llama 2, but did not outperform traditional models. The ensemble model outperformed the individual models, achieving a precision of 90.3%, a recall of 94.2%, and an F1-score of 92.2%. Notably, the ensemble model showed a significant improvement in precision, increasing from a range of 70%-79% to above 90%, compared to the best-performing single model. Error analysis revealed that 63 samples were incorrectly predicted by at least one model; however, only 2 cases (3.2%) were mutual errors across all models, indicating diverse error profiles among them.

Conclusions

LLMs and traditional machine learning models trained using local EHR data exhibited diverse error profiles. The ensemble of these models was found to be complementary, enhancing diagnostic performance. Future research should investigate integrating LLMs with smaller, localized models and incorporating medical data and domain knowledge to enhance performance on specific tasks.

AutoCriteria: a generalizable clinical trial eligibility criteria extraction system powered by large language models

OBJECTIVES

We aim to build a generalizable information extraction system leveraging large language models to extract granular eligibility criteria information for diverse diseases from free text clinical trial protocol documents. We investigate the model's capability to extract criteria entities along with contextual attributes including values, temporality, and modifiers and present the strengths and limitations of this system.

MATERIALS AND METHODS

The clinical trial data were acquired from https://ClinicalTrials.gov/. We developed a system, AutoCriteria, which comprises the following modules: preprocessing, knowledge ingestion, prompt modeling based on GPT, postprocessing, and interim evaluation. The final system evaluation was performed, both quantitatively and qualitatively, on 180 manually annotated trials encompassing 9 diseases.

RESULTS

AutoCriteria achieves an overall F1 score of 89.42 across all 9 diseases in extracting the criteria entities, with the highest being 95.44 for nonalcoholic steatohepatitis and the lowest of 84.10 for breast cancer. Its overall accuracy is 78.95% in identifying all contextual information across all diseases. Our thematic analysis indicated accurate logic interpretation of criteria as one of the strengths and overlooking/neglecting the main criteria as one of the weaknesses of AutoCriteria.

DISCUSSION

AutoCriteria demonstrates strong potential to extract granular eligibility criteria information from trial documents without requiring manual annotations. The prompts developed for AutoCriteria generalize well across different disease areas. Our evaluation suggests that the system handles complex scenarios including multiple arm conditions and logics.

CONCLUSION

AutoCriteria currently encompasses a diverse range of diseases and holds potential to extend to more in the future. This signifies a generalizable and scalable solution, poised to address the complexities of clinical trial application in real-world settings.

Phenotypic and endotypic features of COPD associated with lung cancer development

e13563

Background: Chronic Obstructive Pulmonary Disease (COPD) is a heterogeneous disease with multiple phenotypes and endotypes and associated with lung cancer development. In this study, we analyzed the clinical features of COPD including various phenotypic and endotypic features from the electronic health records (EHR) as the potential risk factors for lung cancer development in a large cohort of COPD patients. Methods: We identified a COPD cohort based on electronic Medical Records and Genomics (eMERGE) network COPD cohort identification algorithms with minor modifications from the Mount Sinai Data Warehouse (2000-2020) and followed the patients on the first diagnosis of lung cancer. The development of lung cancer in COPD patients was confirmed by manual chart review. We retrieved the clinical features from EHRs and conducted Kaplan Meier (KM) analysis and multivariable Cox-regression modeling for hazard ratio (HR) analysis. Results: We found that 3.8 % of COPD patients (824 out of 21,658) developed lung cancer. While COPD patients with emphysema and smoking history (former or current) showed an increased risk of lung cancer onset, patients with concurrent asthma and corticosteroid (ICS) inhalation history showed a reduced risk of lung cancer onset (adjusted HR in Table). Interestingly, COPD patients with higher eosinophil counts showed late onset of lung cancer (>300 cells/ul, cancer rate at 5y=2.4%, p=< 0.002; 150-300 cells/ul, cancer rate at 5y=2.9%, p=0.003) when compared to patients with low eosinophils count (<150 cells/ul, cancer rate at 5y=3.4%) in a KM analysis. A multivariable analysis adjusted for age, gender, race, smoking status, COPD sub-phenotypes, severe exacerbation history, and ICS inhalation history showed a significantly lower risk of lung cancer in COPD patients with higher eosinophils count (150-300 cells/ul; HR: 0.82, 95% CI: 0.69-0.97, p=0.021 and >300 cells/ul; HR:0.72, 95% CI: 0.57-0.89, p=0.003) when compared to those with low eosinophils count (<150 cells/ul). Conclusions: Our study shows that many phenotypic and endotypic features of COPD are differentially associated with lung cancer development. High eosinophil levels, ICS usage, and concurrent asthma in COPD patients may reduce the risk of lung cancer development. [Table: see text]

Modelling and treating GRIN2A developmental and epileptic encephalopathy in mice

NMDA receptors play crucial roles in excitatory synaptic transmission. Rare variants in GRIN2A encoding the GluN2A subunit are associated with a spectrum of disorders, ranging from mild speech and language delay to intractable neurodevelopmental disorders, including but not limited to developmental and epileptic encephalopathy. A de novo missense variant, p.Ser644Gly, was identified in a child with this disorder, and Grin2a knock-in mice were generated to model and extend understanding of this intractable childhood disease. Homozygous and heterozygous mutant mice exhibited altered hippocampal morphology at 2 weeks of age, and all homozygotes exhibited lethal tonic-clonic seizures by mid-third week. Heterozygous adults displayed susceptibility to induced generalized seizures, hyperactivity, repetitive and reduced anxiety behaviours, plus several unexpected features, including significant resistance to electrically-induced limbic seizures and to pentylenetetrazole induced tonic-clonic seizures. Multielectrode recordings of neuronal networks revealed hyperexcitability and altered bursting and synchronicity. In heterologous cells, mutant receptors had enhanced NMDA receptor agonist potency and slow deactivation following rapid removal of glutamate, as occurs at synapses. NMDA receptor-mediated synaptic currents in heterozygous hippocampal slices also showed a prolonged deactivation time course. Standard anti-epileptic drug monotherapy was ineffective in the patient. Introduction of NMDA receptor antagonists was correlated with a decrease in seizure burden. Chronic treatment of homozygous mouse pups with NMDA receptor antagonists significantly delayed the onset of lethal seizures but did not prevent them. These studies illustrate the power of using multiple experimental modalities to model and test therapies for severe neurodevelopmental disorders, while revealing significant biological complexities associated with GRIN2A developmental and epileptic encephalopathy.

ER-mitochondria tethering by PDZD8 regulates Ca2+ dynamics in mammalian neurons

Interfaces between organelles are emerging as critical platforms for many biological responses in eukaryotic cells. In yeast, the ERMES complex is an endoplasmic reticulum (ER)-mitochondria tether composed of four proteins, three of which contain a SMP (synaptotagmin-like mitochondrial-lipid binding protein) domain. No functional ortholog for any ERMES protein has been identified in metazoans. Here, we identified PDZD8 as an ER protein present at ER-mitochondria contacts. The SMP domain of PDZD8 is functionally orthologous to the SMP domain found in yeast Mmm1. PDZD8 was necessary for the formation of ER-mitochondria contacts in mammalian cells. In neurons, PDZD8 was required for calcium ion (Ca²⁺) uptake by mitochondria after synaptically induced Ca²⁺-release from ER and thereby regulated cytoplasmic Ca²⁺ dynamics. Thus, PDZD8 represents a critical ER-mitochondria tethering protein in metazoans. We suggest that ER-mitochondria coupling is involved in the regulation of dendritic Ca²⁺ dynamics in mammalian neurons.

Increased β-catenin activity in the anterior neural plate induces ectopic mid-hindbrain characteristics

BACKGROUND

The early telencephalon shares molecular features with the early mid-hindbrain region. In particular, these two developing brain areas each have a signaling center that secretes FGFs and an adjacent one that secretes WNTs. WNTs and FGFs each play essential roles in regulating cell fates in both the telencephalon and mid-hindbrain. Despite this similarity, telencephalic and mid-hindbrain precursors express distinct genes and ultimately generate different cell types, tissue morphologies, and neural functions.

RESULTS

Here we show that genetically increasing the level of β-catenin, a mediator of canonical WNT signaling, in the anterior neural plate causes a loss of telencephalic characteristics and a gain of mid-hindbrain characteristics.

CONCLUSION

These results, together with previous ones demonstrating that increased WNT signaling in the anterior neural plate increases FGF expression, suggest that the levels of WNT and FGF signaling regulate telencephalic versus mid-hindbrain fates.

β-Catenin-dependent FGF signaling sustains cell survival in the anterior embryonic head by countering Smad4

Growing evidence suggests that FGFs secreted from embryonic signaling centers are key mediators of cell survival. However, the mechanisms regulating FGF-dependent cell survival remain obscure. At the rostral end of the embryo, for example, ablation of FGF signaling leads to the rapid death of the precursor cells that form the anterior head, including the telencephalon. Here, we outline a core genetic circuit that regulates survival in the embryonic mouse head: WNT signaling through β-catenin directly maintains FGF expression and requires FGF function in vivo to oppose proapoptotic TGF-β signaling through SMAD4. Moreover, these antagonistic pathways converge on the transcriptional regulation of apoptosis, and genes such as Cdkn1a, suggesting a mechanism for how signaling centers in the embryonic head regulate cell survival.

Opposing Fgf and Bmp activities regulate the specification of olfactory sensory and respiratory epithelial cell fates

The olfactory sensory epithelium and the respiratory epithelium are derived from the olfactory placode. However, the molecular mechanisms regulating the differential specification of the sensory and the respiratory epithelium have remained undefined. To address this issue, we first identified Msx1/2 and Id3 as markers for respiratory epithelial cells by performing quail chick transplantation studies. Next, we established chick explant and intact chick embryo assays of sensory/respiratory epithelial cell differentiation and analyzed two mice mutants deleted of Bmpr1a;Bmpr1b or Fgfr1;Fgfr2 in the olfactory placode. In this study, we provide evidence that in both chick and mouse, Bmp signals promote respiratory epithelial character, whereas Fgf signals are required for the generation of sensory epithelial cells. Moreover, olfactory placodal cells can switch between sensory and respiratory epithelial cell fates in response to Fgf and Bmp activity, respectively. Our results provide evidence that Fgf activity suppresses and restricts the ability of Bmp signals to induce respiratory cell fate in the nasal epithelium. In addition, we show that in both chick and mouse the lack of Bmp or Fgf activity results in disturbed placodal invagination; however, the fate of cells in the remaining olfactory epithelium is independent of morphological movements related to invagination. In summary, we present a conserved mechanism in amniotes in which Bmp and Fgf signals act in an opposing manner to regulate the respiratory versus sensory epithelial cell fate decision.

FGF signaling is strictly required to maintain early telencephalic precursor cell survival

The FGF family of extracellular signaling factors has been proposed to play multiple roles in patterning the telencephalon, the precursor to the cerebrum. In this study, unlike previous ones, we effectively abolish FGF signaling in the anterior neural plate via deletion of three FGF receptor (FGFR) genes. Triple FGFR mutant mice exhibit a complete loss of the telencephalon, except the dorsal midline. Disruption of FGF signaling prior to and coincident with telencephalic induction reveals that FGFs promote telencephalic character and are strictly required to keep telencephalic cells alive. Moreover, progressively more severe truncations of the telencephalon are observed in FGFR single, double and triple mutants. Together with previous gain-of-function studies showing induction of Foxg1 expression and mirror-image duplications of the cortex by exogenous FGF8, our loss-of-function results suggest that, rather than independently patterning different areas, FGF ligands and receptors act in concert to mediate organizer activity for the whole telencephalon.

FGF signalling generates ventral telencephalic cells independently of SHH

Sonic hedgehog (SHH) is required to generate ventral cell types throughout the central nervous system. Its role in directly specifying ventral cells,however, has recently been questioned because loss of the Shh gene has little effect on ventral development if the Gli3 gene is also mutant. Consequently, another ventral determinant must exist. Here, genetic evidence establishes that FGFs are required for ventral telencephalon development. First, simultaneous deletion of Fgfr1 and Fgfr3specifically in the telencephalon results in the loss of differentiated ventromedial cells; and second, in the Fgfr1;Fgfr2 double mutant, ventral precursor cells are lost, mimicking the phenotype obtained previously with a loss of SHH signalling. Yet, in the Fgfr1;Fgfr2 mutant, Shh remains expressed, as does Gli1, the transcription of which depends on SHH activity, suggesting that FGF signalling acts independently of SHH to generate ventral precursors. Moreover, the Fgfr1;Fgfr2 phenotype, unlike the Shhphenotype, is not rescued by loss of Gli3, further indicating that FGFs act downstream of Shh and Gli3 to generate ventral telencephalic cell types.

Use of capillary electrophoresis with UV detection as a screening method to determine histamine in fish samples

Histamine levels in fish, extracted with methanol, were determined by capillary electrophoresis (CE) using phosphate buffer pH 2.5 and U.V. detection at 210 nm. Histamine was well separated from the other co-extracted components under the given CE condition without any cleanup of the methanol extract. The average recovery of spiked histamine in various types of fish samples was 96%. Using the same methanol extracts from various fish samples, we then compared histamine concentration obtained by CE and fluorometric methods.