Global Standards to Expedite Learning From Medical Research Data

Fine-tuning large language models for rare disease concept normalization

OBJECTIVE

We aim to develop a novel method for rare disease concept normalization by fine-tuning Llama 2, an open-source large language model (LLM), using a domain-specific corpus sourced from the Human Phenotype Ontology (HPO).

METHODS

We developed an in-house template-based script to generate two corpora for fine-tuning. The first (NAME) contains standardized HPO names, sourced from the HPO vocabularies, along with their corresponding identifiers. The second (NAME+SYN) includes HPO names and half of the concept's synonyms as well as identifiers. Subsequently, we fine-tuned Llama 2 (Llama2-7B) for each sentence set and conducted an evaluation using a range of sentence prompts and various phenotype terms.

RESULTS

When the phenotype terms for normalization were included in the fine-tuning corpora, both models demonstrated nearly perfect performance, averaging over 99% accuracy. In comparison, ChatGPT-3.5 has only ∼20% accuracy in identifying HPO IDs for phenotype terms. When single-character typos were introduced in the phenotype terms, the accuracy of NAME and NAME+SYN is 10.2% and 36.1%, respectively, but increases to 61.8% (NAME+SYN) with additional typo-specific fine-tuning. For terms sourced from HPO vocabularies as unseen synonyms, the NAME model achieved 11.2% accuracy, while the NAME+SYN model achieved 92.7% accuracy.

CONCLUSION

Our fine-tuned models demonstrate ability to normalize phenotype terms unseen in the fine-tuning corpus, including misspellings, synonyms, terms from other ontologies, and laymen's terms. Our approach provides a solution for the use of LLMs to identify named medical entities from clinical narratives, while successfully normalizing them to standard concepts in a controlled vocabulary.

Fine-tuning Large Language Models for Rare Disease Concept Normalization

Objective

We aim to develop a novel method for rare disease concept normalization by fine-tuning Llama 2, an open-source large language model (LLM), using a domain-specific corpus sourced from the Human Phenotype Ontology (HPO).

Methods

We developed an in-house template-based script to generate two corpora for fine-tuning. The first (NAME) contains standardized HPO names, sourced from the HPO vocabularies, along with their corresponding identifiers. The second (NAME+SYN) includes HPO names and half of the concept's synonyms as well as identifiers. Subsequently, we fine-tuned Llama2 (Llama2-7B) for each sentence set and conducted an evaluation using a range of sentence prompts and various phenotype terms.

Results

When the phenotype terms for normalization were included in the fine-tuning corpora, both models demonstrated nearly perfect performance, averaging over 99% accuracy. In comparison, ChatGPT-3.5 has only ~20% accuracy in identifying HPO IDs for phenotype terms. When single-character typos were introduced in the phenotype terms, the accuracy of NAME and NAME+SYN is 10.2% and 36.1%, respectively, but increases to 61.8% (NAME+SYN) with additional typo-specific fine-tuning. For terms sourced from HPO vocabularies as unseen synonyms, the NAME model achieved 11.2% accuracy, while the NAME+SYN model achieved 92.7% accuracy.

Conclusion

Our fine-tuned models demonstrate ability to normalize phenotype terms unseen in the fine-tuning corpus, including misspellings, synonyms, terms from other ontologies, and laymen's terms. Our approach provides a solution for the use of LLM to identify named medical entities from the clinical narratives, while successfully normalizing them to standard concepts in a controlled vocabulary.

Use of Clinical Data Interchange Standards Consortium (CDISC) Standards for Real-world Data: Expert Perspectives From a Qualitative Delphi Survey

Background

Real-world data (RWD) and real-world evidence (RWE) are playing increasingly important roles in clinical research and health care decision-making. To leverage RWD and generate reliable RWE, data should be well defined and structured in a way that is semantically interoperable and consistent across stakeholders. The adoption of data standards is one of the cornerstones supporting high-quality evidence for the development of clinical medicine and therapeutics. Clinical Data Interchange Standards Consortium (CDISC) data standards are mature, globally recognized, and heavily used by the pharmaceutical industry for regulatory submissions. The CDISC RWD Connect Initiative aims to better understand the barriers to implementing CDISC standards for RWD and to identify the tools and guidance needed to more easily implement them.

Objective

The aim of this study is to understand the barriers to implementing CDISC standards for RWD and to identify the tools and guidance that may be needed to implement CDISC standards more easily for this purpose.

Methods

We conducted a qualitative Delphi survey involving an expert advisory board with multiple key stakeholders, with 3 rounds of input and review.

Results

Overall, 66 experts participated in round 1, 56 in round 2, and 49 in round 3 of the Delphi survey. Their inputs were collected and analyzed, culminating in group statements. It was widely agreed that the standardization of RWD is highly necessary, and the primary focus should be on its ability to improve data sharing and the quality of RWE. The priorities for RWD standardization included electronic health records, such as data shared using Health Level 7 Fast Health care Interoperability Resources (FHIR), and the data stemming from observational studies. With different standardization efforts already underway in these areas, a gap analysis should be performed to identify the areas where synergies and efficiencies are possible and then collaborate with stakeholders to create or extend existing mappings between CDISC and other standards, controlled terminologies, and models to represent data originating across different sources.

Conclusions

There are many ongoing data standardization efforts around human health data–related activities, each with different definitions, levels of granularity, and purpose. Among these, CDISC has been successful in standardizing clinical trial-based data for regulation worldwide. However, the complexity of the CDISC standards and the fact that they were developed for different purposes, combined with the lack of awareness and incentives to use a new standard and insufficient training and implementation support, are significant barriers to setting up the use of CDISC standards for RWD. The collection and dissemination of use cases, development of tools and support systems for the RWD community, and collaboration with other standards development organizations are potential steps forward. Using CDISC will help link clinical trial data and RWD and promote innovation in health data science.

Addressing the Covid-19 pandemic and future public health challenges through global collaboration and a data-driven systems approach

Covid-19 has already taught us that the greatest public health challenges of our generation will show no respect for national boundaries, will impact lives and health of people of all nations, and will affect economies and quality of life in unprecedented ways. The types of rapid learning envisioned to address Covid-19 and future public health crises require a systems approach that enables sharing of data and lessons learned at scale. Agreement on a systems approach augmented by technology and standards will be foundational to making such learning meaningful and to ensuring its scientific integrity. With this purpose in mind, a group of individuals from Spain, Italy, and the United States have formed a transatlantic collaboration, with the aim of generating a proposed comprehensive standards-based systems approach and data-driven framework for collection, management, and analysis of high-quality data. This framework will inform decisions in managing clinical responses and social measures to overcome the Covid-19 global pandemic and to prepare for future public health crises. We first argue that standardized data of the type now common in global regulated clinical research is the essential fuel that will power a global system for addressing (and preventing) current and future pandemics. We then present a blueprint for a system that will put these data to use in driving a range of key decisions. In the context of this system, we describe and categorize the specific types of data the system will require for different purposes and document the standards currently in use for each of these categories in the three nations participating in this work. In so doing, we anticipate some of the challenges to harmonizing these data but also suggest opportunities for further global standardization and harmonization. While we have scaled this transnational effort to three nations, we hope to stimulate an international dialogue with a culmination of realizing such a system.

Development of electronic medical records for clinical and research purposes: the breast cancer module using an implementation framework in a middle income country- Malaysia

BACKGROUND

Advances in medical domain has led to an increase of clinical data production which offers enhancement opportunities for clinical research sector. In this paper, we propose to expand the scope of Electronic Medical Records in the University Malaya Medical Center (UMMC) using different techniques in establishing interoperability functions between multiple clinical departments involving diagnosis, screening and treatment of breast cancer and building automatic systems for clinical audits as well as for potential data mining to enhance clinical breast cancer research in the future.

RESULTS

Quality Implementation Framework (QIF) was adopted to develop the breast cancer module as part of the in-house EMR system used at UMMC, called i-Pesakit©. The completion of the i-Pesakit© Breast Cancer Module requires management of clinical data electronically, integration of clinical data from multiple internal clinical departments towards setting up of a research focused patient data governance model. The 14 QIF steps were performed in four main phases involved in this study which are (i) initial considerations regarding host setting, (ii) creating structure for implementation, (iii) ongoing structure once implementation begins, and (iv) improving future applications. The architectural framework of the module incorporates both clinical and research needs that comply to the Personal Data Protection Act.

CONCLUSION

The completion of the UMMC i-Pesakit© Breast Cancer Module required populating EMR including management of clinical data access, establishing information technology and research focused governance model and integrating clinical data from multiple internal clinical departments. This multidisciplinary collaboration has enhanced the quality of data capture in clinical service, benefited hospital data monitoring, quality assurance, audit reporting and research data management, as well as a framework for implementing a responsive EMR for a clinical and research organization in a typical middle-income country setting. Future applications include establishing integration with external organization such as the National Registration Department for mortality data, reporting of institutional data for national cancer registry as well as data mining for clinical research. We believe that integration of multiple clinical visit data sources provides a more comprehensive, accurate and real-time update of clinical data to be used for epidemiological studies and audits.