1
|
Liu F. Data Science Methods for Real-World Evidence Generation in Real-World Data. Annu Rev Biomed Data Sci 2024; 7:201-224. [PMID: 38748863 DOI: 10.1146/annurev-biodatasci-102423-113220] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/25/2024]
Abstract
In the healthcare landscape, data science (DS) methods have emerged as indispensable tools to harness real-world data (RWD) from various data sources such as electronic health records, claim and registry data, and data gathered from digital health technologies. Real-world evidence (RWE) generated from RWD empowers researchers, clinicians, and policymakers with a more comprehensive understanding of real-world patient outcomes. Nevertheless, persistent challenges in RWD (e.g., messiness, voluminousness, heterogeneity, multimodality) and a growing awareness of the need for trustworthy and reliable RWE demand innovative, robust, and valid DS methods for analyzing RWD. In this article, I review some common current DS methods for extracting RWE and valuable insights from complex and diverse RWD. This article encompasses the entire RWE-generation pipeline, from study design with RWD to data preprocessing, exploratory analysis, methods for analyzing RWD, and trustworthiness and reliability guarantees, along with data ethics considerations and open-source tools. This review, tailored for an audience that may not be experts in DS, aspires to offer a systematic review of DS methods and assists readers in selecting suitable DS methods and enhancing the process of RWE generation for addressing their specific challenges.
Collapse
Affiliation(s)
- Fang Liu
- Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, Notre Dame, Indiana, USA;
| |
Collapse
|
2
|
Andargoli AE, Ulapane N, Nguyen TA, Shuakat N, Zelcer J, Wickramasinghe N. Intelligent decision support systems for dementia care: A scoping review. Artif Intell Med 2024; 150:102815. [PMID: 38553156 DOI: 10.1016/j.artmed.2024.102815] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2022] [Revised: 12/11/2023] [Accepted: 02/12/2024] [Indexed: 04/02/2024]
Abstract
In the context of dementia care, Artificial Intelligence (AI) powered clinical decision support systems have the potential to enhance diagnosis and management. However, the scope and challenges of applying these technologies remain unclear. This scoping review aims to investigate the current state of AI applications in the development of intelligent decision support systems for dementia care. We conducted a comprehensive scoping review of empirical studies that utilised AI-powered clinical decision support systems in dementia care. The results indicate that AI applications in dementia care primarily focus on diagnosis, with limited attention to other aspects outlined in the World Health Organization (WHO) Global Action Plan on the Public Health Response to Dementia 2017-2025 (GAPD). A trifecta of challenges, encompassing data availability, cost considerations, and AI algorithm performance, emerges as noteworthy barriers in adoption of AI applications in dementia care. To address these challenges and enhance AI reliability, we propose a novel approach: a digital twin-based patient journey model. Future research should address identified gaps in GAPD action areas, navigate data-related obstacles, and explore the implementation of digital twins. Additionally, it is imperative to emphasize that addressing trust and combating the stigma associated with AI in healthcare should be a central focus of future research directions.
Collapse
Affiliation(s)
| | | | - Tuan Anh Nguyen
- Swinburne University of Technology, Melbourne, Australia; National Ageing Research Institute, Australia
| | | | | | | |
Collapse
|
3
|
Vazquez J, Facelli JC. Conformal Prediction in Clinical Medical Sciences. JOURNAL OF HEALTHCARE INFORMATICS RESEARCH 2022; 6:241-252. [PMID: 35898853 PMCID: PMC9309105 DOI: 10.1007/s41666-021-00113-8] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2021] [Revised: 11/30/2021] [Accepted: 12/28/2021] [Indexed: 02/04/2023]
Abstract
The use of machine learning (ML) and artificial intelligence (AI) applications in medicine has attracted a great deal of attention in the medical literature, but little is known about how to use Conformal Predictions (CP) to assess the accuracy of individual predictions in clinical applications. We performed a comprehensive search in SCOPUS® to find papers reporting the use of CP in clinical applications. We identified 14 papers reporting the use of CP for clinical applications, and we briefly describe the methods and results reported in these papers. The literature reviewed shows that CP methods can be used in clinical applications to provide important insight into the accuracy of individual predictions. Unfortunately, the review also shows that most of the studies have been performed in isolation, without input from practicing clinicians, not providing comparisons among different approaches and not considering important socio-technical considerations leading to clinical adoption.
Collapse
Affiliation(s)
- Janette Vazquez
- Department of Biomedical Informatics and Clinical and Translational Science Institute, The University of Utah, Salt Lake City, UT 84108 USA
| | - Julio C. Facelli
- Department of Biomedical Informatics and Clinical and Translational Science Institute, The University of Utah, Salt Lake City, UT 84108 USA
| |
Collapse
|
4
|
Estimation of fault probability in medium voltage feeders through calibration techniques in classification models. Soft comput 2022. [DOI: 10.1007/s00500-022-07194-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/17/2022]
Abstract
AbstractMachine Learning is currently a well-suited approach widely adopted for solving data-driven problems in predictive maintenance. Data-driven approaches can be used as the main building block in risk-based assessment and analysis tools for Transmission and Distribution System Operators in modern Smart Grids. For this purpose, a suitable Decision Support System should be able of providing not only early warnings, such as the detection of faults in real time, but even an accurate probability estimate of outages and failures. In other words, the performance of classification systems, at least in these cases, needs to be assessed even in terms of reliable outputting posterior probabilities, a really important feature that, in general, classifiers very often do not offer. In this paper are compared several state-of-the-art calibration techniques along with a set of simple new proposed techniques, with the aim of calibrating fuzzy scoring values of a custom-made evolutionary-cluster-based hybrid classifier trained on a set of a real-world dataset of faults collected within the power grid that feeds the city of Rome, Italy. Comparison results show that in real-world cases calibration techniques need to be assessed carefully depending on the scores distribution and the proposed techniques are a valid alternative to the ones existing in the technical literature in terms of calibration performance, computational efficiency and flexibility.
Collapse
|
5
|
Mervin LH, Trapotsi MA, Afzal AM, Barrett IP, Bender A, Engkvist O. Probabilistic Random Forest improves bioactivity predictions close to the classification threshold by taking into account experimental uncertainty. J Cheminform 2021; 13:62. [PMID: 34412708 PMCID: PMC8375213 DOI: 10.1186/s13321-021-00539-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2021] [Accepted: 07/30/2021] [Indexed: 11/24/2022] Open
Abstract
Measurements of protein–ligand interactions have reproducibility limits due to experimental errors. Any model based on such assays will consequentially have such unavoidable errors influencing their performance which should ideally be factored into modelling and output predictions, such as the actual standard deviation of experimental measurements (σ) or the associated comparability of activity values between the aggregated heterogenous activity units (i.e., Ki versus IC50 values) during dataset assimilation. However, experimental errors are usually a neglected aspect of model generation. In order to improve upon the current state-of-the-art, we herein present a novel approach toward predicting protein–ligand interactions using a Probabilistic Random Forest (PRF) classifier. The PRF algorithm was applied toward in silico protein target prediction across ~ 550 tasks from ChEMBL and PubChem. Predictions were evaluated by taking into account various scenarios of experimental standard deviations in both training and test sets and performance was assessed using fivefold stratified shuffled splits for validation. The largest benefit in incorporating the experimental deviation in PRF was observed for data points close to the binary threshold boundary, when such information was not considered in any way in the original RF algorithm. For example, in cases when σ ranged between 0.4–0.6 log units and when ideal probability estimates between 0.4–0.6, the PRF outperformed RF with a median absolute error margin of ~ 17%. In comparison, the baseline RF outperformed PRF for cases with high confidence to belong to the active class (far from the binary decision threshold), although the RF models gave errors smaller than the experimental uncertainty, which could indicate that they were overtrained and/or over-confident. Finally, the PRF models trained with putative inactives decreased the performance compared to PRF models without putative inactives and this could be because putative inactives were not assigned an experimental pXC50 value, and therefore they were considered inactives with a low uncertainty (which in practice might not be true). In conclusion, PRF can be useful for target prediction models in particular for data where class boundaries overlap with the measurement uncertainty, and where a substantial part of the training data is located close to the classification threshold.
Collapse
Affiliation(s)
- Lewis H Mervin
- Molecular AI, Discovery Sciences, R&D, AstraZeneca, Cambridge, UK.
| | - Maria-Anna Trapotsi
- Department of Chemistry, Centre for Molecular Informatics, University of Cambridge, Lensfield Road, Cambridge, CB2 1EW, UK
| | - Avid M Afzal
- Data Sciences & Quantitative Biology, Discovery Sciences, R&D, AstraZeneca, Cambridge, UK
| | - Ian P Barrett
- Data Sciences & Quantitative Biology, Discovery Sciences, R&D, AstraZeneca, Cambridge, UK
| | - Andreas Bender
- Department of Chemistry, Centre for Molecular Informatics, University of Cambridge, Lensfield Road, Cambridge, CB2 1EW, UK
| | - Ola Engkvist
- Molecular AI, Discovery Sciences, R&D, AstraZeneca, Gothenburg, Sweden.,Department of Computer Science and Engineering, Chalmers University of Technology, Gothenburg, Sweden
| |
Collapse
|
6
|
Mervin LH, Johansson S, Semenova E, Giblin KA, Engkvist O. Uncertainty quantification in drug design. Drug Discov Today 2020; 26:474-489. [PMID: 33253918 DOI: 10.1016/j.drudis.2020.11.027] [Citation(s) in RCA: 29] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/21/2020] [Revised: 07/13/2020] [Accepted: 11/23/2020] [Indexed: 01/03/2023]
Abstract
Machine learning and artificial intelligence are increasingly being applied to the drug-design process as a result of the development of novel algorithms, growing access, the falling cost of computation and the development of novel technologies for generating chemically and biologically relevant data. There has been recent progress in fields such as molecular de novo generation, synthetic route prediction and, to some extent, property predictions. Despite this, most research in these fields has focused on improving the accuracy of the technologies, rather than on quantifying the uncertainty in the predictions. Uncertainty quantification will become a key component in autonomous decision making and will be crucial for integrating machine learning and chemistry automation to create an autonomous design-make-test-analyse cycle. This review covers the empirical, frequentist and Bayesian approaches to uncertainty quantification, and outlines how they can be used for drug design. We also outline the impact of uncertainty quantification on decision making.
Collapse
Affiliation(s)
- Lewis H Mervin
- Molecular AI, Discovery Sciences, R&D, AstraZeneca, Cambridge, UK.
| | - Simon Johansson
- Department of Computer Science and Engineering, Chalmers University of Technology, Gothenburg, Sweden; Molecular AI, Discovery Sciences, R&D, AstraZeneca, Gothenburg, Sweden
| | - Elizaveta Semenova
- Data Sciences and Quantitative Biology, Discovery Sciences, R&D, AstraZeneca, Cambridge, UK
| | - Kathryn A Giblin
- Medicinal Chemistry, Research and Early Development, Oncology R&D, AstraZeneca, Cambridge, UK
| | - Ola Engkvist
- Molecular AI, Discovery Sciences, R&D, AstraZeneca, Gothenburg, Sweden
| |
Collapse
|
7
|
Mervin LH, Afzal AM, Engkvist O, Bender A. Comparison of Scaling Methods to Obtain Calibrated Probabilities of Activity for Protein–Ligand Predictions. J Chem Inf Model 2020; 60:4546-4559. [DOI: 10.1021/acs.jcim.0c00476] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/16/2022]
Affiliation(s)
- Lewis H. Mervin
- Hit Discovery, Discovery Sciences, R&D, AstraZeneca, Cambridge CB2 0AA, U.K
| | - Avid M. Afzal
- Data Sciences & Quantitative Biology, Discovery Sciences, R&D, AstraZeneca, Cambridge CB2 0AA, U.K
| | - Ola Engkvist
- Hit Discovery, Discovery Sciences, R&D, AstraZeneca, Mölndal SE-431 83, Sweden
| | - Andreas Bender
- Centre for Molecular Informatics, Department of Chemistry, University of Cambridge, Cambridge CB2 1TN, U.K
| |
Collapse
|