Machine Learning Methods in Health Economics and Outcomes Research-The PALISADE Checklist: A Good Practices Report of an ISPOR Task Force

Research Method, Conduct, and Reporting Considerations for Improving the Quality of Non-Hypothesis-Evaluating Treatment Effectiveness Analyses Using Real-World Data: An ISPOR Special Interest Group Report

OBJECTIVES

Numerous real-world evidence guidance documents have been published by many regulatory agencies, health technology assessment agencies, and academic groups. These guidances are largely focused on generating unbiased treatment effect estimates from real-world data (RWD) for use in approval and coverage decisions. One of the most prominently cited documents, is a joint ISPOR/ISPE Task Force guidance on the use of RWD for Hypothesis-Evaluating Treatment Effectiveness (HETE) Studies. However, there are a variety of uses of RWD that existing guidance does not explicitly address. Our goal was to assess existing guidance for its relevance to a range of non-HETE analyses.

METHODS

In this article, we delineate categories of non-HETE applications of RWD. This range of applications includes various types of non-HETE RWD analyses, as well as the use of RWD as inputs into other types of analyses, such as health economic modeling or simulation studies that are not typically considered RWD studies. We then map existing guidance documents to the categories of non-HETE studies to help identify existing resources for analysts interested in conducting these types of analyses.

RESULTS

We identified 13 published guidance documents for detailed review, including documents that outline best practices, provide checklists, or are structured templates. They were selected for their prominence, recency, specific reference to fit-for-purpose real-world evidence generation, or their relevance to specific analysis categories.

CONCLUSIONS

We conclude that existing guidance documents, although they were largely developed for HETE studies, are highly useful for the broad range of non-HETE RWD analyses as well.

How much can we save by applying artificial intelligence in evidence synthesis? Results from a pragmatic review to quantify workload efficiencies and cost savings

Introduction

Researchers are increasingly exploring the use of artificial intelligence (AI) tools in evidence synthesis, a labor-intensive, time-consuming, and costly effort. This review explored and quantified the potential efficiency benefits of using automated tools as part of core evidence synthesis activities compared with human-led methods.

Methods

We searched the MEDLINE and Embase databases for English-language articles published between 2012 and 14 November 2023, and hand-searched the ISPOR presentations database (2020-2023) for articles presenting quantitative results on workload efficiency in systematic literature reviews (SLR) when AI automation tools were utilized. Data on efficiencies (time- and cost-related) were collected.

Results

We identified 25 eligible studies: 13 used machine learning, 10 used natural language processing, and once each used a systematic review automation tool and a non-specified AI tool. In 17 studies, a >50% time reduction was observed, with 5-to 6-fold decreases in abstract review time. When the number of abstracts reviewed was examined, decreases of 55%-64% were noted. Studies examining work saved over sampling at 95% recall reported 6- to 10-fold decreases in workload with automation. No studies quantified the economic impact associated with automation, although one study found that there was an overall labor reduction of >75% over manual methods during dual-screen reviews.

Discussion

AI can reduce both workload and create time efficiencies when applied to evidence gathering efforts in SLRs. These improvements can facilitate the implementation of novel approaches in decision making that consider the real-life value of health technologies. Further research should quantify the economic impact of automation in SLRs.

R WE ready for reimbursement? A round up of developments in real-world evidence relating to health technology assessment: part 17

In this update, we discuss a position statement from the National Institute of Health and Care Excellence (NICE) on the use of artificial intelligence for evidence generation and publications reviewing the use of real-world data as external control arms. Finally, we discuss a number of recent studies investigating the real-world effectiveness of glucagon-like peptide-1 receptor agonists and whether these studies are informative for reimbursement decision making.

Biological age and predicting future health care utilisation

We explore the role of epigenetic biological age in predicting subsequent health care utilisation. We use longitudinal data from the UK Understanding Society panel, capitalising on the availability of baseline epigenetic biological age measures along with data on general practitioner (GP) consultations, outpatient (OP) visits, and hospital inpatient (IP) care collected 5-12 years from baseline. Using least absolute shrinkage and selection operator (LASSO) regression analyses and accounting for participants' pre-existing health conditions, baseline biological underlying health, and socio-economic predictors we find that biological age is selected as a predictor of future GP consultations and IP care, while chronological rather than biological age is selected for future OP visits. Post-selection prediction analysis and Shapley-Shorrocks decompositions, comparing our preferred prediction models to models that replace biological age with chronological age, suggest that biological ageing has a stronger role in the models predicting future IP care as opposed to "gatekeeping" GP consultations.

Incorporating healthcare access and equity in economic evaluations: a scoping review of guidelines

BACKGROUND

International development agendas increasingly push for access to healthcare for all through universal healthcare coverage. Health economic evaluations and health technology assessment (HTA) could provide evidence to support this but do not routinely incorporate consideration of equitable access.

METHODS

We undertook an international scoping review of health economic evaluation and HTA guidelines to examine how well issues of healthcare access and equity are represented, evidence recommendations, and gaps in current guidance to support evidence generation in this area. Guidelines were sourced from guideline repositories and websites of international agencies and organizations providing best practice methods guidance. Articles providing methods guidance for the conduct of HTA, or health economic evaluation, were included, except where they were not available in English and a suitable translation could not be obtained.

RESULTS

The search yielded forty-seven national, four international, and nine independent guidelines, along with eighty-six articles providing specific methods guidance. The inclusion of equity and access considerations in current guidance is extremely limited. Where they do feature, detail on specific methods for providing evidence on these issues is sparse.

DISCUSSION

Economic evaluation could be a valuable tool to provide evidence for the best healthcare strategies that not only maximize health but also ensure equitable access to care for all. Such evidence would be invaluable in supporting progress towards universal healthcare coverage. Clear guidance is required to ensure evaluations provide evidence on the best strategies to support equitable access to healthcare, but such guidance rarely exists in current best practice and guidance documents.

Machine Learning Algorithms to Estimate Propensity Scores in Health Policy Evaluation: A Scoping Review

(1) Background: Quasi-experimental design has been widely used in causal inference for health policy impact evaluation. However, due to the non-randomized treatment used, there is great potential for bias in the assessment of the results, which can be reduced by using propensity score (PS) methods. In this context, this article aims to map the literature concerning the use of machine learning (ML) algorithms for propensity score estimation. (2) Methods: A scoping review was carried out in the PubMed, EMBASE, ACM Digital Library, IEEE Explore, LILACS, Web of Science, Scopus, Compendex, and gray literature (ProQuest and Google Scholar) databases, based on the PRISMA-ScR guidelines. This scoping review aims to identify ML models and their accuracy and the characteristics of studies on causal inference for health policy impacts, with a specific focus on PS estimation using ML. (3) Results: Seven studies were included in the review from 3018 references searched. In general, tree-based ML models were used for PS estimation. Most of the studies did not show or mention the performance metrics of the selected models, focusing instead on discussing the treatment effects under analysis. (4) Conclusions: Despite important aspects of model development and evaluation being under-reported, this scoping review provides insights into the recent use of ML algorithms in health policy impact evaluation.

An Approach for Combining Clinical Judgment with Machine Learning to Inform Medical Decision Making: Analysis of Nonemergency Surgery Strategies for Acute Appendicitis in Patients with Multiple Long-Term Conditions

BACKGROUND

Machine learning (ML) methods can identify complex patterns of treatment effect heterogeneity. However, before ML can help to personalize decision making, transparent approaches must be developed that draw on clinical judgment. We develop an approach that combines clinical judgment with ML to generate appropriate comparative effectiveness evidence for informing decision making.

METHODS

We motivate this approach in evaluating the effectiveness of nonemergency surgery (NES) strategies, such as antibiotic therapy, for people with acute appendicitis who have multiple long-term conditions (MLTCs) compared with emergency surgery (ES). Our 4-stage approach 1) draws on clinical judgment about which patient characteristics and morbidities modify the relative effectiveness of NES; 2) selects additional covariates from a high-dimensional covariate space (P > 500) by applying an ML approach, least absolute shrinkage and selection operator (LASSO), to large-scale administrative data (N = 24,312); 3) generates estimates of comparative effectiveness for relevant subgroups; and 4) presents evidence in a suitable form for decision making.

RESULTS

This approach provides useful evidence for clinically relevant subgroups. We found that overall NES strategies led to increases in the mean number of days alive and out-of-hospital compared with ES, but estimates differed across subgroups, ranging from 21.2 (95% confidence interval: 1.8 to 40.5) for patients with chronic heart failure and chronic kidney disease to -10.4 (-29.8 to 9.1) for patients with cancer and hypertension. Our interactive tool for visualizing ML output allows for findings to be customized according to the specific needs of the clinical decision maker.

CONCLUSIONS

This principled approach of combining clinical judgment with an ML approach can improve trust, relevance, and usefulness of the evidence generated for clinical decision making.

HIGHLIGHTS

Machine learning (ML) methods have many potential applications in medical decision making, but the lack of model interpretability and usability constitutes an important barrier for the wider adoption of ML evidence in practice.We develop a 4-stage approach for integrating clinical judgment into the way an ML approach is used to estimate and report comparative effectiveness.We illustrate the approach in undertaking an evaluation of nonemergency surgery (NES) strategies for acute appendicitis in patients with multiple long-term conditions and find that NES strategies lead to better outcomes compared with emergency surgery and that the effects differ across subgroups.We develop an interactive tool for visualizing the results of this study that allows findings to be customized according to the user's preferences.

Pediatric Blood Pressure and Cardiovascular Health in Adulthood

PURPOSE OF REVIEW

This review summarizes current knowledge on blood pressure in children and adolescents (youth), with a focus on primary hypertension-the most common form of elevated blood pressure in this demographic. We examine its etiology, progression, and long-term cardiovascular implications. The review covers definitions and recommendations of blood pressure classifications, recent developments in measurement, epidemiological trends, findings from observational and clinical studies, and prevention and treatment, while identifying gaps in understanding and suggesting future research directions.

RECENT FINDINGS

Youth hypertension is an escalating global issue, with regional and national variations in prevalence. While the principles of blood pressure measurement have remained largely consistent, challenges in this age group include a scarcity of automated devices that have passed independent validation for accuracy and a generally limited tolerance for ambulatory blood pressure monitoring. A multifaceted interplay of factors contributes to youth hypertension, impacting long-term cardiovascular health. Recent studies, including meta-analysis and sophisticated life-course modelling, reveal an adverse link between youth and life-course blood pressure and subclinical cardiovascular outcomes later in life. New evidence now provides the strongest evidence yet linking youth blood pressure with clinical cardiovascular events in adulthood. Some clinical trials have expanded our understanding of the safety and efficacy of antihypertensive medications in youth, but this remains an area that requires additional attention, particularly regarding varied screening approaches. This review outlines the potential role of preventing and managing blood pressure in youth to reduce future cardiovascular risk. A global perspective is necessary in formulating blood pressure definitions and strategies, considering the specific needs and circumstances in low- and middle-income countries compared to high-income countries.

Core Concepts in Pharmacoepidemiology: Principled Use of Artificial Intelligence and Machine Learning in Pharmacoepidemiology and Healthcare Research

Artificial intelligence (AI) and machine learning (ML) are important tools across many fields of health and medical research. Pharmacoepidemiologists can bring essential methodological rigor and study design expertise to the design and use of these technologies within healthcare settings. AI/ML-based tools also play a role in pharmacoepidemiology research, as we may apply them to answer our own research questions, take responsibility for evaluating medical devices with AI/ML components, or participate in interdisciplinary research to create new AI/ML algorithms. While epidemiologic expertise is essential to deploying AI/ML responsibly and ethically, the rapid advancement of these technologies in the past decade has resulted in a knowledge gap for many in the field. This article provides a brief overview of core AI/ML concepts, followed by a discussion of potential applications of AI/ML in pharmacoepidemiology research, and closes with a review of important concepts across application areas, including interpretability and fairness. This review is intended to provide an accessible, practical overview of AI/ML for pharmacoepidemiology research, with references to further, more detailed resources on fundamental topics.

Comparing Machine Learning and Advanced Methods with Traditional Methods to Generate Weights in Inverse Probability of Treatment Weighting: The INFORM Study

Purpose

Observational research provides valuable insights into treatments used in patient populations in real-world settings. However, confounding is likely to occur if there are differences in patient characteristics associated with both the exposure and outcome between the groups being evaluated. One approach to reduce confounding and facilitate unbiased comparisons is inverse probability of treatment weighting (IPTW) using propensity scores. Machine learning (ML) and entropy balancing can potentially be used in generating propensity scores for IPTW, but there is limited literature on this application. We aimed to assess the feasibility of applying these methods for reducing confounding in observational studies. These methods were assessed in a study comparing cardiovascular outcomes in adults with type 2 diabetes and established atherosclerotic cardiovascular disease taking once-weekly glucagon-like peptide-1 receptor agonists or dipeptidyl peptidase-4 inhibitors.

Methods

We applied advanced methods to generate the propensity scores compared to the original logistic regression method in terms of covariate balance. After calculating weights, a weighted Cox proportional hazards model was used to calculate the sample average treatment effect. Support Vector Classification, Support Vector Regression, XGBoost, and LightGBM were the ML models used. Entropy balancing was also performed on features identified in the original cardiovascular outcomes study.

Results

Accuracy (range: 0.71 to 0.73), area under the curve (0.77 to 0.79), precision (0.53 to 0.60), recall (0.66 to 0.68), and F1 score (0.60 to 0.64) were similar between all of the advanced propensity score methods and traditional logistic regression. Among ML models, only XGBoost achieved balance in all measured baseline characteristics between the two treatment groups, closely approximating the performance of the original logistic regression. Entropy balancing weights provided the best performance among all models in balancing baseline characteristics, achieving near perfect balancing.

Conclusion

Among the advanced methods examined, entropy balancing weights performed the best for optimizing balancing and can produce similar results compared to traditional logistic regression.

Machine Learning Methods to Estimate Individualized Treatment Effects for Use in Health Technology Assessment

BACKGROUND

Recent developments in causal inference and machine learning (ML) allow for the estimation of individualized treatment effects (ITEs), which reveal whether treatment effectiveness varies according to patients' observed covariates. ITEs can be used to stratify health policy decisions according to individual characteristics and potentially achieve greater population health. Little is known about the appropriateness of available ML methods for use in health technology assessment.

METHODS

In this scoping review, we evaluate ML methods available for estimating ITEs, aiming to help practitioners assess their suitability in health technology assessment. We present a taxonomy of ML approaches, categorized by key challenges in health technology assessment using observational data, including handling time-varying confounding and time-to event data and quantifying uncertainty.

RESULTS

We found a wide range of algorithms for simpler settings with baseline confounding and continuous or binary outcomes. Not many ML algorithms can handle time-varying or unobserved confounding, and at the time of writing, no ML algorithm was capable of estimating ITEs for time-to-event outcomes while accounting for time-varying confounding. Many of the ML algorithms that estimate ITEs in longitudinal settings do not formally quantify uncertainty around the point estimates.

LIMITATIONS

This scoping review may not cover all relevant ML methods and algorithms as they are continuously evolving.

CONCLUSIONS

Existing ML methods available for ITE estimation are limited in handling important challenges posed by observational data when used for cost-effectiveness analysis, such as time-to-event outcomes, time-varying and hidden confounding, or the need to estimate sampling uncertainty around the estimates.

IMPLICATIONS

ML methods are promising but need further development before they can be used to estimate ITEs for health technology assessments.

HIGHLIGHTS

Estimating individualized treatment effects (ITEs) using observational data and machine learning (ML) can support personalized treatment advice and help deliver more customized information on the effectiveness and cost-effectiveness of health technologies.ML methods for ITE estimation are mostly designed for handling confounding at baseline but not time-varying or unobserved confounding. The few models that account for time-varying confounding are designed for continuous or binary outcomes, not time-to-event outcomes.Not all ML methods for estimating ITEs can quantify the uncertainty of their predictions.Future work on developing ML that addresses the concerns summarized in this review is needed before these methods can be widely used in clinical and health technology assessment-like decision making.

Utility analysis and demonstration of real-world clinical texts: A case study on Japanese cancer-related EHRs

Real-world data (RWD) in the medical field, such as electronic health records (EHRs) and medication orders, are receiving increasing attention from researchers and practitioners. While structured data have played a vital role thus far, unstructured data represented by text (e.g., discharge summaries) are not effectively utilized because of the difficulty in extracting medical information. We evaluated the information gained by supplementing structured data with clinical concepts extracted from unstructured text by leveraging natural language processing techniques. Using a machine learning-based pretrained named entity recognition tool, we extracted disease and medication names from real discharge summaries in a Japanese hospital and linked them to medical concepts using medical term dictionaries. By comparing the diseases and medications mentioned in the text with medical codes in tabular diagnosis records, we found that: (1) the text data contained richer information on patient symptoms than tabular diagnosis records, whereas the medication-order table stored more injection data than text. In addition, (2) extractable information regarding specific diseases showed surprisingly small intersections among text, diagnosis records, and medication orders. Text data can thus be a useful supplement for RWD mining, which is further demonstrated by (3) our practical application system for drug safety evaluation, which exhaustively visualizes suspicious adverse drug effects caused by the simultaneous use of anticancer drug pairs. We conclude that proper use of textual information extraction can lead to better outcomes in medical RWD mining.

Meta-research on reporting guidelines for artificial intelligence: are authors and reviewers encouraged enough in radiology, nuclear medicine, and medical imaging journals?

PURPOSE

To determine how radiology, nuclear medicine, and medical imaging journals encourage and mandate the use of reporting guidelines for artificial intelligence (AI) in their author and reviewer instructions.

METHODS

The primary source of journal information and associated citation data used was the Journal Citation Reports (June 2023 release for 2022 citation data; Clarivate Analytics, UK). The first- and second-quartile journals indexed in the Science Citation Index Expanded and the Emerging Sources Citation Index were included. The author and reviewer instructions were evaluated by two independent readers, followed by an additional reader for consensus, with the assistance of automatic annotation. Encouragement and submission requirements were systematically analyzed. The reporting guidelines were grouped as AI-specific, related to modeling, and unrelated to modeling.

RESULTS

Out of 102 journals, 98 were included in this study, and all of them had author instructions. Only five journals (5%) encouraged the authors to follow AI-specific reporting guidelines. Among these, three required a filled-out checklist. Reviewer instructions were found in 16 journals (16%), among which one journal (6%) encouraged the reviewers to follow AI-specific reporting guidelines without submission requirements. The proportions of author and reviewer encouragement for AI-specific reporting guidelines were statistically significantly lower compared with those for other types of guidelines (P < 0.05 for all).

CONCLUSION

The findings indicate that AI-specific guidelines are not commonly encouraged and mandated (i.e., requiring a filled-out checklist) by these journals, compared with guidelines related to modeling and unrelated to modeling, leaving vast space for improvement. This meta-research study hopes to contribute to the awareness of the imaging community for AI reporting guidelines and ignite large-scale group efforts by all stakeholders, making AI research less wasteful.

CLINICAL SIGNIFICANCE

This meta-research highlights the need for improved encouragement of AI-specific guidelines in radiology, nuclear medicine, and medical imaging journals. This can potentially foster greater awareness among the AI community and motivate various stakeholders to collaborate to promote more efficient and responsible AI research reporting practices.

Integrating decision modeling and machine learning to inform treatment stratification

There is increasing interest in moving away from "one size fits all (OSFA)" approaches toward stratifying treatment decisions. Understanding how expected effectiveness and cost-effectiveness varies with patient covariates is a key aspect of stratified decision making. Recently proposed machine learning (ML) methods can learn heterogeneity in outcomes without pre-specifying subgroups or functional forms, enabling the construction of decision rules ('policies') that map individual covariates into a treatment decision. However, these methods do not yet integrate ML estimates into a decision modeling framework in order to reflect long-term policy-relevant outcomes and synthesize information from multiple sources. In this paper, we propose a method to integrate ML and decision modeling, when individual patient data is available to estimate treatment-specific survival time. We also propose a novel implementation of policy tree algorithms to define subgroups using decision model output. We demonstrate these methods using the SPRINT (Systolic Blood Pressure Intervention Trial), comparing outcomes for "standard" and "intensive" blood pressure targets. We find that including ML into a decision model can impact the estimate of incremental net health benefit (INHB) for OSFA policies. We also find evidence that stratifying treatment using subgroups defined by a tree-based algorithm can increase the estimates of the INHB.

Advances in Addressing Patient Heterogeneity in Economic Evaluation: A Review of the Methods Literature

Cost-effectiveness analyses commonly use population or sample averages, which can mask key differences across subgroups and may lead to suboptimal resource allocation. Despite there being several new methods developed over the last decade, there is no recent summary of what methods are available to researchers. This review sought to identify advances in methods for addressing patient heterogeneity in economic evaluations and to provide an overview of these methods. A literature search was conducted using the Econlit, Embase and MEDLINE databases to identify studies published after 2011 (date of a previous review on this topic). Eligible studies needed to have an explicit methodological focus, related to how patient heterogeneity can be accounted for within a full economic evaluation. Sixteen studies were included in the review. Methodologies were varied and included regression techniques, model design and value of information analysis. Recent publications have applied methodologies more commonly used in other fields, such as machine learning and causal forests. Commonly noted challenges associated with considering patient heterogeneity included data availability (e.g., sample size), statistical issues (e.g., risk of false positives) and practical factors (e.g., computation time). A range of methods are available to address patient heterogeneity in economic evaluation, with relevant methods differing according to research question, scope of the economic evaluation and data availability. Researchers need to be aware of the challenges associated with addressing patient heterogeneity (e.g., data availability) to ensure findings are meaningful and robust. Future research is needed to assess whether and how methods are being applied in practice.

Assessing Real-World Data From Electronic Health Records for Health Technology Assessment: The SUITABILITY Checklist: A Good Practices Report of an ISPOR Task Force

This ISPOR Good Practices report provides a framework for assessing the suitability of electronic health records data for use in health technology assessments (HTAs). Although electronic health record (EHR) data can fill evidence gaps and improve decisions, several important limitations can affect its validity and relevance. The ISPOR framework includes 2 components: data delineation and data fitness for purpose. Data delineation provides a complete understanding of the data and an assessment of its trustworthiness by describing (1) data characteristics; (2) data provenance; and (3) data governance. Fitness for purpose comprises (1) data reliability items, ie, how accurate and complete the estimates are for answering the question at hand and (2) data relevance items, which assess how well the data are suited to answer the particular question from a decision-making perspective. The report includes a checklist specific to EHR data reporting: the ISPOR SUITABILITY Checklist. It also provides recommendations for HTA agencies and policy makers to improve the use of EHR-derived data over time. The report concludes with a discussion of limitations and future directions in the field, including the potential impact from the substantial and rapid advances in the diffusion and capabilities of large language models and generative artificial intelligence. The report's immediate audiences are HTA evidence developers and users. We anticipate that it will also be useful to other stakeholders, particularly regulators and manufacturers, in the future.

Persistence and heterogeneity of the effects of educating mothers to improve child immunisation uptake: Experimental evidence from Uttar Pradesh in India

Childhood vaccinations are among the most cost-effective health interventions. Yet, in India, where immunisation services are widely available free of charge, a substantial proportion of children remain unvaccinated. We revisit households 30 months after a randomised experiment of a health information intervention designed to educate mothers on the benefits of child vaccination in Uttar Pradesh, India. We find that the large short-term effects on the uptake of diphtheria-pertussis-tetanus and measles vaccination were sustained at 30 months, suggesting the intervention did not simply bring forward vaccinations. We apply causal forests and find that the intervention increased vaccination uptake, but that there was substantial variation in the magnitude of the estimated effects. We conclude that characterising those who benefited most and conversely those who benefited least provides policy-makers with insights on how the intervention worked, and how the targeting of households could be improved.

R WE ready for reimbursement? A round up of developments in real-world evidence relating to health technology assessment: part 15

In this latest update we discuss real-world evidence (RWE) guidance from the leading oncology professional societies, the American Society of Clinical Oncology and the European Society for Medical Oncology, and the PRINCIPLED practical guide on the design and analysis of causal RWE studies.

Acceptability of Using Real-World Data to Estimate Relative Treatment Effects in Health Technology Assessments: Barriers and Future Steps

OBJECTIVES

Evidence about the comparative effects of new treatments is typically collected in randomized controlled trials (RCTs). In some instances, RCTs are not possible, or their value is limited by an inability to capture treatment effects over the longer term or in all relevant population subgroups. In these cases, nonrandomized studies (NRS) using real-world data (RWD) are increasingly used to complement trial evidence on treatment effects for health technology assessment (HTA). However, there have been concerns over a lack of acceptability of this evidence by HTA agencies. This article aims to identify the barriers to the acceptance of NRS and steps that may facilitate increases in the acceptability of NRS in the future.

METHODS

Opinions of the authorship team based on their experience in real-world evidence research in academic, HTA, and industry settings, supported by a critical assessment of existing studies.

RESULTS

Barriers were identified that are applicable to key stakeholder groups, including HTA agencies (eg, the lack of comprehensive methodological guidelines for using RWD), evidence generators (eg, avoidable deviations from best practices), and external stakeholders (eg, data controllers providing timely access to high-quality RWD). Future steps that may facilitate future acceptability of NRS include improvements in the quality, integration, and accessibility of RWD, wider use of demonstration projects to highlight the value and applicability of nonrandomized designs, living, and more detailed HTA guidelines, and improvements in HTA infrastructure relating to RWD.

CONCLUSION

NRS can represent a crucial source of evidence on treatment effects for use in HTA when RCT evidence is limited.

Predicting pressure injury risk in hospitalised patients using machine learning with electronic health records: a US multilevel cohort study

OBJECTIVE

To predict the risk of hospital-acquired pressure injury using machine learning compared with standard care.

DESIGN

We obtained electronic health records (EHRs) to structure a multilevel cohort of hospitalised patients at risk for pressure injury and then calibrate a machine learning model to predict future pressure injury risk. Optimisation methods combined with multilevel logistic regression were used to develop a predictive algorithm of patient-specific shifts in risk over time. Machine learning methods were tested, including random forests, to identify predictive features for the algorithm. We reported the results of the regression approach as well as the area under the receiver operating characteristics (ROC) curve for predictive models.

SETTING

Hospitalised inpatients.

PARTICIPANTS

EHRs of 35 001 hospitalisations over 5 years across 2 academic hospitals.

MAIN OUTCOME MEASURE

Longitudinal shifts in pressure injury risk.

RESULTS

The predictive algorithm with features generated by machine learning achieved significantly improved prediction of pressure injury risk (p<0.001) with an area under the ROC curve of 0.72; whereas standard care only achieved an area under the ROC curve of 0.52. At a specificity of 0.50, the predictive algorithm achieved a sensitivity of 0.75.

CONCLUSIONS

These data could help hospitals conserve resources within a critical period of patient vulnerability of hospital-acquired pressure injury which is not reimbursed by US Medicare; thus, conserving between 30 000 and 90 000 labour-hours per year in an average 500-bed hospital. Hospitals can use this predictive algorithm to initiate a quality improvement programme for pressure injury prevention and further customise the algorithm to patient-specific variation by facility.

Artificial Intelligence to Automate Health Economic Modelling: A Case Study to Evaluate the Potential Application of Large Language Models

BACKGROUND

Current generation large language models (LLMs) such as Generative Pre-Trained Transformer 4 (GPT-4) have achieved human-level performance on many tasks including the generation of computer code based on textual input. This study aimed to assess whether GPT-4 could be used to automatically programme two published health economic analyses.

METHODS

The two analyses were partitioned survival models evaluating interventions in non-small cell lung cancer (NSCLC) and renal cell carcinoma (RCC). We developed prompts which instructed GPT-4 to programme the NSCLC and RCC models in R, and which provided descriptions of each model's methods, assumptions and parameter values. The results of the generated scripts were compared to the published values from the original, human-programmed models. The models were replicated 15 times to capture variability in GPT-4's output.

RESULTS

GPT-4 fully replicated the NSCLC model with high accuracy: 100% (15/15) of the artificial intelligence (AI)-generated NSCLC models were error-free or contained a single minor error, and 93% (14/15) were completely error-free. GPT-4 closely replicated the RCC model, although human intervention was required to simplify an element of the model design (one of the model's fifteen input calculations) because it used too many sequential steps to be implemented in a single prompt. With this simplification, 87% (13/15) of the AI-generated RCC models were error-free or contained a single minor error, and 60% (9/15) were completely error-free. Error-free model scripts replicated the published incremental cost-effectiveness ratios to within 1%.

CONCLUSION

This study provides a promising indication that GPT-4 can have practical applications in the automation of health economic model construction. Potential benefits include accelerated model development timelines and reduced costs of development. Further research is necessary to explore the generalisability of LLM-based automation across a larger sample of models.

American clusters: using machine learning to understand health and health care disparities in the United States

Health and health care access in the United States are plagued by high inequality. While machine learning (ML) is increasingly used in clinical settings to inform health care delivery decisions and predict health care utilization, using ML as a research tool to understand health care disparities in the United States and how these are connected to health outcomes, access to health care, and health system organization is less common. We utilized over 650 variables from 24 different databases aggregated by the Agency for Healthcare Research and Quality in their Social Determinants of Health (SDOH) database. We used k-means-a non-hierarchical ML clustering method-to cluster county-level data. Principal factor analysis created county-level index values for each SDOH domain and 2 health care domains: health care infrastructure and health care access. Logistic regression classification was used to identify the primary drivers of cluster classification. The most efficient cluster classification consists of 3 distinct clusters in the United States; the cluster having the highest life expectancy comprised only 10% of counties. The most efficient ML clusters do not identify the clusters with the widest health care disparities. ML clustering, using county-level data, shows that health care infrastructure and access are the primary drivers of cluster composition.

Transporting Comparative Effectiveness Evidence Between Countries: Considerations for Health Technology Assessments

Internal validity is often the primary concern for health technology assessment agencies when assessing comparative effectiveness evidence. However, the increasing use of real-world data from countries other than a health technology assessment agency's target population in effectiveness research has increased concerns over the external validity, or "transportability", of this evidence, and has led to a preference for local data. Methods have been developed to enable a lack of transportability to be addressed, for example by accounting for cross-country differences in disease characteristics, but their consideration in health technology assessments is limited. This may be because of limited knowledge of the methods and/or uncertainties in how best to utilise them within existing health technology assessment frameworks. This article aims to provide an introduction to transportability, including a summary of its assumptions and the methods available for identifying and adjusting for a lack of transportability, before discussing important considerations relating to their use in health technology assessment settings, including guidance on the identification of effect modifiers, guidance on the choice of target population, estimand, study sample and methods, and how evaluations of transportability can be integrated into health technology assessment submission and decision processes.

Advancements and Applications of Artificial Intelligence in Pharmaceutical Sciences: A Comprehensive Review

Artificial intelligence (AI) has revolutionized the pharmaceutical industry, improving drug discovery, development, and personalized patient care. Through machine learning (ML), deep learning, natural language processing (NLP), and robotic automation, AI has enhanced efficiency, accuracy, and innovation in the field. The purpose of this review is to shed light on the practical applications and potential of AI in various pharmaceutical fields. These fields include medicinal chemistry, pharmaceutics, pharmacology and toxicology, clinical pharmacy, pharmaceutical biotechnology, pharmaceutical nanotechnology, pharmacognosy, and pharmaceutical management and economics. By leveraging AI technologies such as ML, deep learning, NLP, and robotic automation, this review delves into the role of AI in enhancing drug discovery, development processes, and personalized patient care. It analyzes AI's impact in specific areas such as drug synthesis planning, formulation development, toxicology predictions, pharmacy automation, and market analysis. Artificial intelligence integration into pharmaceutical sciences has significantly improved medicinal chemistry, drug discovery, and synthesis planning. In pharmaceutics, AI has advanced personalized medicine and formulation development. In pharmacology and toxicology, AI offers predictive capabilities for drug mechanisms and toxic effects. In clinical pharmacy, AI has facilitated automation and enhanced patient care. Additionally, AI has contributed to protein engineering, gene therapy, nanocarrier design, discovery of natural product therapeutics, and pharmaceutical management and economics, including marketing research and clinical trials management. Artificial intelligence has transformed pharmaceuticals, improving efficiency, accuracy, and innovation. This review highlights AI's role in drug development and personalized care, serving as a reference for professionals. The future promises a revolutionized field with AI-driven methodologies.

Systematic reviews of machine learning in healthcare: a literature review

INTRODUCTION

The increasing availability of data and computing power has made machine learning (ML) a viable approach to faster, more efficient healthcare delivery.

METHODS

A systematic literature review (SLR) of published SLRs evaluating ML applications in healthcare settings published between1 January 2010 and 27 March 2023 was conducted.

RESULTS

In total 220 SLRs covering 10,462 ML algorithms were reviewed. The main application of AI in medicine related to the clinical prediction and disease prognosis in oncology and neurology with the use of imaging data. Accuracy, specificity, and sensitivity were provided in 56%, 28%, and 25% SLRs respectively. Internal and external validation was reported in 53% and less than 1% of the cases respectively. The most common modeling approach was neural networks (2,454 ML algorithms), followed by support vector machine and random forest/decision trees (1,578 and 1,522 ML algorithms, respectively).

EXPERT OPINION

The review indicated considerable reporting gaps in terms of the ML's performance, both internal and external validation. Greater accessibility to healthcare data for developers can ensure the faster adoption of ML algorithms into clinical practice.

Seeing the whole elephant: integrated advanced data analytics in support of RWE for the development and use of innovative pharmaceuticals

INTRODUCTION

The 21st century has brought about significant technological advancement, allowing the collection of new types of data from the real world on an unprecedented scale. The healthcare industry will benefit immensely from this abundance of patient data from electronic health records (EHR), patient-reported outcomes (PROs), laboratory, demographic, social media, digital, and even climate data.

AREAS COVERED

While conventional statistical methods still play a significant role in supporting the drug lifecycle, machine learning (ML) and artificial intelligence (AI) are assuming a more prominent role in the analysis of this 'big data.' Moving forward, conventional statistics and AI/ML will work together to support descriptive, diagnostic, and even predictive analytics to further revolutionize drug discovery and development, regulatory approvals, and payer acceptance. In addition, counterfactual prescriptive analytics, such as causal inference analysis using real-world data (RWD) to generate insights that have cause-and-effect conclusions, will gain momentum as a methodology that can stand up against the rigor of regulatory review.

EXPERT OPINION

Our real-world evidence/health economics and outcomes research (RWE/HEOR) field has evolved in ways that require us to integrate all the methods and data into a single framework that guides a holistic analytic approach and decision-making.

The past, current, and future of neonatal intensive care units with artificial intelligence: a systematic review

Machine learning and deep learning are two subsets of artificial intelligence that involve teaching computers to learn and make decisions from any sort of data. Most recent developments in artificial intelligence are coming from deep learning, which has proven revolutionary in almost all fields, from computer vision to health sciences. The effects of deep learning in medicine have changed the conventional ways of clinical application significantly. Although some sub-fields of medicine, such as pediatrics, have been relatively slow in receiving the critical benefits of deep learning, related research in pediatrics has started to accumulate to a significant level, too. Hence, in this paper, we review recently developed machine learning and deep learning-based solutions for neonatology applications. We systematically evaluate the roles of both classical machine learning and deep learning in neonatology applications, define the methodologies, including algorithmic developments, and describe the remaining challenges in the assessment of neonatal diseases by using PRISMA 2020 guidelines. To date, the primary areas of focus in neonatology regarding AI applications have included survival analysis, neuroimaging, analysis of vital parameters and biosignals, and retinopathy of prematurity diagnosis. We have categorically summarized 106 research articles from 1996 to 2022 and discussed their pros and cons, respectively. In this systematic review, we aimed to further enhance the comprehensiveness of the study. We also discuss possible directions for new AI models and the future of neonatology with the rising power of AI, suggesting roadmaps for the integration of AI into neonatal intensive care units.

Predicting return to work after traumatic brain injury using machine learning and administrative data

BACKGROUND

Accurate patient-specific predictions on return-to-work after traumatic brain injury (TBI) can support both clinical practice and policymaking. The use of machine learning on large administrative data provides interesting opportunities to create such prognostic models.

AIM

The current study assesses whether return-to-work one year after TBI can be predicted accurately from administrative data. Additionally, this study explores how model performance and feature importance change depending on whether a distinction is made between mild and moderate-to-severe TBI.

METHODS

This study used a population-based dataset that combined discharge, claims and social security data of patients hospitalized with a TBI in Belgium during the year 2016. The prediction of TBI was attempted with three algorithms, elastic net logistic regression, random forest and gradient boosting and compared in their performance by their accuracy, sensitivity, specificity and area under the receiver operator curve (ROC AUC).

RESULTS

The distinct modelling algorithms resulted in similar results, with 83% accuracy (ROC AUC 85%) for a binary classification of employed vs. not employed and up to 76% (ROC AUC 82%) for a multiclass operationalization of employment outcome. Modelling mild and moderate-to-severe TBI separately did not result in considerable differences in model performance and feature importance. The features of main importance for return-to-work prediction were related to pre-injury employment.

DISCUSSION

While clearly offering some information beneficial for predicting return-to-work, administrative data needs to be supplemented with additional information to allow further improvement of patient-specific prognose.

Approach to machine learning for extraction of real-world data variables from electronic health records

Background: As artificial intelligence (AI) continues to advance with breakthroughs in natural language processing (NLP) and machine learning (ML), such as the development of models like OpenAI's ChatGPT, new opportunities are emerging for efficient curation of electronic health records (EHR) into real-world data (RWD) for evidence generation in oncology. Our objective is to describe the research and development of industry methods to promote transparency and explainability. Methods: We applied NLP with ML techniques to train, validate, and test the extraction of information from unstructured documents (e.g., clinician notes, radiology reports, lab reports, etc.) to output a set of structured variables required for RWD analysis. This research used a nationwide electronic health record (EHR)-derived database. Models were selected based on performance. Variables curated with an approach using ML extraction are those where the value is determined solely based on an ML model (i.e. not confirmed by abstraction), which identifies key information from visit notes and documents. These models do not predict future events or infer missing information. Results: We developed an approach using NLP and ML for extraction of clinically meaningful information from unstructured EHR documents and found high performance of output variables compared with variables curated by manually abstracted data. These extraction methods resulted in research-ready variables including initial cancer diagnosis with date, advanced/metastatic diagnosis with date, disease stage, histology, smoking status, surgery status with date, biomarker test results with dates, and oral treatments with dates. Conclusion: NLP and ML enable the extraction of retrospective clinical data in EHR with speed and scalability to help researchers learn from the experience of every person with cancer.

Reimagining Patient Data Access for Researchers

OBJECTIVES

Widespread use of electronic health records (EHRs) now makes it feasible to expand beyond health insurance claims data to include full EHR data for health economics and outcomes research (HEOR) studies. We seek to develop ways to maximize researcher access to such data while strongly protecting patients' privacy rights.

METHODS

We analyzed alternative organizational structures and intellectual property rights assignments as they now exist and compared these with structures and intellectual property rights assignments that would maximize access to data for HEOR studies and minimize transactions costs. We analyzed data protection requirements and financial incentives at 3 levels: patient decision making, patients' data aggregators, and final aggregation across patients' data.

RESULTS

Creating new HEOR data systems requires new organizations and funding, while also protecting patients' data privacy rights. The Cures Act enables a new market for trusted third parties (TTPs) to aggregate patients' data. New secondary data aggregators must combine individuals' aggregated EHRs into usable HEOR databases. Maximal patient participation requires complete health insurance coverage of costs that healthcare providers charge for transmitting patients' data to TTPs. The new secondary system to aggregate data from many TTPs into usable HEOR optimally has external funding.

CONCLUSIONS

Important steps remain uncompleted to achieve maximally available HEOR data while protecting patients' privacy rights. HEOR information is a public good, so private incentives to support creation and operation of this new system remain incomplete. Public and private support can expand this system to optimally improve people's health.

Consolidated Reporting Guidelines for Prognostic and Diagnostic Machine Learning Modeling Studies: Development and Validation

BACKGROUND

The reporting of machine learning (ML) prognostic and diagnostic modeling studies is often inadequate, making it difficult to understand and replicate such studies. To address this issue, multiple consensus and expert reporting guidelines for ML studies have been published. However, these guidelines cover different parts of the analytics lifecycle, and individually, none of them provide a complete set of reporting requirements.

OBJECTIVE

We aimed to consolidate the ML reporting guidelines and checklists in the literature to provide reporting items for prognostic and diagnostic ML in in-silico and shadow mode studies.

METHODS

We conducted a literature search that identified 192 unique peer-reviewed English articles that provide guidance and checklists for reporting ML studies. The articles were screened by their title and abstract against a set of 9 inclusion and exclusion criteria. Articles that were filtered through had their quality evaluated by 2 raters using a 9-point checklist constructed from guideline development good practices. The average κ was 0.71 across all quality criteria. The resulting 17 high-quality source papers were defined as having a quality score equal to or higher than the median. The reporting items in these 17 articles were consolidated and screened against a set of 6 inclusion and exclusion criteria. The resulting reporting items were sent to an external group of 11 ML experts for review and updated accordingly. The updated checklist was used to assess the reporting in 6 recent modeling papers in JMIR AI. Feedback from the external review and initial validation efforts was used to improve the reporting items.

RESULTS

In total, 37 reporting items were identified and grouped into 5 categories based on the stage of the ML project: defining the study details, defining and collecting the data, modeling methodology, model evaluation, and explainability. None of the 17 source articles covered all the reporting items. The study details and data description reporting items were the most common in the source literature, with explainability and methodology guidance (ie, data preparation and model training) having the least coverage. For instance, a median of 75% of the data description reporting items appeared in each of the 17 high-quality source guidelines, but only a median of 33% of the data explainability reporting items appeared. The highest-quality source articles tended to have more items on reporting study details. Other categories of reporting items were not related to the source article quality. We converted the reporting items into a checklist to support more complete reporting.

CONCLUSIONS

Our findings supported the need for a set of consolidated reporting items, given that existing high-quality guidelines and checklists do not individually provide complete coverage. The consolidated set of reporting items is expected to improve the quality and reproducibility of ML modeling studies.

Replication of Real-World Evidence in Oncology Using Electronic Health Record Data Extracted by Machine Learning

Meaningful real-world evidence (RWE) generation requires unstructured data found in electronic health records (EHRs) which are often missing from administrative claims; however, obtaining relevant data from unstructured EHR sources is resource-intensive. In response, researchers are using natural language processing (NLP) with machine learning (ML) techniques (i.e., ML extraction) to extract real-world data (RWD) at scale. This study assessed the quality and fitness-for-use of EHR-derived oncology data curated using NLP with ML as compared to the reference standard of expert abstraction. Using a sample of 186,313 patients with lung cancer from a nationwide EHR-derived de-identified database, we performed a series of replication analyses demonstrating some common analyses conducted in retrospective observational research with complex EHR-derived data to generate evidence. Eligible patients were selected into biomarker- and treatment-defined cohorts, first with expert-abstracted then with ML-extracted data. We utilized the biomarker- and treatment-defined cohorts to perform analyses related to biomarker-associated survival and treatment comparative effectiveness, respectively. Across all analyses, the results differed by less than 8% between the data curation methods, and similar conclusions were reached. These results highlight that high-performance ML-extracted variables trained on expert-abstracted data can achieve similar results as when using abstracted data, unlocking the ability to perform oncology research at scale.

Real-world clinical outcomes and cost estimates of metastatic castration-resistant prostate cancer treatment: does sequencing of taxanes and androgen receptor-targeted agents matter?

INTRODUCTION

Health economic outcomes of real-world treatment sequencing of androgen receptor-targeted agents (ARTA) and docetaxel (DOC) remain unclear.

MATERIAL AND METHODS

Data from the electronic Castration-resistant Prostate cancer Australian Database (ePAD) were analyzed including median overall survival (mOS) and median time-to-treatment failure (mTTF). Mean total costs (mTC) and incremental cost-effectiveness ratios (ICER) of treatment sequences were estimated using the average sample method and Zhao and Tian estimator.

RESULTS

Of 752 men, 441 received ARTA, 194 DOC, and 175 both sequentially. Of participants treated with both, first-line DOC followed by ARTA was the more common sequence (n = 125, 71%). mOS for first-line ARTA was 8.38 years (95% CI: 3.48, not-estimated) vs. 3.29 years (95% CI: 2.92, 4.02) for DOC. mTTF was 15.7 months (95% CI: 14.2, 23.7) for the ARTA-DOC sequence and 18.2 months (95% CI: 16.2, 23.2) for DOC-ARTA. In first-line, ARTA cost an additional $13,244 per mTTF month compared to DOC. In second-line, ARTA cost $6726 per mTTF month. The DOC-ARTA sequence saved $2139 per mTTF compared to ARTA-DOC, though not statistically significant.

CONCLUSION

ICERs show ARTA had improved clinical benefit compared to DOC but at higher cost. There were no significant cost differences between combined sequences.

Recommendations to overcome barriers to the use of artificial intelligence-driven evidence in health technology assessment

Background

Artificial intelligence (AI) has attracted much attention because of its enormous potential in healthcare, but uptake has been slow. There are substantial barriers that challenge health technology assessment (HTA) professionals to use AI-generated evidence for decision-making from large real-world databases (e.g., based on claims data). As part of the European Commission-funded HTx H2020 (Next Generation Health Technology Assessment) project, we aimed to put forward recommendations to support healthcare decision-makers in integrating AI into the HTA processes. The barriers, addressed by the paper, are particularly focusing on Central and Eastern European (CEE) countries, where the implementation of HTA and access to health databases lag behind Western European countries.

Methods

We constructed a survey to rank the barriers to using AI for HTA purposes, completed by respondents from CEE jurisdictions with expertise in HTA. Using the results, two members of the HTx consortium from CEE developed recommendations on the most critical barriers. Then these recommendations were discussed in a workshop by a wider group of experts, including HTA and reimbursement decision-makers from both CEE countries and Western European countries, and summarized in a consensus report.

Results

Recommendations have been developed to address the top 15 barriers in areas of (1) human factor-related barriers, focusing on educating HTA doers and users, establishing collaborations and best practice sharing; (2) regulatory and policy-related barriers, proposing increasing awareness and political commitment and improving the management of sensitive information for AI use; (3) data-related barriers, suggesting enhancing standardization and collaboration with data networks, managing missing and unstructured data, using analytical and statistical approaches to address bias, using quality assessment tools and quality standards, improving reporting, and developing better conditions for the use of data; and (4) technological barriers, suggesting sustainable development of AI infrastructure.

Conclusion

In the field of HTA, the great potential of AI to support evidence generation and evaluation has not yet been sufficiently explored and realized. Raising awareness of the intended and unintended consequences of AI-based methods and encouraging political commitment from policymakers is necessary to upgrade the regulatory and infrastructural environment and knowledge base required to integrate AI into HTA-based decision-making processes better.

Predicting Clinical Remission of Chronic Urticaria Using Random Survival Forests: Machine Learning Applied to Real-World Data

INTRODUCTION

The time required to reach clinical remission varies in patients with chronic urticaria (CU). The objective of this study is to develop a predictive model using a machine learning methodology to predict time to clinical remission for patients with CU.

METHODS

Adults with ≥ 2 ICD-9/10 relevant CU diagnosis codes/CU-related treatment > 6 weeks apart were identified in the Optum deidentified electronic health record dataset (January 2007 to June 2019). Clinical remission was defined as ≥ 12 months without CU diagnosis/CU-related treatment. A random survival forest was used to predict time from diagnosis to clinical remission for each patient based on clinical and demographic features available at diagnosis. Model performance was assessed using concordance, which indicates the degree of agreement between observed and predicted time to remission. To characterize clinically relevant groups, features were summarized among cohorts that were defined based on quartiles of predicted time to remission.

RESULTS

Among 112,443 patients, 73.5% reached clinical remission, with a median of 336 days from diagnosis. From 1876 initial features, 176 were retained in the final model, which predicted a median of 318 days to remission. The model showed good performance with a concordance of 0.62. Patients with predicted longer time to remission tended to be older with delayed CU diagnosis, and have more comorbidities, more laboratory tests, higher body mass index, and polypharmacy during the 12-month period before the first CU diagnosis.

CONCLUSIONS

Applying machine learning to real-world data enabled accurate prediction of time to clinical remission and identified multiple relevant demographic and clinical variables with predictive value. Ongoing work aims to further validate and integrate these findings into clinical applications for CU management.

R WE ready for reimbursement? A round up of developments in real-world evidence relating to health technology assessment: part 9

In this latest update we highlight a recent International Society of Pharmacoeconomics and Outcomes Research Good Practice Report on machine learning (ML) for health economics and outcomes research. We specifically discuss use cases of ML that offer opportunities in the generation of evidence using real-world data, including improvements in the identification of study cohorts, confounder identification and adjustment and estimating treatment effect heterogeneity. Barriers to the wider adoption of ML methods are also discussed.