Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients

Integration of Left Atrial Function Assessment, Genetic Risk, and Clinical Risk Factors Improves Prediction of Incident Atrial Fibrillation

BACKGROUND

Integration of imaging, genetic, and clinical markers might improve risk assessment of atrial fibrillation (AF). We explored whether the addition of left atrial (LA) function and polygenic risk score (PRS) to a clinical risk score improves risk prediction of AF.

METHODS

A total of 36 919 individuals without AF who were assessed for LA emptying fraction (evaluated by cardiac magnetic resonance imaging) and PRS were analyzed from the UK Biobank imaging enhancement.

RESULTS

Over a median of 2.9 (2.0-4.2) years, 535 individuals developed incident AF. Per-SD decrease in LA emptying fraction and increase in PRS was associated with a hazard ratio of 2.13 (95% CI, 1.99-2.27) and 1.65 (95% CI, 1.52-1.79) for incident AF, respectively. C-index increase when LA emptying fraction was added to CHARGE-AF (Cohorts for Heart and Aging Research in Genomic Epidemiology Atrial Fibrillation) was 0.036 (95% CI, 0.013-0.057) and when joint LA emptying fraction and PRS was added to CHARGE-AF was 0.057 (95% CI, 0.034-0.081). At a 3-year risk threshold of 3%, the predicted net benefit was 2.48 per 1000 people for a model including LA emptying fraction, PRS, and CHARGE-AF compared with 0.30 per 1000 people for a model including CHARGE-AF only. In addition, a 6.5-fold risk gradient was observed for AF-mediated stroke or other systemic embolism after accounting for LA emptying fraction and PRS.

CONCLUSIONS

Integration of LA emptying fraction and PRS significantly improved risk prediction of incident AF when added to CHARGE-AF.

Machine learning to predict completion of treatment for pancreatic cancer

BACKGROUND

Chemotherapy enhances survival rates for pancreatic cancer (PC) patients postsurgery, yet less than 60% complete adjuvant therapy, with a smaller fraction undergoing neoadjuvant treatment. Our study aimed to predict which patients would complete pre- or postoperative chemotherapy through machine learning (ML).

METHODS

Patients with resectable PC identified in our institutional pancreas database were grouped into two categories: those who completed all intended treatments (i.e., surgery plus either neoadjuvant or adjuvant chemotherapy), and those who did not. We applied logistic regression with lasso penalization and an extreme gradient boosting model for prediction, and further examined it through bootstrapping for sensitivity.

RESULTS

Among 208 patients, the median age was 69, with 49.5% female and 62% white participants. Most had an Eastern Cooperative Oncology Group (ECOG) performance status of ≤2. The PC predominantly affected the pancreatic head. Neoadjuvant and adjuvant chemotherapies were received by 26% and 47.1%, respectively, but only 49% completed all treatments. Incomplete therapy was correlated with older age and lower ECOG status. Negative prognostic factors included worsening diabetes, age, congestive heart failure, high body mass index, family history of PC, initial bilirubin levels, and tumor location in the pancreatic head. The models also flagged other factors, such as jaundice and specific cancer markers, impacting treatment completion. The predictive accuracy (area under the receiver operating characteristic curve) was 0.67 for both models, with performance expected to improve with larger datasets.

CONCLUSIONS

Our findings underscore the potential of ML to forecast PC treatment completion, highlighting the importance of specific preoperative factors. Increasing data volumes may enhance predictive accuracy, offering valuable insights for personalized patient strategies.

Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities

This paper discusses some overlooked challenges faced when working with machine learning models for histopathology and presents a novel opportunity to support "Learning Health Systems" with them. Initially, the authors elaborate on these challenges after separating them according to their mitigation strategies: those that need innovative approaches, time, or future technological capabilities and those that require a conceptual reappraisal from a critical perspective. Then, a novel opportunity to support "Learning Health Systems" by integrating hidden information extracted by ML models from digitalized histopathology slides with other healthcare big data is presented.

Monitoring performance of clinical artificial intelligence in health care: a scoping review

OBJECTIVE

The objective of this review was to provide an overview of the diverse methods described, tested, or implemented for monitoring performance of clinical artificial intelligence (AI) systems, while also summarizing the arguments given for or against these methods.

INTRODUCTION

The integration of AI in clinical decision-making is steadily growing. Performances of AI systems evolve over time, necessitating ongoing performance monitoring. However, the evidence on specific monitoring methods is sparse and heterogeneous. Thus, an overview of the evidence on this topic is warranted to guide further research on clinical AI monitoring.

INCLUSION CRITERIA

We included publications detailing metrics or statistical processes employed in systematic, continuous, or repeated initiatives aimed at evaluating or predicting the clinical performance of AI models with direct implications for patient management in health care. No limitations on language or publication date were enforced.

METHODS

We performed systematic database searches in MEDLINE (Ovid), Embase (Ovid), Scopus, and ProQuest Dissertations and Theses Global, supplemented by backward and forward citation searches and gray literature searches. Two or more independent reviewers conducted title and abstract screening, full-text evaluation, and data extraction using a tool developed by the authors. During extraction, the methods identified were divided into subcategories. The results are presented narratively and summarized in tables and graphs.

RESULTS

Thirty-nine sources of evidence were included in the review, with the most abundant source types being opinion papers/narrative reviews (33%) and simulation studies (33%). One guideline on the topic was identified, offering limited guidance on specific metrics and statistical methods. The number of sources included increased year by year, with almost 4 times as many sources included in 2023 compared with 2019. The most commonly reported performance metrics were traditional metrics from the medical literature, including area under the receiver operating characteristics curve (AUROC), sensitivity, specificity, and predictive values, although few arguments were given supporting these choices. Some studies reported on metrics and statistical processing specifically designed to monitor clinical AI.

CONCLUSION

This review provides a summary of the methods described for monitoring AI in health care. It reveals a relative scarcity of evidence and guidance for specific practical implementation of performance monitoring of clinical AI. This underscores the imperative for further research, discussion, and guidance regarding the specifics of implementing monitoring for clinical AI. The steady increase in the number of relevant sources published per year suggests that this area of research is gaining increased focus, and the amount of evidence and guidance available will likely increase significantly over the coming years.

REVIEW REGISTRATION

Open Science Framework https://osf.io/afkrn.

Machine Learning Operations in Health Care: A Scoping Review

The use of machine learning tools in health care is rapidly expanding. However, the processes that support these tools in deployment, that is, machine learning operations, are still emerging. The purpose of this work was not only to provide a comprehensive synthesis of existing literature in the field but also to identify gaps and offer insights for adoption in clinical practice. A scoping review was conducted using the MEDLINE, PubMed, Google Scholar, Embase, and Scopus databases. We used MeSH and non-MeSH search terms to identify pertinent articles, with the authors performing 2 screening phases and assigning relevance scores: 148 English language articles most salient to the review were eligible for inclusion; 98 offered the most unique information and these were supplemented by 50 additional sources, yielding 148 references. From the 148 references, we distilled 7 key topic areas, based on a synthesis of the available literature and how that aligned with practitioner needs. The 7 topic areas were machine learning model monitoring; automated retraining systems; ethics, equity, and bias; clinical workflow integration; infrastructure, human resources, and technology stack; regulatory considerations; and financial considerations. This review provides an overview of best practices and knowledge gaps of this domain in health care and identifies the strengths and weaknesses of the literature, which may be useful to health care machine learning practitioners and consumers.

European and US Guideline-Based Statin Eligibility, Genetically Predicted Coronary Artery Disease, and the Risk of Major Coronary Events

BACKGROUND

A study was designed to investigate whether the coronary artery disease polygenic risk score (CAD-PRS) may guide lipid-lowering treatment initiation as well as deferral in primary prevention beyond established clinical risk scores.

METHODS AND RESULTS

Participants were 311 799 individuals from the UK Biobank free of atherosclerotic cardiovascular disease, diabetes, chronic kidney disease, and lipid-lowering treatment at baseline. Participants were categorized as statin indicated, statin indication unclear, or statin not indicated as defined by the European and US guidelines on statin use. For a median of 11.9 (11.2-12.6) years, 8196 major coronary events developed. CAD-PRS added to European-Systematic Coronary Risk Evaluation 2 (European-SCORE2) and US-Pooled Cohort Equation (US-PCE) identified 18% and 12% of statin-indication-unclear individuals whose risk of major coronary events were the same as or higher than the average risk of statin-indicated individuals and 16% and 12% of statin-indicated individuals whose major coronary event risks were the same as or lower than the average risk of statin-indication-unclear individuals. For major coronary and atherosclerotic cardiovascular disease events, CAD-PRS improved C-statistics greater among statin-indicated or statin-indication-unclear than statin-not-indicated individuals. For atherosclerotic cardiovascular disease events, CAD-PRS added to the European evaluation and US equation resulted in a net reclassification improvement of 13.6% (95% CI, 11.8-15.5) and 14.7% (95% CI, 13.1-16.3) among statin-indicated, 10.8% (95% CI, 9.6-12.0) and 15.3% (95% CI, 13.2-17.5) among statin-indication-unclear, and 0.9% (95% CI, 0.6-1.3) and 3.6% (95% CI, 3.0-4.2) among statin-not-indicated individuals.

CONCLUSIONS

CAD-PRS may guide statin initiation as well as deferral among statin-indication-unclear or statin-indicated individuals as defined by the European and US guidelines. CAD-PRS had little clinical utility among statin-not-indicated individuals.

Empirical data drift detection experiments on real-world medical imaging data

While it is common to monitor deployed clinical artificial intelligence (AI) models for performance degradation, it is less common for the input data to be monitored for data drift - systemic changes to input distributions. However, when real-time evaluation may not be practical (eg., labeling costs) or when gold-labels are automatically generated, we argue that tracking data drift becomes a vital addition for AI deployments. In this work, we perform empirical experiments on real-world medical imaging to evaluate three data drift detection methods' ability to detect data drift caused (a) naturally (emergence of COVID-19 in X-rays) and (b) synthetically. We find that monitoring performance alone is not a good proxy for detecting data drift and that drift-detection heavily depends on sample size and patient features. Our work discusses the need and utility of data drift detection in various scenarios and highlights gaps in knowledge for the practical application of existing methods.

Longitudinal dynamic clinical phenotypes of in-hospital COVID-19 patients across three dominant virus variants in New York

BACKGROUND

COVID-19 is a challenging disease to characterize given its wide-ranging heterogeneous symptomatology. Several studies have attempted to extract clinical phenotypes but often relied on data from small patient cohorts, usually limited to only one viral variant and utilizing a static snapshot of patient data.

OBJECTIVE

This study aimed to identify clinical phenotypes of hospitalized COVID-19 patients and investigate their longitudinal dynamics throughout the pandemic, with the goal to relate these phenotypes to clinical outcomes and treatment strategies.

METHODS

We utilized routinely collected demographic and clinical data throughout the hospitalization of 38,077 patients admitted between 3/2020 to 5/2022, in 12 New York hospitals. Uniform Manifold Approximation and Projection and agglomerative hierarchical clustering were used to derive the clusters, followed by exploratory data analysis to compare the prevalence of comorbidities and treatments per cluster.

RESULTS

4 distinct clinical phenotypes remained robust in multi-site validation and were associated with different mortality rates. The temporal progression of these phenotypes throughout the COVID-19 pandemic demonstrated increased variability across the waves of the three dominant viral variants (alpha, delta, omicron). Longitudinal analysis evaluating changes in clinical phenotypes of each patient throughout the course of a 4-week hospital stay exemplified the dynamic nature of the disease progression. Factors such as sex, race/ethnicity and specific treatment modalities revealed significant and clinically relevant differences between the observed phenotypes.

CONCLUSIONS

Our proposed methodology has the potential of enabling clinicians and policy makers to draw evidence-based conclusions for guiding treatment modalities in a dynamic fashion.

Automatic correction of performance drift under acquisition shift in medical image classification

Image-based prediction models for disease detection are sensitive to changes in data acquisition such as the replacement of scanner hardware or updates to the image processing software. The resulting differences in image characteristics may lead to drifts in clinically relevant performance metrics which could cause harm in clinical decision making, even for models that generalise in terms of area under the receiver-operating characteristic curve. We propose Unsupervised Prediction Alignment, a generic automatic recalibration method that requires no ground truth annotations and only limited amounts of unlabelled example images from the shifted data distribution. We illustrate the effectiveness of the proposed method to detect and correct performance drift in mammography-based breast cancer screening and on publicly available histopathology data. We show that the proposed method can preserve the expected performance in terms of sensitivity/specificity under various realistic scenarios of image acquisition shift, thus offering an important safeguard for clinical deployment.

The association of clinically relevant variables with chest radiograph lung disease burden quantified in real-time by radiologists upon initial presentation in individuals hospitalized with COVID-19

Objectives

We aimed to correlate lung disease burden on presentation chest radiographs (CXR), quantified at the time of study interpretation, with clinical presentation in patients hospitalized with coronavirus disease 2019 (COVID-19).

Material and methods

This retrospective cross-sectional study included 5833 consecutive adult patients, aged 18 and older, hospitalized with a diagnosis of COVID-19 with a CXR quantified in real-time while hospitalized in 1 of 12 acute care hospitals across a multihospital integrated healthcare network between March 24, 2020, and May 22, 2020. Lung disease burden was quantified in real-time by 118 radiologists on 5833 CXR at the time of exam interpretation with each lung annotated by the degree of lung opacity as clear (0%), mild (1–33%), moderate (34–66%), or severe (67–100%). CXR findings were classified as (1) clear versus disease, (2) unilateral versus bilateral, (3) symmetric versus asymmetric, or (4) not severe versus severe. Lung disease burden was characterized on initial presentation by patient demographics, co-morbidities, vital signs, and lab results with chi-square used for univariate analysis and logistic regression for multivariable analysis.

Results

Patients with severe lung disease were more likely to have oxygen impairment, an elevated respiratory rate, low albumin, high lactate dehydrogenase, and high ferritin compared to non-severe lung disease. A lack of opacities in COVID-19 was associated with a low estimated glomerular filtration rate, hypernatremia, and hypoglycemia.

Conclusions

COVID-19 lung disease burden quantified in real-time on presentation CXR was characterized by demographics, comorbidities, emergency severity index, Charlson Comorbidity Index, vital signs, and lab results on 5833 patients. This novel approach to real-time quantified chest radiograph lung disease burden by radiologists needs further research to understand how this information can be incorporated to improve clinical care for pulmonary-related diseases.. An absence of opacities in COVID-19 may be associated with poor oral intake and a prerenal state as evidenced by the association of clear CXRs with a low eGFR, hypernatremia, and hypoglycemia.

US and Dutch Perspectives on the Use of COVID-19 Clinical Prediction Models: Findings from a Qualitative Analysis

INTRODUCTION

Clinical prediction models (CPMs) for coronavirus disease 2019 (COVID-19) may support clinical decision making, treatment, and communication. However, attitudes about using CPMs for COVID-19 decision making are unknown.

METHODS

Online focus groups and interviews were conducted among health care providers, survivors of COVID-19, and surrogates (i.e., loved ones/surrogate decision makers) in the United States and the Netherlands. Semistructured questions explored experiences about clinical decision making in COVID-19 care and facilitators and barriers for implementing CPMs.

RESULTS

In the United States, we conducted 4 online focus groups with 1) providers and 2) surrogates and survivors of COVID-19 between January 2021 and July 2021. In the Netherlands, we conducted 3 focus groups and 4 individual interviews with 1) providers and 2) surrogates and survivors of COVID-19 between May 2021 and July 2021. Providers expressed concern about CPM validity and the belief that patients may interpret CPM predictions as absolute. They described CPMs as potentially useful for resource allocation, triaging, education, and research. Several surrogates and people who had COVID-19 were not given prognostic estimates but believed this information would have supported and influenced their decision making. A limited number of participants felt the data would not have applied to them and that they or their loved ones may not have survived, as poor prognosis may have suggested withdrawal of treatment.

CONCLUSIONS

Many providers had reservations about using CPMs for people with COVID-19 due to concerns about CPM validity and patient-level interpretation of the outcome predictions. However, several people who survived COVID-19 and their surrogates indicated that they would have found this information useful for decision making. Therefore, information provision may be needed to improve provider-level comfort and patient and surrogate understanding of CPMs.

HIGHLIGHTS

While clinical prediction models (CPMs) may provide an objective means of assessing COVID-19 prognosis, provider concerns about CPM validity and the interpretation of CPM predictions may limit their clinical use.Providers felt that CPMs may be most useful for resource allocation, triage, research, or educational purposes for COVID-19.Several survivors of COVID-19 and their surrogates felt that CPMs would have been informative and may have aided them in making COVID-19 treatment decisions, while others felt the data would not have applied to them.

Prediction of intrapartum fever using continuously monitored vital signs and heart rate variability

Objectives

Neonatal early onset sepsis (EOS), bacterial infection during the first seven days of life, is difficult to diagnose because presenting signs are non-specific, but early diagnosis before birth can direct life-saving treatment for mother and baby. Specifically, maternal fever during labor from placental infection is the strongest predictor of EOS. Alterations in maternal heart rate variability (HRV) may precede development of intrapartum fever, enabling incipient EOS detection. The objective of this work was to build a predictive model for intrapartum fever.

Methods

Continuously measured temperature, heart rate, and beat-to-beat RR intervals were obtained from wireless sensors on women (n = 141) in labor; traditional manual vital signs were taken every 3-6 hours. Validated measures of HRV were calculated in moving 5-minute windows of RR intervals: standard deviation of normal-to-normal intervals (SDNN) and root mean square of successive differences (RMSSD) between normal heartbeats.

Results

Fever (>38.0 °C) was detected by manual or continuous measurements in 48 women. Compared to afebrile mothers, average SDNN and RMSSD in febrile mothers decreased significantly (p < 0.001) at 2 and 3 hours before fever onset, respectively. This observed HRV divergence and raw recorded vitals were applied to a logistic regression model at various time horizons, up to 4-5 hours before fever onset. Model performance increased with decreasing time horizons, and a model built using continuous vital signs as input variables consistently outperformed a model built from episodic vital signs.

Conclusions

HRV-based predictive models could identify mothers at risk for fever and infants at risk for EOS, guiding maternal antibiotic prophylaxis and neonatal monitoring.