A scalable federated learning solution for secondary care using low-cost microcomputing: privacy-preserving development and evaluation of a COVID-19 screening test in UK hospitals

BACKGROUND

Multicentre training could reduce biases in medical artificial intelligence (AI); however, ethical, legal, and technical considerations can constrain the ability of hospitals to share data. Federated learning enables institutions to participate in algorithm development while retaining custody of their data but uptake in hospitals has been limited, possibly as deployment requires specialist software and technical expertise at each site. We previously developed an artificial intelligence-driven screening test for COVID-19 in emergency departments, known as CURIAL-Lab, which uses vital signs and blood tests that are routinely available within 1 h of a patient's arrival. Here we aimed to federate our COVID-19 screening test by developing an easy-to-use embedded system-which we introduce as full-stack federated learning-to train and evaluate machine learning models across four UK hospital groups without centralising patient data.

METHODS

We supplied a Raspberry Pi 4 Model B preloaded with our federated learning software pipeline to four National Health Service (NHS) hospital groups in the UK: Oxford University Hospitals NHS Foundation Trust (OUH; through the locally linked research University, University of Oxford), University Hospitals Birmingham NHS Foundation Trust (UHB), Bedfordshire Hospitals NHS Foundation Trust (BH), and Portsmouth Hospitals University NHS Trust (PUH). OUH, PUH, and UHB participated in federated training, training a deep neural network and logistic regressor over 150 rounds to form and calibrate a global model to predict COVID-19 status, using clinical data from patients admitted before the pandemic (COVID-19-negative) and testing positive for COVID-19 during the first wave of the pandemic. We conducted a federated evaluation of the global model for admissions during the second wave of the pandemic at OUH, PUH, and externally at BH. For OUH and PUH, we additionally performed local fine-tuning of the global model using the sites' individual training data, forming a site-tuned model, and evaluated the resultant model for admissions during the second wave of the pandemic. This study included data collected between Dec 1, 2018, and March 1, 2021; the exact date ranges used varied by site. The primary outcome was overall model performance, measured as the area under the receiver operating characteristic curve (AUROC). Removable micro secure digital (microSD) storage was destroyed on study completion.

FINDINGS

Clinical data from 130 941 patients (1772 COVID-19-positive), routinely collected across three hospital groups (OUH, PUH, and UHB), were included in federated training. The evaluation step included data from 32 986 patients (3549 COVID-19-positive) attending OUH, PUH, or BH during the second wave of the pandemic. Federated training of a global deep neural network classifier improved upon performance of models trained locally in terms of AUROC by a mean of 27·6% (SD 2·2): AUROC increased from 0·574 (95% CI 0·560-0·589) at OUH and 0·622 (0·608-0·637) at PUH using the locally trained models to 0·872 (0·862-0·882) at OUH and 0·876 (0·865-0·886) at PUH using the federated global model. Performance improvement was smaller for a logistic regression model, with a mean increase in AUROC of 13·9% (0·5%). During federated external evaluation at BH, AUROC for the global deep neural network model was 0·917 (0·893-0·942), with 89·7% sensitivity (83·6-93·6) and 76·6% specificity (73·9-79·1). Site-specific tuning of the global model did not significantly improve performance (change in AUROC <0·01).

INTERPRETATION

We developed an embedded system for federated learning, using microcomputing to optimise for ease of deployment. We deployed full-stack federated learning across four UK hospital groups to develop a COVID-19 screening test without centralising patient data. Federation improved model performance, and the resultant global models were generalisable. Full-stack federated learning could enable hospitals to contribute to AI development at low cost and without specialist technical expertise at each site.

FUNDING

The Wellcome Trust, University of Oxford Medical and Life Sciences Translational Fund.

Real-world evaluation of rapid and laboratory-free COVID-19 triage for emergency care: external validation and pilot deployment of artificial intelligence driven screening

Background

Uncertainty in patients' COVID-19 status contributes to treatment delays, nosocomial transmission, and operational pressures in hospitals. However, the typical turnaround time for laboratory PCR remains 12–24 h and lateral flow devices (LFDs) have limited sensitivity. Previously, we have shown that artificial intelligence-driven triage (CURIAL-1.0) can provide rapid COVID-19 screening using clinical data routinely available within 1 h of arrival to hospital. Here, we aimed to improve the time from arrival to the emergency department to the availability of a result, do external and prospective validation, and deploy a novel laboratory-free screening tool in a UK emergency department.

Methods

We optimised our previous model, removing less informative predictors to improve generalisability and speed, developing the CURIAL-Lab model with vital signs and readily available blood tests (full blood count [FBC]; urea, creatinine, and electrolytes; liver function tests; and C-reactive protein) and the CURIAL-Rapide model with vital signs and FBC alone. Models were validated externally for emergency admissions to University Hospitals Birmingham, Bedfordshire Hospitals, and Portsmouth Hospitals University National Health Service (NHS) trusts, and prospectively at Oxford University Hospitals, by comparison with PCR testing. Next, we compared model performance directly against LFDs and evaluated a combined pathway that triaged patients who had either a positive CURIAL model result or a positive LFD to a COVID-19-suspected clinical area. Lastly, we deployed CURIAL-Rapide alongside an approved point-of-care FBC analyser to provide laboratory-free COVID-19 screening at the John Radcliffe Hospital (Oxford, UK). Our primary improvement outcome was time-to-result, and our performance measures were sensitivity, specificity, positive and negative predictive values, and area under receiver operating characteristic curve (AUROC).

Findings

72 223 patients met eligibility criteria across the four validating hospital groups, in a total validation period spanning Dec 1, 2019, to March 31, 2021. CURIAL-Lab and CURIAL-Rapide performed consistently across trusts (AUROC range 0·858–0·881, 95% CI 0·838–0·912, for CURIAL-Lab and 0·836–0·854, 0·814–0·889, for CURIAL-Rapide), achieving highest sensitivity at Portsmouth Hospitals (84·1%, Wilson's 95% CI 82·5–85·7, for CURIAL-Lab and 83·5%, 81·8–85·1, for CURIAL-Rapide) at specificities of 71·3% (70·9–71·8) for CURIAL-Lab and 63·6% (63·1–64·1) for CURIAL-Rapide. When combined with LFDs, model predictions improved triage sensitivity from 56·9% (51·7–62·0) for LFDs alone to 85·6% with CURIAL-Lab (81·6–88·9; AUROC 0·925) and 88·2% with CURIAL-Rapide (84·4–91·1; AUROC 0·919), thereby reducing missed COVID-19 cases by 65% with CURIAL-Lab and 72% with CURIAL-Rapide. For the prospective deployment of CURIAL-Rapide, 520 patients were enrolled for point-of-care FBC analysis between Feb 18 and May 10, 2021, of whom 436 received confirmatory PCR testing and ten (2·3%) tested positive. Median time from arrival to a CURIAL-Rapide result was 45 min (IQR 32–64), 16 min (26·3%) sooner than with LFDs (61 min, 37–99; log-rank p<0·0001), and 6 h 52 min (90·2%) sooner than with PCR (7 h 37 min, 6 h 5 min to 15 h 39 min; p<0·0001). Classification performance was high, with sensitivity of 87·5% (95% CI 52·9–97·8), specificity of 85·4% (81·3–88·7), and negative predictive value of 99·7% (98·2–99·9). CURIAL-Rapide correctly excluded infection for 31 (58·5%) of 53 patients who were triaged by a physician to a COVID-19-suspected area but went on to test negative by PCR.

Interpretation

Our findings show the generalisability, performance, and real-world operational benefits of artificial intelligence-driven screening for COVID-19 over standard-of-care in emergency departments. CURIAL-Rapide provided rapid, laboratory-free screening when used with near-patient FBC analysis, and was able to reduce the number of patients who tested negative for COVID-19 but were triaged to COVID-19-suspected areas.

Funding

The Wellcome Trust, University of Oxford Medical and Life Sciences Translational Fund.

COVID-19 admission risk tools should include multiethnic age structures, multimorbidity and deprivation metrics for air pollution, household overcrowding, housing quality and adult skills

BACKGROUND

Ethnic minorities account for 34% of critically ill patients with COVID-19 despite constituting 14% of the UK population. Internationally, researchers have called for studies to understand deterioration risk factors to inform clinical risk tool development.

METHODS

Multicentre cohort study of hospitalised patients with COVID-19 (n=3671) exploring determinants of health, including Index of Multiple Deprivation (IMD) subdomains, as risk factors for presentation, deterioration and mortality by ethnicity. Receiver operator characteristics were plotted for CURB65 and ISARIC4C by ethnicity and area under the curve (AUC) calculated.

RESULTS

Ethnic minorities were hospitalised with higher Charlson Comorbidity Scores than age, sex and deprivation matched controls and from the most deprived quintile of at least one IMD subdomain: indoor living environment (LE), outdoor LE, adult skills, wider barriers to housing and services. Admission from the most deprived quintile of these deprivation forms was associated with multilobar pneumonia on presentation and ICU admission. AUC did not exceed 0.7 for CURB65 or ISARIC4C among any ethnicity except ISARIC4C among Indian patients (0.83, 95% CI 0.73 to 0.93). Ethnic minorities presenting with pneumonia and low CURB65 (0-1) had higher mortality than White patients (22.6% vs 9.4%; p<0.001); Africans were at highest risk (38.5%; p=0.006), followed by Caribbean (26.7%; p=0.008), Indian (23.1%; p=0.007) and Pakistani (21.2%; p=0.004).

CONCLUSIONS

Ethnic minorities exhibit higher multimorbidity despite younger age structures and disproportionate exposure to unscored risk factors including obesity and deprivation. Household overcrowding, air pollution, housing quality and adult skills deprivation are associated with multilobar pneumonia on presentation and ICU admission which are mortality risk factors. Risk tools need to reflect risks predominantly affecting ethnic minorities.

To What Extent do Social Determinants of Health Modulate Presentation, ITU Admission and Outcomes among Patients with SARS-COV-2 Infection? An Exploration of Household Overcrowding, Air Pollution, Housing Quality, Ethnicity, Comorbidities and Frailty

Background

Internationally, researchers have called for evidence to support tackling health inequalities during the severe acute respiratory syndrome coronavirus 2 (COVID19) pandemic. Despite the 2020 Marmot review highlighting growing health gaps between wealthy and deprived areas, studies have not explored social determinants of health (ethnicity, frailty, comorbidities, household overcrowding, housing quality, air pollution) as modulators of presentation, intensive care unit (ITU) admissions and outcomes among COVID19 patients. There is an urgent need for studies examining social determinants of health including socioenvironmental risk factors in urban areas to inform the national and international landscape.

Methods

An in-depth retrospective cohort study of 408 hospitalized COVID19 patients admitted to the Queen Elizabeth Hospital, Birmingham was conducted. Quantitative data analyses including a two-step cluster analysis were applied to explore the role of social determinants of health as modulators of presentation, ITU admission and outcomes.

Results

Patients admitted from highest Living Environment deprivation indices were at increased risk of presenting with multi-lobar pneumonia and, in turn, ITU admission whilst patients admitted from highest Barriers to Housing and Services (BHS) deprivation Indies were at increased risk of ITU admission. Black, Asian and Minority Ethnic (BAME) patients were more likely, than Caucasians, to be admitted from regions of highest Living Environment and BHS deprivation, present with multi-lobar pneumonia and require ITU admission.

Conclusion

Household overcrowding deprivation and presentation with multi-lobar pneumonia are potential modulators of ITU admission. Air pollution and housing quality deprivation are potential modulators of presentation with multi-lobar pneumonia. BAME patients are demographically at increased risk of exposure to household overcrowding, air pollution and housing quality deprivation, are more likely to present with multi-lobar pneumonia and require ITU admission. Irrespective of deprivation, consideration of the Charlson Comorbidity Score and the Clinical Frailty Score supports clinicians in stratifying high risk patients.