Real-world evaluation of rapid and laboratory-free COVID-19 triage for emergency care: external validation and pilot deployment of artificial intelligence driven screening

The influence of birthweight on mortality and severe neonatal morbidity in late preterm and term infants: an Australian cohort study

Background

The aim of this study was to detail incidence rates and relative risks for severe adverse perinatal outcomes by birthweight centile categories in a large Australian cohort of late preterm and term infants.

Methods

This was a retrospective cohort study of singleton infants (≥34+0 weeks gestation) between 2000 and 2018 in Queensland, Australia. Study outcomes were perinatal mortality, severe neurological morbidity, and other severe morbidity. Categorical outcomes were compared using Chi-squared tests. Continuous outcomes were compared using t-tests. Multinomial logistic regression investigated the effect of birthweight centile on study outcomes.

Findings

The final cohort comprised 991,042 infants. Perinatal mortality occurred in 1944 infants (0.19%). The incidence and risk of perinatal mortality increased as birthweight decreased, peaking for infants <1st centile (perinatal mortality rate 13.2/1000 births, adjusted Relative Risk Ratio (aRRR) of 12.96 (95% CI 10.14, 16.57) for stillbirth and aRRR 7.55 (95% CI 3.78, 15.08) for neonatal death). Severe neurological morbidity occurred in 7311 infants (0.74%), with the highest rate (19.6/1000 live births) in <1st centile cohort. There were 75,243 cases of severe morbidity (7.59% livebirths), with the peak incidence occurring in the <1st centile category (12.3% livebirths). The majority of adverse outcomes occurred in infants with birthweights between 10 and 90th centile. Almost 2 in 3 stillbirths, and approximately 3 in 4 cases of neonatal death, severe neurological morbidity or other severe morbidity occurred within this birthweight range.

Interpretation

Although the incidence and risk of perinatal mortality, severe neurological morbidity and severe morbidity increased at the extremes of birthweight centiles, the majority of these outcomes occurred in infants that were apparently "appropriately grown" (i.e., birthweight 10th-90th centile).

Funding

National Health and Medical Research Council, Mater Foundation, Royal Australian College of Obstetricians and Gynaecologists Women's Health Foundation - Norman Beischer Clinical Research Scholarship, Cerebral Palsy Alliance, University of Queensland Research Scholarship.

A scalable federated learning solution for secondary care using low-cost microcomputing: privacy-preserving development and evaluation of a COVID-19 screening test in UK hospitals

BACKGROUND

Multicentre training could reduce biases in medical artificial intelligence (AI); however, ethical, legal, and technical considerations can constrain the ability of hospitals to share data. Federated learning enables institutions to participate in algorithm development while retaining custody of their data but uptake in hospitals has been limited, possibly as deployment requires specialist software and technical expertise at each site. We previously developed an artificial intelligence-driven screening test for COVID-19 in emergency departments, known as CURIAL-Lab, which uses vital signs and blood tests that are routinely available within 1 h of a patient's arrival. Here we aimed to federate our COVID-19 screening test by developing an easy-to-use embedded system-which we introduce as full-stack federated learning-to train and evaluate machine learning models across four UK hospital groups without centralising patient data.

METHODS

We supplied a Raspberry Pi 4 Model B preloaded with our federated learning software pipeline to four National Health Service (NHS) hospital groups in the UK: Oxford University Hospitals NHS Foundation Trust (OUH; through the locally linked research University, University of Oxford), University Hospitals Birmingham NHS Foundation Trust (UHB), Bedfordshire Hospitals NHS Foundation Trust (BH), and Portsmouth Hospitals University NHS Trust (PUH). OUH, PUH, and UHB participated in federated training, training a deep neural network and logistic regressor over 150 rounds to form and calibrate a global model to predict COVID-19 status, using clinical data from patients admitted before the pandemic (COVID-19-negative) and testing positive for COVID-19 during the first wave of the pandemic. We conducted a federated evaluation of the global model for admissions during the second wave of the pandemic at OUH, PUH, and externally at BH. For OUH and PUH, we additionally performed local fine-tuning of the global model using the sites' individual training data, forming a site-tuned model, and evaluated the resultant model for admissions during the second wave of the pandemic. This study included data collected between Dec 1, 2018, and March 1, 2021; the exact date ranges used varied by site. The primary outcome was overall model performance, measured as the area under the receiver operating characteristic curve (AUROC). Removable micro secure digital (microSD) storage was destroyed on study completion.

FINDINGS

Clinical data from 130 941 patients (1772 COVID-19-positive), routinely collected across three hospital groups (OUH, PUH, and UHB), were included in federated training. The evaluation step included data from 32 986 patients (3549 COVID-19-positive) attending OUH, PUH, or BH during the second wave of the pandemic. Federated training of a global deep neural network classifier improved upon performance of models trained locally in terms of AUROC by a mean of 27·6% (SD 2·2): AUROC increased from 0·574 (95% CI 0·560-0·589) at OUH and 0·622 (0·608-0·637) at PUH using the locally trained models to 0·872 (0·862-0·882) at OUH and 0·876 (0·865-0·886) at PUH using the federated global model. Performance improvement was smaller for a logistic regression model, with a mean increase in AUROC of 13·9% (0·5%). During federated external evaluation at BH, AUROC for the global deep neural network model was 0·917 (0·893-0·942), with 89·7% sensitivity (83·6-93·6) and 76·6% specificity (73·9-79·1). Site-specific tuning of the global model did not significantly improve performance (change in AUROC <0·01).

INTERPRETATION

We developed an embedded system for federated learning, using microcomputing to optimise for ease of deployment. We deployed full-stack federated learning across four UK hospital groups to develop a COVID-19 screening test without centralising patient data. Federation improved model performance, and the resultant global models were generalisable. Full-stack federated learning could enable hospitals to contribute to AI development at low cost and without specialist technical expertise at each site.

FUNDING

The Wellcome Trust, University of Oxford Medical and Life Sciences Translational Fund.

Experiences of using artificial intelligence in healthcare: a qualitative study of UK clinician and key stakeholder perspectives

OBJECTIVES

Artificial intelligence (AI) is a rapidly developing field in healthcare, with tools being developed across various specialties to support healthcare professionals and reduce workloads. It is important to understand the experiences of professionals working in healthcare to ensure that future AI tools are acceptable and effectively implemented. The aim of this study was to gain an in-depth understanding of the experiences and perceptions of UK healthcare workers and other key stakeholders about the use of AI in the National Health Service (NHS).

DESIGN

A qualitative study using semistructured interviews conducted remotely via MS Teams. Thematic analysis was carried out.

SETTING

NHS and UK higher education institutes.

PARTICIPANTS

Thirteen participants were recruited, including clinical and non-clinical participants working for the NHS and researchers working to develop AI tools for healthcare settings.

RESULTS

Four core themes were identified: positive perceptions of AI; potential barriers to using AI in healthcare; concerns regarding AI use and steps needed to ensure the acceptability of future AI tools. Overall, we found that those working in healthcare were generally open to the use of AI and expected it to have many benefits for patients and facilitate access to care. However, concerns were raised regarding the security of patient data, the potential for misdiagnosis and that AI could increase the burden on already strained healthcare staff.

CONCLUSION

This study found that healthcare staff are willing to engage with AI research and incorporate AI tools into care pathways. Going forward, the NHS and AI developers will need to collaborate closely to ensure that future tools are suitable for their intended use and do not negatively impact workloads or patient trust. Future AI studies should continue to incorporate the views of key stakeholders to improve tool acceptability.

TRIAL REGISTRATION NUMBER

NCT05028179; ISRCTN15113915; IRAS ref: 293515.

Deep reinforcement learning for multi-class imbalanced training: applications in healthcare

With the rapid growth of memory and computing power, datasets are becoming increasingly complex and imbalanced. This is especially severe in the context of clinical data, where there may be one rare event for many cases in the majority class. We introduce an imbalanced classification framework, based on reinforcement learning, for training extremely imbalanced data sets, and extend it for use in multi-class settings. We combine dueling and double deep Q-learning architectures, and formulate a custom reward function and episode-training procedure, specifically with the capability of handling multi-class imbalanced training. Using real-world clinical case studies, we demonstrate that our proposed framework outperforms current state-of-the-art imbalanced learning methods, achieving more fair and balanced classification, while also significantly improving the prediction of minority classes.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10994-023-06481-z.

Detection of patients with COVID-19 by the emergency medical services in Lombardy through an operator-based interview and machine learning models

BACKGROUND

The regional emergency medical service (EMS) in Lombardy (Italy) developed clinical algorithms based on operator-based interviews to detect patients with COVID-19 and refer them to the most appropriate hospitals. Machine learning (ML)-based models using additional clinical and geospatial epidemiological data may improve the identification of infected patients and guide EMS in detecting COVID-19 cases before confirmation with SARS-CoV-2 reverse transcriptase PCR (rtPCR).

METHODS

This was an observational, retrospective cohort study using data from October 2020 to July 2021 (training set) and October 2021 to December 2021 (validation set) from patients who underwent a SARS-CoV-2 rtPCR test within 7 days of an EMS call. The performance of an operator-based interview using close contact history and signs/symptoms of COVID-19 was assessed in the training set for its ability to determine which patients had an rtPCR in the 7 days before or after the call. The interview accuracy was compared with four supervised ML models to predict positivity for SARS-CoV-2 within 7 days using readily available prehospital data retrieved from both training and validation sets.

RESULTS

The training set includes 264 976 patients, median age 74 (IQR 55-84). Test characteristics for the detection of COVID-19-positive patients of the operator-based interview were: sensitivity 85.5%, specificity 58.7%, positive predictive value (PPV) 37.5% and negative predictive value (NPV) 93.3%. Contact history, fever and cough showed the highest association with SARS-CoV-2 infection. In the validation set (103 336 patients, median age 73 (IQR 50-84)), the best-performing ML model had an AUC of 0.85 (95% CI 0.84 to 0.86), sensitivity 91.4% (95 CI% 0.91 to 0.92), specificity 44.2% (95% CI 0.44 to 0.45) and accuracy 85% (95% CI 0.84 to 0.85). PPV and NPV were 13.3% (95% CI 0.13 to 0.14) and 98.2% (95% CI 0.98 to 0.98), respectively. Contact history, fever, call geographical distribution and cough were the most important variables in determining the outcome.

CONCLUSION

ML-based models might help EMS identify patients with SARS-CoV-2 infection, and in guiding EMS allocation of hospital resources based on prespecified criteria.

COVID-19 screening in low resource settings using artificial intelligence for chest radiographs and point-of-care blood tests

Artificial intelligence (AI) systems for detection of COVID-19 using chest X-Ray (CXR) imaging and point-of-care blood tests were applied to data from four low resource African settings. The performance of these systems to detect COVID-19 using various input data was analysed and compared with antigen-based rapid diagnostic tests. Participants were tested using the gold standard of RT-PCR test (nasopharyngeal swab) to determine whether they were infected with SARS-CoV-2. A total of 3737 (260 RT-PCR positive) participants were included. In our cohort, AI for CXR images was a poor predictor of COVID-19 (AUC = 0.60), since the majority of positive cases had mild symptoms and no visible pneumonia in the lungs. AI systems using differential white blood cell counts (WBC), or a combination of WBC and C-Reactive Protein (CRP) both achieved an AUC of 0.74 with a suggested optimal cut-off point at 83% sensitivity and 63% specificity. The antigen-RDT tests in this trial obtained 65% sensitivity at 98% specificity. This study is the first to validate AI tools for COVID-19 detection in an African setting. It demonstrates that screening for COVID-19 using AI with point-of-care blood tests is feasible and can operate at a higher sensitivity level than antigen testing.

Perceptions and concerns of emergency medicine practitioners about artificial intelligence in emergency triage management during the pandemic: a national survey-based study

Objective

There have been continuous discussions over the ethics of using AI in healthcare. We sought to identify the ethical issues and viewpoints of Turkish emergency care doctors about the use of AI during epidemic triage.

Materials and methods

Ten emergency specialists were initially enlisted for this project, and their responses to open-ended questions about the ethical issues surrounding AI in the emergency room provided valuable information. A 15-question survey was created based on their input and was refined through a pilot test with 15 emergency specialty doctors. Following that, the updated survey was sent to emergency specialists via email, social media, and private email distribution.

Results

167 emergency medicine specialists participated in the study, with an average age of 38.22 years and 6.79 years of professional experience. The majority agreed that AI could benefit patients (54.50%) and healthcare professionals (70.06%) in emergency department triage during pandemics. Regarding responsibility, 63.47% believed in shared responsibility between emergency medicine specialists and AI manufacturers/programmers for complications. Additionally, 79.04% of participants agreed that the responsibility for complications in AI applications varies depending on the nature of the complication. Concerns about privacy were expressed by 20.36% regarding deep learning-based applications, while 61.68% believed that anonymity protected privacy. Additionally, 70.66% of participants believed that AI systems would be as sensitive as humans in terms of non-discrimination.

Conclusion

The potential advantages of deploying AI programs in emergency department triage during pandemics for patients and healthcare providers were acknowledged by emergency medicine doctors in Turkey. Nevertheless, they expressed notable ethical concerns related to the responsibility and accountability aspects of utilizing AI systems in this context.

Automation in Contemporary Clinical Information Systems: a Survey of AI in Healthcare Settings

AIMS AND OBJECTIVES

To examine the nature and use of automation in contemporary clinical information systems by reviewing studies reporting the implementation and evaluation of artificial intelligence (AI) technologies in healthcare settings.

METHOD

PubMed/MEDLINE, Web of Science, EMBASE, the tables of contents of major informatics journals, and the bibliographies of articles were searched for studies reporting evaluation of AI in clinical settings from January 2021 to December 2022. We documented the clinical application areas and tasks supported, and the level of system autonomy. Reported effects on user experience, decision-making, care delivery and outcomes were summarised.

RESULTS

AI technologies are being applied in a wide variety of clinical areas. Most contemporary systems utilise deep learning, use routinely collected data, support diagnosis and triage, are assistive (requiring users to confirm or approve AI provided information or decisions), and are used by doctors in acute care settings in high-income nations. AI systems are integrated and used within existing clinical information systems including electronic medical records. There is limited support for One Health goals. Evaluation is largely based on quantitative methods measuring effects on decision-making.

CONCLUSION

AI systems are being implemented and evaluated in many clinical areas. There remain many opportunities to understand patterns of routine use and evaluate effects on decision-making, care delivery and patient outcomes using mixed-methods. Support for One Health including integrating data about environmental factors and social determinants needs further exploration.

Algorithmic fairness and bias mitigation for clinical machine learning with deep reinforcement learning

As models based on machine learning continue to be developed for healthcare applications, greater effort is needed to ensure that these technologies do not reflect or exacerbate any unwanted or discriminatory biases that may be present in the data. Here we introduce a reinforcement learning framework capable of mitigating biases that may have been acquired during data collection. In particular, we evaluated our model for the task of rapidly predicting COVID-19 for patients presenting to hospital emergency departments and aimed to mitigate any site (hospital)-specific and ethnicity-based biases present in the data. Using a specialized reward function and training procedure, we show that our method achieves clinically effective screening performances, while significantly improving outcome fairness compared with current benchmarks and state-of-the-art machine learning methods. We performed external validation across three independent hospitals, and additionally tested our method on a patient intensive care unit discharge status task, demonstrating model generalizability.

Machine Learning in Laboratory Medicine: Recommendations of the IFCC Working Group

BACKGROUND

Machine learning (ML) has been applied to an increasing number of predictive problems in laboratory medicine, and published work to date suggests that it has tremendous potential for clinical applications. However, a number of groups have noted the potential pitfalls associated with this work, particularly if certain details of the development and validation pipelines are not carefully controlled.

METHODS

To address these pitfalls and other specific challenges when applying machine learning in a laboratory medicine setting, a working group of the International Federation for Clinical Chemistry and Laboratory Medicine was convened to provide a guidance document for this domain.

RESULTS

This manuscript represents consensus recommendations for best practices from that committee, with the goal of improving the quality of developed and published ML models designed for use in clinical laboratories.

CONCLUSIONS

The committee believes that implementation of these best practices will improve the quality and reproducibility of machine learning utilized in laboratory medicine.

SUMMARY

We have provided our consensus assessment of a number of important practices that are required to ensure that valid, reproducible machine learning (ML) models can be applied to address operational and diagnostic questions in the clinical laboratory. These practices span all phases of model development, from problem formulation through predictive implementation. Although it is not possible to exhaustively discuss every potential pitfall in ML workflows, we believe that our current guidelines capture best practices for avoiding the most common and potentially dangerous errors in this important emerging field.

An adversarial training framework for mitigating algorithmic biases in clinical machine learning

Machine learning is becoming increasingly prominent in healthcare. Although its benefits are clear, growing attention is being given to how these tools may exacerbate existing biases and disparities. In this study, we introduce an adversarial training framework that is capable of mitigating biases that may have been acquired through data collection. We demonstrate this proposed framework on the real-world task of rapidly predicting COVID-19, and focus on mitigating site-specific (hospital) and demographic (ethnicity) biases. Using the statistical definition of equalized odds, we show that adversarial training improves outcome fairness, while still achieving clinically-effective screening performances (negative predictive values >0.98). We compare our method to previous benchmarks, and perform prospective and external validation across four independent hospital cohorts. Our method can be generalized to any outcomes, models, and definitions of fairness.

COVID-19 rapid molecular point-of-care testing is effective and cost-beneficial for the acute care of trauma patients

PURPOSE

To evaluate the accuracy and cost benefit of a rapid molecular point-of-care testing (POCT) device detecting COVID-19 within a traumatological emergency department.

BACKGROUND

Despite continuous withdrawal of COVID-19 restrictions, hospitals will remain particularly vulnerable to local outbreaks which is reflected by a higher institution-specific basic reproduction rate. Patients admitted to the emergency department with unknown COVID-19 infection status due to a- or oligosymptomatic COVID-19 infection put other patients and health care workers at risk, while fast diagnosis and treatment is necessary. Delayed testing results in additional costs to the health care system.

METHODS

From the 8th of April 2021 until 31st of December 2021, all patients admitted to the emergency department were tested with routine RT-PCR and rapid molecular POCT device (Abbott ID NOW™ COVID-19). COVID-19-related additional costs for patients admitted via shock room or emergency department were calculated based on internal cost allocations.

RESULTS

1133 rapid molecular tests resulted in a sensitivity of 83.3% (95% CI 35.9-99.6%), specificity of 99.8% (95% CI 99.4-100%), a positive predictive value of 71.4% (95% CI 29-96.3%) and a negative predictive value of 99.9% (95% CI 99.5-100%) as compared to RT-PCR. Without rapid COVID-19 testing, each emergency department and shock room admission with subsequent surgery showed additional direct costs of 2631.25€, without surgery of 729.01€.

CONCLUSION

Although rapid molecular COVID-19 testing can initially be more expensive than RT-PCR, subsequent cost savings, improved workflows and workforce protection outweigh this effect by far. The data of this study support the use of a rapid molecular POCT device in a traumatological emergency department.

Habitat Imaging Biomarkers for Diagnosis and Prognosis in Cancer Patients Infected with COVID-19

OBJECTIVES

Cancer patients have worse outcomes from the COVID-19 infection and greater need for ventilator support and elevated mortality rates than the general population. However, previous artificial intelligence (AI) studies focused on patients without cancer to develop diagnosis and severity prediction models. Little is known about how the AI models perform in cancer patients. In this study, we aim to develop a computational framework for COVID-19 diagnosis and severity prediction particularly in a cancer population and further compare it head-to-head to a general population.

METHODS

We have enrolled multi-center international cohorts with 531 CT scans from 502 general patients and 420 CT scans from 414 cancer patients. In particular, the habitat imaging pipeline was developed to quantify the complex infection patterns by partitioning the whole lung regions into phenotypically different subregions. Subsequently, various machine learning models nested with feature selection were built for COVID-19 detection and severity prediction.

RESULTS

These models showed almost perfect performance in COVID-19 infection diagnosis and predicting its severity during cross validation. Our analysis revealed that models built separately on the cancer population performed significantly better than those built on the general population and locked to test on the cancer population. This may be because of the significant difference among the habitat features across the two different cohorts.

CONCLUSIONS

Taken together, our habitat imaging analysis as a proof-of-concept study has highlighted the unique radiologic features of cancer patients and demonstrated effectiveness of CT-based machine learning model in informing COVID-19 management in the cancer population.

Privacy-Aware Early Detection of COVID-19 Through Adversarial Training

Early detection of COVID-19 is an ongoing area of research that can help with triage, monitoring and general health assessment of potential patients and may reduce operational strain on hospitals that cope with the coronavirus pandemic. Different machine learning techniques have been used in the literature to detect potential cases of coronavirus using routine clinical data (blood tests, and vital signs measurements). Data breaches and information leakage when using these models can bring reputational damage and cause legal issues for hospitals. In spite of this, protecting healthcare models against leakage of potentially sensitive information is an understudied research area. In this study, two machine learning techniques that aim to predict a patient's COVID-19 status are examined. Using adversarial training, robust deep learning architectures are explored with the aim to protect attributes related to demographic information about the patients. The two models examined in this work are intended to preserve sensitive information against adversarial attacks and information leakage. In a series of experiments using datasets from the Oxford University Hospitals (OUH), Bedfordshire Hospitals NHS Foundation Trust (BH), University Hospitals Birmingham NHS Foundation Trust (UHB), and Portsmouth Hospitals University NHS Trust (PUH), two neural networks are trained and evaluated. These networks predict PCR test results using information from basic laboratory blood tests, and vital signs collected from a patient upon arrival to the hospital. The level of privacy each one of the models can provide is assessed and the efficacy and robustness of the proposed architectures are compared with a relevant baseline. One of the main contributions in this work is the particular focus on the development of effective COVID-19 detection models with built-in mechanisms in order to selectively protect sensitive attributes against adversarial attacks. The results on hold-out test set and external validation confirmed that there was no impact on the generalisibility of the model using adversarial learning.

Machine learning insights concerning inflammatory and liver-related risk comorbidities in non-communicable and viral diseases

The liver is a key organ involved in a wide range of functions, whose damage can lead to chronic liver disease (CLD). CLD accounts for more than two million deaths worldwide, becoming a social and economic burden for most countries. Among the different factors that can cause CLD, alcohol abuse, viruses, drug treatments, and unhealthy dietary patterns top the list. These conditions prompt and perpetuate an inflammatory environment and oxidative stress imbalance that favor the development of hepatic fibrogenesis. High stages of fibrosis can eventually lead to cirrhosis or hepatocellular carcinoma (HCC). Despite the advances achieved in this field, new approaches are needed for the prevention, diagnosis, treatment, and prognosis of CLD. In this context, the scientific com-munity is using machine learning (ML) algorithms to integrate and process vast amounts of data with unprecedented performance. ML techniques allow the integration of anthropometric, genetic, clinical, biochemical, dietary, lifestyle and omics data, giving new insights to tackle CLD and bringing personalized medicine a step closer. This review summarizes the investigations where ML techniques have been applied to study new approaches that could be used in inflammatory-related, hepatitis viruses-induced, and coronavirus disease 2019-induced liver damage and enlighten the factors involved in CLD development.

Machine learning generalizability across healthcare settings: insights from multi-site COVID-19 screening

As patient health information is highly regulated due to privacy concerns, most machine learning (ML)-based healthcare studies are unable to test on external patient cohorts, resulting in a gap between locally reported model performance and cross-site generalizability. Different approaches have been introduced for developing models across multiple clinical sites, however less attention has been given to adopting ready-made models in new settings. We introduce three methods to do this—(1) applying a ready-made model “as-is” (2); readjusting the decision threshold on the model’s output using site-specific data and (3); finetuning the model using site-specific data via transfer learning. Using a case study of COVID-19 diagnosis across four NHS Hospital Trusts, we show that all methods achieve clinically-effective performances (NPV > 0.959), with transfer learning achieving the best results (mean AUROCs between 0.870 and 0.925). Our models demonstrate that site-specific customization improves predictive performance when compared to other ready-made approaches.

Clinical prediction models for diagnosis of COVID-19 among adult patients: a validation and agreement study

BACKGROUND

Since the beginning of the pandemic, hospitals have been constantly overcrowded, with several observed waves of infected cases and hospitalisations. To avoid as much as possible this situation, efficient tools to facilitate the diagnosis of COVID-19 are needed.

OBJECTIVE

To evaluate and compare prediction models to diagnose COVID-19 identified in a systematic review published recently using performance indicators such as discrimination and calibration measures.

METHODS

A total of 1618 adult patients present at two Emergency Department triage centers and for whom qRT-PCR tests had been performed were included in this study. Six previously published models were reconstructed and assessed using diagnostic tests as sensitivity (Se) and negative predictive value (NPV), discrimination (Area Under the Roc Curve (AUROC)) and calibration measures. Agreement was also measured between them using Kappa's coefficient and IntraClass Correlation Coefficient (ICC). A sensitivity analysis has been conducted by waves of patients.

RESULTS

Among the 6 selected models, those based only on symptoms and/or risk exposure were found to be less efficient than those based on biological parameters and/or radiological examination with smallest AUROC values (< 0.80). However, all models showed good calibration and values above > 0.75 for Se and NPV but poor agreement (Kappa and ICC < 0.5) between them. The results of the first wave were similar to those of the second wave.

CONCLUSION

Although quite acceptable and similar results were found between all models, the importance of radiological examination was also emphasized, making it difficult to find an appropriate triage system to classify patients at risk for COVID-19.