Probabilistic forecasting of surgical case duration using machine learning: model development and validation

A novel approach to forecast surgery durations using machine learning techniques

This study presents a methodology for predicting the duration of surgical procedures using Machine Learning (ML). The methodology incorporates a new set of predictors emphasizing the significance of surgical team dynamics and composition, including experience, familiarity, social behavior, and gender diversity. By applying ML techniques to a comprehensive dataset of over 77,000 surgeries, we achieved a 24% improvement in the mean absolute error (MAE) over a model that mimics the current approach of the decision maker. Our results also underscore the critical role of surgeon experience and team composition dynamics in enhancing prediction accuracy. These advancements can lead to more efficient operational planning and resource allocation in hospitals, potentially reducing downtime in operating rooms and improving healthcare delivery.

Long-Term Mortality Predictors Using a Machine-Learning Approach in Patients With Chronic Limb-Threatening Ischemia After Peripheral Vascular Intervention

BACKGROUND

Patients with chronic limb-threatening ischemia (CLTI) face a high long-term mortality risk. Identifying novel mortality predictors and risk profiles would enable individual health care plan design and improved survival. We aimed to leverage a random survival forest machine-learning algorithm to identify long-term all-cause mortality predictors in patients with CLTI undergoing peripheral vascular intervention.

METHODS AND RESULTS

Patients with CLTI undergoing peripheral vascular intervention from 2017 to 2018 were derived from the Medicare-linked VQI (Vascular Quality Initiative) registry. We constructed a random survival forest to rank 66 preprocedural variables according to their relative importance and mean minimal depth for 3-year all-cause mortality. A random survival forest of 2000 trees was built using a training sample (80% of the cohort). Accuracy was assessed in a testing sample (20%) using continuous ranked probability score, Harrell C-index, and out-of-bag error rate. A total of 10 114 patients were included (mean±SD age, 72.0±11.0 years; 59% men). The 3-year mortality rate was 39.1%, with a median survival of 1.4 years (interquartile range, 0.7-2.0 years). The most predictive variables were chronic kidney disease, age, congestive heart failure, dementia, arrhythmias, requiring assisted care, living at home, and body mass index. A total of 41 variables spanning all domains of the biopsychosocial model were ranked as mortality predictors. The accuracy of the model was excellent (continuous ranked probability score, 0.172; Harrell C-index, 0.70; out-of-bag error rate, 29.7%).

CONCLUSIONS

Our random survival forest accurately predicts long-term CLTI mortality, which is driven by demographic, functional, behavioral, and medical comorbidities. Broadening frameworks of risk and refining health care plans to include multidimensional risk factors could improve individualized care for CLTI.

Predicting operative time for metabolic and bariatric surgery using machine learning models: a retrospective observational study

BACKGROUND

Predicting operative time is essential for scheduling surgery and managing the operating room. This study aimed to develop machine learning (ML) models to predict the operative time for metabolic and bariatric surgery (MBS) and to compare each model.

METHODS

The authors used the Metabolic and Bariatric Surgery Accreditation and Quality Improvement Program database between 2016 and 2020 to develop ML models, including linear regression, random forest, support vector machine, gradient-boosted tree, and XGBoost model. Patient characteristics and surgical features were included as variables in the model. The authors used the mean absolute error, root mean square error, and R 2 score to evaluate model performance. The authors identified the 10 most important variables in the best-performing model using the Shapley Additive exPlanations algorithm.

RESULTS

In total, 668 723 patients were included in the study. The XGBoost model outperformed the other ML models, with the lowest root mean square error and highest R 2 score. Random forest performed better than linear regression. The relative performance of the ML algorithms remained consistent across the models, regardless of the surgery type. The surgery type and surgical approach were the most important features to predict the operative time; specifically, sleeve gastrectomy (vs. Roux-en-Y gastric bypass) and the laparoscopic approach (vs. robotic-assisted approach) were associated with a shorter operative time.

CONCLUSIONS

The XGBoost model best predicted the operative time for MBS among the ML models examined. Our findings can be useful in managing the operating room scheduling and in developing software tools to predict the operative times of MBS in clinical settings.

Development, Deployment, and Implementation of a Machine Learning Surgical Case Length Prediction Model and Prospective Evaluation

OBJECTIVE

To implement a machine learning model using only the restricted data available at case creation time to predict surgical case length for multiple services at different locations.

BACKGROUND

The operating room is one of the most expensive resources in a health system, estimated to cost $22 to $133 per minute and generate about 40% of hospital revenue. Accurate prediction of surgical case length is necessary for efficient scheduling and cost-effective utilization of the operating room and other resources.

METHODS

We introduced a similarity cascade to capture the complexity of cases and surgeon influence on the case length and incorporated that into a gradient-boosting machine learning model. The model loss function was customized to improve the balance between over- and under-prediction of the case length. A production pipeline was created to seamlessly deploy and implement the model across our institution.

RESULTS

The prospective analysis showed that the model output was gradually adopted by the schedulers and outperformed the scheduler-predicted case length from August to December 2022. In 33,815 surgical cases across outpatient and inpatient platforms, the operational implementation predicted 11.2% fewer underpredicted cases and 5.9% more cases within 20% of the actual case length compared with the schedulers and only overpredicted 5.3% more. The model assisted schedulers to predict 3.4% more cases within 20% of the actual case length and 4.3% fewer underpredicted cases.

CONCLUSIONS

We created a unique framework that is being leveraged every day to predict surgical case length more accurately at case posting time and could be potentially utilized to deploy future machine learning models.

Machine learning models to predict surgical case duration compared to current industry standards: scoping review

BACKGROUND

Surgical waiting lists have risen dramatically across the UK as a result of the COVID-19 pandemic. The effective use of operating theatres by optimal scheduling could help mitigate this, but this requires accurate case duration predictions. Current standards for predicting the duration of surgery are inaccurate. Artificial intelligence (AI) offers the potential for greater accuracy in predicting surgical case duration. This study aimed to investigate whether there is evidence to support that AI is more accurate than current industry standards at predicting surgical case duration, with a secondary aim of analysing whether the implementation of the models used produced efficiency savings.

METHOD

PubMed, Embase, and MEDLINE libraries were searched through to July 2023 to identify appropriate articles. PRISMA extension for scoping reviews and the Arksey and O'Malley framework were followed. Study quality was assessed using a modified version of the reporting guidelines for surgical AI papers by Farrow et al. Algorithm performance was reported using evaluation metrics.

RESULTS

The search identified 2593 articles: 14 were suitable for inclusion and 13 reported on the accuracy of AI algorithms against industry standards, with seven demonstrating a statistically significant improvement in prediction accuracy (P < 0.05). The larger studies demonstrated the superiority of neural networks over other machine learning techniques. Efficiency savings were identified in a RCT. Significant methodological limitations were identified across most studies.

CONCLUSION

The studies suggest that machine learning and deep learning models are more accurate at predicting the duration of surgery; however, further research is required to determine the best way to implement this technology.

Machine Learning for the Prediction of Procedural Case Durations Developed Using a Large Multicenter Database: Algorithm Development and Validation Study

BACKGROUND

Accurate projections of procedural case durations are complex but critical to the planning of perioperative staffing, operating room resources, and patient communication. Nonlinear prediction models using machine learning methods may provide opportunities for hospitals to improve upon current estimates of procedure duration.

OBJECTIVE

The aim of this study was to determine whether a machine learning algorithm scalable across multiple centers could make estimations of case duration within a tolerance limit because there are substantial resources required for operating room functioning that relate to case duration.

METHODS

Deep learning, gradient boosting, and ensemble machine learning models were generated using perioperative data available at 3 distinct time points: the time of scheduling, the time of patient arrival to the operating or procedure room (primary model), and the time of surgical incision or procedure start. The primary outcome was procedure duration, defined by the time between the arrival and the departure of the patient from the procedure room. Model performance was assessed by mean absolute error (MAE), the proportion of predictions falling within 20% of the actual duration, and other standard metrics. Performance was compared with a baseline method of historical means within a linear regression model. Model features driving predictions were assessed using Shapley additive explanations values and permutation feature importance.

RESULTS

A total of 1,177,893 procedures from 13 academic and private hospitals between 2016 and 2019 were used. Across all procedures, the median procedure duration was 94 (IQR 50-167) minutes. In estimating the procedure duration, the gradient boosting machine was the best-performing model, demonstrating an MAE of 34 (SD 47) minutes, with 46% of the predictions falling within 20% of the actual duration in the test data set. This represented a statistically and clinically significant improvement in predictions compared with a baseline linear regression model (MAE 43 min; P<.001; 39% of the predictions falling within 20% of the actual duration). The most important features in model training were historical procedure duration by surgeon, the word "free" within the procedure text, and the time of day.

CONCLUSIONS

Nonlinear models using machine learning techniques may be used to generate high-performing, automatable, explainable, and scalable prediction models for procedure duration.

An Intraoperative Telemedicine Program to Improve Perioperative Quality Measures: The ACTFAST-3 Randomized Clinical Trial

Importance

Telemedicine for clinical decision support has been adopted in many health care settings, but its utility in improving intraoperative care has not been assessed.

Objective

To pilot the implementation of a real-time intraoperative telemedicine decision support program and evaluate whether it reduces postoperative hypothermia and hyperglycemia as well as other quality of care measures.

Design, Setting, and Participants

This single-center pilot randomized clinical trial (Anesthesiology Control Tower-Feedback Alerts to Supplement Treatments [ACTFAST-3]) was conducted from April 3, 2017, to June 30, 2019, at a large academic medical center in the US. A total of 26 254 adult surgical patients were randomized to receive either usual intraoperative care (control group; n = 12 980) or usual care augmented by telemedicine decision support (intervention group; n = 13 274). Data were initially analyzed from April 22 to May 19, 2021, with updates in November 2022 and February 2023.

Intervention

Patients received either usual care (medical direction from the anesthesia care team) or intraoperative anesthesia care monitored and augmented by decision support from the Anesthesiology Control Tower (ACT), a real-time, live telemedicine intervention. The ACT incorporated remote monitoring of operating rooms by a team of anesthesia clinicians with customized analysis software. The ACT reviewed alerts and electronic health record data to inform recommendations to operating room clinicians.

Main Outcomes and Measures

The primary outcomes were avoidance of postoperative hypothermia (defined as the proportion of patients with a final recorded intraoperative core temperature >36 °C) and hyperglycemia (defined as the proportion of patients with diabetes who had a blood glucose level ≤180 mg/dL on arrival to the postanesthesia recovery area). Secondary outcomes included intraoperative hypotension, temperature monitoring, timely antibiotic redosing, intraoperative glucose evaluation and management, neuromuscular blockade documentation, ventilator management, and volatile anesthetic overuse.

Results

Among 26 254 participants, 13 393 (51.0%) were female and 20 169 (76.8%) were White, with a median (IQR) age of 60 (47-69) years. There was no treatment effect on avoidance of hyperglycemia (7445 of 8676 patients [85.8%] in the intervention group vs 7559 of 8815 [85.8%] in the control group; rate ratio [RR], 1.00; 95% CI, 0.99-1.01) or hypothermia (7602 of 11 447 patients [66.4%] in the intervention group vs 7783 of 11 672 [66.7.%] in the control group; RR, 1.00; 95% CI, 0.97-1.02). Intraoperative glucose measurement was more common among patients with diabetes in the intervention group (RR, 1.07; 95% CI, 1.01-1.15), but other secondary outcomes were not significantly different.

Conclusions and Relevance

In this randomized clinical trial, anesthesia care quality measures did not differ between groups, with high confidence in the findings. These results suggest that the intervention did not affect the targeted care practices. Further streamlining of clinical decision support and workflows may help the intraoperative telemedicine program achieve improvement in targeted clinical measures.

Trial Registration

ClinicalTrials.gov Identifier: NCT02830126.

Surgical procedure prediction using medical ontological information

BACKGROUND AND OBJECTIVE

Predicting the duration of surgical procedures is an important step in scheduling operating rooms. Many factors have been shown to influence the duration of a procedure, in this research we aim to use medical ontological information to improve the predictions.

METHODS

This paper presents two methods for incorporating the medical information about a surgical procedure into the prediction of the duration of the procedure. The first method uses the Systematised Nomenclature of Medicine Clinical Terms to relate different procedures to each other. The second uses simple text fragments. The relationships between types of procedures are included in a regression model for the procedure duration. These methods are applied to data from New Zealand healthcare facilities and the accuracy of the estimations of the durations is compared. In addition a simulation of scheduling the procedures in an operating room is performed.

RESULTS

It is shown that both of the methods provide an improvement in the prediction of procedure durations. When compared to a traditional categorical encoding, the ontological information provides an improvement in the continuous ranked probability scores of the prediction of procedure durations from 18.4 min to 17.1 min, and from 25.3 to 21.3 min for types of procedures that are not performed very often.

CONCLUSIONS

Different methods for encoding medical ontological information in surgery procedure duration predictions are presented, and show an improvement over traditional models. The improvement in duration prediction is shown to improve the efficiency of scheduling in a simple simulation.

Use of artificial intelligence in paediatric anaesthesia: a systematic review

Objectives

Although the development of artificial intelligence (AI) technologies in medicine has been significant, their application to paediatric anaesthesia is not well characterised. As the paediatric operating room is a data-rich environment that requires critical clinical decision-making, this systematic review aims to characterise the current use of AI in paediatric anaesthesia and to identify barriers to the successful integration of such technologies.

Methods

This review was registered with PROSPERO (CRD42022304610), the international registry for systematic reviews. The search strategy was prepared by a librarian and run in five electronic databases (Embase, Medline, Central, Scopus, and Web of Science). Collected articles were screened by two reviewers. Included studies described the use of AI for paediatric anaesthesia (<18 yr old) within the perioperative setting.

Results

From 3313 records identified in the initial search, 40 were included in this review. Identified applications of AI were described for patient risk factor prediction (24 studies; 60%), anaesthetic depth estimation (2; 5%), anaesthetic medication/technique decision guidance (2; 5%), intubation assistance (1; 2.5%), airway device selection (3; 7.5%), physiological variable monitoring (6; 15%), and operating room scheduling (2; 5%). Multiple domains of AI were discussed including machine learning, computer vision, fuzzy logic, and natural language processing.

Conclusion

There is an emerging literature regarding applications of AI for paediatric anaesthesia, and their clinical integration holds potential for ultimately improving patient outcomes. However, multiple barriers to their clinical integration remain including a lack of high-quality input data, lack of external validation/evaluation, and unclear generalisability to diverse settings.

Systematic review protocol

CRD42022304610 (PROSPERO).

New Frontiers of Natural Language Processing in Surgery

The vast and ever-growing volume of electronic health records (EHR) have generated a wealth of information-rich data. Traditional, non-machine learning data extraction techniques are error-prone and laborious, hindering the analytical potential of these massive data sources. Equipped with natural language processing (NLP) tools, surgeons are better able to automate, and customize their review to investigate and implement surgical solutions. We identify current perioperative applications of NLP algorithms as well as research limitations and future avenues to outline the impact and potential of this technology for progressing surgical innovation.

Surgery duration: Optimized prediction and causality analysis

Accurate estimation of duration of surgery (DOS) can lead to cost-effective utilization of surgical staff and operating rooms and decrease patients’ waiting time. In this study, we present a supervised DOS nonlinear regression prediction model whose accuracy outperforms earlier results. In addition, unlike previous studies, we identify the features that influence DOS prediction. Further, in difference from others, we study the causal relationship between the feature set and DOS. The feature sets used in prior studies included a subset of the features presented in this study. This study aimed to derive influential effectors of duration of surgery via optimized prediction and causality analysis. We implemented an array of machine learning algorithms and trained them on datasets comprising surgery-related data, to derive DOS prediction models. The datasets we acquired contain patient, surgical staff, and surgery features. The datasets comprised 23,293 surgery records of eight surgery types performed over a 10-year period in a public hospital. We have introduced new, unstudied features and combined them with features adopted from previous studies to generate a comprehensive feature set. We utilized feature importance methods to identify the influential features, and causal inference methods to identify the causal features. Our model demonstrates superior performance in comparison to DOS prediction models in the art. The performance of our DOS model in terms of the mean absolute error (MAE) was 14.9 minutes. The algorithm that derived the model with the best performance was the gradient boosted trees (GBT). We identified the 10 most influential features and the 10 most causal features. In addition, we showed that 40% of the influential features have a significant (p-value = 0.05) causal relationship with DOS. We developed a DOS prediction model whose accuracy is higher than that of prior models. This improvement is achieved via the introduction of a novel feature set on which the model was trained. Utilizing our prediction model, hospitals can improve the efficiency of surgery schedules, and by exploiting the identified causal relationship, can influence the DOS. Further, the feature importance methods we used can help explain the model’s predictions.

Machine Learning-Based Models Predicting Outpatient Surgery End Time and Recovery Room Discharge at an Ambulatory Surgery Center

Days before surgery, add-ons may be scheduled to fill unused surgical block time at an outpatient surgery center. At times, outpatient surgery centers have time limitations for end of block time and discharge from the postanesthesia care unit (PACU). The objective of our study was to develop machine learning models that predicted the following composite outcome: (1) surgery finished by end of operating room block time and (2) patient was discharged by end of recovery room nursing shift. We compared various machine learning models to logistic regression. By evaluating various performance metrics, including F1 scores, we hypothesized that models using ensemble learning will be superior to logistic regression.

Using Operative Reports to Predict Heart Transplantation Survival

Heart transplantation is a difficult procedure compared with other surgical operations, with a greater outcome uncertainty such as late rejection and death. We can model the success of heart transplants from predicting factors such as the age, sex, diagnosis, etc., of the donor and recipient. Although predictions can mitigate the uncertainty on the transplantation outcome, their accuracy is far from perfect. In this paper, we describe a new method to predict the outcome of a transplantation from textual operative reports instead of traditional tabular data. We carried out an experiment on 300 surgical reports to determine the survival rates at one year and five years. Using a truncated TF-IDF vectorization of the texts and logistic regression, we could reach a macro Fl of 59.1 %, respectively, 54.9% with a five-fold cross validation. While the size of the corpus is relatively small, our experiments show that the operative textual sources can discriminate the transplantation outcomes and could be a valuable additional input to existing prediction systems. Clinical Relevance- Heart transplantation involves a significant number of written reports including in the preoperative examinations and operative documentation. In this paper, we show that these written reports can predict the outcome of the transplantation at one and five years with macro 1s of 59.1 % and 54.9 %, respectively and complement existing prediction methods.

Feasibility of Anesthesiologists Giving Nurse Anesthetists 30-Minute Lunch Breaks and 15-Minute Morning Breaks at a University’s Facilities

Background

Managers of an anesthesia department sought an estimation of how often each anesthesiologist can give lunch breaks and morning breaks to nurse anesthetists to plan staff scheduling. When an anesthesiologist supervising the nurse anesthetists can give a break, it would be preferred because fewer extra nurse anesthetists would be scheduled to facilitate breaks.

Methodology

Our methodological development used retrospective cohort data from the three surgical suites of a single anesthesia department. Surgical times were estimated using three years of data from October 2016 through September 2019, with 95,146 cases. Comparison was made with the next year from October 2019 through September 2020, with 30,987 cases. The 5% lower prediction bounds for surgical time were estimated based on two-parameter, log-normal distributions. The times when two and three sequential rooms had overlapping lower prediction limits were calculated. Sequential rooms were used because that was how anesthesiologists’ assignments were made at the studied department, when feasible given constraints. Percentages of cases were reported with 15 minutes available starting sometime between 9:00 and 10:30 and 30 minutes starting sometime between 11:15 and 12:45, times characteristic for the studied department. At the studied university’s facilities, the nurse anesthetists were independent practitioners (e.g., an anesthesiologist supervising two nurse anesthetists each with a long case could give a break to one of the two rooms).

Results

The percentage of days for which an anesthesiologist could give a lunch break (11:15-12:45) was close to the percentage of cases when an anesthesiologist could give the same-length break anytime throughout the workday. In other words, the length of the break was important, not the time of the day of the break. The absolute percentages also depended on how many rooms the anesthesiologist supervised, the duration of cases, and facility. For example, among anesthesiologists at the adult surgical suite supervising three nurse anesthetists, a lunch break could be given by the anesthesiologist on at most one-third of the days without affecting workflow.

Conclusions

Our results show that the feasibility of an anesthesiologist clinically supervising one, two, or three rooms to give lunch breaks to the nurse anesthetists in the rooms depends principally on how many rooms are supervised, the duration of the break, and the facility’s percentage of cases with surgical times longer than that duration. The specific numerical results will differ among departments. Our methodology would be useful to other departments where anesthesiologists are clinically supervising independent practitioners, sometimes during cases long enough for a break, and there is anesthesiologist backup help. Such departments can use our methodology to plan their staff scheduling for additional nurse anesthetists to give the remaining breaks.

Continuous real-time prediction of surgical case duration using a modular artificial neural network

BACKGROUND

Real-time prediction of surgical duration can inform perioperative decisions and reduce surgical costs. We developed a machine learning approach that continuously incorporates preoperative and intraoperative information for forecasting surgical duration.

METHODS

Preoperative (e.g. procedure name) and intraoperative (e.g. medications and vital signs) variables were retrieved from anaesthetic records of surgeries performed between March 1, 2019 and October 31, 2019. A modular artificial neural network was developed and compared with a Bayesian approach and the scheduled surgical duration. Continuous ranked probability score (CRPS) was used as a measure of time error to assess model accuracy. For evaluating clinical performance, accuracy for each approach was assessed in identifying cases that ran beyond 15:00 (commonly scheduled end of shift), thus identifying opportunities to avoid overtime labour costs.

RESULTS

The analysis included 70 826 cases performed at eight hospitals. The modular artificial neural network had the lowest time error (CRPS: mean=13.8; standard deviation=35.4 min), which was significantly better (mean difference=6.4 min [95% confidence interval: 6.3-6.5]; P<0.001) than the Bayesian approach. The modular artificial neural network also had the highest accuracy in identifying operating theatres that would overrun 15:00 (accuracy at 1 h prior=89%) compared with the Bayesian approach (80%) and a naïve approach using the scheduled duration (78%).

CONCLUSIONS

A real-time neural network model using preoperative and intraoperative data had significantly better performance than a Bayesian approach or scheduled duration, offering opportunities to avoid overtime labour costs and reduce the cost of surgery by providing superior real-time information for perioperative decision support.

Similarities Between Pediatric and General Hospitals Based on Fundamental Attributes of Surgery Including Cases Per Surgeon Per Workday

Introduction

Operating room (OR) management decision-making at both pediatric and adult hospitals is determined, in large part, by the same fundamental attributes of surgery and other considerations related to case duration prediction. These include the non-preemptive nature of surgeries, wide prediction limits for case duration, and constraints to moving or resequencing cases on the day of surgery. Another attribute fundamentally affecting OR management is the median number of cases a surgeon performs on their OR days. Most adult surgeons have short lists of cases (i.e., one or two cases per day). Similarly, at adult hospitals, growth in caseloads is mostly due to the subset of those surgeons who also operate just once or twice per week. It is unknown if these characteristics of surgery apply to pediatric surgeons and pediatric hospitals as well.

Methods

Our retrospective cohort study included all elective surgical cases performed at the six pediatric hospitals in Florida during 2018 and 2019 (n = 71,340 cases). We calculated the percentages of combinations of surgeon, date, and hospital (lists) comprising one or two cases, or just one case, and determined if the values were statistically >50% (i.e., indicative of “most”). We determined if most of the growth in caseload and intraoperative work relative value units (wRVUs) at the pediatric hospitals between 2018 and 2019 accrued from low-caseload surgeons. Results are reported as mean ± standard error of the mean.

Results

Averaging among the six pediatric hospitals, the non-holiday weekday lists of most surgeons at each facility had just one or two elective cases, inpatient and/or ambulatory (68.1%; p = 0.016 vs. 50%, n = 27,557 lists). Growth in surgical caseloads from 2018 to 2019 was mostly attributable to surgeons who in 2018 averaged ≤2.0 cases per week (76.3% ± 5.4%, p = 0.0085 vs. 50%). Similarly, growth in wRVUs was mostly attributable to these low-caseload surgeons (73.8% ± 5.4%, p = 0.017 vs. 50%).

Conclusions

Like adult hospitals, most pediatric surgeons’ lists of cases consist of only one or two cases per day, with many lists containing a single case. Similarly, growth at pediatric hospitals accrued from low-caseload surgeons who performed one or two cases per week in the preceding year. These findings indicate that hospitals desiring to increase their surgical caseload should ensure that low-caseload surgeons are provided access to the OR schedule. Additionally, since percent-adjusted utilization and raw utilization cannot be accurately measured for low-caseload surgeons, neither metric should be used to allocate OR time to individual surgeons. Since most adult and pediatric surgeons have low caseloads, this is a fundamental attribute of surgery.

How New Mexico Leveraged a COVID-19 Case Forecasting Model to Preemptively Address the Health Care Needs of the State: Quantitative Analysis

BACKGROUND

Prior to the COVID-19 pandemic, US hospitals relied on static projections of future trends for long-term planning and were only beginning to consider forecasting methods for short-term planning of staffing and other resources. With the overwhelming burden imposed by COVID-19 on the health care system, an emergent need exists to accurately forecast hospitalization needs within an actionable timeframe.

OBJECTIVE

Our goal was to leverage an existing COVID-19 case and death forecasting tool to generate the expected number of concurrent hospitalizations, occupied intensive care unit (ICU) beds, and in-use ventilators 1 day to 4 weeks in the future for New Mexico and each of its five health regions.

METHODS

We developed a probabilistic model that took as input the number of new COVID-19 cases for New Mexico from Los Alamos National Laboratory's COVID-19 Forecasts Using Fast Evaluations and Estimation tool, and we used the model to estimate the number of new daily hospital admissions 4 weeks into the future based on current statewide hospitalization rates. The model estimated the number of new admissions that would require an ICU bed or use of a ventilator and then projected the individual lengths of hospital stays based on the resource need. By tracking the lengths of stay through time, we captured the projected simultaneous need for inpatient beds, ICU beds, and ventilators. We used a postprocessing method to adjust the forecasts based on the differences between prior forecasts and the subsequent observed data. Thus, we ensured that our forecasts could reflect a dynamically changing situation on the ground.

RESULTS

Forecasts made between September 1 and December 9, 2020, showed variable accuracy across time, health care resource needs, and forecast horizon. Forecasts made in October, when new COVID-19 cases were steadily increasing, had an average accuracy error of 20.0%, while the error in forecasts made in September, a month with low COVID-19 activity, was 39.7%. Across health care use categories, state-level forecasts were more accurate than those at the regional level. Although the accuracy declined as the forecast was projected further into the future, the stated uncertainty of the prediction improved. Forecasts were within 5% of their stated uncertainty at the 50% and 90% prediction intervals at the 3- to 4-week forecast horizon for state-level inpatient and ICU needs. However, uncertainty intervals were too narrow for forecasts of state-level ventilator need and all regional health care resource needs.

CONCLUSIONS

Real-time forecasting of the burden imposed by a spreading infectious disease is a crucial component of decision support during a public health emergency. Our proposed methodology demonstrated utility in providing near-term forecasts, particularly at the state level. This tool can aid other stakeholders as they face COVID-19 population impacts now and in the future.