The use of a validated obstetrical risk score for maternal morbidity to predict postpartum hemorrhage for women in labor

External validation and comparison of four prediction scores for severe maternal morbidity

BACKGROUND

Severe maternal morbidity (SMM) is increasing in the United States. Several tools and scores exist to stratify an individual's risk of SMM.

OBJECTIVE

We sought to examine and compare the validity of four scoring systems for predicting SMM.

STUDY DESIGN

This was a retrospective cohort study of all individuals in the Consortium on Safe Labor dataset, which was conducted from 2002 to 2008. Individuals were excluded if they had missing information on risk factors. SMM was defined based on the Centers for Disease Control and Prevention excluding blood transfusion. Blood transfusion was excluded due to concerns regarding the specificity of International Classification of Diseases codes for this indicator and its variable clinical significance. Risk scores were calculated for each participant using the Assessment of Perinatal Excellence (APEX), California Maternal Quality Care Collaborative (CMQCC), Obstetric Comorbidity Index (OB-CMI), and modified OB-CMI. We calculated the probability of SMM according to the risk scores. The discriminative performance of the prediction score was examined by the areas under receiver operating characteristic curves and their 95% confidence intervals (95% CI). The area under the curve for each score was compared using the bootstrap resampling. Calibration plots were developed for each score to examine the goodness-of-fit. The concordance probability method was used to define an optimal cutoff point for the best-performing score.

RESULTS

Of 153, 463 individuals, 1115 (0.7%) had SMM. The CMQCC scoring system had a significantly higher area under the curve (95% CI) (0.78 [0.77-0.80]) compared to the APEX scoring system, OB-CMI, and modified OB-CMI scoring systems (0.75 [0.73-0.76], 0.67 [0.65-0.68], 0.66 [0.70-0.73]; P<.001). Calibration plots showed excellent concordance between the predicted and actual SMM for the APEX scoring system and OB-CMI (both Hosmer-Lemeshow test P values=1.00, suggesting goodness-of-fit).

CONCLUSION

This study validated four risk-scoring systems to predict SMM. Both CMQCC and APEX scoring systems had good discrimination to predict SMM. The APEX score and the OB-CMI had goodness-of-fit. At ideal calculated cut-off points, the APEX score had the highest sensitivity of the four scores at 71%, indicating that better scoring systems are still needed for predicting SMM.

The impact of birth settings on pregnancy outcomes in the United States

In the United States, 98.3% of patients give birth in hospitals, 1.1% give birth at home, and 0.5% give birth in freestanding birth centers. This review investigated the impact of birth settings on birth outcomes in the United States. Presently, there are insufficient data to evaluate levels of maternal mortality and severe morbidity according to place of birth. Out-of-hospital births are associated with fewer interventions such as episiotomies, epidural anesthesia, operative deliveries, and cesarean deliveries. When compared with hospital births, there are increased rates of avoidable adverse perinatal outcomes in out-of-hospital births in the United States, both for those with and without risk factors. In one recent study, the neonatal mortality rates were significantly elevated for all planned home births: 13.66 per 10,000 live births (242/177,156; odds ratio, 4.19; 95% confidence interval, 3.62-4.84; P<.0001) vs 3.27 per 10,000 live births for in-hospital Certified Nurse-Midwife-attended births (745/2,280,044; odds ratio, 1). These differences increased further when patients were stratified by recognized risk factors such as breech presentation, multiple gestations, nulliparity, advanced maternal age, and postterm pregnancy. Causes of the increased perinatal morbidity and mortality include deliveries of patients with increased risks, absence of standardized criteria to exclude high-risk deliveries, and that most midwives attending out-of-hospital births in the United States do not meet the gold standard for midwifery regulation, the International Confederation of Midwives' Global Standards for Midwifery Education. As part of the informed consent process, pregnant patients interested in out-of-hospital births should be informed of its increased perinatal risks. Hospital births should be supported for all patients, especially those with increased risks.

Development of a risk prediction score for acute postpartum care utilization

BACKGROUND

Acute postpartum care utilization and readmissions are increasing in the United States and contribute significantly to maternal morbidity, mortality, and healthcare costs. Currently, there are limited data on the prediction of patients who will require acute postpartum care utilization.

OBJECTIVE

To develop and validate a risk prediction model for acute postpartum care utilization.

STUDY DESIGN

A retrospective cohort study of delivery hospitalizations with a linked birth certificate and discharge records in California from 2011 to 2015 was divided into a training and testing set for analysis and validation. Predictive models for acute postpartum care utilization using demographic, comorbidity, obstetrical complication, and other factors were developed using a backward stepwise logistic regression on training data. A risk score for acute postpartum care utilization was developed using beta coefficients from the factors remaining in the final multivariable model. Risk scores were validated using the testing dataset.

RESULTS

The final sample included 2,045,988 delivery hospitalizations with an acute postpartum care utilization rate of 7.6% in both training and testing cohorts. Twenty-two risk factors were identified for the final multivariable model, including several that were associated with two or more increased odds of acute care utilization (public insurance, postpartum hemorrhage, extremes of maternal age). The mean risk score was 2.45, conferring a 15 times higher risk of acute postpartum care utilization compared to those with a risk score <1 (RR 15.4, 95% CI: 11.0, 21.7). Demographics and test performance characteristics were comparably similar in predictive capability in both models (0.67 in both the training and testing cohorts).

CONCLUSION

Risk factors that are identifiable before discharge can be used to create a cumulative risk score to stratify patients at the lowest and highest risk of acute postpartum care utilization with satisfactory accuracy. External validation and the addition of other granular clinical variables are necessary to validate the feasibility of use.

Natural language processing of admission notes to predict severe maternal morbidity during the delivery encounter

BACKGROUND

Severe maternal morbidity and mortality remain public health priorities in the United States, given their high rates relative to other high-income countries and the notable racial and ethnic disparities that exist. In general, accurate risk stratification methods are needed to help patients, providers, hospitals, and health systems plan for and potentially avert adverse outcomes.

OBJECTIVE

Our objective was to understand if machine learning methods with natural language processing of history and physical notes could identify a group of patients at high risk of maternal morbidity on admission for delivery without relying on any additional patient information (eg, demographics and diagnosis codes).

STUDY DESIGN

This was a retrospective study of people admitted for delivery at 2 hospitals (hospitals A and B) in a single healthcare system between July 1, 2016, and June 30, 2020. The primary outcome was severe maternal morbidity, as defined by the Centers for Disease Control and Prevention; furthermore, we examined nontransfusion severe maternal morbidity. Clinician documents designated as history and physical notes were extracted from the electronic health record for processing and analysis. A bag-of-words approach was used for this natural language processing analysis (ie, each history or physical note was converted into a matrix of counts of individual words (or phrases) that occurred within the document). The least absolute shrinkage and selection operator models were used to generate prediction probabilities for severe maternal morbidity and nontransfusion severe maternal morbidity for each note. Model discrimination was assessed via the area under the receiver operating curve. Discrimination was compared between models using the DeLong test. Calibration plots were generated to assess model calibration. Moreover, the natural language processing models with history and physical note texts were compared with validated obstetrical comorbidity risk scores based on diagnosis codes.

RESULTS

There were 13,572 delivery encounters with history and physical notes from hospital A, split between training (A_train, n=10,250) and testing (A_test, n=3,322) datasets for model derivation and internal validation. There were 23,397 delivery encounters with history and physical notes from hospital B (B_valid) used for external validation. For the outcome of severe maternal morbidity, the natural language processing model had an area under the receiver operating curve of 0.67 (95% confidence interval, 0.63-0.72) and 0.72 (95% confidence interval, 0.70-0.74) in the A_test and B_valid datasets, respectively. For the outcome of nontransfusion severe maternal morbidity, the area under the receiver operating curve was 0.72 (95% confidence interval, 0.65-0.80) and 0.76 (95% confidence interval, 0.73-0.79) in the A_test and B_valid datasets, respectively. The calibration plots demonstrated the bag-of-words model's ability to distinguish a group of individuals at a substantially higher risk of severe maternal morbidity and nontransfusion severe maternal morbidity, notably those in the top deciles of predicted risk. Areas under the receiver operating curve in the natural language processing-based models were similar to those generated using a validated, retrospectively derived, diagnosis code-based comorbidity score.

CONCLUSION

In this practical application of machine learning, we demonstrated the capabilities of natural language processing for the prediction of severe maternal morbidity based on provider documentation inherently generated at the time of admission. This work should serve as a catalyst for providers, hospitals, and electronic health record systems to explore ways that artificial intelligence can be incorporated into clinical practice and evaluated rigorously for their ability to improve health.

The potential of big data for obstetrics discovery

PURPOSE OF REVIEW

The purpose of this article is to introduce the concept of 'Big Data' and review its potential to advance scientific discovery in obstetrics.

RECENT FINDINGS

Big Data is now ubiquitous in medicine, being used in many specialties to understand the pathophysiology, risk factors, and treatment for many diseases. Big Data analyses often employ machine learning methods to understand the complex relationships that may exist within these sources. We review the basic principles of supervised and unsupervised machine learning methods, including deep learning. We highlight how these methods have been used to study genetic risk factors for preterm birth, interpreting electronic fetal heart rate tracings, and predict adverse maternal and neonatal outcomes during pregnancy and delivery. Despite its promise, there are challenges with using Big Data, including data integrity, generalizability (namely the concerns about perpetuating inequalities), and confidentiality.

SUMMARY

The combination of new data and enhanced methods present a synergistic opportunity to explore the complex relationships common to human illness and medical practice, including obstetrics. With prediction as a primary objective instead of the more familiar goals of hypothesis testing, these analytic methods can capture multifaceted, rare, and nuanced relationships between exposures and outcomes that exist within these large data sets.