Comparison of diagnostic usefulness of predictive models in preliminary differentiation of adnexal masses

Multiclass risk models for ovarian malignancy: an illustration of prediction uncertainty due to the choice of algorithm

BACKGROUND

Assessing malignancy risk is important to choose appropriate management of ovarian tumors. We compared six algorithms to estimate the probabilities that an ovarian tumor is benign, borderline malignant, stage I primary invasive, stage II-IV primary invasive, or secondary metastatic.

METHODS

This retrospective cohort study used 5909 patients recruited from 1999 to 2012 for model development, and 3199 patients recruited from 2012 to 2015 for model validation. Patients were recruited at oncology referral or general centers and underwent an ultrasound examination and surgery ≤ 120 days later. We developed models using standard multinomial logistic regression (MLR), Ridge MLR, random forest (RF), XGBoost, neural networks (NN), and support vector machines (SVM). We used nine clinical and ultrasound predictors but developed models with or without CA125.

RESULTS

Most tumors were benign (3980 in development and 1688 in validation data), secondary metastatic tumors were least common (246 and 172). The c-statistic (AUROC) to discriminate benign from any type of malignant tumor ranged from 0.89 to 0.92 for models with CA125, from 0.89 to 0.91 for models without. The multiclass c-statistic ranged from 0.41 (SVM) to 0.55 (XGBoost) for models with CA125, and from 0.42 (SVM) to 0.51 (standard MLR) for models without. Multiclass calibration was best for RF and XGBoost. Estimated probabilities for a benign tumor in the same patient often differed by more than 0.2 (20% points) depending on the model. Net Benefit for diagnosing malignancy was similar for algorithms at the commonly used 10% risk threshold, but was slightly higher for RF at higher thresholds. Comparing models, between 3% (XGBoost vs. NN, with CA125) and 30% (NN vs. SVM, without CA125) of patients fell on opposite sides of the 10% threshold.

CONCLUSION

Although several models had similarly good performance, individual probability estimates varied substantially.

Usefulness of the HE4 biomarker as a second-line test in the assessment of suspicious ovarian tumors

Purpose

The aim of our study was the evaluation of HE4 usefulness as a test in assessment of ovarian tumors which are suspicious and difficult to classify correctly via subjective ultrasound examination.

Methods

In this retrospective cohort study 253 women diagnosed with adnexal masses were examined preoperatively. Suspicious tumors (n = 145) were divided into groups of: “probably benign” (n = 70), “uncertain” (n = 34), and “probably malignant” (n = 41). “Uncertain” tumors were also assessed as “benign” (n = 11) or “malignant” (n = 23). The logistic regression model was performed to analyze if the serum marker improves the prediction of a malignant finding and net reclassification improvement (NRI) was calculated to measure diagnostic improvement.

Results

Within the analyzed group 85 (58.6 %) benign and 60 (41.4 %) malignant tumors were confirmed histopathologically. The comparison of HE4 with subjective ultrasound assessment showed lowered NRI in the entire analyzed group as well as in the groups of tumors classified as “probably benign” or “probably malignant” (NRI = −0.16; P = 0.0139 and NRI = −0.133; P = 0.0489, respectively). The analysis of logistic regression model confirmed that biomarkers do not improve diagnostic accuracy. The difference between areas under ROC for HE4 (0.891) and CA125 (0.902) was not statistically significant (P = 0.760).

Conclusions

After subjective ultrasound assessment, the addition of the second-line test—HE4 as well as CA125 serum level does not improve diagnostic performance. However, HE4 evaluation satisfies the clinical expectations of diagnostic tools for ovarian tumors and, thus, may be useful to less experienced sonographers.

Mathematical Models to Discriminate Between Benign and Malignant Adnexal Masses: Potential Diagnostic Improvement Using Ovarian HistoScanning

Purpose:

Accurate preoperative clinical assessment of adnexal masses can optimize outcomes by ensuring appropriate and timely surgery. This article addresses whether a new technology, ovarian HistoScanning, has an additional diagnostic value in mathematical models developed for the differential diagnosis of adnexal masses.

Patients and Methods:

Transvaginal sonography-based morphological variables were obtained through blinded analysis of archived images in 199 women enrolled in a prospective study to assess the performance of ovarian HistoScanning. Logistic regression (LR) and neural network (NN) models including these variables and clinical and patient data along with the HistoScanning score (HSS) (range, 0-125; based on mathematical algorithms) were developed in a learning set (60% patients). The remaining 40% patients (evaluation set) were used to assess model performance.

Results:

Of all morphological and clinical variables tested, serum CA-125, presence of a solid component, and HSS were most significant and used to develop the LR model. The NN model included all variables. The novel variable, HSS, offered significant improvement in the LR and NN models' performance. The LR and NN models in an independent evaluation set were found to have area under the receiver operating characteristic curve = 0.97 (95% confidence interval [CI], 94-99) and 0.93 (95% CI, 88-98), sensitivities = 83% (95% CI, 71%-91%) and 80% (95% CI, 67%-89%), and specificities = 98% (95% CI, 89%-99%) and 86% (95% CI, 72%-95%), respectively. In addition, these models showed an improved performance when compared with 3 other existing models (allP< 0.05).

Conclusions:

This initial report shows a clear benefit of including ovarian HistoScanning into mathematical models used for discriminating benign from malignant ovarian masses. These models may be specifically helpful to the less experienced examiner. Future research should assess performance of these models in prospective clinical trials in different populations.

The accuracy of risk scores in predicting ovarian malignancy: a systematic review

OBJECTIVE

To perform a systematic review of the literature on the accuracy of prediction models in the preoperative assessment of adnexal masses.

DATA SOURCES

Studies were identified through the MEDLINE and EMBASE databases from inception to March 2008. The MEDLINE search was performed using the keywords ["ovarian neoplasms"[MeSH] NOT "therapeutics"[MeSH] AND "model"] and ["ovarian neoplasms"[MeSH] NOT "therapeutics"[MeSH] AND "prediction"]. The Embase search was performed using the keywords [ovary tumor AND prediction], [ovary tumor AND Mathematical model], and [ovary tumor AND statistical model].

METHODS OF STUDY SELECTION

The search detected 1,161 publications; from the cross-references, another 116 studies were identified. Language restrictions were not applied. Eligible studies contained data on the accuracy of models predicting the risk of malignancy in ovarian masses. Models were required to combine at least two parameters.

TABULATION, INTEGRATION, AND RESULTS

Two independent reviewers selected studies and extracted study characteristics, study quality, and test accuracy. There were 109 accuracy studies that met the selection criteria. Accuracy data were used to form two-by-two contingency tables of the results of the risk score compared with definitive histology. We used bivariate meta-analysis to estimate pooled sensitivities and specificities and to fit summary receiver operating characteristic curves.Studies included in our analysis reported on 83 different prediction models. The model developed by Sassone was the most evaluated prediction model. All models has acceptable sensitivity and specificity. However, the Risk of Malignancy Index I and the Risk of Malignancy Index II, which use the product of the serum CA 125 level, an ultrasound scan result, and the menopausal state, were the best predictors. When 200 was used as the cutoff level, the pooled estimate for sensitivity was 78% for a specificity of 87%.

CONCLUSION

Based on our review, the Risk of Malignancy Index should be the prediction model of choice in the preoperative assessment of the adnexal mass.

Comparison of the risk of malignancy index and self-constructed logistic regression models in preoperative evaluation of adnexal masses

OBJECTIVE

The aim of this study was to evaluate women with adnexal masses in the preoperative period by creating 2 logistic regression models, 1 including sonographic morphologic characteristics and the other including both morphologic and color Doppler characteristics, to compare the diagnostic accuracy of these 2 models with the risk of malignancy index (RMI).

METHODS

This prospective study included 38 malignant, 7 borderline, and 244 benign ovarian masses. The menopausal status, presence of septa, presence of papillary projections, location of the tumor, presence of ascites, presence of metastases, cancer antigen 125 level, tumor volume, septa thickness, and percentage of the solid component were included in the initial analysis. A second regression analysis was performed with the addition of Doppler parameters (location of blood flow and lowest resistive index) in the data set. Diagnostic performance of the 2 regression models and RMI were described and compared by generating receiver operating characteristic curves for each model.

RESULTS

The area under the curve values for the morphologic model (model 1), Doppler model (model 2), and RMI were 0.907, 0.971, and 0.889, respectively. Significance levels of model 1 and the RMI were similar (P = .23), whereas model 2 had a significantly higher area under the curve compared with both model 1 (P = .037) and the RMI (P = .018).

CONCLUSIONS

The addition of Doppler parameters in the regression model significantly increases the predictive performance. Nevertheless, in low-resource settings, the RMI remains the method of choice for distinguishing adnexal masses and referral to gynecologic oncology clinics.

Factors influencing the use and accuracy of frozen section diagnosis of epithelial ovarian tumors

OBJECTIVE

The objective of the study was to study factors influencing the use and accuracy of frozen section diagnosis (FSD) of ovarian tumors.

STUDY DESIGN

Surgery was performed in 414 patients with epithelial ovarian tumors between 2001 and 2006. Factors were identified by univariate and multivariate analysis.

RESULTS

FSD was requested in 274 patients: 152 benign, 55 borderline, and 67 malignant tumors. Age 50 years or older, tumor size 10 cm or greater, and preoperative evidence of malignancy were associated with FSD request. The sensitivity and specificity of FSD for benign, borderline, and malignant tumors were 97% and 81%, 62% and 96%, and 88% and 99%, respectively. The histologic type (mucinous), tumor size (less than 10 cm), the borderline component (less than 10%), and the pathologist's experience predicted misdiagnosis of borderline tumors. Spread outside the ovary was the only significant predictor of accurate FSD of malignant tumors.

CONCLUSION

FSD is less accurate for borderline than benign and malignant ovarian tumors. The pathologist's experience is a major determinant of diagnostic accuracy.