The NILS Study Protocol: A Retrospective Validation Study of an Artificial Neural Network Based Preoperative Decision-Making Tool for Noninvasive Lymph Node Staging in Women with Primary Breast Cancer (ISRCTN14341750)

Prediction model protocols indicate better adherence to recommended guidelines for study conduct and reporting

BACKGROUND AND OBJECTIVE

Protocols are invaluable documents for any research study, especially for prediction model studies. However, the mere existence of a protocol is insufficient if key details are omitted. We reviewed the reporting content and details of the proposed design and methods reported in published protocols for prediction model research.

METHODS

We searched MEDLINE, Embase, and the Web of Science Core Collection for protocols for studies developing or validating a diagnostic or prognostic model using any modeling approach in any clinical area. We screened protocols published between Jan 1, 2022 and June 30, 2022. We used the abstract, introduction, methods, and discussion sections of The Transparent Reporting of a multivariable prediction model of Individual Prognosis Or Diagnosis (TRIPOD) statement to inform data extraction.

RESULTS

We identified 30 protocols, of which 28 were describing plans for model development and six for model validation. All protocols were open access, including a preprint. 15 protocols reported prospectively collecting data. 21 protocols planned to use clustered data, of which one-third planned methods to account for it. A planned sample size was reported for 93% development and 67% validation analyses. 16 protocols reported details of study registration, but all protocols reported a statement on ethics approval. Plans for data sharing were reported in 13 protocols.

CONCLUSION

Protocols for prediction model studies are uncommon, and few are made publicly available. Those that are available were reasonably well-reported and often described their methods following current prediction model research recommendations, likely leading to better reporting and methods in the actual study.

Retrospective validation study of an artificial neural network-based preoperative decision-support tool for noninvasive lymph node staging (NILS) in women with primary breast cancer (ISRCTN14341750)

BACKGROUND

Surgical sentinel lymph node biopsy (SLNB) is routinely used to reliably stage axillary lymph nodes in early breast cancer (BC). However, SLNB may be associated with postoperative arm morbidities. For most patients with BC undergoing SLNB, the findings are benign, and the procedure is currently questioned. A decision-support tool for the prediction of benign sentinel lymph nodes based on preoperatively available data has been developed using artificial neural network modelling.

METHODS

This was a retrospective geographical and temporal validation study of the noninvasive lymph node staging (NILS) model, based on preoperatively available data from 586 women consecutively diagnosed with primary BC at two sites. Ten preoperative clinicopathological characteristics from each patient were entered into the web-based calculator, and the probability of benign lymph nodes was predicted. The performance of the NILS model was assessed in terms of discrimination with the area under the receiver operating characteristic curve (AUC) and calibration, that is, comparison of the observed and predicted event rates of benign axillary nodal status (N0) using calibration slope and intercept. The primary endpoint was axillary nodal status (discrimination, benign [N0] vs. metastatic axillary nodal status [N+]) determined by the NILS model compared to nodal status by definitive pathology.

RESULTS

The mean age of the women in the cohort was 65 years, and most of them (93%) had luminal cancers. Approximately three-fourths of the patients had no metastases in SLNB (N0 74% and 73%, respectively). The AUC for the predicted probabilities for the whole cohort was 0.6741 (95% confidence interval: 0.6255-0.7227). More than one in four patients (n = 151, 26%) were identified as candidates for SLNB omission when applying the predefined cut-off for lymph node-negative status from the development cohort. The NILS model showed the best calibration in patients with a predicted high probability of healthy axilla.

CONCLUSION

The performance of the NILS model was satisfactory. In approximately every fourth patient, SLNB could potentially be omitted. Considering the shift from postoperatively to preoperatively available predictors in this validation study, we have demonstrated the robustness of the NILS model. The clinical usability of the web interface will be evaluated before its clinical implementation.

TRIAL REGISTRATION

Registered in the ISRCTN registry with study ID ISRCTN14341750. Date of registration 23/11/2018.

Noninvasive Staging of Lymph Node Status in Breast Cancer Using Machine Learning: External Validation and Further Model Development

BACKGROUND

Most patients diagnosed with breast cancer present with a node-negative disease. Sentinel lymph node biopsy (SLNB) is routinely used for axillary staging, leaving patients with healthy axillary lymph nodes without therapeutic effects but at risk of morbidities from the intervention. Numerous studies have developed nodal status prediction models for noninvasive axillary staging using postoperative data or imaging features that are not part of the diagnostic workup. Lymphovascular invasion (LVI) is a top-ranked predictor of nodal metastasis; however, its preoperative assessment is challenging.

OBJECTIVE

This paper aimed to externally validate a multilayer perceptron (MLP) model for noninvasive lymph node staging (NILS) in a large population-based cohort (n=18,633) and develop a new MLP in the same cohort. Data were extracted from the Swedish National Quality Register for Breast Cancer (NKBC, 2014-2017), comprising only routinely and preoperatively available documented clinicopathological variables. A secondary aim was to develop and validate an LVI MLP for imputation of missing LVI status to increase the preoperative feasibility of the original NILS model.

METHODS

Three nonoverlapping cohorts were used for model development and validation. A total of 4 MLPs for nodal status and 1 LVI MLP were developed using 11 to 12 routinely available predictors. Three nodal status models were used to account for the different availabilities of LVI status in the cohorts and external validation in NKBC. The fourth nodal status model was developed for 80% (14,906/18,663) of NKBC cases and validated in the remaining 20% (3727/18,663). Three alternatives for imputation of LVI status were compared. The discriminatory capacity was evaluated using the validation area under the receiver operating characteristics curve (AUC) in 3 of the nodal status models. The clinical feasibility of the models was evaluated using calibration and decision curve analyses.

RESULTS

External validation of the original NILS model was performed in NKBC (AUC 0.699, 95% CI 0.690-0.708) with good calibration and the potential of sparing 16% of patients with node-negative disease from SLNB. The LVI model was externally validated (AUC 0.747, 95% CI 0.694-0.799) with good calibration but did not improve the discriminatory performance of the nodal status models. A new nodal status model was developed in NKBC without information on LVI (AUC 0.709, 95% CI: 0.688-0.729), with excellent calibration in the holdout internal validation cohort, resulting in the potential omission of 24% of patients from unnecessary SLNBs.

CONCLUSIONS

The NILS model was externally validated in NKBC, where the imputation of LVI status did not improve the model's discriminatory performance. A new nodal status model demonstrated the feasibility of using register data comprising only the variables available in the preoperative setting for NILS using machine learning. Future steps include ongoing preoperative validation of the NILS model and extending the model with, for example, mammography images.

The implementation of NILS: A web-based artificial neural network decision support tool for noninvasive lymph node staging in breast cancer

Objective

To implement artificial neural network (ANN) algorithms for noninvasive lymph node staging (NILS) to a decision support tool and facilitate the option to omit surgical axillary staging in breast cancer patients with low-risk of nodal metastasis.

Methods

The NILS tool is a further development of an ANN prototype for the prediction of nodal status. Training and internal validation of the original algorithm included 15 clinical and tumor-related variables from a consecutive cohort of 800 breast cancer cases. The updated NILS tool included 10 top-ranked input variables from the original prototype. A workflow with four ANN pathways was additionally developed to allow different combinations of missing preoperative input values. Predictive performances were assessed by area under the receiver operating characteristics curves (AUC) and sensitivity/specificity values at defined cut-points. Clinical utility was presented by estimating possible sentinel lymph node biopsy (SLNB) reduction rates. The principles of user-centered design were applied to develop an interactive web-interface to predict the patient's probability of healthy lymph nodes. A technical validation of the interface was performed using data from 100 test patients selected to cover all combinations of missing histopathological input values.

Results

ANN algorithms for the prediction of nodal status have been implemented into the web-based NILS tool for personalized, noninvasive nodal staging in breast cancer. The estimated probability of healthy lymph nodes using the interface showed a complete concordance with estimations from the reference algorithm except in two cases that had been wrongly included (ineligible for the technical validation). NILS predictive performance to distinguish node-negative from node-positive disease, also with missing values, displayed AUC ranged from 0.718 (95% CI, 0.687-0.748) to 0.735 (95% CI, 0.704-0.764), with good calibration. Sensitivity 90% and specificity 34% were demonstrated. The potential to abstain from axillary surgery was observed in 26% of patients using the NILS tool, acknowledging a false negative rate of 10%, which is clinically accepted for the standard SLNB technique.

Conclusions

The implementation of NILS into a web-interface are expected to provide the health care with decision support and facilitate preoperative identification of patients who could be good candidates to avoid unnecessary surgical axillary staging.

The implementation of a noninvasive lymph node staging (NILS) preoperative prediction model is cost effective in primary breast cancer

Purpose

The need for sentinel lymph node biopsy (SLNB) in clinically node-negative (cN0) patients is currently questioned. Our objective was to investigate the cost-effectiveness of a preoperative noninvasive lymph node staging (NILS) model (an artificial neural network model) for predicting pathological nodal status in patients with cN0 breast cancer (BC).

Methods

A health-economic decision-analytic model was developed to evaluate the utility of the NILS model in reducing the proportion of cN0 patients with low predicted risk undergoing SLNB. The model used information from a national registry and published studies, and three sensitivity/specificity scenarios of the NILS model were evaluated. Subgroup analysis explored the outcomes of breast-conserving surgery (BCS) or mastectomy. The results are presented as cost (€) and quality-adjusted life years (QALYs) per 1000 patients.

Results

All three scenarios of the NILS model reduced total costs (–€93,244 to –€398,941 per 1000 patients). The overall health benefit allowing for the impact of SLNB complications was a net health gain (7.0–26.9 QALYs per 1000 patients). Sensitivity analyses disregarding reduced quality of life from lymphedema showed a small loss in total health benefits (0.4–4.0 QALYs per 1000 patients) because of the reduction in total life years (0.6–6.5 life years per 1000 patients) after reduced adjuvant treatment. Subgroup analyses showed greater cost reductions and QALY gains in patients undergoing BCS.

Conclusion

Implementing the NILS model to identify patients with low risk for nodal metastases was associated with substantial cost reductions and likely overall health gains, especially in patients undergoing BCS.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10549-022-06636-x.