Defining 'optimal' test cut-off using global test metrics: evidence from a cognitive screening instrument

Contextual Validation of the Prediction of Postoperative Complications of Colorectal Surgery by the "ACS NSQIP ® Risk Calculator" in a Tunisian Center

CONTEXT

Models for predicting individual risks of surgical complications are advantageous for operative decision making and the nature of postoperative management procedures.

OBJECTIVE

Validate the "ACS NSQIP® Risk Calculator" in the prediction of postoperative complications during colorectal cancer surgery, operated during the years 2015 to 2019.

METHODS

this is a prognostic validation study of the "ACS NSQIP®" applied retrospectively to patients operated on for colorectal cancer in the surgical department of Farhat Hached hospital, during the 2015 and 2019 5-year term. Three levels of adjustment. Discrimination and calibration were carried out mainly by ROC curves (AUC ⩾ 0.8).

RESULTS

In this study, 129 patients were included with a sex ratio of 1.22 and a median age of 62 years. The most common operative procedure was low segmental colectomy with colorectal anastomosis. Thirty-seven patients (28.7%) had at least one postoperative complication. The prediction and cuts-off points values of mortality (AUC = 0.858; CI95% [0.570-0.960]; Cuts-off points = 1.8%), cardiac complications (AUC = 0.824; CI95% [0.658-0.990]; Cuts-off points = 1.8%), thromboembolic complications (AUC = 0.802; CI95% [0.617-0.987]; Cuts-off point = 3.1%), and renal insufficiency (AUC = 0.802; CI95% [ 0.623-0.981]; Cuts-off point = 1.2%) were adjusted according to level 1 of the calculator.

CONCLUSION

This work contextualized the prediction of postoperative complications in colorectal surgery in the university general surgery department of Farhat Hached in Sousse (Tunisia), making it possible to improve the quality and safety of surgical care. The application of the Tunisian mini calculator is recommended as well as the generalization of validation following the development of a generic calculator for all operating procedures.

Accuracy of cognitive screening instruments reconsidered: overall, balanced or unbiased accuracy?

Aim: To examine three different accuracy metrics for evaluation of cognitive screening instruments: overall correct classification accuracy (Acc), the sum of true positives and negatives divided by the total number tested; balanced accuracy (balanced Acc), half of the sum of sensitivity and specificity; and unbiased accuracy (unbiased Acc), removing biasing effects of random associations between test results and disease prevalence. Materials & methods: Data from a prospective test accuracy study of Mini-Addenbrooke's Cognitive Examination were used to calculate and plot the Acc measures. Results: Each Acc metric resulted in a similar pattern of results across the range of Mini-Addenbrooke's Cognitive Examination cut-offs for diagnosis of both dementia and mild cognitive impairment. Acc and balanced Acc gave more optimistic outcomes (closer to possible maximum value of 1) than unbiased Acc. Conclusion: Unbiased Acc may have advantages over Acc and balanced Acc by removing biasing effects of random associations between test result and disease prevalence.

Receiver operating characteristic plot and area under the curve with binary classifiers: pragmatic analysis of cognitive screening instruments

Aim: To examine whether receiver operating characteristic plots and area under the curve (AUC) values may be potentially misleading when assessing cognitive screening instruments as binary predictors rather than as categorical or continuous scales. Materials & methods: AUC was calculated using different methods (rank-sum, diagnostic odds ratio) using data from test accuracy studies of two binary classifiers of cognitive status (applause sign, attended with sign), a screener producing categorical data (Codex), and a continuous scale screening test (Mini-Addenbrooke's Cognitive Examination). Results: For all screeners, AUC calculated using diagnostic odds ratio method was greater than using rank-sum method. When Codex and Mini-Addenbrooke's Cognitive Examination were analyzed as binary (single fixed threshold) tests, AUC using rank-sum method was lower than when screeners were analyzed as categorical or continuous scales, respectively. Conclusion: If cognitive screeners producing categorical or continuous measures are dichotomized, calculated AUC may be an underestimate, thus affecting screening test accuracy.