A critical appraisal of the clinical applicability and risk of bias of the predictive models for mortality and recurrence in patients with oropharyngeal cancer: Systematic review

Individualized Dynamic Prediction Model for Patient-Reported Voice Quality in Early-Stage Glottic Cancer

OBJECTIVE

Early-stage glottic cancer (ESGC) is a malignancy of the head and neck. Besides disease control, preservation and improvement of voice quality are essential. To enable expectation management and well-informed decision-making, patients should be sufficiently counseled with individualized information on expected voice quality. This study aims to develop an individualized dynamic prediction model for patient-reported voice quality. This model should be able to provide individualized predictions at every time point from intake to the end of follow-up.

STUDY DESIGN

Longitudinal cohort study.

SETTING

Tertiary cancer center.

METHODS

Patients treated for ESGC were included in this study (N = 294). The Voice Handicap Index was obtained prospectively. The framework of mixed and joint models was used. The prognostic factors used are treatment, age, gender, comorbidity, performance score, smoking, T-stage, and involvement of the anterior commissure. The overall performance of these models was assessed during an internal cross-validation procedure and presentation of absolute errors using box plots.

RESULTS

The mean age in this cohort was 67 years and 81.3% are male. Patients were treated with transoral CO2 laser microsurgery (57.8%), single vocal cord irradiation up to (24.5), or local radiotherapy (17.5%). The mean follow-up was 43.4 months (SD 21.5). Including more measurements during prediction improves predictive performance. Including more clinical and demographic variables did not provide better predictions. Little differences in predictive performance between models were found.

CONCLUSION

We developed a dynamic individualized prediction model for patient-reported voice quality. This model has the potential to empower patients and professionals in making well-informed decisions and enables tailor-made counseling.

Predictive value of machine learning for breast cancer recurrence: a systematic review and meta-analysis

PURPOSE

Recurrence of breast cancer leads to a high lifetime risk and a low 5 year survival rate. Researchers have used machine learning to predict the risk of recurrence in patients with breast cancer, but the predictive performance of machine learning remains controversial. Hence, this study aimed to explore the accuracy of machine learning in predicting breast cancer recurrence risk and aggregate predictive variables to provide guidance for the development of subsequent risk scoring systems.

METHODS

We searched Pubmed, EMBASE, Cochrane, and Web of Science. The risk of bias in the included studies was evaluated using prediction model risk of bias assessment tool (PROBAST). Meta-regression was adopted to explore whether there was a significant difference in the recurrence time by machine learning.

RESULTS

Thirty-four studies involving 67,560 subjects were included, among whom 8695 experienced breast cancer recurrence. The c-index of prediction models was 0.814 (95%CI 0.802-0.826) and 0.770 (95%CI 0.737-0.803) in the training and validation sets, respectively; the sensitivity and specificity were 0.69 (95% CI 0.64-0.74), 0.89 (95% CI 0.86-0.92) in the training, and 0.64 (95% CI 0.58-0.70), 0.88 (95% CI 0.82-0.92) in the validation, respectively. Age, histological grading, and lymph node status are the most commonly used variables in model construction. Attention should be paid to unhealthy lifestyles such as drinking, smoking and BMI as modeling variables. Risk prediction models based on machine learning have long-term monitoring value for breast cancer population, and subsequent studies should consider using large-sample and multi-center data to establish risk equations for verification.

CONCLUSION

Machine learning may be used as a predictive tool for breast cancer recurrence. Currently, there is a lack of effective and universally applicable machine learning models in clinical practice. We expect to incorporate multi-center studies in the future and attempt to develop tools for predicting breast cancer recurrence risk, so as to effectively identify populations at high risk of recurrence and develop personalized follow-up strategies and prognostic interventions to reduce the risk of recurrence.

Outcome prediction models incorporating clinical variables for Head and Neck Squamous cell Carcinoma: A systematic review of methodological conduct and risk of bias

Multiple outcome prediction models have been developed for Head and Neck Squamous Cell Carcinoma (HNSCC). This systematic review aimed to identify HNSCC outcome prediction model studies, assess their methodological quality and identify those with potential utility for clinical practice. Inclusion criteria were mucosal HNSCC prognostic prediction model studies (development or validation) incorporating clinically available variables accessible at time of treatment decision making and predicting tumour-related outcomes. Eligible publications were identified from PubMed and Embase. Methodological quality and risk of bias were assessed using the checklist for critical appraisal and data extraction for systematic reviews of prediction modelling studies (CHARMS) and prediction model risk of bias assessment tool (PROBAST). Eligible publications were categorised by study type for reporting. 64 eligible publications were identified; 55 reported model development, 37 external validations, with 28 reporting both. CHARMS checklist items relating to participants, predictors, outcomes, handling of missing data, and some model development and evaluation procedures were generally well-reported. Less well-reported were measures accounting for model overfitting and model performance measures, especially model calibration. Full model information was poorly reported (3/55 model developments), specifically model intercept, baseline survival or full model code. Most publications (54/55 model developments, 28/37 external validations) were found to have high risk of bias, predominantly due to methodological issues in the PROBAST analysis domain. The identified methodological issues may affect prediction model accuracy in heterogeneous populations. Independent external validation studies in the local population and demonstration of clinical impact are essential for the clinical implementation of outcome prediction models.

Mortality prediction models after radical cystectomy for bladder tumour: A systematic review and critical appraisal

INTRODUCTION

To identify risk-predictive models for bladder-specific cancer mortality in patients undergoing radical cystectomy and assess their clinical utility and risk of bias.

METHODS

Systematic review (CRD42021224626:PROSPERO) in Medline and EMBASE (from their creation until 31/10/2021) was screened to include articles focused on the development and internal validation of a predictive model of specific cancer mortality in patients undergoing radical cystectomy. CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) and Prediction model Risk Of Bias ASsessment Tool (PROBAST) were applied.

RESULTS

Nineteen observational studies were included. The main predictors were sociodemographic variables, such as age (18 studies, 94.7%) and sex (17, 89.5% studies), tumour characteristics (TNM stage (18 studies, 94.7%), histological subtype/grade (15 studies, 78.9%), lymphovascular invasion (10 studies, 52.6%) and treatment with chemotherapy (13 studies, 68.4%). C-index values were presented in 14 studies. The overall risk of bias assessed using PROBAST led to 100% of studies being classified as high risk (the analysis domain was rated to be at high risk of bias in all the studies), and 52.6% showed low applicability. Only 5 studies (26.3%) included an external validation and 2 (10.5%) included a prospective study design.

CONCLUSIONS

Using clinical predictors to assess the risk of bladder-specific cancer mortality is a feasibility alternative. However, the studies showed a high risk of bias and their applicability is uncertain. Studies should improve the conducting and reporting, and subsequent external validation studies should be developed.

Machine learning for lymph node metastasis prediction of in patients with gastric cancer: A systematic review and meta-analysis

Objective

To evaluate the diagnostic performance of machine learning (ML) in predicting lymph node metastasis (LNM) in patients with gastric cancer (GC) and to identify predictors applicable to the models.

Methods

PubMed, EMBASE, Web of Science, and Cochrane Library were searched from inception to March 16, 2022. The pooled c-index and accuracy were used to assess the diagnostic accuracy. Subgroup analysis was performed based on ML types. Meta-analyses were performed using random-effect models. Risk of bias assessment was conducted using PROBAST tool.

Results

A total of 41 studies (56182 patients) were included, and 33 of the studies divided the participants into a training set and a test set, while the rest of the studies only had a training set. The c-index of ML for LNM prediction in training set and test set was 0.837 [95%CI (0.814, 0.859)] and 0.811 [95%CI (0.785-0.838)], respectively. The pooled accuracy was 0.781 [(95%CI (0.756-0.805)] in training set and 0.753 [95%CI (0.721-0.783)] in test set. Subgroup analysis for different ML algorithms and staging of GC showed no significant difference. In contrast, in the subgroup analysis for predictors, in the training set, the model that included radiomics had better accuracy than the model with only clinical predictors (F = 3.546, p = 0.037). Additionally, cancer size, depth of cancer invasion and histological differentiation were the three most commonly used features in models built for prediction.

Conclusion

ML has shown to be of excellent diagnostic performance in predicting the LNM of GC. One of the models covering radiomics and its ML algorithms showed good accuracy for the risk of LNM in GC. However, the results revealed some methodological limitations in the development process. Future studies should focus on refining and improving existing models to improve the accuracy of LNM prediction.

Systematic Review Registration

https://www.crd.york.ac.uk/PROSPERO/, identifier CRD42022320752

Prediction models for gastric cancer risk in the general population: a systematic review

Risk prediction models for gastric cancer (GC) could identify high-risk individuals in the general population. The objective of this study was to systematically review the available evidence about the construction and verification of GC predictive models. We searched PubMed, Embase, and Cochrane Library databases for articles that developed or validated GC risk prediction models up to November 2021. Data extracted included study characteristics, predictor selection, missing data, and evaluation metrics. Risk of bias (ROB) was assessed using the Prediction model study Risk Of Bias Assessment Tool (PROBAST). We identified a total of 12 original risk prediction models that fulfilled the criteria for analysis. The area under the receiver operating characteristic curve ranged from 0.73 to 0.93 in derivation sets (n=6), 0.68 to 0.90 in internal validation sets (n=5), 0.71 to 0.92 in external validation sets (n=7). The higher-performing models usually include age, salt preference, Helicobacter pylori, smoking, BMI, family history, pepsinogen and sex. According to PROBAST, at least one domain with a high ROB was present in all studies mainly due to methodologic limitations in the analysis domain. In conclusion, although some risk prediction models including similar predictors have displayed sufficient discriminative abilities, many have a high ROB due to methodological limitations and are not externally validated efficiently. Future prediction models should adherence to well-established standards and guidelines to benefit GC screening.

A narrative review of prognosis prediction models for non-small cell lung cancer: what kind of predictors should be selected and how to improve models?

Objective

To discover potential predictors and explore how to build better models by summarizing the existing prognostic prediction models of non-small cell lung cancer (NSCLC).

Background

Research on clinical prediction models of NSCLC has experienced explosive growth in recent years. As more predictors of prognosis are discovered, the choice of predictors to build models is particularly important, and in the background of more applications of next-generation sequencing technology, gene-related predictors are widely used. As it is more convenient to obtain samples and follow-up data, the prognostic model is preferred by researchers.

Methods

PubMed and the Cochrane Library were searched using the items “NSCLC”, “prognostic model”, “prognosis prediction”, and “survival prediction” from 1 January 1980 to 5 May 2021. Reference lists from articles were reviewed and relevant articles were identified.

Conclusions

The performance of gene-related models has not obviously improved. Relative to the innovation and diversity of predictors, it is more important to establish a highly stable model that is convenient for clinical application. Most of the prevalent models are highly biased and referring to PROBAST at the beginning of the study may be able to significantly control the bias. Existing models should be validated in a large external dataset to make a meaningful comparison.

Risk Prediction Models for Inadvertent Intraoperative Hypothermia: A Systematic Review

PURPOSES

Inadvertent intraoperative hypothermia (core temperature <36°C) is a common surgical complication with several adverse events. Hypothermia prediction models can be a tool for providing the healthcare staff with information on the risk of inadvertent hypothermia. Our systematic review aimed to identify, demonstrate, and evaluate the available intraoperative hypothermia risk prediction models in surgical populations.

DESIGN

This study is a systematic review of literature.

METHODS

We systematically searched multiple databases (Ovid MEDLINE, Web of Science, Embase, and Cochrane Center Register of Controlled Trials). Two reviewers independently examined abstracts and the full text for eligibility. Data collection was guided by the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS checklist), and methodological quality and applicability were assessed by the Prediction model Risk Of Bias ASsessment Tool (PROBAST).

FINDINGS

A total of 3,672 references were screened, of which eight articles were included in this study. All the models had a high risk of bias since most of them lacked model validation. Also, they failed to report the model performance and final model presentations, which restricted their clinical application.

CONCLUSIONS

The researchers should present models in a more standard way and improve the existing models to increase their predictive values for clinical application.

A critical appraisal of the prognostic predictive models for patients with sepsis: Which model can be applied in clinical practice?

BACKGROUND

Sepsis is associated with high mortality and predictive models can help in clinical decision-making. The objective of this study was to carry out a systematic review of these models.

METHODS

In 2019, we conducted a systematic review in MEDLINE and EMBASE (CDR42018111121:PROSPERO) of articles that developed predictive models for mortality in septic patients (inclusion criteria). We followed the CHARMS recommendations (Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies), extracting the information from its 11 domains (Source of data, Participants, etc). We determined the risk of bias and applicability (participants, outcome, predictors and analysis) through PROBAST (Prediction model Risk Of Bias ASsessment Tool).

RESULTS

A total of 14 studies were included. In the CHARMS extraction, the models found showed great variability in its 11 domains. Regarding the PROBAST checklist, only one article had an unclear risk of bias as it did not indicate how missing data were handled while the others all had a high risk of bias. This was mainly due to the statistical analysis (inadequate sample size, handling of continuous predictors, missing data and selection of predictors), since 13 studies had a high risk of bias. Applicability was satisfactory in six articles. Most of the models integrate predictors from routine clinical practice. Discrimination and calibration were assessed for almost all the models, with the area under the ROC curve ranging from 0.59 to 0.955 and no lack of calibration. Only three models were externally validated and their maximum discrimination values in the derivation were from 0.712 and 0.84. One of them (Osborn) had undergone multiple validation studies.

DISCUSSION

Despite most of the studies showing a high risk of bias, we very cautiously recommend applying the Osborn model, as this has been externally validated various times.

Prediction Models for Prognosis of Cervical Cancer: Systematic Review and Critical Appraisal

Objective: This work aims to systematically identify, describe, and appraise all prognostic models for cervical cancer and provide a reference for clinical practice and future research. Methods: We systematically searched PubMed, EMBASE, and Cochrane library databases up to December 2020 and included studies developing, validating, or updating a prognostic model for cervical cancer. Two reviewers extracted information based on the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modeling Studies checklist and assessed the risk of bias using the Prediction model Risk Of Bias ASsessment Tool. Results: Fifty-six eligible articles were identified, describing the development of 77 prognostic models and 27 external validation efforts. The 77 prognostic models focused on three types of cervical cancer patients at different stages, i.e., patients with early-stage cervical cancer (n = 29; 38%), patients with locally advanced cervical cancer (n = 27; 35%), and all-stage cervical cancer patients (n = 21; 27%). Among the 77 models, the most frequently used predictors were lymph node status (n = 57; 74%), the International Federation of Gynecology and Obstetrics stage (n = 42; 55%), histological types (n = 38; 49%), and tumor size (n = 37; 48%). The number of models that applied internal validation, presented a full equation, and assessed model calibration was 52 (68%), 16 (21%), and 45 (58%), respectively. Twenty-four models were externally validated, among which three were validated twice. None of the models were assessed with an overall low risk of bias. The Prediction Model of Failure in Locally Advanced Cervical Cancer model was externally validated twice, with acceptable performance, and seemed to be the most reliable. Conclusions: Methodological details including internal validation, sample size, and handling of missing data need to be emphasized on, and external validation is needed to facilitate the application and generalization of models for cervical cancer.

Machine Learning Prediction Models for Gestational Diabetes Mellitus: A meta- analysis (Preprint)

Background

Gestational diabetes mellitus (GDM) is a common endocrine metabolic disease, involving a carbohydrate intolerance of variable severity during pregnancy. The incidence of GDM-related complications and adverse pregnancy outcomes has declined, in part, due to early screening. Machine learning (ML) models are increasingly used to identify risk factors and enable the early prediction of GDM.

Objective

The aim of this study was to perform a meta-analysis and comparison of published prognostic models for predicting the risk of GDM and identify predictors applicable to the models.

Methods

Four reliable electronic databases were searched for studies that developed ML prediction models for GDM in the general population instead of among high-risk groups only. The novel Prediction Model Risk of Bias Assessment Tool (PROBAST) was used to assess the risk of bias of the ML models. The Meta-DiSc software program (version 1.4) was used to perform the meta-analysis and determination of heterogeneity. To limit the influence of heterogeneity, we also performed sensitivity analyses, a meta-regression, and subgroup analysis.

Results

A total of 25 studies that included women older than 18 years without a history of vital disease were analyzed. The pooled area under the receiver operating characteristic curve (AUROC) for ML models predicting GDM was 0.8492; the pooled sensitivity was 0.69 (95% CI 0.68-0.69; P<.001; I²=99.6%) and the pooled specificity was 0.75 (95% CI 0.75-0.75; P<.001; I²=100%). As one of the most commonly employed ML methods, logistic regression achieved an overall pooled AUROC of 0.8151, while non–logistic regression models performed better, with an overall pooled AUROC of 0.8891. Additionally, maternal age, family history of diabetes, BMI, and fasting blood glucose were the four most commonly used features of models established by the various feature selection methods.

Conclusions

Compared to current screening strategies, ML methods are attractive for predicting GDM. To expand their use, the importance of quality assessments and unified diagnostic criteria should be further emphasized.

A general presentation on how to carry out a CHARMS analysis for prognostic multivariate models

The CHARMS (critical appraisal and data extraction for systematic reviews of prediction modelling studies) checklist was created to provide methodological appraisals of predictive models, based on the best available scientific evidence and through systematic reviews. Our purpose is to give a general presentation on how to carry out a CHARMS analysis for prognostic multivariate models, making clear what the steps are and how they are applied individually to the studies included in the systematic review. This tutorial is aimed at providing such a resource. In addition to this explanation, we will apply the method to a real case: predictive models of atrial fibrillation in the community. This methodology could be applied to other predictive models using the steps provided in our review so as to have complete information for each included model and determine whether it can be implemented in daily clinical practice.