The majority of 922 prediction models supporting breast cancer decision-making are at high risk of bias

Risk-Prediction Models for Clinical Decision-Making in Sarcoma Care: An International Survey Among Soft-Tissue Sarcoma Clinicians

INTRODUCTION

Risk prediction models (RPMs) are statistical tools that predict outcomes on the basis of clinical characteristics and can thereby support (shared) decision-making. With the shift toward personalized medicine, the number of RPMs has increased exponentially, including in multimodal sarcoma care. However, their integration into routine soft-tissue sarcoma (STS) care remains largely unknown. Therefore, we inventoried RPM use in sarcoma care during tumor board discussions and patient consultations as well as the attitudes toward the use of RPMs to support (shared) decision-making among STS clinicians.

MATERIALS AND METHODS

A 29-item survey was disseminated online to members of international sarcoma societies.

RESULTS

This study enrolled 278 respondents. Respectively, 68% and 65% of the clinicians reported using RPMs during tumor board discussions and/or patient consultations. During tumor board discussions, RPMs were used primarily to assess the potential benefits of (neo)adjuvant chemotherapy. During patient consultations, RPMs were used to predict patient prognosis upon request and to assist in decision-making regarding (neo)adjuvant therapies. The reliability of patient risk predicted by RPMs and the absence of guidelines regarding the use of RPMs were identified as barriers. Additionally, some clinicians questioned the applicability of estimates from RPMs to individual patients and expressed concerns about causing unnecessary anxiety when discussing prognostic outcomes.

CONCLUSIONS

Responding STS clinicians frequently use RPMs to support decision-making about (neo)adjuvant therapies. However, they expressed concerns about the applicability of RPM estimates to individual patients and reported challenges in communicating prognostic outcomes with patients. These findings highlight the difficulties clinicians face when integrating RPMs into patient consultations.

How Outcome Prediction Could Aid Clinical Practice

Predictive algorithms have myriad potential clinical decision-making implications from prognostic counselling to improving clinical trial efficiency. Large observational (or "real world") cohorts are a common data source for the development and evaluation of such tools. There is significant optimism regarding the benefits and use cases for risk-based care, but there is a notable disparity between the volume of clinical prediction models published and implementation into healthcare systems that drive and realise patient benefit. Considering the perspective of a clinician or clinical researcher that may encounter clinical predictive algorithms in the near future as a user or developer, this editorial: (1) discusses the ways in which prediction models built using observational data could inform better clinical decisions; (2) summarises the main steps in producing a model with special focus on key appraisal factors; and (3) highlights recent work driving evolution in the ways that we should conceptualise, build and evaluate these tools.

Bridging the Gap: A Mixed-Methods Study on Factors Influencing Breast Cancer Clinicians' Decisions to Use Clinical Prediction Models

Background. Clinical prediction models provide tailored risk estimates that can help guide decisions in breast cancer care. Despite their potential, few models are widely used in clinical practice. We aimed to identify the factors influencing breast cancer clinicians' decisions to adopt prediction models and assess their relative importance. Methods. We conducted a mixed-methods study, beginning with semi-structured interviews, followed by a nationwide online survey. Thematic analysis was used to qualitatively summarize the interviews and identify key factors. For the survey, we used descriptive analysis to characterize the sample and Mann-Whitney U and Kruskal-Wallis tests to explore differences in score (0 = not important to 10 = very important) distributions. Results. Interviews (N = 16) identified eight key factors influencing model use. Practical/methodological factors included accessibility, cost, understandability, objective accuracy, actionability, and clinical relevance. Perceptual factors included acceptability, subjective accuracy, and risk communication. In the survey (N = 146; 137 model users), clinicians ranked online accessibility (median score = 9 [interquartile range = 8-10]) as most important. Cost was also highly rated, with preferences for freely available models (9 [8-10]) and those with reimbursable tests (8 [8-10]). Formal regulatory approval (7 [5-8]) and direct integration with electronic health records (6 [3-8]) were considered less critical. Subgroup analysis revealed differences in score distributions; for example, clinicians from general hospitals prioritized inclusion of new biomarkers more than those in academic settings. Conclusions. Breast cancer clinicians' decisions to initiate use of prediction models are influenced by practical and perceptual factors, extending beyond technical metrics such as discrimination and calibration. Addressing these factors more holistically through collaborative efforts between model developers, clinicians, and communication and implementation experts, for instance, by developing clinician-friendly online tools that prioritize usability and local adaptability, could increase model uptake.

Highlights

Accessibility, cost, and practical considerations, such as ease of use and clinical utility, were prioritized slightly more than technical validation metrics, such as discrimination and calibration, when deciding to start using a clinical prediction model.Most breast cancer clinicians valued models with clear inputs (e.g., variable definitions, cutoffs) and outputs; few were interested in the exact model specifications.Perceptual or subjective factors, including perceived accuracy and peer acceptability, also influenced model adoption but were secondary to practical considerations.Sociodemographic variables, such as clinical specialization and hospital setting, influenced the importance of factors for model use.

Cost-effective prognostic evaluation of breast cancer: using a STAR nomogram model based on routine blood tests

Background

Breast cancer (BC) is the most common and prominent deadly disease among women. Predicting BC survival mainly relies on TNM staging, molecular profiling and imaging, hampered by subjectivity and expenses. This study aimed to establish an economical and reliable model using the most common preoperative routine blood tests (RT) data for survival and surveillance strategy management.

Methods

We examined 2863 BC patients, dividing them into training and validation cohorts (7:3). We collected demographic features, pathomics characteristics and preoperative 24-item RT data. BC risk factors were identified through Cox regression, and a predictive nomogram was established. Its performance was assessed using C-index, area under curves (AUC), calibration curve and decision curve analysis. Kaplan-Meier curves stratified patients into different risk groups. We further compared the STAR model (utilizing HE and RT methodologies) with alternative nomograms grounded in molecular profiling (employing second-generation short-read sequencing methodologies) and imaging (utilizing PET-CT methodologies).

Results

The STAR nomogram, incorporating subtype, TNM stage, age and preoperative RT data (LYM, LYM%, EOSO%, RDW-SD, P-LCR), achieved a C-index of 0.828 in the training cohort and impressive AUCs (0.847, 0.823 and 0.780) for 3-, 5- and 7-year OS rates, outperforming other nomograms. The validation cohort showed similar impressive results. The nomogram calculates a patient's total score by assigning values to each risk factor, higher scores indicating a poor prognosis. STAR promises potential cost savings by enabling less intensive surveillance in around 90% of BC patients. Compared to nomograms based on molecular profiling and imaging, STAR presents a more cost-effective, with potential savings of approximately $700-800 per breast cancer patient.

Conclusion

Combining appropriate RT parameters, STAR nomogram could help in the detection of patient anemia, coagulation function, inflammation and immune status. Practical implementation of the STAR nomogram in a clinical setting is feasible, and its potential clinical impact lies in its ability to provide an early, economical and reliable tool for survival prediction and surveillance strategy management. However, our model still has limitations and requires external data validation. In subsequent studies, we plan to mitigate the potential impact on model robustness by further updating and adjusting the data and model.

Improving Prediction of Complications Post-Proton Therapy in Lung Cancer Using Large Language Models and Meta-Analysis

PURPOSE

This study enhances the efficiency of predicting complications in lung cancer patients receiving proton therapy by utilizing large language models (LLMs) and meta-analytical techniques for literature quality assessment.

MATERIALS AND METHODS

We integrated systematic reviews with LLM evaluations, sourcing studies from Web of Science, PubMed, and Scopus, managed via EndNote X20. Inclusion and exclusion criteria ensured literature relevance. Techniques included meta-analysis, heterogeneity assessment using Cochran's Q test and I2 statistics, and subgroup analyses for different complications. Quality and bias risk were assessed using the PROBAST tool and further analyzed with models such as ChatGPT-4, Llama2-13b, and Llama3-8b. Evaluation metrics included AUC, accuracy, precision, recall, F1 score, and time efficiency (WPM).

RESULTS

The meta-analysis revealed an overall effect size of 0.78 for model predictions, with high heterogeneity observed (I2 = 72.88%, P < 0.001). Subgroup analysis for radiation-induced esophagitis and pneumonitis revealed predictive effect sizes of 0.79 and 0.77, respectively, with a heterogeneity index (I2) of 0%, indicating that there were no significant differences among the models in predicting these specific complications. A literature assessment using LLMs demonstrated that ChatGPT-4 achieved the highest accuracy at 90%, significantly outperforming the Llama3 and Llama2 models, which had accuracies ranging from 44% to 62%. Additionally, LLM evaluations were conducted 3229 times faster than manual assessments were, markedly enhancing both efficiency and accuracy. The risk assessment results identified nine studies as high risk, three as low risk, and one as unknown, confirming the robustness of the ChatGPT-4 across various evaluation metrics.

CONCLUSION

This study demonstrated that the integration of large language models with meta-analysis techniques can significantly increase the efficiency of literature evaluations and reduce the time required for assessments, confirming that there are no significant differences among models in predicting post proton therapy complications in lung cancer patients.

Development and internal-external validation of statistical and machine learning models for breast cancer prognostication: cohort study

OBJECTIVE

To develop a clinically useful model that estimates the 10 year risk of breast cancer related mortality in women (self-reported female sex) with breast cancer of any stage, comparing results from regression and machine learning approaches.

DESIGN

Population based cohort study.

SETTING

QResearch primary care database in England, with individual level linkage to the national cancer registry, Hospital Episodes Statistics, and national mortality registers.

PARTICIPANTS

141 765 women aged 20 years and older with a diagnosis of invasive breast cancer between 1 January 2000 and 31 December 2020.

MAIN OUTCOME MEASURES

Four model building strategies comprising two regression (Cox proportional hazards and competing risks regression) and two machine learning (XGBoost and an artificial neural network) approaches. Internal-external cross validation was used for model evaluation. Random effects meta-analysis that pooled estimates of discrimination and calibration metrics, calibration plots, and decision curve analysis were used to assess model performance, transportability, and clinical utility.

RESULTS

During a median 4.16 years (interquartile range 1.76-8.26) of follow-up, 21 688 breast cancer related deaths and 11 454 deaths from other causes occurred. Restricting to 10 years maximum follow-up from breast cancer diagnosis, 20 367 breast cancer related deaths occurred during a total of 688 564.81 person years. The crude breast cancer mortality rate was 295.79 per 10 000 person years (95% confidence interval 291.75 to 299.88). Predictors varied for each regression model, but both Cox and competing risks models included age at diagnosis, body mass index, smoking status, route to diagnosis, hormone receptor status, cancer stage, and grade of breast cancer. The Cox model's random effects meta-analysis pooled estimate for Harrell's C index was the highest of any model at 0.858 (95% confidence interval 0.853 to 0.864, and 95% prediction interval 0.843 to 0.873). It appeared acceptably calibrated on calibration plots. The competing risks regression model had good discrimination: pooled Harrell's C index 0.849 (0.839 to 0.859, and 0.821 to 0.876, and evidence of systematic miscalibration on summary metrics was lacking. The machine learning models had acceptable discrimination overall (Harrell's C index: XGBoost 0.821 (0.813 to 0.828, and 0.805 to 0.837); neural network 0.847 (0.835 to 0.858, and 0.816 to 0.878)), but had more complex patterns of miscalibration and more variable regional and stage specific performance. Decision curve analysis suggested that the Cox and competing risks regression models tested may have higher clinical utility than the two machine learning approaches.

CONCLUSION

In women with breast cancer of any stage, using the predictors available in this dataset, regression based methods had better and more consistent performance compared with machine learning approaches and may be worthy of further evaluation for potential clinical use, such as for stratified follow-up.

External validation of 87 clinical prediction models supporting clinical decisions for breast cancer patients

INTRODUCTION

Numerous prediction models have been developed to support treatment-related decisions for breast cancer patients. External validation, a prerequisite for implementation in clinical practice, has been performed for only a few models. This study aims to externally validate published clinical prediction models using population-based Dutch data.

METHODS

Patient-, tumor- and treatment-related data were derived from the Netherlands Cancer Registry (NCR). Model performance was assessed using the area under the receiver operating characteristic curve (AUC), scaled Brier score, and model calibration. Net benefit across applicable risk thresholds was evaluated with decision curve analysis.

RESULTS

After assessing 922 models, 87 (9%) were included for validation. Models were excluded due to an incomplete model description (n = 262 (28%)), lack of required data (n = 521 (57%)), previously validated or developed with NCR data (n = 45 (5%)), or the associated NCR sample size was insufficient (n = 7 (1%)). The included models predicted survival (33 (38%) overall, 27 (31%) breast cancer-specific, and 3 (3%) other cause-specific), locoregional recurrence (n = 7 (8%)), disease free survival (n = 7 (8%)), metastases (n = 5 (6%)), lymph node involvement (n = 3 (3%)), pathologic complete response (n = 1 (1%)), and surgical margins (n = 1 (1%)). Seven models (8%) showed poor (AUC<0.6), 39 (45%) moderate (AUC:0.6-0.7), 38 (46%) good (AUC:0.7-0.9), and 3 (3%) excellent (AUC≥0.9) discrimination. Using the scaled Brier score, worse performance than an uninformative model was found in 34 (39%) models.

CONCLUSION

Comprehensive registry data supports broad validation of published prediction models. Model performance varies considerably in new patient populations, affirming the importance of external validation studies before applying models in clinical practice. Well performing models could be clinically useful in a Dutch setting after careful impact evaluation.