Publications poorly report the essential RadiOmics ParametERs (PROPER): A meta-research on quality of reporting

Tumor grade-titude: XGBoost radiomics paves the way for RCC classification

This study aimed to develop and evaluate a non-invasive XGBoost-based machine learning model using radiomic features extracted from pre-treatment CT images to differentiate grade 4 renal cell carcinoma (RCC) from lower-grade tumours. A total of 102 RCC patients who underwent contrast-enhanced CT scans were included in the analysis. Radiomic features were extracted, and a two-step feature selection methodology was applied to identify the most relevant features for classification. The XGBoost model demonstrated high performance in both training (AUC = 0.87) and testing (AUC = 0.92) sets, with no significant difference between the two (p = 0.521). The model also exhibited high sensitivity, specificity, positive predictive value, and negative predictive value. The selected radiomic features captured both the distribution of intensity values and spatial relationships, which may provide valuable insights for personalized treatment decision-making. Our findings suggest that the XGBoost model has the potential to be integrated into clinical workflows to facilitate personalized adjuvant immunotherapy decision-making, ultimately improving patient outcomes. Further research is needed to validate the model in larger, multicentre cohorts and explore the potential of combining radiomic features with other clinical and molecular data.

CLEAR guideline for radiomics: Early insights into current reporting practices endorsed by EuSoMII

PURPOSE

This study aims to evaluate current reporting practices in radiomics research, with a focus on CheckList for EvaluAtion of Radiomics research (CLEAR).

METHODS

We conducted a citation search using Google Scholar to collect original research articles on radiomics citing the CLEAR guideline up to June 17, 2024. We examined the adoption of the guideline, adherence scores per publication, item-wise adherence rates, and self-reporting practices. An expert panel from the European Society of Medical Imaging Informatics Radiomics Auditing Group conducted a detailed item-by-item confirmation analysis of the self-reported CLEAR checklists.

RESULTS

Out of 100 unique citations from 104 records, 48 original research papers on radiomics were included. The overall adoption rate in the literature was 2 %. Among the citing articles, 94 % (45/48) adopted CLEAR for reporting purposes, applying it to both hand-crafted radiomics (89 %) and deep learning (24 %). Self-reported checklists were included in 58 % (26/45) of these papers. Median study-wise adherence score for self-reported data was 91 % (interquartile range = 18 %). Mean confirmed adherence score was 66 % (standard deviation = 14 %). Difference between these scores was statistically significant, (mean = 21 %; standard deviation = 11 %), p < 0.001. Using an arbitrary 50 % adherence cut-off, the number of items with poor adherence increased from 3 to 15 after confirmation analysis, mostly comprised of open science-related items. In addition, several items were frequently misreported.

CONCLUSION

This study revealed significant discrepancies between self-reported and confirmed adherence to the CLEAR guideline in radiomics research, indicating a need for improved reporting accuracy and verification practices.

Self-reported checklists and quality scoring tools in radiomics: a meta-research

OBJECTIVE

To evaluate the use of reporting checklists and quality scoring tools for self-reporting purposes in radiomics literature.

METHODS

Literature search was conducted in PubMed (date, April 23, 2023). The radiomics literature was sampled at random after a sample size calculation with a priori power analysis. A systematic assessment for self-reporting, including the use of documentation such as completed checklists or quality scoring tools, was conducted in original research papers. These eligible papers underwent independent evaluation by a panel of nine readers, with three readers assigned to each paper. Automatic annotation was used to assist in this process. Then, a detailed item-by-item confirmation analysis was carried out on papers with checklist documentation, with independent evaluation of two readers.

RESULTS

The sample size calculation yielded 117 papers. Most of the included papers were retrospective (94%; 110/117), single-center (68%; 80/117), based on their private data (89%; 104/117), and lacked external validation (79%; 93/117). Only seven papers (6%) had at least one self-reported document (Radiomics Quality Score (RQS), Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD), or Checklist for Artificial Intelligence in Medical Imaging (CLAIM)), with a statistically significant binomial test (p < 0.001). Median rate of confirmed items for all three documents was 81% (interquartile range, 6). For quality scoring tools, documented scores were higher than suggested scores, with a mean difference of - 7.2 (standard deviation, 6.8).

CONCLUSION

Radiomic publications often lack self-reported checklists or quality scoring tools. Even when such documents are provided, it is essential to be cautious, as the accuracy of the reported items or scores may be questionable.

CLINICAL RELEVANCE STATEMENT

Current state of radiomic literature reveals a notable absence of self-reporting with documentation and inaccurate reporting practices. This critical observation may serve as a catalyst for motivating the radiomics community to adopt and utilize such tools appropriately, thereby fostering rigor, transparency, and reproducibility of their research, moving the field forward.

KEY POINTS

• In radiomics literature, there has been a notable absence of self-reporting with documentation. • Even if such documents are provided, it is critical to exercise caution because the accuracy of the reported items or scores may be questionable. • Radiomics community needs to be motivated to adopt and appropriately utilize the reporting checklists and quality scoring tools.

Applying oversampling before cross-validation will lead to high bias in radiomics

Class imbalance is often unavoidable for radiomic data collected from clinical routine. It can create problems during classifier training since the majority class could dominate the minority class. Consequently, resampling methods like oversampling or undersampling are applied to the data to class-balance the data. However, the resampling must not be applied upfront to all data because it would lead to data leakage and, therefore, to erroneous results. This study aims to measure the extent of this bias. Five-fold cross-validation with 30 repeats was performed using a set of 15 radiomic datasets to train predictive models. The training involved two scenarios: first, the models were trained correctly by applying the resampling methods during the cross-validation. Second, the models were trained incorrectly by performing the resampling on all the data before cross-validation. The bias was defined empirically as the difference between the best-performing models in both scenarios in terms of area under the receiver operating characteristic curve (AUC), sensitivity, specificity, balanced accuracy, and the Brier score. In addition, a simulation study was performed on a randomly generated dataset for verification. The results demonstrated that incorrectly applying the oversampling methods to all data resulted in a large positive bias (up to 0.34 in AUC, 0.33 in sensitivity, 0.31 in specificity, and 0.37 in balanced accuracy). The bias depended on the data balance, and approximately an increase of 0.10 in the AUC was observed for each increase in imbalance. The models also showed a bias in calibration measured using the Brier score, which differed by up to -0.18 between the correctly and incorrectly trained models. The undersampling methods were not affected significantly by bias. These results emphasize that any resampling method should be applied correctly only to the training data to avoid data leakage and, subsequently, biased model performance and calibration.