Breaking down the RECIST 1.1 double read variability in lung trials: What do baseline assessments tell us?

A CT-based deep learning-driven tool for automatic liver tumor detection and delineation in patients with cancer

Liver tumors, whether primary or metastatic, significantly impact the outcomes of patients with cancer. Accurate identification and quantification are crucial for effective patient management, including precise diagnosis, prognosis, and therapy evaluation. We present SALSA (system for automatic liver tumor segmentation and detection), a fully automated tool for liver tumor detection and delineation. Developed on 1,598 computed tomography (CT) scans and 4,908 liver tumors, SALSA demonstrates superior accuracy in tumor identification and volume quantification, outperforming state-of-the-art models and inter-reader agreement among expert radiologists. SALSA achieves a patient-wise detection precision of 99.65%, and 81.72% at lesion level, in the external validation cohorts. Additionally, it exhibits good overlap, achieving a dice similarity coefficient (DSC) of 0.760, outperforming both state-of-the-art and the inter-radiologist assessment. SALSA's automatic quantification of tumor volume proves to have prognostic value across various solid tumors (p = 0.028). SALSA's robust capabilities position it as a potential medical device for automatic cancer detection, staging, and response evaluation.

A call for objectivity: Radiologists' proposed wishlist for response evaluation in solid tumors (RECIST 1.1)

The Response Evaluation in Solid Tumors (RECIST) 1.1 provides key guidance for performing imaging response assessment and defines image-based outcome metrics in oncology clinical trials, including progression free survival. In this framework, tumors identified on imaging are designated as either target lesions, non-target disease or new lesions and a structured categorical response is assigned at each imaging time point. While RECIST provides definitions for these categories, it specifically and objectively defines only the target disease. Predefined thresholds of size change provide unbiased metrics for determining objective response and disease progression of the target lesions. However, worsening of non-target disease or emergence of new lesions is given the same importance in determining disease progression despite these being qualitatively assessed and less rigorously defined. The subjective assessment of non-target and new disease contributes to reader variability, which can impact the quality of image interpretation and even the determination of progression free survival. The RECIST Working Group has made significant efforts in developing RECIST 1.1 beyond its initial publication, particularly in its application to targeted agents and immunotherapy. A review of the literature highlights that the Working Group has occasionally employed or adopted objective measures for assessing non-target and new lesions in their evaluation of RECIST-based outcome measures. Perhaps a prospective evaluation of these more objective definitions for non-target and new lesions within the framework of RECIST 1.1 might improve reader interpretation. Ideally, these changes could also better align with clinically meaningful outcome measures of patient survival or quality of life.

Reproducing RECIST lesion selection via machine learning: Insights into intra and inter-radiologist variation

Background

The Response Evaluation Criteria in Solid Tumors (RECIST) aims to provide a standardized approach to assess treatment response in solid tumors. However, discrepancies in the selection of measurable and target lesions among radiologists using these criteria pose a significant limitation to their reproducibility and accuracy. This study aimed to understand the factors contributing to this variability.

Methods

Machine learning models were used to replicate, in parallel, the selection process of measurable and target lesions by two radiologists in a cohort of 40 patients from an internal pan-cancer dataset. The models were trained on lesion characteristics such as size, shape, texture, rank, and proximity to other lesions. Ablation experiments were conducted to evaluate the impact of lesion diameter, volume, and rank on the selection process.

Results

The models successfully reproduced the selection of measurable lesions, relying primarily on size-related features. Similarly, the models reproduced target lesion selection, relying mostly on lesion rank. Beyond these features, the importance placed by different radiologists on different visual characteristics can vary, specifically when choosing target lesions. Worth noting that substantial variability was still observed between radiologists in both measurable and target lesion selection.

Conclusions

Despite the successful replication of lesion selection, our results still revealed significant inter-radiologist disagreement. This underscores the necessity for more precise guidelines to standardize lesion selection processes and minimize reliance on individual interpretation and experience as a means to bridge existing ambiguities.

Can we predict discordant RECIST 1.1 evaluations in double read clinical trials?

Background

In lung clinical trials with imaging, blinded independent central review with double reads is recommended to reduce evaluation bias and the Response Evaluation Criteria In Solid Tumor (RECIST) is still widely used. We retrospectively analyzed the inter-reader discrepancies rate over time, the risk factors for discrepancies related to baseline evaluations, and the potential of machine learning to predict inter-reader discrepancies.

Materials and methods

We retrospectively analyzed five BICR clinical trials for patients on immunotherapy or targeted therapy for lung cancer. Double reads of 1724 patients involving 17 radiologists were performed using RECIST 1.1. We evaluated the rate of discrepancies over time according to four endpoints: progressive disease declared (PDD), date of progressive disease (DOPD), best overall response (BOR), and date of the first response (DOFR). Risk factors associated with discrepancies were analyzed, two predictive models were evaluated.

Results

At the end of trials, the discrepancy rates between trials were not different. On average, the discrepancy rates were 21.0%, 41.0%, 28.8%, and 48.8% for PDD, DOPD, BOR, and DOFR, respectively. Over time, the discrepancy rate was higher for DOFR than DOPD, and the rates increased as the trial progressed, even after accrual was completed. It was rare for readers to not find any disease, for less than 7% of patients, at least one reader selected non-measurable disease only (NTL). Often the readers selected some of their target lesions (TLs) and NTLs in different organs, with ranges of 36.0-57.9% and 60.5-73.5% of patients, respectively. Rarely (4-8.1%) two readers selected all their TLs in different locations. Significant risk factors were different depending on the endpoint and the trial being considered. Prediction had a poor performance but the positive predictive value was higher than 80%. The best classification was obtained with BOR.

Conclusion

Predicting discordance rates necessitates having knowledge of patient accrual, patient survival, and the probability of discordances over time. In lung cancer trials, although risk factors for inter-reader discrepancies are known, they are weakly significant, the ability to predict discrepancies from baseline data is limited. To boost prediction accuracy, it would be necessary to enhance baseline-derived features or create new ones, considering other risk factors and looking into optimal reader associations.