Multi-stage thresholded region classification for whole-body PET-CT lymphoma studies

Multimodal Spatial Attention Module for Targeting Multimodal PET-CT Lung Tumor Segmentation

Multimodal positron emission tomography-computed tomography (PET-CT) is used routinely in the assessment of cancer. PET-CT combines the high sensitivity for tumor detection of PET and anatomical information from CT. Tumor segmentation is a critical element of PET-CT but at present, the performance of existing automated methods for this challenging task is low. Segmentation tends to be done manually by different imaging experts, which is labor-intensive and prone to errors and inconsistency. Previous automated segmentation methods largely focused on fusing information that is extracted separately from the PET and CT modalities, with the underlying assumption that each modality contains complementary information. However, these methods do not fully exploit the high PET tumor sensitivity that can guide the segmentation. We introduce a deep learning-based framework in multimodal PET-CT segmentation with a multimodal spatial attention module (MSAM). The MSAM automatically learns to emphasize regions (spatial areas) related to tumors and suppress normal regions with physiologic high-uptake from the PET input. The resulting spatial attention maps are subsequently employed to target a convolutional neural network (CNN) backbone for segmentation of areas with higher tumor likelihood from the CT image. Our experimental results on two clinical PET-CT datasets of non-small cell lung cancer (NSCLC) and soft tissue sarcoma (STS) validate the effectiveness of our framework in these different cancer types. We show that our MSAM, with a conventional U-Net backbone, surpasses the state-of-the-art lung tumor segmentation approach by a margin of 7.6% in Dice similarity coefficient (DSC).

Combining Superpixels and Deep Learning Approaches to Segment Active Organs in Metastatic Breast Cancer PET Images

Semi-automatic measurements are performed on ¹⁸FDG PET-CT images to monitor the evolution of metastatic sites in the clinical follow-up of metastatic breast cancer patients. Apart from being time-consuming and prone to subjective approximation, semi-automatic tools cannot make the difference between cancerous regions and active organs, presenting a high ¹⁸FDG uptake.In this work, we combine a deep learning-based approach with a superpixel segmentation method to segment the main active organs (brain, heart, bladder) from full-body PET images. In particular, we integrate a superpixel SLIC algorithm at different levels of a convolutional network. Results are compared with a deep learning segmentation network alone. The methods are cross-validated on full-body PET images of 36 patients and tested on the acquisitions of 24 patients from a different study center, in the context of the ongoing EPICUREseinmeta study. The similarity between the manually defined organ masks and the results is evaluated with the Dice score. Moreover, the amount of false positives is evaluated through the positive predictive value (PPV).According to the computed Dice scores, all approaches allow to accurately segment the target organs. However, the networks integrating superpixels are better suited to transfer knowledge across datasets acquired on multiple sites (domain adaptation) and are less likely to segment structures outside of the target organs, according to the PPV.Hence, combining deep learning with superpixels allows to segment organs presenting a high ¹⁸FDG uptake on PET images without selecting cancerous lesion, and thus improves the precision of the semi-automatic tools monitoring the evolution of breast cancer metastasis.Clinical relevance- We demonstrate the utility of combining deep learning and superpixel segmentation methods to accurately find the contours of active organs from metastatic breast cancer images, to different dataset distributions.

Convolutional Neural Networks for Automated PET/CT Detection of Diseased Lymph Node Burden in Patients with Lymphoma

PURPOSE

To automatically detect lymph nodes involved in lymphoma on fluorine 18 (¹⁸F) fluorodeoxyglucose (FDG) PET/CT images using convolutional neural networks (CNNs).

MATERIALS AND METHODS

In this retrospective study, baseline disease of 90 patients with lymphoma was segmented on ¹⁸F-FDG PET/CT images (acquired between 2005 and 2011) by a nuclear medicine physician. An ensemble of three-dimensional patch-based, multiresolution pathway CNNs was trained using fivefold cross-validation. Performance was assessed using the true-positive rate (TPR) and number of false-positive (FP) findings. CNN performance was compared with agreement between physicians by comparing the annotations of a second nuclear medicine physician to the first reader in 20 of the patients. Patient TPR was compared using Wilcoxon signed rank tests.

RESULTS

Across all 90 patients, a range of 0-61 nodes per patient was detected. At an average of four FP findings per patient, the method achieved a TPR of 85% (923 of 1087 nodes). Performance varied widely across patients (TPR range, 33%-100%; FP range, 0-21 findings). In the 20 patients labeled by both physicians, a range of 1-49 nodes per patient was detected and labeled. The second reader identified 96% (210 of 219) of nodes with an additional 3.7 per patient compared with the first reader. In the same 20 patients, the CNN achieved a 90% (197 of 219) TPR at 3.7 FP findings per patient.

CONCLUSION

An ensemble of three-dimensional CNNs detected lymph nodes at a performance nearly comparable to differences between two physicians' annotations. This preliminary study is a first step toward automated PET/CT assessment for lymphoma.© RSNA, 2020.

Co-Learning Feature Fusion Maps from PET-CT Images of Lung Cancer

The analysis of multi-modality positron emission tomography and computed tomography (PET-CT) images for computer aided diagnosis applications (e.g., detection and segmentation) requires combining the sensitivity of PET to detect abnormal regions with anatomical localization from CT. Current methods for PET-CT image analysis either process the modalities separately or fuse information from each modality based on knowledge about the image analysis task. These methods generally do not consider the spatially varying visual characteristics that encode different information across the different modalities, which have different priorities at different locations. For example, a high abnormal PET uptake in the lungs is more meaningful for tumor detection than physiological PET uptake in the heart. Our aim is to improve fusion of the complementary information in multi-modality PET-CT with a new supervised convolutional neural network (CNN) that learns to fuse complementary information for multi-modality medical image analysis. Our CNN first encodes modality-specific features and then uses them to derive a spatially varying fusion map that quantifies the relative importance of each modality's features across different spatial locations. These fusion maps are then multiplied with the modality-specific feature maps to obtain a representation of the complementary multi-modality information at different locations, which can then be used for image analysis. We evaluated the ability of our CNN to detect and segment multiple regions (lungs, mediastinum, tumors) with different fusion requirements using a dataset of PET-CT images of lung cancer. We compared our method to baseline techniques for multi-modality image fusion (fused inputs (FS), multi-branch (MB) techniques, and multichannel (MC) techniques) and segmentation. Our findings show that our CNN had a significantly higher foreground detection accuracy (99.29%, p < 0:05) than the fusion baselines (FS: 99.00%, MB: 99.08%, TC: 98.92%) and a significantly higher Dice score (63.85%) than recent PET-CT tumor segmentation methods.

Body region localization in whole-body low-dose CT images of PET/CT scans using virtual landmarks

PURPOSE

Radiological imaging and image interpretation for clinical decision making are mostly specific to each body region such as head and neck, thorax, abdomen, pelvis, and extremities. In this study, we present a new solution to trim automatically the given axial image stack into image volumes satisfying the given body region definition.

METHODS

The proposed approach consists of the following steps. First, a set of reference objects is selected and roughly segmented. Virtual landmarks (VLs) for the objects are then identified by using principal component analysis and recursive subdivision of the object via the principal axes system. The VLs can be defined based on just the binary objects or objects with gray values also considered. The VLs may lie anywhere with respect to the object, inside or outside, and rarely on the object surface, and are tethered to the object. Second, a classic neural network regressor is configured to learn the geometric mapping relationship between the VLs and the boundary locations of each body region. The trained network is then used to predict the locations of the body region boundaries. In this study, we focus on three body regions - thorax, abdomen, and pelvis, and predict their superior and inferior axial locations denoted by TS(I), TI(I), AS(I), AI(I), PS(I), and PI(I), respectively, for any given volume image I. Two kinds of reference objects - the skeleton and the lungs and airways, are employed to test the localization performance of the proposed approach.

RESULTS

Our method is tested by using low-dose unenhanced computed tomography (CT) images of 180 near whole-body ¹⁸ F-fluorodeoxyglucose-positron emission tomography/computed tomography (FDG-PET/CT) scans (including 34 whole-body scans) which are randomly divided into training and testing sets with a ratio of 85%:15%. The procedure is repeated six times and three times for the case of lungs and skeleton, respectively, with different divisions of the entire data set at this proportion. For the case of using skeleton as a reference object, the overall mean localization error for the six locations expressed as number of slices (nS) and distance (dS) in mm, is found to be nS: 3.4, 4.7, 4.1, 5.2, 5.2, and 3.9; dS: 13.4, 18.9, 16.5, 20.8, 20.8, and 15.5 mm for binary objects; nS: 4.1, 5.7, 4.3, 5.9, 5.9, and 4.0; dS: 16.2, 22.7, 17.2, 23.7, 23.7, and 16.1 mm for gray objects, respectively. For the case of using lungs and airways as a reference object, the corresponding results are, nS: 4.0, 5.3, 4.1, 6.9, 6.9, and 7.4; dS: 15.0, 19.7, 15.3, 26.2, 26.2, and 27.9 mm for binary objects; nS: 3.9, 5.4, 3.6, 7.2, 7.2, and 7.6; dS: 14.6, 20.1, 13.7, 27.3, 27.3, and 28.6 mm for gray objects, respectively.

CONCLUSIONS

Precise body region identification automatically in whole-body or body region tomographic images is vital for numerous medical image analysis and analytics applications. Despite its importance, this issue has received very little attention in the literature. We present a solution to this problem in this study using the concept of virtual landmarks. The method achieves localization accuracy within 2-3 slices, which is roughly comparable to the variation found in localization by experts. As long as the reference objects can be roughly segmented, the method with its learned VLs-to-boundary location relationship and predictive ability is transferable from one image modality to another.