Preliminary Assessment of an Optical Flow Method (OFM) for Nonrigid Registration and Temporal Subtraction (TS) of Serial CT Examinations to Facilitate Evaluation of Interval Change in Metastatic Lung Nodules

Using deep learning-derived image features in radiologic time series to make personalised predictions: proof of concept in colonic transit data

OBJECTIVES

Siamese neural networks (SNN) were used to classify the presence of radiopaque beads as part of a colonic transit time study (CTS). The SNN output was then used as a feature in a time series model to predict progression through a CTS.

METHODS

This retrospective study included all patients undergoing a CTS in a single institution from 2010 to 2020. Data were partitioned in an 80/20 Train/Test split. Deep learning models based on a SNN architecture were trained and tested to classify images according to the presence, absence, and number of radiopaque beads and to output the Euclidean distance between the feature representations of the input images. Time series models were used to predict the total duration of the study.

RESULTS

In total, 568 images of 229 patients (143, 62% female, mean age 57) patients were included. For the classification of the presence of beads, the best performing model (Siamese DenseNET trained with a contrastive loss with unfrozen weights) achieved an accuracy, precision, and recall of 0.988, 0.986, and 1. A Gaussian process regressor (GPR) trained on the outputs of the SNN outperformed both GPR using only the number of beads and basic statistical exponential curve fitting with MAE of 0.9 days compared to 2.3 and 6.3 days (p < 0.05) respectively.

CONCLUSIONS

SNNs perform well at the identification of radiopaque beads in CTS. For time series prediction our methods were superior at identifying progression through the time series compared to statistical models, enabling more accurate personalised predictions.

CLINICAL RELEVANCE STATEMENT

Our radiologic time series model has potential clinical application in use cases where change assessment is critical (e.g. nodule surveillance, cancer treatment response, and screening programmes) by quantifying change and using it to make more personalised predictions.

KEY POINTS

• Time series methods have improved but application to radiology lags behind computer vision. Colonic transit studies are a simple radiologic time series measuring function through serial radiographs. • We successfully employed a Siamese neural network (SNN) to compare between radiographs at different points in time and then used the output of SNN as a feature in a Gaussian process regression model to predict progression through the time series. • This novel use of features derived from a neural network on medical imaging data to predict progression has potential clinical application in more complex use cases where change assessment is critical such as in oncologic imaging, monitoring for treatment response, and screening programmes.

Optical flow estimation of coronary angiography sequences based on semi-supervised learning

Optical flow is widely used in medical image processing, such as image registration, segmentation, 3D reconstruction, and temporal super-resolution. However, high-precision optical flow training datasets for medical images are challenging to produce. The current optical flow estimation models trained on these non-medical datasets, such as KITTI, Sintel, and FlyingChairs are unsuitable for medical images. In this work, we propose a semi-supervised learning mechanism to estimate the optical flow of coronary angiography. Our proposed method only needs the original medical images, segmentation results of regions of interest, and pre-trained models based on other optical flow datasets to train a new optical flow estimation model suitable for medical images. First, we use the coronary segmentation results to perform image enhancement processing on the coronary vascular region to improve the image contrast between the vascular region and the surrounding tissues. Then, we extract the high-precision optical flow of coronary arteries based on the coronary-enhanced images and the pre-trained optical flow estimation model. After estimating the optical flow, we take it and its corresponding original coronary angiography images as the training dataset to train the optical flow estimation network. Furthermore, we generate a large-scale synthetic Flying-artery dataset based on coronary artery segmentation results and original coronary angiography images, which is used to improve and evaluate the accuracy of optical flow estimation for coronary angiography. The experimental results on the coronary angiography datasets demonstrate that our proposed method can significantly improve the optical flow estimation accuracy of coronary angiography sequences compared with other methods.

Optical Flow Methods for Lung Nodule Segmentation on LIDC-IDRI Images

Lung nodule segmentation is an essential step in any CAD system for lung cancer detection and diagnosis. Traditional approaches for image segmentation are mainly morphology based or intensity based. Motion-based segmentation techniques tend to use the temporal information along with the morphology and intensity information to perform segmentation of regions of interest in videos. CT scans comprise of a sequence of dicom 2-D image slices similar to videos which also comprise of a sequence of image frames ordered on a timeline. In this work, Farneback, Horn-Schunck and Lucas-Kanade optical flow methods have been used for processing the dicom slices. The novelty of this work lies in the usage of optical flow methods, generally used in motion-based segmentation tasks, for the segmentation of nodules from CT images. Since thin-sliced CT scans are the imaging modality considered, they closely approximate the motion videos and are the primary motivation for using optical flow for lung nodule segmentation. This paper also provides a detailed comparative analysis and validates the effectiveness of using optical flow methods for segmentation. Finally, we propose methods to further improve the efficiency of segmentation using optical flow methods on CT scans.