Spatio-temporal physics-informed learning: A novel approach to CT perfusion analysis in acute ischemic stroke

CT perfusion imaging is important in the imaging workup of acute ischemic stroke for evaluating affected cerebral tissue. CT perfusion analysis software produces cerebral perfusion maps from commonly noisy spatio-temporal CT perfusion data. High levels of noise can influence the results of CT perfusion analysis, necessitating software tuning. This work proposes a novel approach for CT perfusion analysis that uses physics-informed learning, an optimization framework that is robust to noise. In particular, we propose SPPINN: Spatio-temporal Perfusion Physics-Informed Neural Network and research spatio-temporal physics-informed learning. SPPINN learns implicit neural representations of contrast attenuation in CT perfusion scans using the spatio-temporal coordinates of the data and employs these representations to estimate a continuous representation of the cerebral perfusion parameters. We validate the approach on simulated data to quantify perfusion parameter estimation performance. Furthermore, we apply the method to in-house patient data and the public Ischemic Stroke Lesion Segmentation 2018 benchmark data to assess the correspondence between the perfusion maps and reference standard infarct core segmentations. Our method achieves accurate perfusion parameter estimates even with high noise levels and differentiates healthy tissue from infarcted tissue. Moreover, SPPINN perfusion maps accurately correspond with reference standard infarct core segmentations. Hence, we show that using spatio-temporal physics-informed learning for cerebral perfusion estimation is accurate, even in noisy CT perfusion data. The code for this work is available at https://github.com/lucasdevries/SPPINN.

PC-Reg: A pyramidal prediction-correction approach for large deformation image registration

Deformable image registration plays an important role in medical image analysis. Deep neural networks such as VoxelMorph and TransMorph are fast, but limited to small deformations and face challenges in the presence of large deformations. To tackle large deformations in medical image registration, we propose PC-Reg, a pyramidal Prediction and Correction method for deformable registration, which treats multi-scale registration akin to solving an ordinary differential equation (ODE) across scales. Starting with a zero-initialized deformation at the coarse level, PC-Reg follows the predictor-corrector regime and progressively predicts a residual flow and a correction flow to update the deformation vector field through different scales. The prediction in each scale can be regarded as a single step of ODE integration. PC-Reg can be easily extended to diffeomorphic registration and is able to alleviate the multiscale accumulated upsampling and diffeomorphic integration error. Further, to transfer details from full resolution to low scale, we introduce a distillation loss, where the output is used as the target label for intermediate outputs. Experiments on inter-patient deformable registration show that the proposed method significantly improves registration not only for large but also for small deformations.

PerfU-Net: Baseline infarct estimation from CT perfusion source data for acute ischemic stroke

CT perfusion imaging is commonly used for infarct core quantification in acute ischemic stroke patients. The outcomes and perfusion maps of CT perfusion software, however, show many discrepancies between vendors. We aim to perform infarct core segmentation directly from CT perfusion source data using machine learning, excluding the need to use the perfusion maps from standard CT perfusion software. To this end, we present a symmetry-aware spatio-temporal segmentation model that encodes the micro-perfusion dynamics in the brain, while decoding a static segmentation map for infarct core assessment. Our proposed spatio-temporal PerfU-Net employs an attention module on the skip-connections to match the dimensions of the encoder and decoder. We train and evaluate the method on 94 and 62 scans, respectively, using the Ischemic Stroke Lesion Segmentation (ISLES) 2018 challenge data. We achieve state-of-the-art results compared to methods that only use CT perfusion source imaging with a Dice score of 0.46. We are almost on par with methods that use perfusion maps from third party software, whilst it is known that there is a large variation in these perfusion maps from various vendors. Moreover, we achieve improved performance compared to simple perfusion map analysis, which is used in clinical practice.

Abstract PD11-03: Deep learning applied on resection specimen tissue slides of ‘pure’ ductal carcinoma in situ predicts ipsilateral invasive breast cancer recurrence

Clinical problem. Ductal Carcinoma in Situ (DCIS) is a potential precursor for invasive breast cancer (IBC). Therefore, DCIS is currently treated with surgical excision, often supplemented with radiotherapy to prevent progression to ipsilateral IBC (iIBC). However, many DCIS lesions will never do so. Estimating the risk of progression is a grand challenge, as neither the histopathological grade of the DCIS lesion nor other biological markers are conclusively associated with the disease outcome. Aim. We aimed to develop a deep-learning based pipeline for estimating the risk of iIBC recurrence following DCIS using a dataset of 235 H&E-stained whole-slide images (WSIs) of the primary DCIS lesions with corresponding 10-year follow up metadata of DCIS recurrence collected at the Netherlands Cancer Institute. The patients included in our dataset did not receive radiotherapy and experienced recurrence in 167 of the cases. Results. We developed a two-step pipeline that is able to find predictive features for 10-year iIBC prediction. First, tissue regions on WSIs were divided into equally sized tiles. Mammary ducts were detected in the tiles using a RetinaNet with ResNet101 backbone that was implemented in Detectron2 and pre-trained on ImageNet. Selecting only tiles containing ducts served to reduce the input dimensionality of typically giga-pixel WSIs for the second step. Here, DCIS recurrence was predicted by a weakly-supervised multi-instance learning (MIL) classification model where the label of the WSI was determined by average weighting of duct labels. The performance of this model was enhanced by pre-training it with SimCLR, a self-supervised learning method, on image data from the histopathology domain. Our proposed model achieved an AUC of .93 ± .005, with a sensitivity of .83 ± .27 and a specificity of .85± .09. These results show that the model was able to correctly distinguish patients with low subsequent IBC-risk from those with a substantially higher risk. An active research pursuit of our group is now to develop a model which is able to predict iIBC progression for patients treated with radiotherapy. This poses additional challenges to the model, as the effect of radiotherapy, as well as disease outcome must be predicted. Conclusion and impact. Our method opens up an avenue for identifying biologically relevant features for estimating DCIS progression risk into invasive breast cancer. This knowledge may be used for appropriately choosing a personalized DCIS management option for patients - which may be active surveillance rather than surgical removal of the lesion. JW and JT were equal senior co-authors on this project. This work was supported by Cancer Research UK and by KWF Dutch Cancer Society (ref.C38317/A24043)

Citation Format: Shannon Doyle, Francesco Dal Canton, Timo Koostra, Maartje van Seijen, Emilie Groen, Efstratios Gavves, Hugo Horlings, Esther Lips, Jelle Wesseling, Jonas Teuwen, Grand Challenge PRECISION Consortium. Deep learning applied on resection specimen tissue slides of ‘pure’ ductal carcinoma in situ predicts ipsilateral invasive breast cancer recurrence [abstract]. In: Proceedings of the 2021 San Antonio Breast Cancer Symposium; 2021 Dec 7-10; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2022;82(4 Suppl):Abstract nr PD11-03.

Domain- and task-specific transfer learning for medical segmentation tasks

BACKGROUND AND OBJECTIVES

Transfer learning is a valuable approach to perform medical image segmentation in settings with limited cases available for training convolutional neural networks (CNN). Both the source task and the source domain influence transfer learning performance on a given target medical image segmentation task. This study aims to assess transfer learning-based medical segmentation task performance for various source task and domain combinations.

METHODS

CNNs were pre-trained on classification, segmentation, and self-supervised tasks on two domains: natural images and T1 brain MRI. Next, these CNNs were fine-tuned on three target T1 brain MRI segmentation tasks: stroke lesion, MS lesions, and brain anatomy segmentation. In all experiments, the CNN architecture and transfer learning strategy were the same. The segmentation accuracy on all target tasks was evaluated using the mIOU or Dice coefficients. The detection accuracy was evaluated for the stroke and MS lesion target tasks only.

RESULTS

CNNs pre-trained on a segmentation task on the same domain as the target tasks resulted in higher or similar segmentation accuracy compared to other source task and domain combinations. Pre-training a CNN on ImageNet resulted in a comparable, but not consistently higher lesion detection rate, despite the amount of training data used being 10 times larger.

CONCLUSIONS

This study suggests that optimal transfer learning for medical segmentation is achieved with a similar task and domain for pre-training. As a result, CNNs can be effectively pre-trained on smaller datasets by selecting a source domain and task similar to the target domain and task.

Automated Final Lesion Segmentation in Posterior Circulation Acute Ischemic Stroke Using Deep Learning

Final lesion volume (FLV) is a surrogate outcome measure in anterior circulation stroke (ACS). In posterior circulation stroke (PCS), this relation is plausibly understudied due to a lack of methods that automatically quantify FLV. The applicability of deep learning approaches to PCS is limited due to its lower incidence compared to ACS. We evaluated strategies to develop a convolutional neural network (CNN) for PCS lesion segmentation by using image data from both ACS and PCS patients. We included follow-up non-contrast computed tomography scans of 1018 patients with ACS and 107 patients with PCS. To assess whether an ACS lesion segmentation generalizes to PCS, a CNN was trained on ACS data (ACS-CNN). Second, to evaluate the performance of only including PCS patients, a CNN was trained on PCS data. Third, to evaluate the performance when combining the datasets, a CNN was trained on both datasets. Finally, to evaluate the performance of transfer learning, the ACS-CNN was fine-tuned using PCS patients. The transfer learning strategy outperformed the other strategies in volume agreement with an intra-class correlation of 0.88 (95% CI: 0.83-0.92) vs. 0.55 to 0.83 and a lesion detection rate of 87% vs. 41-77 for the other strategies. Hence, transfer learning improved the FLV quantification and detection rate of PCS lesions compared to the other strategies.

Unsharp Mask Guided Filtering

The goal of this paper is guided image filtering, which emphasizes the importance of structure transfer during filtering by means of an additional guidance image. Where classical guided filters transfer structures using hand-designed functions, recent guided filters have been considerably advanced through parametric learning of deep networks. The state-of-the-art leverages deep networks to estimate the two core coefficients of the guided filter. In this work, we posit that simultaneously estimating both coefficients is suboptimal, resulting in halo artifacts and structure inconsistencies. Inspired by unsharp masking, a classical technique for edge enhancement that requires only a single coefficient, we propose a new and simplified formulation of the guided filter. Our formulation enjoys a filtering prior from a low-pass filter and enables explicit structure transfer by estimating a single coefficient. Based on our proposed formulation, we introduce a successive guided filtering network, which provides multiple filtering results from a single network, allowing for a trade-off between accuracy and efficiency. Extensive ablations, comparisons and analysis show the effectiveness and efficiency of our formulation and network, resulting in state-of-the-art results across filtering tasks like upsampling, denoising, and cross-modality filtering. Code is available at https://github.com/shizenglin/Unsharp-Mask-Guided-Filtering.

Automatic Triage of 12-Lead ECGs Using Deep Convolutional Neural Networks

BACKGROUND

The correct interpretation of the ECG is pivotal for the accurate diagnosis of many cardiac abnormalities, and conventional computerized interpretation has not been able to reach physician‐level accuracy in detecting (acute) cardiac abnormalities. This study aims to develop and validate a deep neural network for comprehensive automated ECG triage in daily practice.

METHODS AND RESULTS

We developed a 37‐layer convolutional residual deep neural network on a data set of free‐text physician‐annotated 12‐lead ECGs. The deep neural network was trained on a data set with 336.835 recordings from 142.040 patients and validated on an independent validation data set (n=984), annotated by a panel of 5 cardiologists electrophysiologists. The 12‐lead ECGs were acquired in all noncardiology departments of the University Medical Center Utrecht. The algorithm learned to classify these ECGs into the following 4 triage categories: normal, abnormal not acute, subacute, and acute. Discriminative performance is presented with overall and category‐specific concordance statistics, polytomous discrimination indexes, sensitivities, specificities, and positive and negative predictive values. The patients in the validation data set had a mean age of 60.4 years and 54.3% were men. The deep neural network showed excellent overall discrimination with an overall concordance statistic of 0.93 (95% CI, 0.92–0.95) and a polytomous discriminatory index of 0.83 (95% CI, 0.79–0.87).

CONCLUSIONS

This study demonstrates that an end‐to‐end deep neural network can be accurately trained on unstructured free‐text physician annotations and used to consistently triage 12‐lead ECGs. When further fine‐tuned with other clinical outcomes and externally validated in clinical practice, the demonstrated deep learning–based ECG interpretation can potentially improve time to treatment and decrease healthcare burden.

Action Recognition with Dynamic Image Networks

We introduce the concept of dynamic image, a novel compact representation of videos useful for video analysis, particularly in combination with convolutional neural networks (CNNs). A dynamic image encodes temporal data such as RGB or optical flow videos by using the concept of 'rank pooling'. The idea is to learn a ranking machine that captures the temporal evolution of the data and to use the parameters of the latter as a representation. We call the resulting representation dynamic image because it summarizes the video dynamics in addition to appearance. This powerful idea allows to convert any video to an image so that existing CNN models pre-trained with still images can be immediately extended to videos. We also present an efficient approximate rank pooling operator that runs two orders of magnitude faster than the standard ones with any loss in ranking performance and can be formulated as a CNN layer. To demonstrate the power of the representation, we introduce a novel four stream CNN architecture which can learn from RGB and optical flow frames as well as from their dynamic image representations. We show that the proposed network achieves state-of-the-art performance, 95.5 and 72.5 percent accuracy, in the UCF101 and HMDB51, respectively.

Reflectance and Natural Illumination from Single-Material Specular Objects Using Deep Learning

In this paper, we present a method that estimates reflectance and illumination information from a single image depicting a single-material specular object from a given class under natural illumination. We follow a data-driven, learning-based approach trained on a very large dataset, but in contrast to earlier work we do not assume one or more components (shape, reflectance, or illumination) to be known. We propose a two-step approach, where we first estimate the object's reflectance map, and then further decompose it into reflectance and illumination. For the first step, we introduce a Convolutional Neural Network (CNN) that directly predicts a reflectance map from the input image itself, as well as an indirect scheme that uses additional supervision, first estimating surface orientation and afterwards inferring the reflectance map using a learning-based sparse data interpolation technique. For the second step, we suggest a CNN architecture to reconstruct both Phong reflectance parameters and high-resolution spherical illumination maps from the reflectance map. We also propose new datasets to train these CNNs. We demonstrate the effectiveness of our approach for both steps by extensive quantitative and qualitative evaluation in both synthetic and real data as well as through numerous applications, that show improvements over the state-of-the-art.

Rank Pooling for Action Recognition

We propose a function-based temporal pooling method that captures the latent structure of the video sequence data - e.g., how frame-level features evolve over time in a video. We show how the parameters of a function that has been fit to the video data can serve as a robust new video representation. As a specific example, we learn a pooling function via ranking machines. By learning to rank the frame-level features of a video in chronological order, we obtain a new representation that captures the video-wide temporal dynamics of a video, suitable for action recognition. Other than ranking functions, we explore different parametric models that could also explain the temporal changes in videos. The proposed functional pooling methods, and rank pooling in particular, is easy to interpret and implement, fast to compute and effective in recognizing a wide variety of actions. We evaluate our method on various benchmarks for generic action, fine-grained action and gesture recognition. Results show that rank pooling brings an absolute improvement of 7-10 average pooling baseline. At the same time, rank pooling is compatible with and complementary to several appearance and local motion based methods and features, such as improved trajectories and deep learning features.