Federated learning enables big data for rare cancer boundary detection

Although machine learning (ML) has shown promise across disciplines, out-of-sample generalizability is concerning. This is currently addressed by sharing multi-site data, but such centralization is challenging/infeasible to scale due to various limitations. Federated ML (FL) provides an alternative paradigm for accurate and generalizable ML, by only sharing numerical model updates. Here we present the largest FL study to-date, involving data from 71 sites across 6 continents, to generate an automatic tumor boundary detector for the rare disease of glioblastoma, reporting the largest such dataset in the literature (n = 6, 314). We demonstrate a 33% delineation improvement for the surgically targetable tumor, and 23% for the complete tumor extent, over a publicly trained model. We anticipate our study to: 1) enable more healthcare studies informed by large diverse data, ensuring meaningful results for rare diseases and underrepresented populations, 2) facilitate further analyses for glioblastoma by releasing our consensus model, and 3) demonstrate the FL effectiveness at such scale and task-complexity as a paradigm shift for multi-site collaborations, alleviating the need for data-sharing.

The federated tumor segmentation (FeTS) tool: an open-source solution to further solid tumor research

Objective.De-centralized data analysis becomes an increasingly preferred option in the healthcare domain, as it alleviates the need for sharing primary patient data across collaborating institutions. This highlights the need for consistent harmonized data curation, pre-processing, and identification of regions of interest based on uniform criteria.Approach.Towards this end, this manuscript describes theFederatedTumorSegmentation (FeTS) tool, in terms of software architecture and functionality.Main results.The primary aim of the FeTS tool is to facilitate this harmonized processing and the generation of gold standard reference labels for tumor sub-compartments on brain magnetic resonance imaging, and further enable federated training of a tumor sub-compartment delineation model across numerous sites distributed across the globe, without the need to share patient data.Significance.Building upon existing open-source tools such as the Insight Toolkit and Qt, the FeTS tool is designed to enable training deep learning models targeting tumor delineation in either centralized or federated settings. The target audience of the FeTS tool is primarily the computational researcher interested in developing federated learning models, and interested in joining a global federation towards this effort. The tool is open sourced athttps://github.com/FETS-AI/Front-End.

Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data

Several studies underscore the potential of deep learning in identifying complex patterns, leading to diagnostic and prognostic biomarkers. Identifying sufficiently large and diverse datasets, required for training, is a significant challenge in medicine and can rarely be found in individual institutions. Multi-institutional collaborations based on centrally-shared patient data face privacy and ownership challenges. Federated learning is a novel paradigm for data-private multi-institutional collaborations, where model-learning leverages all available data without sharing data between institutions, by distributing the model-training to the data-owners and aggregating their results. We show that federated learning among 10 institutions results in models reaching 99% of the model quality achieved with centralized data, and evaluate generalizability on data from institutions outside the federation. We further investigate the effects of data distribution across collaborating institutions on model quality and learning patterns, indicating that increased access to data through data private multi-institutional collaborations can benefit model quality more than the errors introduced by the collaborative method. Finally, we compare with other collaborative-learning approaches demonstrating the superiority of federated learning, and discuss practical implementation considerations. Clinical adoption of federated learning is expected to lead to models trained on datasets of unprecedented size, hence have a catalytic impact towards precision/personalized medicine.

Systematic Evaluation of Image Tiling Adverse Effects on Deep Learning Semantic Segmentation

Convolutional neural network (CNN) models obtain state of the art performance on image classification, localization, and segmentation tasks. Limitations in computer hardware, most notably memory size in deep learning accelerator cards, prevent relatively large images, such as those from medical and satellite imaging, from being processed as a whole in their original resolution. A fully convolutional topology, such as U-Net, is typically trained on down-sampled images and inferred on images of their original size and resolution, by simply dividing the larger image into smaller (typically overlapping) tiles, making predictions on these tiles, and stitching them back together as the prediction for the whole image. In this study, we show that this tiling technique combined with translationally-invariant nature of CNNs causes small, but relevant differences during inference that can be detrimental in the performance of the model. Here we quantify these variations in both medical (i.e., BraTS) and non-medical (i.e., satellite) images and show that training a 2D U-Net model on the whole image substantially improves the overall model performance. Finally, we compare 2D and 3D semantic segmentation models to show that providing CNN models with a wider context of the image in all three dimensions leads to more accurate and consistent predictions. Our results suggest that tiling the input to CNN models-while perhaps necessary to overcome the memory limitations in computer hardware-may lead to undesirable and unpredictable errors in the model's output that can only be adequately mitigated by increasing the input of the model to the largest possible tile size.

NIMG-59. ADVERSE EFFECTS OF IMAGE TILING FOR AUTOMATIC DEEP LEARNING GLIOMA SEGMENTATION IN MRI

BACKGROUND

Application of deep learning to neuro-oncology has shown promising clinically relevant results for tumor classification, localization, and segmentation. Hardware limitations, typically memory size of graphics cards, prevent magnetic resonance imaging (MRI) volumes from being processed as a whole, and hence they are divided into smaller, overlapping tiles. Deep learning algorithms (e.g., U-Net) can then be trained and applied for predictions on such tiles, followed by their combination/stitching as the final prediction for the whole volume. We investigate the hypothesis that image tiling options, such as tile placing, size, overlap, and stitching, introduce variations with adverse effects on predictions, both in terms of inconsistency and accuracy.

METHODS

We utilized the publicly available BraTS 2018 dataset of 285 baseline pre-operative MRI glioma scans, with corresponding expert tumor boundary annotations. We implemented a 3D U-Net to predict boundaries of the whole tumor extent, by virtue of the abnormal hyper-intense signal of T2-FLAIR scans.

RESULTS

Simply flipping the tile horizontally, or translating it by one voxel, produces different predictions. Use of small tiles (64x64x64 voxels) yields substantially more false positive predictions than when using larger tile size (i.e., 128x128x128 voxels). Overlapping tiles produce conflicting predictions, leading to ambiguous interpretations upon their stitching. In areas of overlapping tiles, rounding followed by averaging the overlapping predictions produce superior results to the inverse sequence. All these are particularly noticeable in the margins of the abnormal signal and in areas of large contrast variation.

CONCLUSIONS

Although tiling is a workaround for hardware limitations, it introduces variations detrimental to accuracy. Tiling of neuro-oncology scans for computational analysis using deep learning leads to non-generalizable, non-reproducible results, thereby affecting the performance and potential clinical translatability of such algorithms. Careful considerations and standardization recommendations should be established and appropriately documented for performing such analyses, in order to avoid misinterpretation of results.

Differential Representation of Perception and Action in the Frontal Cortex

A motor illusion was created to separate human subjects' perception of arm movement from their actual movement during figure drawing. Trajectories constructed from cortical activity recorded in monkeys performing the same task showed that the actual movement was represented in the primary motor cortex, whereas the visualized, presumably perceived, trajectories were found in the ventral premotor cortex. Perception and action representations can be differentially recognized in the brain and may be contained in separate structures.

Eye-hand coupling during closed-loop drawing: evidence of shared motor planning?

Previous paradigms have used reaching movements to study coupling of eye-hand kinematics. In the present study, we investigated eye-hand kinematics as curved trajectories were drawn at normal speeds. Eye and hand movements were tracked as a monkey traced ellipses and circles with the hand in free space while viewing the hand's position on a computer monitor. The results demonstrate that the movement of the hand was smooth and obeyed the 2/3 power law. Eye position, however, was restricted to 2-3 clusters along the hand's trajectory and fixed approximately 80% of the time in one of these clusters. The eye remained stationary as the hand moved away from the fixation for up to 200 ms and saccaded ahead of the hand position to the next fixation along the trajectory. The movement from one fixation cluster to another consistently occurred just after the tangential hand velocity had reached a local minimum, but before the next segment of the hand's trajectory began. The next fixation point was close to an area of high curvature along the hand's trajectory even though the hand had not reached that point along the path. A visuo-motor illusion of hand movement demonstrated that the eye movement was influenced by hand movement and not simply by visual input. During the task, neural activity of pre-motor cortex (area F4) was recorded using extracellular electrodes and used to construct a population vector of the hand's trajectory. The results suggest that the saccade onset is correlated in time with maximum curvature in the population vector trajectory for the hand movement. We hypothesize that eye and arm movements may have common, or shared, information in forming their motor plans.

On the relationship between joint angular velocity and motor cortical discharge during reaching

Single-unit activity in area M1 was recorded in awake, behaving monkeys during a three-dimensional (3D) reaching task performed in a virtual reality environment. This study compares motor cortical discharge rate to both the hand's velocity and the arm's joint angular velocities. Hand velocity is considered a parameter of extrinsic space because it is measured in the Cartesian coordinate system of the monkey's workspace. Joint angular velocity is considered a parameter of intrinsic space because it is measured relative to adjacent arm/body segments. In the initial analysis, velocity was measured as the difference in hand position or joint posture between the beginning and ending of the reach. Cortical discharge rate was taken as the mean activity between these two times. This discharge rate was compared through a regression analysis to either an extrinsic-coordinate model based on the three components of hand velocity or to an intrinsic-coordinate model based on seven joint angular velocities. The model showed that velocities about four degrees-of-freedom (elbow flexion/extension, shoulder flexion/extension, shoulder internal/external rotation, and shoulder adduction/abduction) were those best represented in the sampled population of recorded activity. Patterns of activity recorded across the cortical population at each point in time throughout the task were used in a second analysis to predict the temporal profiles of joint angular velocity and hand velocity. The population of cortical units from area M1 matched the hand velocity and three of the four major joint angular velocities. However, shoulder adduction/abduction could not be predicted even though individual cells showed good correlation to movement on this axis. This was also the only major degree-of-freedom not well correlated to hand velocity, suggesting that the other apparent relations between joint angular velocity and neuronal activity may be due to intrinsic-extrinsic correlations inherent in reaching movements.