Feature-Based Representation Improves Color Decomposition and Nuclear Detection Using a Convolutional Neural Network

Multi-Aperture Transformers for 3D (MAT3D) Segmentation of Clinical and Microscopic Images

3D segmentation of biological structures is critical in biomedical imaging, offering significant insights into structures and functions. This paper introduces a novel segmentation of biological images that couples Multi-Aperture representation with Transformers for 3D (MAT3D) segmentation. Our method integrates the global context-awareness of Transformer networks with the local feature extraction capabilities of Convolutional Neural Networks (CNNs), providing a comprehensive solution for accurately delineating complex biological structures. First, we evaluated the performance of the proposed technique on two public clinical datasets of ACDC and Synapse multi-organ segmentation, rendering superior Dice scores of 93.34±0.05 and 89.73±0.04, respectively, with fewer parameters compared to the published literature. Next, we assessed the performance of our technique on an organoid dataset comprising four breast cancer subtypes. The proposed method achieved a Dice 95.12±0.02 and a PQ score of 97.01±0.01, respectively. MAT3D also significantly reduces the parameters to 40 million. The code is available on https://github.com/sohaibcs1/MAT3D.

3D-Organoid-SwinNet: High-Content Profiling of 3D Organoids

Profiling of Patient-Derived organoids is necessary for drug screening and precision medicine. This step requires accurate segmentation of three-dimensional cellular structures followed by protein readouts. While fully Convolutional Neural Networks are widely used in medical image segmentation, they struggle to capture long-range dependencies necessary for accurate segmentation. On the other hand, transformer models have shown promise in capturing long-range dependencies and self-similarities. Motivated by this, we present 3D-Organoid-SwinNet, a unique segmentation model explicitly designed for organoid semantic segmentation. We evaluated the performance of our technique using an Organoid dataset from four breast cancer subtypes. We demonstrated consistent top-tier performance in both the validation and testing phases, achieving a Dice score of 94.91 while reducing the number of parameters to 21 million. Our findings indicate that the proposed model offers a foundation for transformer-based models designed for high-content profiling of organoid models.

Small Blob Detector Using Bi-Threshold Constrained Adaptive Scales

Recent advances in medical imaging technology bring great promises for medicine practices. Imaging biomarkers are discovered to inform disease diagnosis, prognosis, and treatment assessment. Detecting and segmenting objects from images are often the first steps in quantitative measurement of these biomarkers. The challenges of detecting objects in images, particularly small objects known as blobs, include low image resolution, image noise and overlap among the blobs. This research proposes a Bi-Threshold Constrained Adaptive Scale (BTCAS) blob detector to uncover the relationship between the U-Net threshold and the Difference of Gaussian (DoG) scale to derive a multi-threshold, multi-scale small blob detector. With lower and upper bounds on the probability thresholds from U-Net, two binarized maps of the distance are rendered between blob centers. Each blob is transformed to a DoG space with an adaptively identified local optimum scale. A Hessian convexity map is rendered using the adaptive scale, and the under-segmentation typical of the U-Net is resolved. To validate the performance of the proposed BTCAS, a 3D simulated dataset (n = 20) of blobs, a 3D MRI dataset of human kidneys and a 3D MRI dataset of mouse kidneys, are studied. BTCAS is compared against four state-of-the-art methods: HDoG, U-Net with standard thresholding, U-Net with optimal thresholding, and UH-DoG using precision, recall, F-score, Dice and IoU. We conclude that BTCAS statistically outperforms the compared detectors.

Developing image analysis pipelines of whole-slide images: Pre- and post-processing

Deep learning has pushed the scope of digital pathology beyond simple digitization and telemedicine. The incorporation of these algorithms in routine workflow is on the horizon and maybe a disruptive technology, reducing processing time, and increasing detection of anomalies. While the newest computational methods enjoy much of the press, incorporating deep learning into standard laboratory workflow requires many more steps than simply training and testing a model. Image analysis using deep learning methods often requires substantial pre- and post-processing order to improve interpretation and prediction. Similar to any data processing pipeline, images must be prepared for modeling and the resultant predictions need further processing for interpretation. Examples include artifact detection, color normalization, image subsampling or tiling, removal of errant predictions, etc. Once processed, predictions are complicated by image file size - typically several gigabytes when unpacked. This forces images to be tiled, meaning that a series of subsamples from the whole-slide image (WSI) are used in modeling. Herein, we review many of these methods as they pertain to the analysis of biopsy slides and discuss the multitude of unique issues that are part of the analysis of very large images.

Towards pixel-to-pixel deep nucleus detection in microscopy images

BACKGROUND

Nucleus is a fundamental task in microscopy image analysis and supports many other quantitative studies such as object counting, segmentation, tracking, etc. Deep neural networks are emerging as a powerful tool for biomedical image computing; in particular, convolutional neural networks have been widely applied to nucleus/cell detection in microscopy images. However, almost all models are tailored for specific datasets and their applicability to other microscopy image data remains unknown. Some existing studies casually learn and evaluate deep neural networks on multiple microscopy datasets, but there are still several critical, open questions to be addressed.

RESULTS

We analyze the applicability of deep models specifically for nucleus detection across a wide variety of microscopy image data. More specifically, we present a fully convolutional network-based regression model and extensively evaluate it on large-scale digital pathology and microscopy image datasets, which consist of 23 organs (or cancer diseases) and come from multiple institutions. We demonstrate that for a specific target dataset, training with images from the same types of organs might be usually necessary for nucleus detection. Although the images can be visually similar due to the same staining technique and imaging protocol, deep models learned with images from different organs might not deliver desirable results and would require model fine-tuning to be on a par with those trained with target data. We also observe that training with a mixture of target and other/non-target data does not always mean a higher accuracy of nucleus detection, and it might require proper data manipulation during model training to achieve good performance.

CONCLUSIONS

We conduct a systematic case study on deep models for nucleus detection in a wide variety of microscopy images, aiming to address several important but previously understudied questions. We present and extensively evaluate an end-to-end, pixel-to-pixel fully convolutional regression network and report a few significant findings, some of which might have not been reported in previous studies. The model performance analysis and observations would be helpful to nucleus detection in microscopy images.

Fusion of encoder-decoder deep networks improves delineation of multiple nuclear phenotypes

BACKGROUND

Nuclear segmentation is an important step for profiling aberrant regions of histology sections. If nuclear segmentation can be resolved, then new biomarkers of nuclear phenotypes and their organization can be predicted for the application of precision medicine. However, segmentation is a complex problem as a result of variations in nuclear geometry (e.g., size, shape), nuclear type (e.g., epithelial, fibroblast), nuclear phenotypes (e.g., vesicular, aneuploidy), and overlapping nuclei. The problem is further complicated as a result of variations in sample preparation (e.g., fixation, staining). Our hypothesis is that (i) deep learning techniques can learn complex phenotypic signatures that rise in tumor sections, and (ii) fusion of different representations (e.g., regions, boundaries) contributes to improved nuclear segmentation.

RESULTS

We have demonstrated that training of deep encoder-decoder convolutional networks overcomes complexities associated with multiple nuclear phenotypes, where we evaluate alternative architecture of deep learning for an improved performance against the simplicity of the design. In addition, improved nuclear segmentation is achieved by color decomposition and combining region- and boundary-based features through a fusion network. The trained models have been evaluated against approximately 19,000 manually annotated nuclei, and object-level Precision, Recall, F1-score and Standard Error are reported with the best F1-score being 0.91. Raw training images, annotated images, processed images, and source codes are released as a part of the Additional file 1.

CONCLUSIONS

There are two intrinsic barriers in nuclear segmentation to H&E stained images, which correspond to the diversity of nuclear phenotypes and perceptual boundaries between adjacent cells. We demonstrate that (i) the encoder-decoder architecture can learn complex phenotypes that include the vesicular type; (ii) delineation of overlapping nuclei is enhanced by fusion of region- and edge-based networks; (iii) fusion of ENets produces an improved result over the fusion of UNets; and (iv) fusion of networks is better than multitask learning. We suggest that our protocol enables processing a large cohort of whole slide images for applications in precision medicine.