Nerve Segmentation with Deep Learning from Label-Free Endoscopic Images Obtained Using Coherent Anti-Stokes Raman Scattering

Rapid azoospermia classification by stimulated Raman scattering and second harmonic generation microscopy

Disease diagnosis and classification pose significant challenges due to the limited capabilities of traditional methods to obtain molecular information with spatial distribution. Optical imaging techniques, utilizing (auto)fluorescence and nonlinear optical signals, introduce new dimensions for biomarkers exploration that can improve diagnosis and classification. Nevertheless, these signals often cover only a limited number of species, impeding a comprehensive assessment of the tissue microenvironment, which is crucial for effective disease diagnosis and therapy. To address this challenge, we developed a multimodal platform, termed stimulated Raman scattering and second harmonic generation microscopy (SRASH), capable of simultaneously providing both chemical bonds and structural information of tissues. Applying SRASH imaging to azoospermia patient samples, we successfully identified lipids, protein, and collagen contrasts, unveiling molecular and structural signatures for non-obstructive azoospermia. This achievement is facilitated by LiteBlendNet-Dx (LBNet-Dx), our diagnostic algorithm, which achieved an outstanding 100% sample-level accuracy in classifying azoospermia, surpassing conventional imaging modalities. As a label-free technique, SRASH imaging eliminates the requirement for sample pre-treatment, demonstrating great potential for clinical translation and enabling molecular imaging-based diagnosis and therapy.

Expectation-maximization algorithm leads to domain adaptation for a perineural invasion and nerve extraction task in whole slide digital pathology images

In addition to lymphatic and vascular channels, tumor cells can also spread via nerves, i.e., perineural invasion (PNI). PNI serves as an independent prognostic indicator in many malignancies. As a result, identifying and determining the extent of PNI is an important yet extremely tedious task in surgical pathology. In this work, we present a computational approach to extract nerves and PNI from whole slide histopathology images. We make manual annotations on selected prostate cancer slides once but then apply the trained model for nerve segmentation to both prostate cancer slides and head and neck cancer slides. For the purpose of multi-domain learning/prediction and investigation on the generalization capability of deep neural network, an expectation-maximization (EM)-based domain adaptation approach is proposed to improve the segmentation performance, in particular for the head and neck cancer slides. Experiments are conducted to demonstrate the segmentation performances. The average Dice coefficient for prostate cancer slides is 0.82 and 0.79 for head and neck cancer slides. Comparisons are then made for segmentations with and without the proposed EM-based domain adaptation on prostate cancer and head and neck cancer whole slide histopathology images from The Cancer Genome Atlas (TCGA) database and significant improvements are observed.

A Framework for Biosensors Assisted by Multiphoton Effects and Machine Learning

The ability to interpret information through automatic sensors is one of the most important pillars of modern technology. In particular, the potential of biosensors has been used to evaluate biological information of living organisms, and to detect danger or predict urgent situations in a battlefield, as in the invasion of SARS-CoV-2 in this era. This work is devoted to describing a panoramic overview of optical biosensors that can be improved by the assistance of nonlinear optics and machine learning methods. Optical biosensors have demonstrated their effectiveness in detecting a diverse range of viruses. Specifically, the SARS-CoV-2 virus has generated disturbance all over the world, and biosensors have emerged as a key for providing an analysis based on physical and chemical phenomena. In this perspective, we highlight how multiphoton interactions can be responsible for an enhancement in sensibility exhibited by biosensors. The nonlinear optical effects open up a series of options to expand the applications of optical biosensors. Nonlinearities together with computer tools are suitable for the identification of complex low-dimensional agents. Machine learning methods can approximate functions to reveal patterns in the detection of dynamic objects in the human body and determine viruses, harmful entities, or strange kinetics in cells.

DXM-TransFuse U-net: Dual cross-modal transformer fusion U-net for automated nerve identification

Accurate nerve identification is critical during surgical procedures to prevent damage to nerve tissues. Nerve injury can cause long-term adverse effects for patients, as well as financial overburden. Birefringence imaging is a noninvasive technique derived from polarized images that have successfully identified nerves that can assist during intraoperative surgery. Furthermore, birefringence images can be processed under 20 ms with a GPGPU implementation, making it a viable image modality option for real-time processing. In this study, we first comprehensively investigate the usage of birefringence images combined with deep learning, which can automatically detect nerves with gains upwards of 14% over its color image-based (RGB) counterparts on the F2 score. Additionally, we develop a deep learning network framework using the U-Net architecture with a Transformer based fusion module at the bottleneck that leverages both birefringence and RGB modalities. The dual-modality framework achieves 76.12 on the F2 score, a gain of 19.6 % over single-modality networks using only RGB images. By leveraging and extracting the feature maps of each modality independently and using each modality's information for cross-modal interactions, we aim to provide a solution that would further increase the effectiveness of imaging systems for enabling noninvasive intraoperative nerve identification.

Deep-Learning-Based Cerebral Artery Semantic Segmentation in Neurosurgical Operating Microscope Vision Using Indocyanine Green Fluorescence Videoangiography

There have been few anatomical structure segmentation studies using deep learning. Numbers of training and ground truth images applied were small and the accuracies of which were low or inconsistent. For a surgical video anatomy analysis, various obstacles, including a variable fast-changing view, large deformations, occlusions, low illumination, and inadequate focus occur. In addition, it is difficult and costly to obtain a large and accurate dataset on operational video anatomical structures, including arteries. In this study, we investigated cerebral artery segmentation using an automatic ground-truth generation method. Indocyanine green (ICG) fluorescence intraoperative cerebral videoangiography was used to create a ground-truth dataset mainly for cerebral arteries and partly for cerebral blood vessels, including veins. Four different neural network models were trained using the dataset and compared. Before augmentation, 35,975 training images and 11,266 validation images were used. After augmentation, 260,499 training and 90,129 validation images were used. A Dice score of 79% for cerebral artery segmentation was achieved using the DeepLabv3+ model trained using an automatically generated dataset. Strict validation in different patient groups was conducted. Arteries were also discerned from the veins using the ICG videoangiography phase. We achieved fair accuracy, which demonstrated the appropriateness of the methodology. This study proved the feasibility of operating field view of the cerebral artery segmentation using deep learning, and the effectiveness of the automatic blood vessel ground truth generation method using ICG fluorescence videoangiography. Using this method, computer vision can discern blood vessels and arteries from veins in a neurosurgical microscope field of view. Thus, this technique is essential for neurosurgical field vessel anatomy-based navigation. In addition, surgical assistance, safety, and autonomous surgery neurorobotics that can detect or manipulate cerebral vessels would require computer vision to identify blood vessels and arteries.