Fluorescence Image Histology Pattern Transformation Using Image Style Transfer

Deep learning in neurosurgery: a systematic literature review with a structured analysis of applications across subspecialties

Objective

This study systematically reviewed deep learning (DL) applications in neurosurgical practice to provide a comprehensive understanding of DL in neurosurgery. The review process included a systematic overview of recent developments in DL technologies, an examination of the existing literature on their applications in neurosurgery, and insights into the future of neurosurgery. The study also summarized the most widely used DL algorithms, their specific applications in neurosurgical practice, their limitations, and future directions.

Materials and methods

An advanced search using medical subject heading terms was conducted in Medline (via PubMed), Scopus, and Embase databases restricted to articles published in English. Two independent neurosurgically experienced reviewers screened selected articles.

Results

A total of 456 articles were initially retrieved. After screening, 162 were found eligible and included in the study. Reference lists of all 162 articles were checked, and 19 additional articles were found eligible and included in the study. The 181 included articles were divided into 6 categories according to the subspecialties: general neurosurgery (n = 64), neuro-oncology (n = 49), functional neurosurgery (n = 32), vascular neurosurgery (n = 17), neurotrauma (n = 9), and spine and peripheral nerve (n = 10). The leading procedures in which DL algorithms were most commonly used were deep brain stimulation and subthalamic and thalamic nuclei localization (n = 24) in the functional neurosurgery group; segmentation, identification, classification, and diagnosis of brain tumors (n = 29) in the neuro-oncology group; and neuronavigation and image-guided neurosurgery (n = 13) in the general neurosurgery group. Apart from various video and image datasets, computed tomography, magnetic resonance imaging, and ultrasonography were the most frequently used datasets to train DL algorithms in all groups overall (n = 79). Although there were few studies involving DL applications in neurosurgery in 2016, research interest began to increase in 2019 and has continued to grow in the 2020s.

Conclusion

DL algorithms can enhance neurosurgical practice by improving surgical workflows, real-time monitoring, diagnostic accuracy, outcome prediction, volumetric assessment, and neurosurgical education. However, their integration into neurosurgical practice involves challenges and limitations. Future studies should focus on refining DL models with a wide variety of datasets, developing effective implementation techniques, and assessing their affect on time and cost efficiency.

Confocal Laser Endomicroscopy: Enhancing Intraoperative Decision Making in Neurosurgery

Brain tumors, both primary and metastatic, represent a significant global health burden due to their high incidence, mortality, and the severe neurological deficits they frequently cause. Gliomas, especially high-grade gliomas (HGGs), rank among the most aggressive and lethal neoplasms, with only modest gains in long-term survival despite extensive molecular research and established standard therapies. In neurosurgical practice, maximizing the extent of safe resection is a principal strategy for improving clinical outcomes. Yet, the infiltrative nature of gliomas often complicates the accurate delineation of tumor margins. Confocal laser endomicroscopy (CLE), originally introduced in gastroenterology, has recently gained prominence in neuro-oncology by enabling real-time, high-resolution cellular imaging during surgery. This technique allows for intraoperative tumor characterization and reduces dependence on time-consuming frozen-section analyses. Recent technological advances, including device miniaturization and second-generation CLE systems, have substantially improved image quality and diagnostic utility. Furthermore, integration with deep learning algorithms and telepathology platforms fosters automated image interpretation and remote expert consultations, thereby accelerating surgical decision making and enhancing diagnostic consistency. Future work should address remaining challenges, such as mitigating motion artifacts, refining training protocols, and broadening the range of applicable fluorescent probes, to solidify CLE's role as a critical intraoperative adjunct in neurosurgical oncology.

Intraoperative in vivo confocal endomicroscopy of the glioma margin: performance assessment of image interpretation by neurosurgeon users

Objectives

Confocal laser endomicroscopy (CLE) is an intraoperative real-time cellular resolution imaging technology that images brain tumor histoarchitecture. Previously, we demonstrated that CLE images may be interpreted by neuropathologists to determine the presence of tumor infiltration at glioma margins. In this study, we assessed neurosurgeons' ability to interpret CLE images from glioma margins and compared their assessments to those of neuropathologists.

Methods

In vivo CLE images acquired at the glioma margins that were previously reviewed by CLE-experienced neuropathologists were interpreted by four CLE-experienced neurosurgeons. A numerical scoring system from 0 to 5 and a dichotomous scoring system based on pathological features were used. Scores from assessments of hematoxylin and eosin (H&E)-stained sections and CLE images by neuropathologists from a previous study were used for comparison. Neurosurgeons' scores were compared to the H&E findings. The inter-rater agreement and diagnostic performance based on neurosurgeons' scores were calculated. The concordance between dichotomous and numerical scores was determined.

Results

In all, 4275 images from 56 glioma margin regions of interest (ROIs) were included in the analysis. With the numerical scoring system, the inter-rater agreement for neurosurgeons interpreting CLE images was moderate for all ROIs (mean agreement, 61%), which was significantly better than the inter-rater agreement for the neuropathologists (mean agreement, 48%) (p < 0.01). The inter-rater agreement for neurosurgeons using the dichotomous scoring system was 83%. The concordance between the numerical and dichotomous scoring systems was 93%. The overall sensitivity, specificity, positive predictive value, and negative predictive value were 78%, 32%, 62%, and 50%, respectively, using the numerical scoring system and 80%, 27%, 61%, and 48%, respectively, using the dichotomous scoring system. No statistically significant differences in diagnostic performance were found between the neurosurgeons and neuropathologists.

Conclusion

Neurosurgeons' performance in interpreting CLE images was comparable to that of neuropathologists. These results suggest that CLE could be used as an intraoperative guidance tool with neurosurgeons interpreting the images with or without assistance of the neuropathologists. The dichotomous scoring system is robust yet simple and may streamline rapid, simultaneous interpretation of CLE images during imaging.

Exceeding the limit for microscopic image translation with a deep learning-based unified framework

Deep learning algorithms have been widely used in microscopic image translation. The corresponding data-driven models can be trained by supervised or unsupervised learning depending on the availability of paired data. However, general cases are where the data are only roughly paired such that supervised learning could be invalid due to data unalignment, and unsupervised learning would be less ideal as the roughly paired information is not utilized. In this work, we propose a unified framework (U-Frame) that unifies supervised and unsupervised learning by introducing a tolerance size that can be adjusted automatically according to the degree of data misalignment. Together with the implementation of a global sampling rule, we demonstrate that U-Frame consistently outperforms both supervised and unsupervised learning in all levels of data misalignments (even for perfectly aligned image pairs) in a myriad of image translation applications, including pseudo-optical sectioning, virtual histological staining (with clinical evaluations for cancer diagnosis), improvement of signal-to-noise ratio or resolution, and prediction of fluorescent labels, potentially serving as new standard for image translation.

Unpaired mesh-to-image translation for 3D fluorescent microscopy images of neurons

While Generative Adversarial Networks (GANs) can now reliably produce realistic images in a multitude of imaging domains, they are ill-equipped to model thin, stochastic textures present in many large 3D fluorescent microscopy (FM) images acquired in biological research. This is especially problematic in neuroscience where the lack of ground truth data impedes the development of automated image analysis algorithms for neurons and neural populations. We therefore propose an unpaired mesh-to-image translation methodology for generating volumetric FM images of neurons from paired ground truths. We start by learning unique FM styles efficiently through a Gramian-based discriminator. Then, we stylize 3D voxelized meshes of previously reconstructed neurons by successively generating slices. As a result, we effectively create a synthetic microscope and can acquire realistic FM images of neurons with control over the image content and imaging configurations. We demonstrate the feasibility of our architecture and its superior performance compared to state-of-the-art image translation architectures through a variety of texture-based metrics, unsupervised segmentation accuracy, and an expert opinion test. In this study, we use 2 synthetic FM datasets and 2 newly acquired FM datasets of retinal neurons.

Characterization of ex vivo and in vivo intraoperative neurosurgical confocal laser endomicroscopy imaging

Background

The new US Food and Drug Administration-cleared fluorescein sodium (FNa)-based confocal laser endomicroscopy (CLE) imaging system allows for intraoperative on-the-fly cellular level imaging. Two feasibility studies have been completed with intraoperative use of this CLE system in ex vivo and in vivo modalities. This study quantitatively compares the image quality and diagnostic performance of ex vivo and in vivo CLE imaging.

Methods

Images acquired from two prospective CLE clinical studies, one ex vivo and one in vivo, were analyzed quantitatively. Two image quality parameters – brightness and contrast – were measured using Fiji software and compared between ex vivo and in vivo images for imaging timing from FNa dose and in glioma, meningioma, and intracranial metastatic tumor cases. The diagnostic performance of the two studies was compared.

Results

Overall, the in vivo images have higher brightness and contrast than the ex vivo images (p < 0.001). A weak negative correlation exists between image quality and timing of imaging after FNa dose for the ex vivo images, but not the in vivo images. In vivo images have higher image quality than ex vivo images (p < 0.001) in glioma, meningioma, and intracranial metastatic tumor cases. In vivo imaging yielded higher sensitivity and negative predictive value than ex vivo imaging.

Conclusions

In our setting, in vivo CLE optical biopsy outperforms ex vivo CLE by producing higher quality images and less image deterioration, leading to better diagnostic performance. These results support the in vivo modality as the modality of choice for intraoperative CLE imaging.

Deep learning-based image processing in optical microscopy

Optical microscopy has emerged as a key driver of fundamental research since it provides the ability to probe into imperceptible structures in the biomedical world. For the detailed investigation of samples, a high-resolution image with enhanced contrast and minimal damage is preferred. To achieve this, an automated image analysis method is preferable over manual analysis in terms of both speed of acquisition and reduced error accumulation. In this regard, deep learning (DL)-based image processing can be highly beneficial. The review summarises and critiques the use of DL in image processing for the data collected using various optical microscopic techniques. In tandem with optical microscopy, DL has already found applications in various problems related to image classification and segmentation. It has also performed well in enhancing image resolution in smartphone-based microscopy, which in turn enablse crucial medical assistance in remote places.

Graphical abstract

Fluorescein-Guided Panendoscopy for Head and Neck Cancer Using Handheld Probe-Based Confocal Laser Endomicroscopy: A Pilot Study

BACKGROUND

White-light endoscopy and microscopy combined with histological analysis is currently the mainstay for intraprocedural tissue diagnosis during panendoscopy for head and neck cancer. However, taking biopsies leads to selection bias, ex vivo histopathology is time-consuming, and the advantages of in-vivo intraoperative decision making cannot be used. Confocal laser endomicroscopy (CLE) has the potential for a rapid and histological assessment in the head and neck operating room.

METHODS

Between July 2019 and January 2020, 13 patients (69% male, median age: 61 years) with newly diagnosed head and neck cancer (T3/T4: 46%) underwent fluorescein-guided panendoscopy. CLE was performed from both the tumor and margins followed by biopsies from the CLE spots. The biopsies were processed for histopathology. The CLE images were ex vivo classified blinded with a CLE cancer score (DOC score). The classification was compared to the histopathological results.

RESULTS

Median additional time for CLE during surgery was 9 min. A total of 2,565 CLE images were taken (median CLE images: 178 per patient; 68 per biopsy; evaluable 87.5%). The concordance between histopathology and CLE images varied between the patients from 82.5 to 98.6%. The sensitivity, specificity, and accuracy to detect cancer using the classified CLE images was 87.5, 80.0, and 84.6%, respectively. The positive and negative predictive values were 87.0 and 80.0%, respectively.

CONCLUSION

CLE with a rigid handheld probe is easy and intuitive to handle during panendoscopy. As next step, the high accuracy of ex vivo CLE image classification for tumor tissue suggests the validation of CLE in vivo. This will evolve CLE as a complementary tool for in vivo intraoperative diagnosis during panendoscopy.

Deep Neural Network for Differentiation of Brain Tumor Tissue Displayed by Confocal Laser Endomicroscopy

Background

Reliable on site classification of resected tumor specimens remains a challenge. Implementation of high-resolution confocal laser endoscopic techniques (CLEs) during fluorescence-guided brain tumor surgery is a new tool for intraoperative tumor tissue visualization. To overcome observer dependent errors, we aimed to predict tumor type by applying a deep learning model to image data obtained by CLE.

Methods

Human brain tumor specimens from 25 patients with brain metastasis, glioblastoma, and meningioma were evaluated within this study. In addition to routine histopathological analysis, tissue samples were stained with fluorescein ex vivo and analyzed with CLE. We trained two convolutional neural networks and built a predictive level for the outputs.

Results

Multiple CLE images were obtained from each specimen with a total number of 13,972 fluorescein based images. Test accuracy of 90.9% was achieved after applying a two-class prediction for glioblastomas and brain metastases with an area under the curve (AUC) value of 0.92. For three class predictions, our model achieved a ratio of correct predicted label of 85.8% in the test set, which was confirmed with five-fold cross validation, without definition of confidence. Applying a confidence rate of 0.999 increased the prediction accuracy to 98.6% when images with substantial artifacts were excluded before the analysis. 36.3% of total images met the output criteria.

Conclusions

We trained a residual network model that allows automated, on site analysis of resected tumor specimens based on CLE image datasets. Further in vivo studies are required to assess the clinical benefit CLE can have.

Confocal Laser Microscopy in Neurosurgery: State of the Art of Actual Clinical Applications

Achievement of complete resections is of utmost importance in brain tumor surgery, due to the established correlation among extent of resection and postoperative survival. Various tools have recently been included in current clinical practice aiming to more complete resections, such as neuronavigation and fluorescent-aided techniques, histopathological analysis still remains the gold-standard for diagnosis, with frozen section as the most used, rapid and precise intraoperative histopathological method that permits an intraoperative differential diagnosis. Unfortunately, due to the various limitations linked to this technique, it is still unsatisfactorily for obtaining real-time intraoperative diagnosis. Confocal laser technology has been recently suggested as a promising method to obtain near real-time intraoperative histological data in neurosurgery, due to its established use in other non-neurosurgical fields. Still far to be widely implemented in current neurosurgical clinical practice, this technology was initially studied in preclinical experiences confirming its utility in identifying brain tumors, microvasculature and tumor margins. Hence, ex vivo and in vivo clinical studies evaluated the possibility with this technology of identifying and classifying brain neoplasms, discerning between normal and pathologic tissue, showing very promising results. This systematic review has the main objective of presenting a state-of-the-art summary on actual clinical applications of confocal laser imaging in neurosurgical practice.

Intraoperative Confocal Laser Endomicroscopy Ex Vivo Examination of Tissue Microstructure During Fluorescence-Guided Brain Tumor Surgery

Background

Noninvasive intraoperative optical biopsy that provides real-time imaging of histoarchitectural (cell resolution) features of brain tumors, especially at the margin of invasive tumors, would be of great value. To assess clinical-grade confocal laser endomicroscopy (CLE) and to prepare for its use intraoperatively in vivo, we performed an assessment of CLE ex vivo imaging in brain lesions.

Methods

Tissue samples from patients who underwent intracranial surgeries with fluorescein sodium (FNa)–based wide-field fluorescence guidance were acquired for immediate intraoperative ex vivo optical biopsies with CLE. Hematoxylin-eosin–stained frozen section analysis of the same specimens served as the gold standard for blinded neuropathology comparison. FNa 2 to 5 mg/kg was administered upon induction of anesthesia, and FNa 5 mg/kg was injected for CLE contrast improvement. Histologic features were identified, and the diagnostic accuracy of CLE was assessed.

Results

Of 77 eligible patients, 47 patients with 122 biopsies were enrolled, including 32 patients with gliomas and 15 patients with other intracranial lesions. The positive predictive value of CLE optical biopsies was 97% for all specimens and 98% for gliomas. The specificity of CLE was 90% for all specimens and 94% for gliomas. The second FNa injection in seven patients, a mean of 2.6 h after the first injection, improved image quality and increased the percentage of accurately diagnosed images from 67% to 93%. Diagnostic CLE features of lesional glioma biopsies and normal brain were identified. Seventeen histologic features were identified.

Conclusions

Results demonstrated high specificity and positive predictive value of ex vivo intraoperative CLE optical biopsies and justify an in vivo intraoperative trial. This new portable, noninvasive intraoperative imaging technique provides diagnostic features to discriminate lesional tissue with high specificity and is feasible for incorporation into the fluorescence-guided surgery workflow, particularly for patients with invasive brain tumors.