Real-time intraoperative diagnosis by deep neural network driven multiphoton virtual histology

Diagnosis of Fibrotic Interstitial Lung Diseases Based on the Combination of Label-Free Quantitative Multiphoton Fiber Histology and Machine Learning

Interstitial lung disease (ILD), characterized by inflammation and fibrosis, often suffers from low diagnostic accuracy and consistency. Traditional hematoxylin and eosin (H&E) staining primarily reveals cellular inflammation with limited detail on fibrosis. To address these issues, we introduce a pioneering label-free quantitative multiphoton fiber histology (MPFH) technique that delineates the intricate characteristics of collagen and elastin fibers for ILD diagnosis. We acquired colocated multiphoton and H&E-stained images from a single tissue slice. Multiphoton imaging was performed on the deparaffinized section to obtain fibrotic tissue information, followed by H&E staining to capture cellular information. This approach was tested in a blinded diagnostic trial among 7 pathologists involving 14 patients with relatively normal lung and 31 patients with ILD (11 idiopathic pulmonary fibrosis/usual interstitial pneumonia, 14 nonspecific interstitial pneumonia, and 6 pleuroparenchymal fibroelastosis). A customized algorithm extracted quantitative fiber indicators from multiphoton images. These indicators, combined with clinical and radiologic features, were used to develop an automatic multiclass ILD classifier. Using MPFH, we can acquire high-quality, colocalized images of collagen fibers, elastin fibers, and cells. We found that the type, distribution, and degree of fibrotic proliferation can effectively distinguish between different subtypes. The blind study showed that MPFH enhanced diagnostic consistency (κ values from 0.56 to 0.72) and accuracy (from 73.0% to 82.5%, P = .0090). The combination of quantitative fiber indicators effectively distinguished between different tissues, with areas under the receiver operating characteristic curves exceeding 0.92. The automatic classifier achieved 93.8% accuracy, closely paralleling the 92.2% accuracy of expert pathologists. The outcomes of our research underscore the transformative potential of MPFH in the field of fibrotic-ILD diagnostics. By integrating quantitative analysis of fiber characteristics with advanced machine learning algorithms, MPFH facilitates the automatic and accurate identification of various fibrotic disease subtypes, showcasing a significant leap forward in precision diagnostics.

Deep and dynamic metabolic and structural imaging in living tissues

Label-free imaging through two-photon autofluorescence of NAD(P)H allows for nondestructive, high-resolution visualization of cellular activities in living systems. However, its application to thick tissues has been restricted by its limited penetration depth within 300 μm, largely due to light scattering. Here, we demonstrate that the imaging depth for NAD(P)H can be extended to more than 700 μm in living engineered human multicellular microtissues by adopting multimode fiber-based, low repetition rate, high peak power, three-photon excitation of NAD(P)H at 1100 nm. This is achieved by having more than 0.5 megawatts peak power at the band of 1100 ± 25 nm through adaptively modulating multimodal nonlinear pulse propagation with a compact fiber shaper. Moreover, the eightfold increase in pulse energy enables faster imaging of monocyte behaviors in the living multicellular models. These results represent a substantial advance for deep and dynamic imaging of intact living biosystems. The modular design is anticipated to allow wide adoption for demanding imaging applications, including cancer research, immune responses, and tissue engineering.

Towards next-generation diagnostic pathology: AI-empowered label-free multiphoton microscopy

Diagnostic pathology, historically dependent on visual scrutiny by experts, is essential for disease detection. Advances in digital pathology and developments in computer vision technology have led to the application of artificial intelligence (AI) in this field. Despite these advancements, the variability in pathologists' subjective interpretations of diagnostic criteria can lead to inconsistent outcomes. To meet the need for precision in cancer therapies, there is an increasing demand for accurate pathological diagnoses. Consequently, traditional diagnostic pathology is evolving towards "next-generation diagnostic pathology", prioritizing on the development of a multi-dimensional, intelligent diagnostic approach. Using nonlinear optical effects arising from the interaction of light with biological tissues, multiphoton microscopy (MPM) enables high-resolution label-free imaging of multiple intrinsic components across various human pathological tissues. AI-empowered MPM further improves the accuracy and efficiency of diagnosis, holding promise for providing auxiliary pathology diagnostic methods based on multiphoton diagnostic criteria. In this review, we systematically outline the applications of MPM in pathological diagnosis across various human diseases, and summarize common multiphoton diagnostic features. Moreover, we examine the significant role of AI in enhancing multiphoton pathological diagnosis, including aspects such as image preprocessing, refined differential diagnosis, and the prognostication of outcomes. We also discuss the challenges and perspectives faced by the integration of MPM and AI, encompassing equipment, datasets, analytical models, and integration into the existing clinical pathways. Finally, the review explores the synergy between AI and label-free MPM to forge novel diagnostic frameworks, aiming to accelerate the adoption and implementation of intelligent multiphoton pathology systems in clinical settings.

Inspiring a convergent engineering approach to measure and model the tissue microenvironment

Understanding the molecular and physical complexity of the tissue microenvironment (TiME) in the context of its spatiotemporal organization has remained an enduring challenge. Recent advances in engineering and data science are now promising the ability to study the structure, functions, and dynamics of the TiME in unprecedented detail; however, many advances still occur in silos that rarely integrate information to study the TiME in its full detail. This review provides an integrative overview of the engineering principles underlying chemical, optical, electrical, mechanical, and computational science to probe, sense, model, and fabricate the TiME. In individual sections, we first summarize the underlying principles, capabilities, and scope of emerging technologies, the breakthrough discoveries enabled by each technology and recent, promising innovations. We provide perspectives on the potential of these advances in answering critical questions about the TiME and its role in various disease and developmental processes. Finally, we present an integrative view that appreciates the major scientific and educational aspects in the study of the TiME.

In vivo organoid growth monitoring by stimulated Raman histology

Patient-derived tumor organoids have emerged as a crucial tool for assessing the efficacy of chemotherapy and conducting preclinical drug screenings. However, the conventional histological investigation of these organoids necessitates their devitalization through fixation and slicing, limiting their utility to a single-time analysis. Here, we use stimulated Raman histology (SRH) to demonstrate non-destructive, label-free virtual staining of 3D organoids, while preserving their viability and growth. This novel approach provides contrast similar to conventional staining methods, allowing for the continuous monitoring of organoids over time. Our results demonstrate that SRH transforms organoids from one-time use products into repeatable models, facilitating the efficient selection of effective drug combinations. This advancement holds promise for personalized cancer treatment, allowing for the dynamic assessment and optimization of chemotherapy treatments in patient-specific contexts.

Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions

Artificial intelligence (AI) in oncology is advancing beyond algorithm development to integration into clinical practice. This review describes the current state of the field, with a specific focus on clinical integration. AI applications are structured according to cancer type and clinical domain, focusing on the four most common cancers and tasks of detection, diagnosis, and treatment. These applications encompass various data modalities, including imaging, genomics, and medical records. We conclude with a summary of existing challenges, evolving solutions, and potential future directions for the field.

SIGNIFICANCE

AI is increasingly being applied to all aspects of oncology, where several applications are maturing beyond research and development to direct clinical integration. This review summarizes the current state of the field through the lens of clinical translation along the clinical care continuum. Emerging areas are also highlighted, along with common challenges, evolving solutions, and potential future directions for the field.

Deep UV-excited fluorescence microscopy installed with CycleGAN-assisted image translation enhances precise detection of lymph node metastasis towards rapid intraoperative diagnosis

Rapid and precise intraoperative diagnosing systems are required for improving surgical outcomes and patient prognosis. Because of the poor quality and time-intensive process of the prevalent frozen section procedure, various intraoperative diagnostic imaging systems have been explored. Microscopy with ultraviolet surface excitation (MUSE) is an inexpensive, maintenance-free, and rapid imaging technique that yields images like thin-sectioned samples without sectioning. However, pathologists find it nearly impossible to assign diagnostic labels to MUSE images of unfixed specimens; thus, AI for intraoperative diagnosis cannot be trained in a supervised learning manner. In this study, we propose a deep-learning pipeline model for lymph node metastasis detection, in which CycleGAN translate MUSE images of unfixed lymph nodes to formalin-fixed paraffin-embedded (FFPE) sample, and diagnostic prediction is performed using deep convolutional neural network trained on FFPE sample images. Our pipeline yielded an average accuracy of 84.6% when using each of the three deep convolutional neural networks, which is a 18.3% increase over the classification-only model without CycleGAN. The modality translation to FFPE sample images using CycleGAN can be applied to various intraoperative diagnostic imaging systems and eliminate the difficulty for pathologists in labeling new modality images in clinical sites. We anticipate our pipeline to be a starting point for accurate rapid intraoperative diagnostic systems for new imaging modalities, leading to healthcare quality improvement.

Label-free biomedical optical imaging

Label-free optical imaging employs natural and nondestructive approaches for the visualisation of biomedical samples for both biological assays and clinical diagnosis. Currently, this field revolves around multiple broad technology-oriented communities, each with a specific focus on a particular modality despite the existence of shared challenges and applications. As a result, biologists or clinical researchers who require label-free imaging are often not aware of the most appropriate modality to use. This manuscript presents a comprehensive review of and comparison among different label-free imaging modalities and discusses common challenges and applications. We expect this review to facilitate collaborative interactions between imaging communities, push the field forward and foster technological advancements, biophysical discoveries, as well as clinical detection, diagnosis, and monitoring of disease.

A deep learning-based stripe self-correction method for stitched microscopic images

Stitched fluorescence microscope images inevitably exist in various types of stripes or artifacts caused by uncertain factors such as optical devices or specimens, which severely affects the image quality and downstream quantitative analysis. Here, we present a deep learning-based Stripe Self-Correction method, so-called SSCOR. Specifically, we propose a proximity sampling scheme and adversarial reciprocal self-training paradigm that enable SSCOR to utilize stripe-free patches sampled from the stitched microscope image itself to correct their adjacent stripe patches. Comparing to off-the-shelf approaches, SSCOR can not only adaptively correct non-uniform, oblique, and grid stripes, but also remove scanning, bubble, and out-of-focus artifacts, achieving the state-of-the-art performance across different imaging conditions and modalities. Moreover, SSCOR does not require any physical parameter estimation, patch-wise manual annotation, or raw stitched information in the correction process. This provides an intelligent prior-free image restoration solution for microscopists or even microscope companies, thus ensuring more precise biomedical applications for researchers.

Nonlinear Imaging Histopathology: A Pipeline to Correlate Gold-Standard Hematoxylin and Eosin Staining With Modern Nonlinear Microscopy

Hematoxylin and eosin (H&E) staining, the century-old technique, has been the gold standard tool for pathologists to detect anomalies in tissues and diseases such as cancer. H&E staining is a cumbersome, time-consuming process that delays and wastes precious minutes during an intraoperative diagnosis. However, even in the modern era, real-time label-free imaging techniques such as simultaneous label-free autofluorescence multiharmonic (SLAM) microscopy have delivered several more layers of information to characterize a tissue with high precision. Still, they have yet to translate to the clinic. The slow translation rate can be attributed to the lack of direct comparisons between the old and new techniques. Our approach to solving this problem is to: 1) reduce dimensions by pre-sectioning the tissue in 500 μm slices, and 2) produce fiducial laser markings which appear in both SLAM and histological imaging. High peak-power femtosecond laser pulses enable ablation in a controlled and contained manner. We perform laser marking on a grid of points encompassing the SLAM region of interest. We optimize laser power, numerical aperture, and timing to produce axially extended marking, hence multilayered fiducial markers, with minimal damage to the surrounding tissues. We performed this co-registration over an area of 3 × 3 mm2 of freshly excised mouse kidney and intestine, followed by standard H&E staining. Reduced dimensionality and the use of laser markings provided a comparison of the old and new techniques, giving a wealth of correlative information and elevating the potential of translating nonlinear microscopy to the clinic for rapid pathological assessment.

Toward next-generation endoscopes integrating biomimetic video systems, nonlinear optical microscopy, and deep learning

According to the World Health Organization, the proportion of the world's population over 60 years will approximately double by 2050. This progressive increase in the elderly population will lead to a dramatic growth of age-related diseases, resulting in tremendous pressure on the sustainability of healthcare systems globally. In this context, finding more efficient ways to address cancers, a set of diseases whose incidence is correlated with age, is of utmost importance. Prevention of cancers to decrease morbidity relies on the identification of precursor lesions before the onset of the disease, or at least diagnosis at an early stage. In this article, after briefly discussing some of the most prominent endoscopic approaches for gastric cancer diagnostics, we review relevant progress in three emerging technologies that have significant potential to play pivotal roles in next-generation endoscopy systems: biomimetic vision (with special focus on compound eye cameras), non-linear optical microscopies, and Deep Learning. Such systems are urgently needed to enhance the three major steps required for the successful diagnostics of gastrointestinal cancers: detection, characterization, and confirmation of suspicious lesions. In the final part, we discuss challenges that lie en route to translating these technologies to next-generation endoscopes that could enhance gastrointestinal imaging, and depict a possible configuration of a system capable of (i) biomimetic endoscopic vision enabling easier detection of lesions, (ii) label-free in vivo tissue characterization, and (iii) intelligently automated gastrointestinal cancer diagnostic.

Automatic diagnosis and classification of breast surgical samples with dynamic full-field OCT and machine learning

Purpose

The adoption of emerging imaging technologies in the medical community is often hampered when they provide a new unfamiliar contrast that requires experience to be interpreted. Dynamic full-field optical coherence tomography (D-FF-OCT) microscopy is such an emerging technique. It provides fast, high-resolution images of excised tissues with a contrast comparable to H&E histology but without any tissue preparation and alteration.

Approach

We designed and compared two machine learning approaches to support interpretation of D-FF-OCT images of breast surgical specimens and thus provide tools to facilitate medical adoption. We conducted a pilot study on 51 breast lumpectomy and mastectomy surgical specimens and more than 1000 individual 1.3 × 1.3    mm 2 images and compared with standard H&E histology diagnosis.

Results

Using our automatic diagnosis algorithms, we obtained an accuracy above 88% at the image level (1.3 × 1.3    mm 2 ) and above 96% at the specimen level (above cm 2 ).

Conclusions

Altogether, these results demonstrate the high potential of D-FF-OCT coupled to machine learning to provide a rapid, automatic, and accurate histopathology diagnosis with minimal sample alteration.

Biological insights and novel biomarker discovery through deep learning approaches in breast cancer histopathology

Breast cancer remains a highly prevalent disease with considerable inter- and intra-tumoral heterogeneity complicating prognostication and treatment decisions. The utilization and depth of genomic, transcriptomic and proteomic data for cancer has exploded over recent times and the addition of spatial context to this information, by understanding the correlating morphologic and spatial patterns of cells in tissue samples, has created an exciting frontier of research, histo-genomics. At the same time, deep learning (DL), a class of machine learning algorithms employing artificial neural networks, has rapidly progressed in the last decade with a confluence of technical developments - including the advent of modern graphic processing units (GPU), allowing efficient implementation of increasingly complex architectures at scale; advances in the theoretical and practical design of network architectures; and access to larger datasets for training - all leading to sweeping advances in image classification and object detection. In this review, we examine recent developments in the application of DL in breast cancer histology with particular emphasis of those producing biologic insights or novel biomarkers, spanning the extraction of genomic information to the use of stroma to predict cancer recurrence, with the aim of suggesting avenues for further advancing this exciting field.

Intraoperative cytological diagnosis of brain tumours: A preliminary study using a deep learning model

BACKGROUND

Intraoperative pathological diagnosis of central nervous system (CNS) tumours is essential to planning patient management in neuro-oncology. Frozen section slides and cytological preparations provide architectural and cellular information that is analysed by pathologists to reach an intraoperative diagnosis. Progress in the fields of artificial intelligence and machine learning means that AI systems have significant potential for the provision of highly accurate real-time diagnosis in cytopathology.

OBJECTIVE

To investigate the efficiency of machine-learning models in the intraoperative cytological diagnosis of CNS tumours.

MATERIALS AND METHODS

We trained a deep neural network to classify biopsy material for intraoperative tissue diagnosis of four major brain lesions. Overall, 205 medical images were obtained from squash smear slides of histologically correlated cases, with 18 high-grade and 11 low-grade gliomas, 17 metastatic carcinomas, and 9 non-neoplastic pathological brain tissue samples. The neural network model was trained and evaluated using 5-fold cross-validation.

RESULTS

The model achieved 95% and 97% diagnostic accuracy in the patch-level classification and patient-level classification tasks, respectively.

CONCLUSIONS

We conclude that deep learning-based classification of cytological preparations may be a promising complementary method for the rapid and accurate intraoperative diagnosis of CNS tumours.

Squamous Cell Carcinoma of Skin Cancer Margin Classification From Digital Histopathology Images Using Deep Learning

Objectives

Now a days, squamous cell carcinoma (SCC) margin assessment is done by examining histopathology images and inspection of whole slide images (WSI) using a conventional microscope. This is time-consuming, tedious, and depends on experts’ experience which may lead to misdiagnosis and mistreatment plans. This study aims to develop a system for the automatic diagnosis of skin cancer margin for squamous cell carcinoma from histopathology microscopic images by applying deep learning techniques.

Methods

The system was trained, validated, and tested using histopathology images of SCC cancer locally acquired from Jimma Medical Center Pathology Department from seven different skin sites using an Olympus digital microscope. All images were preprocessed and trained with transfer learning pre-trained models by fine-tuning the hyper-parameter of the selected models.

Results

The overall best training accuracy of the models become 95.3%, 97.1%, 89.8%, and 89.9% on EffecientNetB0, MobileNetv2, ResNet50, VGG16 respectively. In addition to this, the best validation accuracy of the models was 94.7%, 91.8%, 87.8%, and 86.7% respectively. The best testing accuracy of the models at the same epoch was 95.2%, 91.5%, 87%, and 85.5% respectively. From these models, EfficientNetB0 showed the best average training and testing accuracy than the other models.

Conclusions

The system assists the pathologist during the margin assessment of SCC by decreasing the diagnosis time from an average of 25 minutes to less than a minute.

Evaluation of histopathological response to neoadjuvant therapy in rectal cancer using slide-free, stain-free multimodal multiphoton microscopy

Neoadjuvant therapy has become a standard treatment for patients with locally advanced rectal cancer to achieve better prognostic outcomes. The response to treatment has been shown to correlate closely with the prognosis. However, current evaluation systems only provide coarse assessment on limited information, due to the lack of accurate and reproducible approach for quantitation of different types of responses. In this study, a novel stain-free, slide-free multimodal multiphoton microscopy imaging technique was applied to image rectal cancer tissues after neoadjuvant therapies with high resolution and contrast. Qualitative and quantitative evaluation of tumor, stromal, and inflammatory responses were demonstrated which are consistent with current tumor regression grading system using American Joint Committee on Cancer criteria, showing the great potential of such approach to build a more informative grading system for accurate and standardizable assessment of neoadjuvant therapy in rectal cancer.

Automated classification of otitis media with OCT: augmenting pediatric image datasets with gold-standard animal model data

Otitis media (OM) is an extremely common disease that affects children worldwide. Optical coherence tomography (OCT) has emerged as a noninvasive diagnostic tool for OM, which can detect the presence and quantify the properties of middle ear fluid and biofilms. Here, the use of OCT data from the chinchilla, the gold-standard OM model for the human disease, is used to supplement a human image database to produce diagnostically relevant conclusions in a machine learning model. Statistical analysis shows the datatypes are compatible, with a blended-species model reaching ∼95% accuracy and F1 score, maintaining performance while additional human data is collected.

Label-free multimodal nonlinear optical imaging of needle biopsy cores for intraoperative cancer diagnosis

SIGNIFICANCE

Needle biopsy (NB) procedures are important for the initial diagnosis of many types of cancer. However, the possibility of NB specimens being unable to provide diagnostic information, (i.e., non-diagnostic sampling) and the time-consuming histological evaluation process can cause delays in diagnoses that affect patient care.

AIM

We aim to demonstrate the advantages of this label-free multimodal nonlinear optical imaging (NLOI) technique as a non-destructive point-of-procedure evaluation method for NB tissue cores, for the visualization and characterization of the tissue microenvironment.

APPROACH

A portable, label-free, multimodal NLOI system combined second-harmonic generation (SHG) and third-harmonic generation and two- and three-photon autofluorescence (2PF, 3PF) microscopy. It was used for intraoperative imaging of fresh NB tissue cores acquired during canine cancer surgeries, which involved liver, lung, and mammary tumors as well as soft-tissue sarcoma; in total, eight canine patients were recruited. An added tissue culture chamber enabled the use of this NLOI system for longitudinal imaging of fresh NB tissue cores taken from an induced rat mammary tumor and healthy mouse livers.

RESULTS

The intraoperative NLOI system was used to assess fresh canine NB specimens during veterinary cancer surgeries. Histology-like morphological features were visualized by the combination of four NLOI modalities at the point-of-procedure. The NLOI results provided quantitative information on the tissue microenvironment such as the collagen fiber orientation using Fourier-domain SHG analysis and metabolic profiling by optical redox ratio (ORR) defined by 2PF/(2PF + 3PF). The analyses showed that the canine mammary tumor had more randomly oriented collagen fibers compared to the tumor margin, and hepatocarcinoma had a wider distribution of ORR with a lower mean value compared to the liver fibrosis and the normal-appearing liver. Moreover, the loss of metabolic information during tissue degradation of fresh murine NB specimens was shown by overall intensity decreases in all channels and an increase of mean ORR from 0.94 (standard deviation 0.099) to 0.97 (standard deviation 0.077) during 1-h longitudinal imaging of a rat mammary tumor NB specimen. The tissue response to staurosporine (STS), an apoptotic inducer, from fresh murine liver NB specimens was also observed. The mean ORR decreased from 0.86 to 0.74 in the first 40 min and then increased to 0.8 during the rest of the hour of imaging, compared to the imaging results without the addition of STS, which showed a continuous increase of ORR from 0.72 to 0.75.

CONCLUSIONS

A label-free, multimodal NLOI platform reveals microstructural and metabolic information of the fresh NB cores during intraoperative cancer imaging. This system has been demonstrated on animal models to show its potential to provide a more comprehensive histological assessment and a better understanding of the unperturbed tumor microenvironment. Considering tissue degradation, or loss of viability upon fixation, this intraoperative NLOI system has the advantage of immediate assessment of freshly excised tissue specimens at the point of procedure.

Electrical Impedance-Based Characterization of Hepatic Tissue with Early-Stage Fibrosis

Liver fibrosis is a key pathological precondition for hepatocellular carcinoma in which the severity is confidently correlated with liver cancer. Liver fibrosis, characterized by gradual cell loss and excessive extracellular matrix deposition, can be reverted if detected at the early stage. The gold standard for staging and diagnosis of liver fibrosis is undoubtedly biopsy. However, this technique needs careful sample preparation and expert analysis. In the present work, an ex vivo, minimally destructive, label-free characterization of liver biopsies is presented. Through a custom-made experimental setup, liver biopsies of bile-duct-ligated and sham-operated mice were measured at 8, 15, and 21 days after the procedure. Changes in impedance were observed with the progression of fibrosis, and through data fitting, tissue biopsies were approximated to an equivalent RC circuit model. The model was validated by means of 3D hepatic cell culture measurement, in which the capacitive part of impedance was proportionally associated with cell number and the resistive one was proportionally associated with the extracellular matrix. While the sham-operated samples presented a decrease in resistance with time, the bile-duct-ligated ones exhibited an increase in this parameter with the evolution of fibrosis. Moreover, since the largest difference in resistance between healthy and fibrotic tissue, of around 2 kΩ, was found at 8 days, this method presents great potential for the study of fibrotic tissue at early stages. Our data point out the great potential of exploiting the proposed needle setup in clinical applications.

Biopsy-free in vivo virtual histology of skin using deep learning

An invasive biopsy followed by histological staining is the benchmark for pathological diagnosis of skin tumors. The process is cumbersome and time-consuming, often leading to unnecessary biopsies and scars. Emerging noninvasive optical technologies such as reflectance confocal microscopy (RCM) can provide label-free, cellular-level resolution, in vivo images of skin without performing a biopsy. Although RCM is a useful diagnostic tool, it requires specialized training because the acquired images are grayscale, lack nuclear features, and are difficult to correlate with tissue pathology. Here, we present a deep learning-based framework that uses a convolutional neural network to rapidly transform in vivo RCM images of unstained skin into virtually-stained hematoxylin and eosin-like images with microscopic resolution, enabling visualization of the epidermis, dermal-epidermal junction, and superficial dermis layers. The network was trained under an adversarial learning scheme, which takes ex vivo RCM images of excised unstained/label-free tissue as inputs and uses the microscopic images of the same tissue labeled with acetic acid nuclear contrast staining as the ground truth. We show that this trained neural network can be used to rapidly perform virtual histology of in vivo, label-free RCM images of normal skin structure, basal cell carcinoma, and melanocytic nevi with pigmented melanocytes, demonstrating similar histological features to traditional histology from the same excised tissue. This application of deep learning-based virtual staining to noninvasive imaging technologies may permit more rapid diagnoses of malignant skin neoplasms and reduce invasive skin biopsies.

Application of a deep-learning technique to non-linear images from human tissue biopsies for shedding new light on breast cancer diagnosis

The development of label-free non-invasive techniques to be used as diagnostic tools in cancer research is of great importance for improving the quality of life for millions of patients. Previous studies have demonstrated that Third Harmonic Generation (THG) imaging could differentiate malignant from benign unlabeled human breast biopsies and distinguish the different grades of cancer. Towards the application of such technologies to clinic, in the present report, a deep learning technique was applied to THG images recorded from breast cancer tissues of grades 0, I, II and III. By the implementation of a convolutional neural network (CNN) model, the differentiation of malignant from benign breast tissue samples and the discrimination of the different grades of cancer in a fast and accurate way were achieved. The obtained results provide a step ahead towards the use of optical diagnostic tools in conjunction with the CNN image classifier for the reliable and rapid malignancy diagnosis in clinic.

Deep Learning in Biomedical Optics

This article reviews deep learning applications in biomedical optics with a particular emphasis on image formation. The review is organized by imaging domains within biomedical optics and includes microscopy, fluorescence lifetime imaging, in vivo microscopy, widefield endoscopy, optical coherence tomography, photoacoustic imaging, diffuse tomography, and functional optical brain imaging. For each of these domains, we summarize how deep learning has been applied and highlight methods by which deep learning can enable new capabilities for optics in medicine. Challenges and opportunities to improve translation and adoption of deep learning in biomedical optics are also summarized. Lasers Surg. Med. © 2021 Wiley Periodicals LLC.

Synthetic polarization-sensitive optical coherence tomography by deep learning

Polarization-sensitive optical coherence tomography (PS-OCT) is a high-resolution label-free optical biomedical imaging modality that is sensitive to the microstructural architecture in tissue that gives rise to form birefringence, such as collagen or muscle fibers. To enable polarization sensitivity in an OCT system, however, requires additional hardware and complexity. We developed a deep-learning method to synthesize PS-OCT images by training a generative adversarial network (GAN) on OCT intensity and PS-OCT images. The synthesis accuracy was first evaluated by the structural similarity index (SSIM) between the synthetic and real PS-OCT images. Furthermore, the effectiveness of the computational PS-OCT images was validated by separately training two image classifiers using the real and synthetic PS-OCT images for cancer/normal classification. The similar classification results of the two trained classifiers demonstrate that the predicted PS-OCT images can be potentially used interchangeably in cancer diagnosis applications. In addition, we applied the trained GAN models on OCT images collected from a separate OCT imaging system, and the synthetic PS-OCT images correlate well with the real PS-OCT image collected from the same sample sites using the PS-OCT imaging system. This computational PS-OCT imaging method has the potential to reduce the cost, complexity, and need for hardware-based PS-OCT imaging systems.

Intraoperative Label-Free Multimodal Nonlinear Optical Imaging for Point-of-Procedure Cancer Diagnostics

Intraoperative imaging in surgical oncology can provide information about the tumor microenvironment as well as information about the tumor margin. Visualizing microstructural features and molecular and functional dynamics may provide important diagnostic and prognostic information, especially when obtained in real-time at the point-of-procedure. A majority of current intraoperative optical techniques are based on the use of the labels, such as fluorescent dyes. However, these exogenous agents disrupt the natural microenvironment, perturb biological processes, and alter the endogenous optical signatures that cells and the microenvironment can provide. Portable nonlinear imaging systems have enabled intraoperative imaging for real-time detection and diagnosis of tissue. We review the development of a label-free multimodal nonlinear optical imaging technique that was adapted into a portable imaging system for intraoperative optical assessment of resected human breast tissue. New developments have applied this technology to assessing needle-biopsy specimens. Needle-biopsy procedures most always precede surgical resection and serve as the first sampling of suspicious masses for diagnosis. We demonstrate the diagnostic feasibility of imaging core needle-biopsy specimens during veterinary cancer surgeries. This intraoperative label-free multimodal nonlinear optical imaging technique can potentially provide a powerful tool to assist in cancer diagnosis at the point-of-procedure.

Modeling and Synthesis of Breast Cancer Optical Property Signatures With Generative Models

Is it possible to find deterministic relationships between optical measurements and pathophysiology in an unsupervised manner and based on data alone? Optical property quantification is a rapidly growing biomedical imaging technique for characterizing biological tissues that shows promise in a range of clinical applications, such as intraoperative breast-conserving surgery margin assessment. However, translating tissue optical properties to clinical pathology information is still a cumbersome problem due to, amongst other things, inter- and intrapatient variability, calibration, and ultimately the nonlinear behavior of light in turbid media. These challenges limit the ability of standard statistical methods to generate a simple model of pathology, requiring more advanced algorithms. We present a data-driven, nonlinear model of breast cancer pathology for real-time margin assessment of resected samples using optical properties derived from spatial frequency domain imaging data. A series of deep neural network models are employed to obtain sets of latent embeddings that relate optical data signatures to the underlying tissue pathology in a tractable manner. These self-explanatory models can translate absorption and scattering properties measured from pathology, while also being able to synthesize new data. The method was tested on a total of 70 resected breast tissue samples containing 137 regions of interest, achieving rapid optical property modeling with errors only limited by current semi-empirical models, allowing for mass sample synthesis and providing a systematic understanding of dataset properties, paving the way for deep automated margin assessment algorithms using structured light imaging or, in principle, any other optical imaging technique seeking modeling. Code is available.

Label-Free Deep Profiling of the Tumor Microenvironment

Label-free nonlinear microscopy enables nonperturbative visualization of structural and metabolic contrast within living cells in their native tissue microenvironment. Here a computational pipeline was developed to provide a quantitative view of the microenvironmental architecture within cancerous tissue from label-free nonlinear microscopy images. To enable single-cell and single-extracellular vesicle (EV) analysis, individual cells, including tumor cells and various types of stromal cells, and EVs were segmented by a multiclass pixelwise segmentation neural network and subsequently analyzed for their metabolic status and molecular structure in the context of the local cellular neighborhood. By comparing cancer tissue with normal tissue, extensive tissue reorganization and formation of a patterned cell-EV neighborhood was observed in the tumor microenvironment. The proposed analytic pipeline is expected to be useful in a wide range of biomedical tasks that benefit from single-cell, single-EV, and cell-to-EV analysis. SIGNIFICANCE: The proposed computational framework allows label-free microscopic analysis that quantifies the complexity and heterogeneity of the tumor microenvironment and opens possibilities for better characterization and utilization of the evolving cancer landscape.

Diagnosing Hirschsprung disease by detecting intestinal ganglion cells using label-free hyperspectral microscopy

Hirschsprung disease (HD) is a congenital disorder in the distal colon that is characterized by the absence of nerve ganglion cells in the diseased tissue. The primary treatment for HD is surgical intervention with resection of the aganglionic bowel. The accurate identification of the aganglionic segment depends on the histologic evaluation of multiple biopsies to determine the absence of ganglion cells in the tissue, which can be a time-consuming procedure. We investigate the feasibility of using a combination of label-free optical modalities, second harmonic generation (SHG); two-photon excitation autofluorescence (2PAF); and Raman spectroscopy (RS), to accurately locate and identify ganglion cells in murine intestinal tissue without the use of exogenous labels or dyes. We show that the image contrast provided by SHG and 2PAF signals allows for the visualization of the overall tissue morphology and localization of regions that may contain ganglion cells, while RS provides detailed multiplexed molecular information that can be used to accurately identify specific ganglion cells. Support vector machine, principal component analysis and linear discriminant analysis classification models were applied to the hyperspectral Raman data and showed that ganglion cells can be identified with a classification accuracy higher than 95%. Our findings suggest that a near real-time intraoperative histology method can be developed using these three optical modalities together that can aid pathologists and surgeons in rapid, accurate identification of ganglion cells to guide surgical decisions with minimal human intervention.

Simultaneous two-photon activation and imaging of neural activity based on spectral-temporal modulation of supercontinuum light

SIGNIFICANCE

Recent advances in nonlinear optics in neuroscience have focused on using two ultrafast lasers for activity imaging and optogenetic stimulation. Broadband femtosecond light sources can obviate the need for multiple lasers by spectral separation for chromatically targeted excitation.

AIM

We present a photonic crystal fiber (PCF)-based supercontinuum source for spectrally resolved two-photon (2P) imaging and excitation of GCaMP6s and C1V1-mCherry, respectively.

APPROACH

A PCF is pumped using a 20-MHz repetition rate femtosecond laser to generate a supercontinuum of light, which is spectrally separated, compressed, and recombined to image GCaMP6s (930 nm excitation) and stimulate the optogenetic protein, C1V1-mCherry (1060 nm excitation). Galvanometric spiral scanning is employed on a single-cell level for multiphoton excitation and high-speed resonant scanning is employed for imaging of calcium activity.

RESULTS

Continuous wave lasers were used to verify functionality of optogenetic activation followed by directed 2P excitation. Results from these experiments demonstrate the utility of a supercontinuum light source for simultaneous, single-cell excitation and calcium imaging.

CONCLUSIONS

A PCF-based supercontinuum light source was employed for simultaneous imaging and excitation of calcium dynamics in brain tissue. Pumped PCFs can serve as powerful light sources for imaging and activation of neural activity, and overcome the limited spectra and space associated with multilaser approaches.

Deep-Hipo: Multi-scale receptive field deep learning for histopathological image analysis

Digitizing whole-slide imaging in digital pathology has led to the advancement of computer-aided tissue examination using machine learning techniques, especially convolutional neural networks. A number of convolutional neural network-based methodologies have been proposed to accurately analyze histopathological images for cancer detection, risk prediction, and cancer subtype classification. Most existing methods have conducted patch-based examinations, due to the extremely large size of histopathological images. However, patches of a small window often do not contain sufficient information or patterns for the tasks of interest. It corresponds that pathologists also examine tissues at various magnification levels, while checking complex morphological patterns in a microscope. We propose a novel multi-task based deep learning model for HIstoPatholOgy (named Deep-Hipo) that takes multi-scale patches simultaneously for accurate histopathological image analysis. Deep-Hipo extracts two patches of the same size in both high and low magnification levels, and captures complex morphological patterns in both large and small receptive fields of a whole-slide image. Deep-Hipo has outperformed the current state-of-the-art deep learning methods. We assessed the proposed method in various types of whole-slide images of the stomach: well-differentiated, moderately-differentiated, and poorly-differentiated adenocarcinoma; poorly cohesive carcinoma, including signet-ring cell features; and normal gastric mucosa. The optimally trained model was also applied to histopathological images of The Cancer Genome Atlas (TCGA), Stomach Adenocarcinoma (TCGA-STAD) and TCGA Colon Adenocarcinoma (TCGA-COAD), which show similar pathological patterns with gastric carcinoma, and the experimental results were clinically verified by a pathologist. The source code of Deep-Hipo is publicly available athttp://dataxlab.org/deep-hipo.