Biopsy-free in vivo virtual histology of skin using deep learning

AI-driven 3D bioprinting for regenerative medicine: From bench to bedside

In recent decades, 3D bioprinting has garnered significant research attention due to its ability to manipulate biomaterials and cells to create complex structures precisely. However, due to technological and cost constraints, the clinical translation of 3D bioprinted products (BPPs) from bench to bedside has been hindered by challenges in terms of personalization of design and scaling up of production. Recently, the emerging applications of artificial intelligence (AI) technologies have significantly improved the performance of 3D bioprinting. However, the existing literature remains deficient in a methodological exploration of AI technologies' potential to overcome these challenges in advancing 3D bioprinting toward clinical application. This paper aims to present a systematic methodology for AI-driven 3D bioprinting, structured within the theoretical framework of Quality by Design (QbD). This paper commences by introducing the QbD theory into 3D bioprinting, followed by summarizing the technology roadmap of AI integration in 3D bioprinting, including multi-scale and multi-modal sensing, data-driven design, and in-line process control. This paper further describes specific AI applications in 3D bioprinting's key elements, including bioink formulation, model structure, printing process, and function regulation. Finally, the paper discusses current prospects and challenges associated with AI technologies to further advance the clinical translation of 3D bioprinting.

Acetic acid enabled nuclear contrast enhancement in epi-mode quantitative phase imaging

Significance

The acetowhitening effect of acetic acid (AA) enhances light scattering of cell nuclei, an effect that has been widely leveraged to facilitate tissue inspection for (pre)cancerous lesions. Here, we show that a concomitant effect of acetowhitening-changes in refractive index composition-yields nuclear contrast enhancement in quantitative phase imaging (QPI) of thick tissue samples.

Aim

We aim to explore how changes in refractive index composition during acetowhitening can be captured through a novel epi-mode 3D QPI technique called quantitative oblique back-illumination microscopy (qOBM). We also aim to demonstrate the potential of using a machine learning-based approach to convert qOBM images of fresh tissues into virtually AA-stained images.

Approach

We implemented qOBM, an imaging technique that allows for epi-mode 3D QPI to observe phase changes induced by AA in thick tissue samples. We focus on detecting nuclear contrast changes caused by AA in mouse brain samples. As a proof of concept, we also applied a Cycle-GAN algorithm to convert the acquired qOBM images into virtually AA-stained images, simulating the effect of AA staining.

Results

Our findings demonstrate that AA-induced acetowhitening leads to significant nuclear contrast enhancement in qOBM images of thick tissue samples. In addition, the Cycle-GAN algorithm successfully converted qOBM images into virtually AA-stained images, further facilitating the nuclear enhancement process without any physical stains.

Conclusions

We show that the acetowhitening effect of acetic acid induces changes in refractive index composition that significantly enhance nuclear contrast in QPI. The application of qOBM with AA, along with the use of a Cycle-GAN algorithm to virtually stain tissues, highlights the potential of this approach for advancing label-free and slide-free, ex vivo, and in vivo histology.

Virtual Gram staining of label-free bacteria using dark-field microscopy and deep learning

Gram staining has been a frequently used staining protocol in microbiology. It is vulnerable to staining artifacts due to, e.g., operator errors and chemical variations. Here, we introduce virtual Gram staining of label-free bacteria using a trained neural network that digitally transforms dark-field images of unstained bacteria into their Gram-stained equivalents matching bright-field image contrast. After a one-time training, the virtual Gram staining model processes an axial stack of dark-field microscopy images of label-free bacteria (never seen before) to rapidly generate Gram staining, bypassing several chemical steps involved in the conventional staining process. We demonstrated the success of virtual Gram staining on label-free bacteria samples containing Escherichia coli and Listeria innocua by quantifying the staining accuracy of the model and comparing the chromatic and morphological features of the virtually stained bacteria against their chemically stained counterparts. This virtual bacterial staining framework bypasses the traditional Gram staining protocol and its challenges, including stain standardization, operator errors, and sensitivity to chemical variations.

Skin Cancer Detection Using Transfer Learning and Deep Attention Mechanisms

Background/Objectives: Early and accurate diagnosis of skin cancer improves survival rates; however, dermatologists often struggle with lesion detection due to similar pigmentation. Deep learning and transfer learning models have shown promise in diagnosing skin cancers through image processing. Integrating attention mechanisms (AMs) with deep learning has further enhanced the accuracy of medical image classification. While significant progress has been made, further research is needed to improve the detection accuracy. Previous studies have not explored the integration of attention mechanisms with the pre-trained Xception transfer learning model for binary classification of skin cancer. This study aims to investigate the impact of various attention mechanisms on the Xception model's performance in detecting benign and malignant skin lesions. Methods: We conducted four experiments on the HAM10000 dataset. Three models integrated self-attention (SL), hard attention (HD), and soft attention (SF) mechanisms, while the fourth model used the standard Xception without attention mechanisms. Each mechanism analyzed features from the Xception model uniquely: self-attention examined the input relationships, hard-attention selected elements sparsely, and soft-attention distributed the focus probabilistically. Results: Integrating AMs into the Xception architecture effectively enhanced its performance. The accuracy of the Xception alone was 91.05%. With AMs, the accuracy increased to 94.11% using self-attention, 93.29% with soft attention, and 92.97% with hard attention. Moreover, the proposed models outperformed previous studies in terms of the recall metrics, which are crucial for medical investigations. Conclusions: These findings suggest that AMs can enhance performance in relation to complex medical imaging tasks, potentially supporting earlier diagnosis and improving treatment outcomes.

Artificial intelligence in surgical pathology - Where do we stand, where do we go?

Surgical and neuropathologists continuously search for new and disease-specific features, such as independent predictors of tumor prognosis or determinants of tumor entities and sub-entities. This is a task where artificial intelligence (AI)/machine learning (ML) systems could significantly contribute to help with tumor outcome prediction and the search for new diagnostic or treatment stratification biomarkers. AI systems are increasingly integrated into routine pathology workflows to improve accuracy, reproducibility, productivity and to reveal difficult-to-see features in complicated histological slides, including the quantification of important markers for tumor grading and staging. In this article, we review the infrastructure needed to facilitate digital and computational pathology. We address the barriers for its full deployment in the clinical setting and describe the use of AI in intraoperative or postoperative settings were frozen or formalin-fixed, paraffin-embedded materials are used. We also summarize quality assessment issues of slide digitization, new spatial biology approaches, and the determination of specific gene-expression from whole slide images. Finally, we highlight new innovative and future technologies, such as large language models, optical biopsies, and mass spectrometry imaging.

Evaluation of tumor budding with virtual panCK stains generated by novel multi-model CNN framework

As the global incidence of cancer continues to rise rapidly, the need for swift and precise diagnoses has become increasingly pressing. Pathologists commonly rely on H&E-panCK stain pairs for various aspects of cancer diagnosis, including the detection of occult tumor cells and the evaluation of tumor budding. Nevertheless, conventional chemical staining methods suffer from notable drawbacks, such as time-intensive processes and irreversible staining outcomes. The virtual stain technique, leveraging generative adversarial network (GAN), has emerged as a promising alternative to chemical stains. This approach aims to transform biopsy scans (often H&E) into other stain types. Despite achieving notable progress in recent years, current state-of-the-art virtual staining models confront challenges that hinder their efficacy, particularly in achieving accurate staining outcomes under specific conditions. These limitations have impeded the practical integration of virtual staining into diagnostic practices. To address the goal of producing virtual panCK stains capable of replacing chemical panCK, we propose an innovative multi-model framework. Our approach involves employing a combination of Mask-RCNN (for cell segmentation) and GAN models to extract cytokeratin distribution from chemical H&E images. Additionally, we introduce a tailored dynamic GAN model to convert H&E images into virtual panCK stains, integrating the derived cytokeratin distribution. Our framework is motivated by the fact that the unique pattern of the panCK is derived from cytokeratin distribution. As a proof of concept, we employ our virtual panCK stains to evaluate tumor budding in 45 H&E whole-slide images taken from breast cancer-invaded lymph nodes . Through thorough validation by both pathologists and the QuPath software, our virtual panCK stains demonstrate a remarkable level of accuracy. In stark contrast, the accuracy of state-of-the-art single cycleGAN virtual panCK stains is negligible. To our best knowledge, this is the first instance of a multi-model virtual panCK framework and the utilization of virtual panCK for tumor budding assessment. Our framework excels in generating dependable virtual panCK stains with significantly improved efficiency, thereby considerably reducing turnaround times in diagnosis. Furthermore, its outcomes are easily comprehensible even to pathologists who may not be well-versed in computer technology. We firmly believe that our framework has the potential to advance the field of virtual stain, thereby making significant strides towards improved cancer diagnosis.

Single color digital H&E staining with In-and-Out Net

Digital staining streamlines traditional staining procedures by digitally generating stained images from unstained or differently stained images. While conventional staining methods involve time-consuming chemical processes, digital staining offers an efficient and low-infrastructure alternative. Researchers can expedite tissue analysis without physical sectioning by leveraging microscopy-based techniques, such as confocal microscopy. However, interpreting grayscale or pseudo-color microscopic images remains challenging for pathologists and surgeons accustomed to traditional histologically stained images. To fill this gap, various studies explore digitally simulating staining to mimic targeted histological stains. This paper introduces a novel network, In-and-Out Net, designed explicitly for digital staining tasks. Based on Generative Adversarial Networks (GAN), our model efficiently transforms Reflectance Confocal Microscopy (RCM) images into Hematoxylin and Eosin (H&E) stained images. Using aluminum chloride preprocessing for skin tissue, we enhance nuclei contrast in RCM images. We trained the model with digital H&E labels featuring two fluorescence channels, eliminating the need for image registration and providing pixel-level ground truth. Our contributions include proposing an optimal training strategy, conducting a comparative analysis demonstrating state-of-the-art performance, validating the model through an ablation study, and collecting perfectly matched input and ground truth images without registration. In-and-Out Net showcases promising results, offering a valuable tool for digital staining tasks and advancing the field of histological image analysis.

Metabolic light absorption, scattering, and emission (MetaLASE) microscopy

Optical imaging of metabolism can provide key information about health and disease progression in cells and tissues; however, current methods have lacked gold-standard information about histological structure. Conversely, histology and virtual histology methods have lacked metabolic contrast. Here, we present metabolic light absorption, scattering, and emission (MetaLASE) microscopy, which rapidly provides a virtual histology and optical metabolic readout simultaneously. Hematoxylin-like nucleic contrast and eosin-like cytoplasmic contrast are obtained using photoacoustic remote sensing and ultraviolet reflectance microscopy, respectively. The same ultraviolet source excites endogenous Nicotinamide adenine dinucleotide (phosphate), flavin adenine dinucleotide, and collagen autofluorescence, providing a map of optical redox ratios to visualize metabolic variations including in areas of invasive carcinoma. Benign chronic inflammation and glands also are seen to exhibit hypermetabolism. MetaLASE microscopy offers promise for future applications in intraoperative margin analysis and in research applications where greater insights into metabolic activity could be correlated with cell and tissue types.

Cutaneous Imaging Techniques

Cutaneous imaging is a central tenant to the practice of dermatology. In this article, the authors explore various noninvasive and invasive skin imaging techniques, as well as the latest deployment of these technologies in conjunction with the use artificial intelligence and machine learning. The authors also provide insight into the benefits, limitations, and challenges around integrating these technologies into dermatologic practice.

Accelerating histopathology workflows with generative AI-based virtually multiplexed tumour profiling

Understanding the spatial heterogeneity of tumours and its links to disease initiation and progression is a cornerstone of cancer biology. Presently, histopathology workflows heavily rely on hematoxylin and eosin and serial immunohistochemistry staining, a cumbersome, tissue-exhaustive process that results in non-aligned tissue images. We propose the VirtualMultiplexer, a generative artificial intelligence toolkit that effectively synthesizes multiplexed immunohistochemistry images for several antibody markers (namely AR, NKX3.1, CD44, CD146, p53 and ERG) from only an input hematoxylin and eosin image. The VirtualMultiplexer captures biologically relevant staining patterns across tissue scales without requiring consecutive tissue sections, image registration or extensive expert annotations. Thorough qualitative and quantitative assessment indicates that the VirtualMultiplexer achieves rapid, robust and precise generation of virtually multiplexed imaging datasets of high staining quality that are indistinguishable from the real ones. The VirtualMultiplexer is successfully transferred across tissue scales and patient cohorts with no need for model fine-tuning. Crucially, the virtually multiplexed images enabled training a graph transformer that simultaneously learns from the joint spatial distribution of several proteins to predict clinically relevant endpoints. We observe that this multiplexed learning scheme was able to greatly improve clinical prediction, as corroborated across several downstream tasks, independent patient cohorts and cancer types. Our results showcase the clinical relevance of artificial intelligence-assisted multiplexed tumour imaging, accelerating histopathology workflows and cancer biology.

Deep learning for rapid virtual H&E staining of label-free glioma tissue from hyperspectral images

Hematoxylin and eosin (H&E) staining is a crucial technique for diagnosing glioma, allowing direct observation of tissue structures. However, the H&E staining workflow necessitates intricate processing, specialized laboratory infrastructures, and specialist pathologists, rendering it expensive, labor-intensive, and time-consuming. In view of these considerations, we combine the deep learning method and hyperspectral imaging technique, aiming at accurately and rapidly converting the hyperspectral images into virtual H&E staining images. The method overcomes the limitations of H&E staining by capturing tissue information at different wavelengths, providing comprehensive and detailed tissue composition information as the realistic H&E staining. In comparison with various generator structures, the Unet exhibits substantial overall advantages, as evidenced by a mean structure similarity index measure (SSIM) of 0.7731 and a peak signal-to-noise ratio (PSNR) of 23.3120, as well as the shortest training and inference time. A comprehensive software system for virtual H&E staining, which integrates CCD control, microscope control, and virtual H&E staining technology, is developed to facilitate fast intraoperative imaging, promote disease diagnosis, and accelerate the development of medical automation. The platform reconstructs large-scale virtual H&E staining images of gliomas at a high speed of 3.81 mm2/s. This innovative approach will pave the way for a novel, expedited route in histological staining.

Line-Field Confocal Optical Coherence Tomography for the Diagnosis of Skin Tumors: A Systematic Review and Meta-Analysis

A line-field confocal optical coherence tomography (LC-OCT) combines confocal microscopy and optical coherence tomography into a single, rapid, easy-to-use device. This meta-analysis was performed to determine the reliability of LC-OCT for diagnosing malignant skin tumors. PubMed, EMBASE, Web of Science databases, and the Cochrane Library were searched for research studies in the English language from inception till December 2023. To assess quality and the risk of bias, the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) was used. The sensitivity and specificity of each study were calculated. The bivariate summary sensitivity and specificity were calculated using the linear mixed model. Five studies with 904 reported per lesion analyses in our study; the specificity and sensitivity ranged from 67% to 97% and 72% to 92%, respectively. The pooled specificity and sensitivity were 91% (95% CI: 76-97%) and 86.9% (95% CI: 81.8-90.8%), respectively. The summary sensitivity and specificity from the bivariate approach are 86.9% (95% CI: 81.8-90.8%) and 91.1% (95% CI: 76.7-97.0%), respectively. The area under the curve is 0.914. LC-OCT shows great sensitivity and specificity in diagnosing malignant skin tumors. However, due to the limited number of studies included in our meta-analysis, it is premature to elucidate the true potential of LC-OCT.

In vivo organoid growth monitoring by stimulated Raman histology

Patient-derived tumor organoids have emerged as a crucial tool for assessing the efficacy of chemotherapy and conducting preclinical drug screenings. However, the conventional histological investigation of these organoids necessitates their devitalization through fixation and slicing, limiting their utility to a single-time analysis. Here, we use stimulated Raman histology (SRH) to demonstrate non-destructive, label-free virtual staining of 3D organoids, while preserving their viability and growth. This novel approach provides contrast similar to conventional staining methods, allowing for the continuous monitoring of organoids over time. Our results demonstrate that SRH transforms organoids from one-time use products into repeatable models, facilitating the efficient selection of effective drug combinations. This advancement holds promise for personalized cancer treatment, allowing for the dynamic assessment and optimization of chemotherapy treatments in patient-specific contexts.

Deep learning-based virtual H& E staining from label-free autofluorescence lifetime images

Label-free autofluorescence lifetime is a unique feature of the inherent fluorescence signals emitted by natural fluorophores in biological samples. Fluorescence lifetime imaging microscopy (FLIM) can capture these signals enabling comprehensive analyses of biological samples. Despite the fundamental importance and wide application of FLIM in biomedical and clinical sciences, existing methods for analysing FLIM images often struggle to provide rapid and precise interpretations without reliable references, such as histology images, which are usually unavailable alongside FLIM images. To address this issue, we propose a deep learning (DL)-based approach for generating virtual Hematoxylin and Eosin (H&E) staining. By combining an advanced DL model with a contemporary image quality metric, we can generate clinical-grade virtual H&E-stained images from label-free FLIM images acquired on unstained tissue samples. Our experiments also show that the inclusion of lifetime information, an extra dimension beyond intensity, results in more accurate reconstructions of virtual staining when compared to using intensity-only images. This advancement allows for the instant and accurate interpretation of FLIM images at the cellular level without the complexities associated with co-registering FLIM and histology images. Consequently, we are able to identify distinct lifetime signatures of seven different cell types commonly found in the tumour microenvironment, opening up new opportunities towards biomarker-free tissue histology using FLIM across multiple cancer types.

Perspective on quantitative phase imaging to improve precision cancer medicine

Significance

Quantitative phase imaging (QPI) offers a label-free approach to non-invasively characterize cellular processes by exploiting their refractive index based intrinsic contrast. QPI captures this contrast by translating refractive index associated phase shifts into intensity-based quantifiable data with nanoscale sensitivity. It holds significant potential for advancing precision cancer medicine by providing quantitative characterization of the biophysical properties of cells and tissue in their natural states.

Aim

This perspective aims to discuss the potential of QPI to increase our understanding of cancer development and its response to therapeutics. It also explores new developments in QPI methods towards advancing personalized cancer therapy and early detection.

Approach

We begin by detailing the technical advancements of QPI, examining its implementations across transmission and reflection geometries and phase retrieval methods, both interferometric and non-interferometric. The focus then shifts to QPI's applications in cancer research, including dynamic cell mass imaging for drug response assessment, cancer risk stratification, and in-vivo tissue imaging.

Results

QPI has emerged as a crucial tool in precision cancer medicine, offering insights into tumor biology and treatment efficacy. Its sensitivity to detecting nanoscale changes holds promise for enhancing cancer diagnostics, risk assessment, and prognostication. The future of QPI is envisioned in its integration with artificial intelligence, morpho-dynamics, and spatial biology, broadening its impact in cancer research.

Conclusions

QPI presents significant potential in advancing precision cancer medicine and redefining our approach to cancer diagnosis, monitoring, and treatment. Future directions include harnessing high-throughput dynamic imaging, 3D QPI for realistic tumor models, and combining artificial intelligence with multi-omics data to extend QPI's capabilities. As a result, QPI stands at the forefront of cancer research and clinical application in cancer care.

Virtual Staining of Nonfixed Tissue Histology

Surgical pathology workflow involves multiple labor-intensive steps, such as tissue removal, fixation, embedding, sectioning, staining, and microscopic examination. This process is time-consuming and costly and requires skilled technicians. In certain clinical scenarios, such as intraoperative consultations, there is a need for faster histologic evaluation to provide real-time surgical guidance. Currently, frozen section techniques involving hematoxylin and eosin (H&E) staining are used for intraoperative pathology consultations. However, these techniques have limitations, including a turnaround time of 20 to 30 minutes, staining artifacts, and potential tissue loss, negatively impacting accurate diagnosis. To address these challenges, researchers are exploring alternative optical imaging modalities for rapid microscopic tissue imaging. These modalities differ in optical characteristics, tissue preparation requirements, imaging equipment, and output image quality and format. Some of these imaging methods have been combined with computational algorithms to generate H&E-like images, which could greatly facilitate their adoption by pathologists. Here, we provide a comprehensive, organ-specific review of the latest advancements in emerging imaging modalities applied to nonfixed human tissue. We focused on studies that generated H&E-like images evaluated by pathologists. By presenting up-to-date research progress and clinical utility, this review serves as a valuable resource for scholars and clinicians, covering some of the major technical developments in this rapidly evolving field. It also offers insights into the potential benefits and drawbacks of alternative imaging modalities and their implications for improving patient care.

Noninvasive virtual biopsy using micro-registered optical coherence tomography (OCT) in human subjects

Histological hematoxylin and eosin-stained (H&E) tissue sections are used as the gold standard for pathologic detection of cancer, tumor margin detection, and disease diagnosis. Producing H&E sections, however, is invasive and time-consuming. While deep learning has shown promise in virtual staining of unstained tissue slides, true virtual biopsy requires staining of images taken from intact tissue. In this work, we developed a micron-accuracy coregistration method [micro-registered optical coherence tomography (OCT)] that can take a two-dimensional (2D) H&E slide and find the exact corresponding section in a 3D OCT image taken from the original fresh tissue. We trained a conditional generative adversarial network using the paired dataset and showed high-fidelity conversion of noninvasive OCT images to virtually stained H&E slices in both 2D and 3D. Applying these trained neural networks to in vivo OCT images should enable physicians to readily incorporate OCT imaging into their clinical practice, reducing the number of unnecessary biopsy procedures.

Exceeding the limit for microscopic image translation with a deep learning-based unified framework

Deep learning algorithms have been widely used in microscopic image translation. The corresponding data-driven models can be trained by supervised or unsupervised learning depending on the availability of paired data. However, general cases are where the data are only roughly paired such that supervised learning could be invalid due to data unalignment, and unsupervised learning would be less ideal as the roughly paired information is not utilized. In this work, we propose a unified framework (U-Frame) that unifies supervised and unsupervised learning by introducing a tolerance size that can be adjusted automatically according to the degree of data misalignment. Together with the implementation of a global sampling rule, we demonstrate that U-Frame consistently outperforms both supervised and unsupervised learning in all levels of data misalignments (even for perfectly aligned image pairs) in a myriad of image translation applications, including pseudo-optical sectioning, virtual histological staining (with clinical evaluations for cancer diagnosis), improvement of signal-to-noise ratio or resolution, and prediction of fluorescent labels, potentially serving as new standard for image translation.

Virtual histological staining of unlabeled autopsy tissue

Traditional histochemical staining of post-mortem samples often confronts inferior staining quality due to autolysis caused by delayed fixation of cadaver tissue, and such chemical staining procedures covering large tissue areas demand substantial labor, cost and time. Here, we demonstrate virtual staining of autopsy tissue using a trained neural network to rapidly transform autofluorescence images of label-free autopsy tissue sections into brightfield equivalent images, matching hematoxylin and eosin (H&E) stained versions of the same samples. The trained model can effectively accentuate nuclear, cytoplasmic and extracellular features in new autopsy tissue samples that experienced severe autolysis, such as COVID-19 samples never seen before, where the traditional histochemical staining fails to provide consistent staining quality. This virtual autopsy staining technique provides a rapid and resource-efficient solution to generate artifact-free H&E stains despite severe autolysis and cell death, also reducing labor, cost and infrastructure requirements associated with the standard histochemical staining.

Dual contrastive learning based image-to-image translation of unstained skin tissue into virtually stained H&E images

Staining is a crucial step in histopathology that prepares tissue sections for microscopic examination. Hematoxylin and eosin (H&E) staining, also known as basic or routine staining, is used in 80% of histopathology slides worldwide. To enhance the histopathology workflow, recent research has focused on integrating generative artificial intelligence and deep learning models. These models have the potential to improve staining accuracy, reduce staining time, and minimize the use of hazardous chemicals, making histopathology a safer and more efficient field. In this study, we introduce a novel three-stage, dual contrastive learning-based, image-to-image generative (DCLGAN) model for virtually applying an "H&E stain" to unstained skin tissue images. The proposed model utilizes a unique learning setting comprising two pairs of generators and discriminators. By employing contrastive learning, our model maximizes the mutual information between traditional H&E-stained and virtually stained H&E patches. Our dataset consists of pairs of unstained and H&E-stained images, scanned with a brightfield microscope at 20 × magnification, providing a comprehensive set of training and testing images for evaluating the efficacy of our proposed model. Two metrics, Fréchet Inception Distance (FID) and Kernel Inception Distance (KID), were used to quantitatively evaluate virtual stained slides. Our analysis revealed that the average FID score between virtually stained and H&E-stained images (80.47) was considerably lower than that between unstained and virtually stained slides (342.01), and unstained and H&E stained (320.4) indicating a similarity virtual and H&E stains. Similarly, the mean KID score between H&E stained and virtually stained images (0.022) was significantly lower than the mean KID score between unstained and H&E stained (0.28) or unstained and virtually stained (0.31) images. In addition, a group of experienced dermatopathologists evaluated traditional and virtually stained images and demonstrated an average agreement of 78.8% and 90.2% for paired and single virtual stained image evaluations, respectively. Our study demonstrates that the proposed three-stage dual contrastive learning-based image-to-image generative model is effective in generating virtual stained images, as indicated by quantified parameters and grader evaluations. In addition, our findings suggest that GAN models have the potential to replace traditional H&E staining, which can reduce both time and environmental impact. This study highlights the promise of virtual staining as a viable alternative to traditional staining techniques in histopathology.

Factors for Improving Diagnosis of Skin Tumors

Background

Although several studies have investigated the accuracy of clinical diagnoses of skin tumors, specific ways to improve diagnostic accuracy have not been identified. This study investigated factors that influence the accuracy of clinical skin tumor diagnostic methods and discusses strategies to improve accuracy.

Methods

Study 1 retrospectively analyzed 657 skin tumors excised at our hospital between March 2001 and March 2011. Data were extracted from surgical records to establish a diagnostic template for further research. Study 2 prospectively applied this template to aid clinical diagnosis at four facilities between April 2011 and March 2013. The clinical diagnoses were compared with the histological ones and the concordance was determined.

Results

A total of 448 and 209 benign and malignant tumors, respectively, were included in Study 1. The overall diagnostic accuracy was 79.0%. In Study 2, 310 patients were clinically diagnosed using a standardized template, which did not affect the diagnostic accuracy. Age, sex, duration of disease, tumor size and location, skin tone, mobility, stiffness, and years of diagnostic experience did not significantly affect diagnostic accuracy. A high proportion of pathologically malignant tumors were clinically misdiagnosed as benign (16/22; 72%). Other clinical examinations were performed in only 35 cases.

Conclusions

Auxiliary diagnostic tools such as dermoscopy and biopsies should be used to accurately diagnose malignant tumors.

Multi-channel feature extraction for virtual histological staining of photon absorption remote sensing images

Accurate and fast histological staining is crucial in histopathology, impacting diagnostic precision and reliability. Traditional staining methods are time-consuming and subjective, causing delays in diagnosis. Digital pathology plays a vital role in advancing and optimizing histology processes to improve efficiency and reduce turnaround times. This study introduces a novel deep learning-based framework for virtual histological staining using photon absorption remote sensing (PARS) images. By extracting features from PARS time-resolved signals using a variant of the K-means method, valuable multi-modal information is captured. The proposed multi-channel cycleGAN model expands on the traditional cycleGAN framework, allowing the inclusion of additional features. Experimental results reveal that specific combinations of features outperform the conventional channels by improving the labeling of tissue structures prior to model training. Applied to human skin and mouse brain tissue, the results underscore the significance of choosing the optimal combination of features, as it reveals a substantial visual and quantitative concurrence between the virtually stained and the gold standard chemically stained hematoxylin and eosin images, surpassing the performance of other feature combinations. Accurate virtual staining is valuable for reliable diagnostic information, aiding pathologists in disease classification, grading, and treatment planning. This study aims to advance label-free histological imaging and opens doors for intraoperative microscopy applications.

A perspective on computer vision in biosensing

Computer vision has become a powerful tool in the field of biosensing, aiding in the development of innovative and precise systems for the analysis and interpretation of biological data. This interdisciplinary approach harnesses the capabilities of computer vision algorithms and techniques to extract valuable information from various biosensing applications, including medical diagnostics, environmental monitoring, and food health. Despite years of development, there is still significant room for improvement in this area. In this perspective, we outline how computer vision is applied to raw sensor data in biosensors and its advantages to biosensing applications. We then discuss ongoing research and developments in the field and subsequently explore the challenges and opportunities that computer vision faces in biosensor applications. We also suggest directions for future work, ultimately underscoring the significant impact of computer vision on advancing biosensing technologies and their applications.

[Non-invasive automated methods for the diagnosis of periorbital skin tumors]

Malignant skin tumors are the most common type of cancer in both Russia and globally. Malignant skin tumors located in the periorbital region, particularly basal cell carcinoma, pose a significant threat to the visual organ due to the high risk of local invasion, highlighting the need for early diagnosis and timely treatment. This review discusses the main methods of non-invasive instrumental diagnosis of skin tumors in the periorbital region. Key stages in the development of these methods are briefly outlined, and their most significant advantages and disadvantages are noted. The article also considers the automation of diagnostic studies, and potential challenges with its practical implementation.

Complete digital pathology transition: A large multi-center experience

INTRODUCTION

Transitioning from glass slide pathology to digital pathology for primary diagnostics requires an appropriate laboratory information system, an image management system, and slide scanners; it also reinforces the need for sophisticated pathology informatics including synoptic reporting. Previous reports have discussed the transition itself and relevant considerations for it, but not the selection criteria and considerations for the infrastructure.

OBJECTIVE

To describe the process used to evaluate slide scanners, image management systems, and synoptic reporting systems for a large multisite institution.

METHODS

Six network hospitals evaluated six slide scanners, three image management systems, and three synoptic reporting systems. Scanners were evaluated based on the quality of image, speed, ease of operation, and special capabilities (including z-stacking, fluorescence and others). Image management and synoptic reporting systems were evaluated for their ease of use and capacity.

RESULTS

Among the scanners evaluated, the Leica GT450 produced the highest quality images, while the 3DHistech Pannoramic provided fluorescence and superior z-stacking. The newest generation of scanners, released relatively recently, performed better than slightly older scanners from major manufacturers Although the Olympus VS200 was not fully vetted due to not meeting all inclusion criteria, it is discussed herein due to its exceptional versatility. For Image Management Software, the authors believe that Sectra is, at the time of writing the best developed option, but this could change in the very near future as other systems improve their capabilities. All synoptic reporting systems performed impressively.

CONCLUSIONS

Specifics regarding quality and abilities of different components will change rapidly with time, but large pathology practices considering such a transition should be aware of the issues discussed and evaluate the most current generation to arrive at appropriate conclusions.

Label- and slide-free tissue histology using 3D epi-mode quantitative phase imaging and virtual hematoxylin and eosin staining

Histological staining of tissue biopsies, especially hematoxylin and eosin (H&E) staining, serves as the benchmark for disease diagnosis and comprehensive clinical assessment of tissue. However, the typical formalin-fixation, paraffin-embedding (FFPE) process is laborious and time consuming, often limiting its usage in time-sensitive applications such as surgical margin assessment. To address these challenges, we combine an emerging 3D quantitative phase imaging technology, termed quantitative oblique back illumination microscopy (qOBM), with an unsupervised generative adversarial network pipeline to map qOBM phase images of unaltered thick tissues (i.e., label- and slide-free) to virtually stained H&E-like (vH&E) images. We demonstrate that the approach achieves high-fidelity conversions to H&E with subcellular detail using fresh tissue specimens from mouse liver, rat gliosarcoma, and human gliomas. We also show that the framework directly enables additional capabilities such as H&E-like contrast for volumetric imaging. The quality and fidelity of the vH&E images are validated using both a neural network classifier trained on real H&E images and tested on virtual H&E images, and a user study with neuropathologists. Given its simple and low-cost embodiment and ability to provide real-time feedback in vivo, this deep-learning-enabled qOBM approach could enable new workflows for histopathology with the potential to significantly save time, labor, and costs in cancer screening, detection, treatment guidance, and more.

Toward cell nuclei precision between OCT and H&E images translation using signal-to-noise ratio cycle-consistency

Medical image-to-image translation is often difficult and of limited effectiveness due to the differences in image acquisition mechanisms and the diverse structure of biological tissues. This work presents an unpaired image translation model between in-vivo optical coherence tomography (OCT) and ex-vivo Hematoxylin and eosin (H&E) stained images without the need for image stacking, registration, post-processing, and annotation. The model can generate high-quality and highly accurate virtual medical images, and is robust and bidirectional. Our framework introduces random noise to (1) blur redundant features, (2) defend against self-adversarial attacks, (3) stabilize inverse conversion, and (4) mitigate the impact of OCT speckles. We also demonstrate that our model can be pre-trained and then fine-tuned using images from different OCT systems in just a few epochs. Qualitative and quantitative comparisons with traditional image-to-image translation models show the robustness of our proposed signal-to-noise ratio (SNR) cycle-consistency method.

Roadmap on Label-Free Super-Resolution Imaging

Label-free super-resolution (LFSR) imaging relies on light-scattering processes in nanoscale objects without a need for fluorescent (FL) staining required in super-resolved FL microscopy. The objectives of this Roadmap are to present a comprehensive vision of the developments, the state-of-the-art in this field, and to discuss the resolution boundaries and hurdles which need to be overcome to break the classical diffraction limit of the LFSR imaging. The scope of this Roadmap spans from the advanced interference detection techniques, where the diffraction-limited lateral resolution is combined with unsurpassed axial and temporal resolution, to techniques with true lateral super-resolution capability which are based on understanding resolution as an information science problem, on using novel structured illumination, near-field scanning, and nonlinear optics approaches, and on designing superlenses based on nanoplasmonics, metamaterials, transformation optics, and microsphere-assisted approaches. To this end, this Roadmap brings under the same umbrella researchers from the physics and biomedical optics communities in which such studies have often been developing separately. The ultimate intent of this paper is to create a vision for the current and future developments of LFSR imaging based on its physical mechanisms and to create a great opening for the series of articles in this field.

Deep learning-enabled realistic virtual histology with ultraviolet photoacoustic remote sensing microscopy

The goal of oncologic surgeries is complete tumor resection, yet positive margins are frequently found postoperatively using gold standard H&E-stained histology methods. Frozen section analysis is sometimes performed for rapid intraoperative margin evaluation, albeit with known inaccuracies. Here, we introduce a label-free histological imaging method based on an ultraviolet photoacoustic remote sensing and scattering microscope, combined with unsupervised deep learning using a cycle-consistent generative adversarial network for realistic virtual staining. Unstained tissues are scanned at rates of up to 7 mins/cm2, at resolution equivalent to 400x digital histopathology. Quantitative validation suggests strong concordance with conventional histology in benign and malignant prostate and breast tissues. In diagnostic utility studies we demonstrate a mean sensitivity and specificity of 0.96 and 0.91 in breast specimens, and respectively 0.87 and 0.94 in prostate specimens. We also find virtual stain quality is preferred (P = 0.03) compared to frozen section analysis in a blinded survey of pathologists.

Digital staining facilitates biomedical microscopy

Traditional staining of biological specimens for microscopic imaging entails time-consuming, laborious, and costly procedures, in addition to producing inconsistent labeling and causing irreversible sample damage. In recent years, computational "virtual" staining using deep learning techniques has evolved into a robust and comprehensive application for streamlining the staining process without typical histochemical staining-related drawbacks. Such virtual staining techniques can also be combined with neural networks designed to correct various microscopy aberrations, such as out-of-focus or motion blur artifacts, and improve upon diffracted-limited resolution. Here, we highlight how such methods lead to a host of new opportunities that can significantly improve both sample preparation and imaging in biomedical microscopy.

Comprehensive Surface Histology of Fresh Resection Margins With Rapid Open-Top Light-Sheet (OTLS) Microscopy

OBJECTIVE

For tumor resections, margin status typically correlates with patient survival but positive margin rates are generally high (up to 45% for head and neck cancer). Frozen section analysis (FSA) is often used to intraoperatively assess the margins of excised tissue, but suffers from severe under-sampling of the actual margin surface, inferior image quality, slow turnaround, and tissue destructiveness.

METHODS

Here, we have developed an imaging workflow to generate en face histologic images of freshly excised surgical margin surfaces based on open-top light-sheet (OTLS) microscopy. Key innovations include (1) the ability to generate false-colored H&E-mimicking images of tissue surfaces stained for < 1 min with a single fluorophore, (2) rapid OTLS surface imaging at a rate of 15 min/cm2 followed by real-time post-processing of datasets within RAM at a rate of 5 min/cm2, and (3) rapid digital surface extraction to account for topological irregularities at the tissue surface.

RESULTS

In addition to the performance metrics listed above, we show that the image quality generated by our rapid surface-histology method approaches that of gold-standard archival histology.

CONCLUSION

OTLS microscopy has the feasibility to provide intraoperative guidance of surgical oncology procedures.

SIGNIFICANCE

The reported methods can potentially improve tumor-resection procedures, thereby improving patient outcomes and quality of life.

Deep learning-enabled virtual histological staining of biological samples

Histological staining is the gold standard for tissue examination in clinical pathology and life-science research, which visualizes the tissue and cellular structures using chromatic dyes or fluorescence labels to aid the microscopic assessment of tissue. However, the current histological staining workflow requires tedious sample preparation steps, specialized laboratory infrastructure, and trained histotechnologists, making it expensive, time-consuming, and not accessible in resource-limited settings. Deep learning techniques created new opportunities to revolutionize staining methods by digitally generating histological stains using trained neural networks, providing rapid, cost-effective, and accurate alternatives to standard chemical staining methods. These techniques, broadly referred to as virtual staining, were extensively explored by multiple research groups and demonstrated to be successful in generating various types of histological stains from label-free microscopic images of unstained samples; similar approaches were also used for transforming images of an already stained tissue sample into another type of stain, performing virtual stain-to-stain transformations. In this Review, we provide a comprehensive overview of the recent research advances in deep learning-enabled virtual histological staining techniques. The basic concepts and the typical workflow of virtual staining are introduced, followed by a discussion of representative works and their technical innovations. We also share our perspectives on the future of this emerging field, aiming to inspire readers from diverse scientific fields to further expand the scope of deep learning-enabled virtual histological staining techniques and their applications.

Diffractive interconnects: all-optical permutation operation using diffractive networks

Permutation matrices form an important computational building block frequently used in various fields including, e.g., communications, information security, and data processing. Optical implementation of permutation operators with relatively large number of input-output interconnections based on power-efficient, fast, and compact platforms is highly desirable. Here, we present diffractive optical networks engineered through deep learning to all-optically perform permutation operations that can scale to hundreds of thousands of interconnections between an input and an output field-of-view using passive transmissive layers that are individually structured at the wavelength scale. Our findings indicate that the capacity of the diffractive optical network in approximating a given permutation operation increases proportional to the number of diffractive layers and trainable transmission elements in the system. Such deeper diffractive network designs can pose practical challenges in terms of physical alignment and output diffraction efficiency of the system. We addressed these challenges by designing misalignment tolerant diffractive designs that can all-optically perform arbitrarily selected permutation operations, and experimentally demonstrated, for the first time, a diffractive permutation network that operates at THz part of the spectrum. Diffractive permutation networks might find various applications in, e.g., security, image encryption, and data processing, along with telecommunications; especially with the carrier frequencies in wireless communications approaching THz-bands, the presented diffractive permutation networks can potentially serve as channel routing and interconnection panels in wireless networks.

Confocal Microscopy for Diagnosis and Management of Cutaneous Malignancies: Clinical Impacts and Innovation

Cutaneous malignancies are common malignancies worldwide, with rising incidence. Most skin cancers, including melanoma, can be cured if diagnosed correctly at an early stage. Thus, millions of biopsies are performed annually, posing a major economic burden. Non-invasive skin imaging techniques can aid in early diagnosis and save unnecessary benign biopsies. In this review article, we will discuss in vivo and ex vivo confocal microscopy (CM) techniques that are currently being utilized in dermatology clinics for skin cancer diagnosis. We will discuss their current applications and clinical impact. Additionally, we will provide a comprehensive review of the advances in the field of CM, including multi-modal approaches, the integration of fluorescent targeted dyes, and the role of artificial intelligence for improved diagnosis and management.

Skin Cancer Pathobiology at a Glance: A Focus on Imaging Techniques and Their Potential for Improved Diagnosis and Surveillance in Clinical Cohorts

Early diagnosis is essential for completely eradicating skin cancer and maximizing patients' clinical benefits. Emerging optical imaging modalities such as reflectance confocal microscopy (RCM), optical coherence tomography (OCT), magnetic resonance imaging (MRI), near-infrared (NIR) bioimaging, positron emission tomography (PET), and their combinations provide non-invasive imaging data that may help in the early detection of cutaneous tumors and surgical planning. Hence, they seem appropriate for observing dynamic processes such as blood flow, immune cell activation, and tumor energy metabolism, which may be relevant for disease evolution. This review discusses the latest technological and methodological advances in imaging techniques that may be applied for skin cancer detection and monitoring. In the first instance, we will describe the principle and prospective clinical applications of the most commonly used imaging techniques, highlighting the challenges and opportunities of their implementation in the clinical setting. We will also highlight how imaging techniques may complement the molecular and histological approaches in sharpening the non-invasive skin characterization, laying the ground for more personalized approaches in skin cancer patients.

Label-Free Virtual HER2 Immunohistochemical Staining of Breast Tissue using Deep Learning

The immunohistochemical (IHC) staining of the human epidermal growth factor receptor 2 (HER2) biomarker is widely practiced in breast tissue analysis, preclinical studies, and diagnostic decisions, guiding cancer treatment and investigation of pathogenesis. HER2 staining demands laborious tissue treatment and chemical processing performed by a histotechnologist, which typically takes one day to prepare in a laboratory, increasing analysis time and associated costs. Here, we describe a deep learning-based virtual HER2 IHC staining method using a conditional generative adversarial network that is trained to rapidly transform autofluorescence microscopic images of unlabeled/label-free breast tissue sections into bright-field equivalent microscopic images, matching the standard HER2 IHC staining that is chemically performed on the same tissue sections. The efficacy of this virtual HER2 staining framework was demonstrated by quantitative analysis, in which three board-certified breast pathologists blindly graded the HER2 scores of virtually stained and immunohistochemically stained HER2 whole slide images (WSIs) to reveal that the HER2 scores determined by inspecting virtual IHC images are as accurate as their immunohistochemically stained counterparts. A second quantitative blinded study performed by the same diagnosticians further revealed that the virtually stained HER2 images exhibit a comparable staining quality in the level of nuclear detail, membrane clearness, and absence of staining artifacts with respect to their immunohistochemically stained counterparts. This virtual HER2 staining framework bypasses the costly, laborious, and time-consuming IHC staining procedures in laboratory and can be extended to other types of biomarkers to accelerate the IHC tissue staining used in life sciences and biomedical workflow.

Flexible method for generating needle-shaped beams and its application in optical coherence tomography

Needle-shaped beams (NBs) featuring a long depth-of-focus (DOF) can drastically improve the resolution of microscopy systems. However, thus far, the implementation of a specific NB has been onerous due to the lack of a common, flexible generation method. Here we develop a spatially multiplexed phase pattern that creates many axially closely spaced foci as a universal platform for customizing various NBs, allowing flexible manipulations of beam length and diameter, uniform axial intensity, and sub-diffraction-limit beams. NBs designed via this method successfully extended the DOF of our optical coherence tomography (OCT) system. It revealed clear individual epidermal cells of the entire human epidermis, fine structures of human dermal-epidermal junction in a large depth range, and a high-resolution dynamic heartbeat of alive Drosophila larvae.

Quantitative phase imaging through an ultra-thin lensless fiber endoscope

Quantitative phase imaging (QPI) is a label-free technique providing both morphology and quantitative biophysical information in biomedicine. However, applying such a powerful technique to in vivo pathological diagnosis remains challenging. Multi-core fiber bundles (MCFs) enable ultra-thin probes for in vivo imaging, but current MCF imaging techniques are limited to amplitude imaging modalities. We demonstrate a computational lensless microendoscope that uses an ultra-thin bare MCF to perform quantitative phase imaging with microscale lateral resolution and nanoscale axial sensitivity of the optical path length. The incident complex light field at the measurement side is precisely reconstructed from the far-field speckle pattern at the detection side, enabling digital refocusing in a multi-layer sample without any mechanical movement. The accuracy of the quantitative phase reconstruction is validated by imaging the phase target and hydrogel beads through the MCF. With the proposed imaging modality, three-dimensional imaging of human cancer cells is achieved through the ultra-thin fiber endoscope, promising widespread clinical applications.

Virtual histological staining of label-free total absorption photoacoustic remote sensing (TA-PARS)

Histopathological visualizations are a pillar of modern medicine and biological research. Surgical oncology relies exclusively on post-operative histology to determine definitive surgical success and guide adjuvant treatments. The current histology workflow is based on bright-field microscopic assessment of histochemical stained tissues and has some major limitations. For example, the preparation of stained specimens for brightfield assessment requires lengthy sample processing, delaying interventions for days or even weeks. Therefore, there is a pressing need for improved histopathology methods. In this paper, we present a deep-learning-based approach for virtual label-free histochemical staining of total-absorption photoacoustic remote sensing (TA-PARS) images of unstained tissue. TA-PARS provides an array of directly measured label-free contrasts such as scattering and total absorption (radiative and non-radiative), ideal for developing H&E colorizations without the need to infer arbitrary tissue structures. We use a Pix2Pix generative adversarial network to develop visualizations analogous to H&E staining from label-free TA-PARS images. Thin sections of human skin tissue were first virtually stained with the TA-PARS, then were chemically stained with H&E producing a one-to-one comparison between the virtual and chemical staining. The one-to-one matched virtually- and chemically- stained images exhibit high concordance validating the digital colorization of the TA-PARS images against the gold standard H&E. TA-PARS images were reviewed by four dermatologic pathologists who confirmed they are of diagnostic quality, and that resolution, contrast, and color permitted interpretation as if they were H&E. The presented approach paves the way for the development of TA-PARS slide-free histological imaging, which promises to dramatically reduce the time from specimen resection to histological imaging.

Fourier ptychography multi-parameunter neural network with composite physical priori optimization

Fourier ptychography microscopy(FPM) is a recently developed computational imaging approach for microscopic super-resolution imaging. By turning on each light-emitting-diode (LED) located on different position on the LED array sequentially and acquiring the corresponding images that contain different spatial frequency components, high spatial resolution and quantitative phase imaging can be achieved in the case of large field-of-view. Nevertheless, FPM has high requirements for the system construction and data acquisition processes, such as precise LEDs position, accurate focusing and appropriate exposure time, which brings many limitations to its practical applications. In this paper, inspired by artificial neural network, we propose a Fourier ptychography multi-parameter neural network (FPMN) with composite physical prior optimization. A hybrid parameter determination strategy combining physical imaging model and data-driven network training is proposed to recover the multi layers of the network corresponding to different physical parameters, including sample complex function, system pupil function, defocus distance, LED array position deviation and illumination intensity fluctuation, etc. Among these parameters, LED array position deviation is recovered based on the features of brightfield to darkfield transition low-resolution images while the others are recovered in the process of training of the neural network. The feasibility and effectiveness of FPMN are verified through simulations and actual experiments. Therefore FPMN can evidently reduce the requirement for practical applications of FPM.

Horizontal Histopathology Correlation with In Vivo Reflectance Confocal Microscopy in Inflammatory Skin Diseases: A Review

Horizontal histopathological sections (HHSs) have been reported to show a strong correlation with images obtained via in vivo reflectance confocal microscopy (RCM), as both reflect the same horizontal plane of the skin. Although vertical histopathology remains the diagnostic gold standard for most neoplastic and inflammatory skin diseases, HHSs represent a useful tool to validate the RCM features of some inflammatory disorders, including psoriasis, discoid lupus erythematosus, and eczema. The aim of the present review is to summarize the state of the art on the existing correlations between HHS and RCM in this field and to emphasize that RCM may represent a useful diagnostic tool to discriminate between diseases with similar clinical presentations.