End-to-End Adversarial Retinal Image Synthesis

AI image generation technology in ophthalmology: Use, misuse and future applications

BACKGROUND

AI-powered image generation technology holds the potential to reshape medical practice, yet it remains an unfamiliar technology for both medical researchers and clinicians alike. Given the adoption of this technology relies on clinician understanding and acceptance, we sought to demystify its use in ophthalmology. To this end, we present a literature review on image generation technology in ophthalmology, examining both its theoretical applications and future role in clinical practice.

METHODS

First, we consider the key model designs used for image synthesis, including generative adversarial networks, autoencoders, and diffusion models. We then perform a survey of the literature for image generation technology in ophthalmology prior to September 2024, presenting both the type of model used and its clinical application. Finally, we discuss the limitations of this technology, the risks of its misuse and the future directions of research in this field.

RESULTS

Applications of this technology include improving AI diagnostic models, inter-modality image transformation, more accurate treatment and disease prognostication, image denoising, and individualised education. Key barriers to its adoption include bias in generative models, risks to patient data security, computational and logistical barriers to development, challenges with model explainability, inconsistent use of validation metrics between studies and misuse of synthetic images. Looking forward, researchers are placing a further emphasis on clinically grounded metrics, the development of image generation foundation models and the implementation of methods to ensure data provenance.

CONCLUSION

Compared to other medical applications of AI, image generation is still in its infancy. Yet, it holds the potential to revolutionise ophthalmology across research, education and clinical practice. This review aims to guide ophthalmic researchers wanting to leverage this technology, while also providing an insight for clinicians on how it may change ophthalmic practice in the future.

Unsupervised translation of vascular masks to NIR-II fluorescence images using Attention-Guided generative adversarial networks

The second near-infrared window (NIR-II) fluorescence imaging is a crucial technology for investigating the structure and functionality of blood vessels. However, challenges arise from privacy concerns and the significant effort needed for data annotation, complicating the acquisition of near-infrared vascular imaging datasets. To tackle these issues, methods based on deep learning for data synthesis have demonstrated promise in generating high-quality synthetic images. In this paper, we propose an unsupervised generative adversarial network (GAN) approach for translating vascular masks into realistic NIR-II fluorescence vascular images. Leveraging an attention mechanism integrated into the loss function, our model focuses on essential features during the generation process, resulting in high-quality NIRII images without the need for supervision. Our method significantly outperforms eight baseline techniques in both visual quality and quantitative metrics, demonstrating its potential to address the challenge of limited datasets in NIR-II medical imaging. This work not only enhances the applications of NIR-II imaging but also facilitates downstream tasks by providing abundant, high-fidelity synthetic data.

Generative artificial intelligence in ophthalmology

Generative artificial intelligence (AI) has revolutionized medicine over the past several years. A generative adversarial network (GAN) is a deep learning framework that has become a powerful technique in medicine, particularly in ophthalmology for image analysis. In this paper we review the current ophthalmic literature involving GANs, and highlight key contributions in the field. We briefly touch on ChatGPT, another application of generative AI, and its potential in ophthalmology. We also explore the potential uses for GANs in ocular imaging, with a specific emphasis on 3 primary domains: image enhancement, disease identification, and generating of synthetic data. PubMed, Ovid MEDLINE, Google Scholar were searched from inception to October 30, 2022, to identify applications of GAN in ophthalmology. A total of 40 papers were included in this review. We cover various applications of GANs in ophthalmic-related imaging including optical coherence tomography, orbital magnetic resonance imaging, fundus photography, and ultrasound; however, we also highlight several challenges that resulted in the generation of inaccurate and atypical results during certain iterations. Finally, we examine future directions and considerations for generative AI in ophthalmology.

Standardization and clinical applications of retinal imaging biomarkers for cardiovascular disease: a Roadmap from an NHLBI workshop

The accessibility of the retina with the use of non-invasive and relatively low-cost ophthalmic imaging techniques and analytics provides a unique opportunity to improve the detection, diagnosis and monitoring of systemic diseases. The National Heart, Lung, and Blood Institute conducted a workshop in October 2022 to examine this concept. On the basis of the discussions at that workshop, this Roadmap describes current knowledge gaps and new research opportunities to evaluate the relationships between the eye (in particular, retinal biomarkers) and the risk of cardiovascular diseases, including coronary artery disease, heart failure, stroke, hypertension and vascular dementia. Identified gaps include the need to simplify and standardize the capture of high-quality images of the eye by non-ophthalmic health workers and to conduct longitudinal studies using multidisciplinary networks of diverse at-risk populations with improved implementation and methods to protect participant and dataset privacy. Other gaps include improving the measurement of structural and functional retinal biomarkers, determining the relationship between microvascular and macrovascular risk factors, improving multimodal imaging 'pipelines', and integrating advanced imaging with 'omics', lifestyle factors, primary care data and radiological reports, by using artificial intelligence technology to improve the identification of individual-level risk. Future research on retinal microvascular disease and retinal biomarkers might additionally provide insights into the temporal development of microvascular disease across other systemic vascular beds.

Medical Image Segmentation Review: The Success of U-Net

Automatic medical image segmentation is a crucial topic in the medical domain and successively a critical counterpart in the computer-aided diagnosis paradigm. U-Net is the most widespread image segmentation architecture due to its flexibility, optimized modular design, and success in all medical image modalities. Over the years, the U-Net model has received tremendous attention from academic and industrial researchers who have extended it to address the scale and complexity created by medical tasks. These extensions are commonly related to enhancing the U-Net's backbone, bottleneck, or skip connections, or including representation learning, or combining it with a Transformer architecture, or even addressing probabilistic prediction of the segmentation map. Having a compendium of different previously proposed U-Net variants makes it easier for machine learning researchers to identify relevant research questions and understand the challenges of the biological tasks that challenge the model. In this work, we discuss the practical aspects of the U-Net model and organize each variant model into a taxonomy. Moreover, to measure the performance of these strategies in a clinical application, we propose fair evaluations of some unique and famous designs on well-known datasets. Furthermore, we provide a comprehensive implementation library with trained models. In addition, for ease of future studies, we created an online list of U-Net papers with their possible official implementation.

Vision language models in ophthalmology

PURPOSE OF REVIEW

Vision Language Models are an emerging paradigm in artificial intelligence that offers the potential to natively analyze both image and textual data simultaneously, within a single model. The fusion of these two modalities is of particular relevance to ophthalmology, which has historically involved specialized imaging techniques such as angiography, optical coherence tomography, and fundus photography, while also interfacing with electronic health records that include free text descriptions. This review then surveys the fast-evolving field of Vision Language Models as they apply to current ophthalmologic research and practice.

RECENT FINDINGS

Although models incorporating both image and text data have a long provenance in ophthalmology, effective multimodal Vision Language Models are a recent development exploiting advances in technologies such as transformer and autoencoder models.

SUMMARY

Vision Language Models offer the potential to assist and streamline the existing clinical workflow in ophthalmology, whether previsit, during, or post-visit. There are, however, also important challenges to be overcome, particularly regarding patient privacy and explainability of model recommendations.

Adversarial robustness improvement for X-ray bone segmentation using synthetic data created from computed tomography scans

Deep learning-based image analysis offers great potential in clinical practice. However, it faces mainly two challenges: scarcity of large-scale annotated clinical data for training and susceptibility to adversarial data in inference. As an example, an artificial intelligence (AI) system could check patient positioning, by segmenting and evaluating relative positions of anatomical structures in medical images. Nevertheless, data to train such AI system might be highly imbalanced with mostly well-positioned images being available. Thus, we propose the use of synthetic X-ray images and annotation masks forward projected from 3D photon-counting CT volumes to create realistic non-optimally positioned X-ray images for training. An open-source model (TotalSegmentator) was used to annotate the clavicles in 3D CT volumes. We evaluated model robustness with respect to the internal (simulated) patient rotation α on real-data-trained models and real&synthetic-data-trained models. Our results showed that real&synthetic- data-trained models have Dice score percentage improvements of 3% to 15% across different α groups compared to the real-data-trained model. Therefore, we demonstrated that synthetic data could be supplementary used to train and enrich heavily underrepresented conditions to increase model robustness.

AI-ADC: Channel and Spatial Attention-Based Contrastive Learning to Generate ADC Maps from T2W MRI for Prostate Cancer Detection

BACKGROUND/OBJECTIVES

Apparent Diffusion Coefficient (ADC) maps in prostate MRI can reveal tumor characteristics, but their accuracy can be compromised by artifacts related with patient motion or rectal gas associated distortions. To address these challenges, we propose a novel approach that utilizes a Generative Adversarial Network to synthesize ADC maps from T2-weighted magnetic resonance images (T2W MRI).

METHODS

By leveraging contrastive learning, our model accurately maps axial T2W MRI to ADC maps within the cropped region of the prostate organ boundary, capturing subtle variations and intricate structural details by learning similar and dissimilar pairs from two imaging modalities. We trained our model on a comprehensive dataset of unpaired T2-weighted images and ADC maps from 506 patients. In evaluating our model, named AI-ADC, we compared it against three state-of-the-art methods: CycleGAN, CUT, and StyTr2.

RESULTS

Our model demonstrated a higher mean Structural Similarity Index (SSIM) of 0.863 on a test dataset of 3240 2D MRI slices from 195 patients, compared to values of 0.855, 0.797, and 0.824 for CycleGAN, CUT, and StyTr2, respectively. Similarly, our model achieved a significantly lower Fréchet Inception Distance (FID) value of 31.992, compared to values of 43.458, 179.983, and 58.784 for the other three models, indicating its superior performance in generating ADC maps. Furthermore, we evaluated our model on 147 patients from the publicly available ProstateX dataset, where it demonstrated a higher SSIM of 0.647 and a lower FID of 113.876 compared to the other three models.

CONCLUSIONS

These results highlight the efficacy of our proposed model in generating ADC maps from T2W MRI, showcasing its potential for enhancing clinical diagnostics and radiological workflows.

Diverse Data Generation for Retinal Layer Segmentation With Potential Structure Modeling

Accurate retinal layer segmentation on optical coherence tomography (OCT) images is hampered by the challenges of collecting OCT images with diverse pathological characterization and balanced distribution. Current generative models can produce high-realistic images and corresponding labels without quantitative limitations by fitting distributions of real collected data. Nevertheless, the diversity of their generated data is still limited due to the inherent imbalance of training data. To address these issues, we propose an image-label pair generation framework that generates diverse and balanced potential data from imbalanced real samples. Specifically, the framework first generates diverse layer masks, and then generates plausible OCT images corresponding to these layer masks using two customized diffusion probabilistic models respectively. To learn from imbalanced data and facilitate balanced generation, we introduce pathological-related conditions to guide the generation processes. To enhance the diversity of the generated image-label pairs, we propose a potential structure modeling technique that transfers the knowledge of diverse sub-structures from lowly- or non-pathological samples to highly pathological samples. We conducted extensive experiments on two public datasets for retinal layer segmentation. Firstly, our method generates OCT images with higher image quality and diversity compared to other generative methods. Furthermore, based on the extensive training with the generated OCT images, downstream retinal layer segmentation tasks demonstrate improved results. The code is publicly available at: https://github.com/nicetomeetu21/GenPSM.

Skeleton-guided multi-scale dual-coordinate attention aggregation network for retinal blood vessel segmentation

Deep learning plays a pivotal role in retinal blood vessel segmentation for medical diagnosis. Despite their significant efficacy, these techniques face two major challenges. Firstly, they often neglect the severe class imbalance in fundus images, where thin vessels in the foreground are proportionally minimal. Secondly, they are susceptible to poor image quality and blurred vessel edges, resulting in discontinuities or breaks in vascular structures. In response, this paper proposes the Skeleton-guided Multi-scale Dual-coordinate Attention Aggregation (SMDAA) network for retinal vessel segmentation. SMDAA comprises three innovative modules: Dual-coordinate Attention (DCA), Unbalanced Pixel Amplifier (UPA), and Vessel Skeleton Guidance (VSG). DCA, integrating Multi-scale Coordinate Feature Aggregation (MCFA) and Scale Coordinate Attention Decoding (SCAD), meticulously analyzes vessel structures across various scales, adept at capturing intricate details, thereby significantly enhancing segmentation accuracy. To address class imbalance, we introduce UPA, dynamically allocating more attention to misclassified pixels, ensuring precise extraction of thin and small blood vessels. Moreover, to preserve vessel structure continuity, we integrate vessel anatomy and develop the VSG module to connect fragmented vessel segments. Additionally, a Feature-level Contrast (FCL) loss is introduced to capture subtle nuances within the same category, enhancing the fidelity of retinal blood vessel segmentation. Extensive experiments on three public datasets (DRIVE, STARE, and CHASE_DB1) demonstrate superior performance in comparison to current methods. The code is available at https://github.com/wangwxr/SMDAA_NET.

Histopathology in focus: a review on explainable multi-modal approaches for breast cancer diagnosis

Precision and timeliness in breast cancer detection are paramount for improving patient outcomes. Traditional diagnostic methods have predominantly relied on unimodal approaches, but recent advancements in medical data analytics have enabled the integration of diverse data sources beyond conventional imaging techniques. This review critically examines the transformative potential of integrating histopathology images with genomic data, clinical records, and patient histories to enhance diagnostic accuracy and comprehensiveness in multi-modal diagnostic techniques. It explores early, intermediate, and late fusion methods, as well as advanced deep multimodal fusion techniques, including encoder-decoder architectures, attention-based mechanisms, and graph neural networks. An overview of recent advancements in multimodal tasks such as Visual Question Answering (VQA), report generation, semantic segmentation, and cross-modal retrieval is provided, highlighting the utilization of generative AI and visual language models. Additionally, the review delves into the role of Explainable Artificial Intelligence (XAI) in elucidating the decision-making processes of sophisticated diagnostic algorithms, emphasizing the critical need for transparency and interpretability. By showcasing the importance of explainability, we demonstrate how XAI methods, including Grad-CAM, SHAP, LIME, trainable attention, and image captioning, enhance diagnostic precision, strengthen clinician confidence, and foster patient engagement. The review also discusses the latest XAI developments, such as X-VARs, LeGrad, LangXAI, LVLM-Interpret, and ex-ILP, to demonstrate their potential utility in multimodal breast cancer detection, while identifying key research gaps and proposing future directions for advancing the field.

Creating realistic anterior segment optical coherence tomography images using generative adversarial networks

AIMS

To develop a generative adversarial network (GAN) capable of generating realistic high-resolution anterior segment optical coherence tomography (AS-OCT) images.

METHODS

This study included 142 628 AS-OCT B-scans from the American University of Beirut Medical Center. The Style and WAvelet based GAN architecture was trained to generate realistic AS-OCT images and was evaluated through the Fréchet Inception Distance (FID) Score and a blinded assessment by three refractive surgeons who were asked to distinguish between real and generated images. To assess the suitability of the generated images for machine learning tasks, a convolutional neural network (CNN) was trained using a dataset of real and generated images over a classification task. The generated AS-OCT images were then upsampled using an enhanced super-resolution GAN (ESRGAN) to achieve high resolution.

RESULTS

The generated images exhibited visual and quantitative similarity to real AS-OCT images. Quantitative similarity assessed using FID scored an average of 6.32. Surgeons scored 51.7% in identifying real versus generated images which was not significantly better than chance (p value >0.3). The CNN accuracy improved from 78% to 100% when synthetic images were added to the dataset. The ESRGAN upsampled images were objectively more realistic and accurate compared with traditional upsampling techniques by scoring a lower Learned Perceptual Image Patch Similarity of 0.0905 compared with 0.4244 of bicubic interpolation.

CONCLUSIONS

This study successfully developed and leveraged GANs capable of generating high-definition synthetic AS-OCT images that are realistic and suitable for machine learning and image analysis tasks.

VSG-GAN: A high-fidelity image synthesis method with semantic manipulation in retinal fundus image

In recent years, advancements in retinal image analysis, driven by machine learning and deep learning techniques, have enhanced disease detection and diagnosis through automated feature extraction. However, challenges persist, including limited data set diversity due to privacy concerns and imbalanced sample pairs, hindering effective model training. To address these issues, we introduce the vessel and style guided generative adversarial network (VSG-GAN), an innovative algorithm building upon the foundational concept of GAN. In VSG-GAN, a generator and discriminator engage in an adversarial process to produce realistic retinal images. Our approach decouples retinal image generation into distinct modules: the vascular skeleton and background style. Leveraging style transformation and GAN inversion, our proposed hierarchical variational autoencoder module generates retinal images with diverse morphological traits. In addition, the spatially adaptive denormalization module ensures consistency between input and generated images. We evaluate our model on MESSIDOR and RITE data sets using various metrics, including structural similarity index measure, inception score, Fréchet inception distance, and kernel inception distance. Our results demonstrate the superiority of VSG-GAN, outperforming existing methods across all evaluation assessments. This underscores its effectiveness in addressing data set limitations and imbalances. Our algorithm provides a novel solution to challenges in retinal image analysis by offering diverse and realistic retinal image generation. Implementing the VSG-GAN augmentation approach on downstream diabetic retinopathy classification tasks has shown enhanced disease diagnosis accuracy, further advancing the utility of machine learning in this domain.

Generating Synthetic Data for Medical Imaging

Artificial intelligence (AI) models for medical imaging tasks, such as classification or segmentation, require large and diverse datasets of images. However, due to privacy and ethical issues, as well as data sharing infrastructure barriers, these datasets are scarce and difficult to assemble. Synthetic medical imaging data generated by AI from existing data could address this challenge by augmenting and anonymizing real imaging data. In addition, synthetic data enable new applications, including modality translation, contrast synthesis, and professional training for radiologists. However, the use of synthetic data also poses technical and ethical challenges. These challenges include ensuring the realism and diversity of the synthesized images while keeping data unidentifiable, evaluating the performance and generalizability of models trained on synthetic data, and high computational costs. Since existing regulations are not sufficient to guarantee the safe and ethical use of synthetic images, it becomes evident that updated laws and more rigorous oversight are needed. Regulatory bodies, physicians, and AI developers should collaborate to develop, maintain, and continually refine best practices for synthetic data. This review aims to provide an overview of the current knowledge of synthetic data in medical imaging and highlights current key challenges in the field to guide future research and development.

Development of oculomics artificial intelligence for cardiovascular risk factors: A case study in fundus oculomics for HbA1c assessment and clinically relevant considerations for clinicians

Artificial Intelligence (AI) is transforming healthcare, notably in ophthalmology, where its ability to interpret images and data can significantly enhance disease diagnosis and patient care. Recent developments in oculomics, the integration of ophthalmic features to develop biomarkers for systemic diseases, have demonstrated the potential for providing rapid, non-invasive methods of screening leading to enhance in early detection and improve healthcare quality, particularly in underserved areas. However, the widespread adoption of such AI-based technologies faces challenges primarily related to the trustworthiness of the system. We demonstrate the potential and considerations needed to develop trustworthy AI in oculomics through a pilot study for HbA1c assessment using an AI-based approach. We then discuss various challenges, considerations, and solutions that have been developed for powerful AI technologies in the past in healthcare and subsequently apply these considerations to the oculomics pilot study. Building upon the observations in the study we highlight the challenges and opportunities for advancing trustworthy AI in oculomics. Ultimately, oculomics presents as a powerful and emerging technology in ophthalmology and understanding how to optimize transparency prior to clinical adoption is of utmost importance.

Multi-scale consistent self-training network for semi-supervised orbital tumor segmentation

PURPOSE

Segmentation of orbital tumors in CT images is of great significance for orbital tumor diagnosis, which is one of the most prevalent diseases of the eye. However, the large variety of tumor sizes and shapes makes the segmentation task very challenging, especially when the available annotation data is limited.

METHODS

To this end, in this paper, we propose a multi-scale consistent self-training network (MSCINet) for semi-supervised orbital tumor segmentation. Specifically, we exploit the semantic-invariance features by enforcing the consistency between the predictions of different scales of the same image to make the model more robust to size variation. Moreover, we incorporate a new self-training strategy, which adopts iterative training with an uncertainty filtering mechanism to filter the pseudo-labels generated by the model, to eliminate the accumulation of pseudo-label error predictions and increase the generalization of the model.

RESULTS

For evaluation, we have built two datasets, the orbital tumor binary segmentation dataset (Orbtum-B) and the orbital multi-organ segmentation dataset (Orbtum-M). Experimental results on these two datasets show that our proposed method can both achieve state-of-the-art performance. In our datasets, there are a total of 55 patients containing 602 2D images.

CONCLUSION

In this paper, we develop a new semi-supervised segmentation method for orbital tumors, which is designed for the characteristics of orbital tumors and exhibits excellent performance compared to previous semi-supervised algorithms.

Patient-specific placental vessel segmentation with limited data

A major obstacle in applying machine learning for medical fields is the disparity between the data distribution of the training images and the data encountered in clinics. This phenomenon can be explained by inconsistent acquisition techniques and large variations across the patient spectrum. The result is poor translation of the trained models to the clinic, which limits their implementation in medical practice. Patient-specific trained networks could provide a potential solution. Although patient-specific approaches are usually infeasible because of the expenses associated with on-the-fly labeling, the use of generative adversarial networks enables this approach. This study proposes a patient-specific approach based on generative adversarial networks. In the presented training pipeline, the user trains a patient-specific segmentation network with extremely limited data which is supplemented with artificial samples generated by generative adversarial models. This approach is demonstrated in endoscopic video data captured during fetoscopic laser coagulation, a procedure used for treating twin-to-twin transfusion syndrome by ablating the placental blood vessels. Compared to a standard deep learning segmentation approach, the pipeline was able to achieve an intersection over union score of 0.60 using only 20 annotated images compared to 100 images using a standard approach. Furthermore, training with 20 annotated images without the use of the pipeline achieves an intersection over union score of 0.30, which, therefore, corresponds to a 100% increase in performance when incorporating the pipeline. A pipeline using GANs was used to generate artificial data which supplements the real data, this allows patient-specific training of a segmentation network. We show that artificial images generated using GANs significantly improve performance in vessel segmentation and that training patient-specific models can be a viable solution to bring automated vessel segmentation to the clinic.

Synthetic Optical Coherence Tomography Angiographs for Detailed Retinal Vessel Segmentation Without Human Annotations

Optical coherence tomography angiography (OCTA) is a non-invasive imaging modality that can acquire high-resolution volumes of the retinal vasculature and aid the diagnosis of ocular, neurological and cardiac diseases. Segmenting the visible blood vessels is a common first step when extracting quantitative biomarkers from these images. Classical segmentation algorithms based on thresholding are strongly affected by image artifacts and limited signal-to-noise ratio. The use of modern, deep learning-based segmentation methods has been inhibited by a lack of large datasets with detailed annotations of the blood vessels. To address this issue, recent work has employed transfer learning, where a segmentation network is trained on synthetic OCTA images and is then applied to real data. However, the previously proposed simulations fail to faithfully model the retinal vasculature and do not provide effective domain adaptation. Because of this, current methods are unable to fully segment the retinal vasculature, in particular the smallest capillaries. In this work, we present a lightweight simulation of the retinal vascular network based on space colonization for faster and more realistic OCTA synthesis. We then introduce three contrast adaptation pipelines to decrease the domain gap between real and artificial images. We demonstrate the superior segmentation performance of our approach in extensive quantitative and qualitative experiments on three public datasets that compare our method to traditional computer vision algorithms and supervised training using human annotations. Finally, we make our entire pipeline publicly available, including the source code, pretrained models, and a large dataset of synthetic OCTA images.

Optical coherence tomography choroidal enhancement using generative deep learning

Spectral-domain optical coherence tomography (SDOCT) is the gold standard of imaging the eye in clinics. Penetration depth with such devices is, however, limited and visualization of the choroid, which is essential for diagnosing chorioretinal disease, remains limited. Whereas swept-source OCT (SSOCT) devices allow for visualization of the choroid these instruments are expensive and availability in praxis is limited. We present an artificial intelligence (AI)-based solution to enhance the visualization of the choroid in OCT scans and allow for quantitative measurements of choroidal metrics using generative deep learning (DL). Synthetically enhanced SDOCT B-scans with improved choroidal visibility were generated, leveraging matching images to learn deep anatomical features during the training. Using a single-center tertiary eye care institution cohort comprising a total of 362 SDOCT-SSOCT paired subjects, we trained our model with 150,784 images from 410 healthy, 192 glaucoma, and 133 diabetic retinopathy eyes. An independent external test dataset of 37,376 images from 146 eyes was deployed to assess the authenticity and quality of the synthetically enhanced SDOCT images. Experts' ability to differentiate real versus synthetic images was poor (47.5% accuracy). Measurements of choroidal thickness, area, volume, and vascularity index, from the reference SSOCT and synthetically enhanced SDOCT, showed high Pearson's correlations of 0.97 [95% CI: 0.96-0.98], 0.97 [0.95-0.98], 0.95 [0.92-0.98], and 0.87 [0.83-0.91], with intra-class correlation values of 0.99 [0.98-0.99], 0.98 [0.98-0.99], and 0.95 [0.96-0.98], 0.93 [0.91-0.95], respectively. Thus, our DL generative model successfully generated realistic enhanced SDOCT data that is indistinguishable from SSOCT images providing improved visualization of the choroid. This technology enabled accurate measurements of choroidal metrics previously limited by the imaging depth constraints of SDOCT. The findings open new possibilities for utilizing affordable SDOCT devices in studying the choroid in both healthy and pathological conditions.

Cross2SynNet: cross-device-cross-modal synthesis of routine brain MRI sequences from CT with brain lesion

OBJECTIVES

CT and MR are often needed to determine the location and extent of brain lesions collectively to improve diagnosis. However, patients with acute brain diseases cannot complete the MRI examination within a short time. The aim of the study is to devise a cross-device and cross-modal medical image synthesis (MIS) method Cross2SynNet for synthesizing routine brain MRI sequences of T1WI, T2WI, FLAIR, and DWI from CT with stroke and brain tumors.

MATERIALS AND METHODS

For the retrospective study, the participants covered four different diseases of cerebral ischemic stroke (CIS-cohort), cerebral hemorrhage (CH-cohort), meningioma (M-cohort), glioma (G-cohort). The MIS model Cross2SynNet was established on the basic architecture of conditional generative adversarial network (CGAN), of which, the fully convolutional Transformer (FCT) module was adopted into generator to capture the short- and long-range dependencies between healthy and pathological tissues, and the edge loss function was to minimize the difference in gradient magnitude between synthetic image and ground truth. Three metrics of mean square error (MSE), peak signal-to-noise ratio (PSNR), and structure similarity index measure (SSIM) were used for evaluation.

RESULTS

A total of 230 participants (mean patient age, 59.77 years ± 13.63 [standard deviation]; 163 men [71%] and 67 women [29%]) were included, including CIS-cohort (95 participants between Dec 2019 and Feb 2022), CH-cohort (69 participants between Jan 2020 and Dec 2021), M-cohort (40 participants between Sep 2018 and Dec 2021), and G-cohort (26 participants between Sep 2019 and Dec 2021). The Cross2SynNet achieved averaged values of MSE = 0.008, PSNR = 21.728, and SSIM = 0.758 when synthesizing MRIs from CT, outperforming the CycleGAN, pix2pix, RegGAN, Pix2PixHD, and ResViT. The Cross2SynNet could synthesize the brain lesion on pseudo DWI even if the CT image did not exhibit clear signal in the acute ischemic stroke patients.

CONCLUSIONS

Cross2SynNet could achieve routine brain MRI synthesis of T1WI, T2WI, FLAIR, and DWI from CT with promising performance given the brain lesion of stroke and brain tumor.

Applications of artificial intelligence in the utilisation of imaging modalities in dentistry: A systematic review and meta-analysis of in-vitro studies

Background

In the past, dentistry heavily relied on manual image analysis and diagnostic procedures, which could be time-consuming and prone to human error. The advent of artificial intelligence (AI) has brought transformative potential to the field, promising enhanced accuracy and efficiency in various dental imaging tasks. This systematic review and meta-analysis aimed to comprehensively evaluate the applications of AI in dental imaging modalities, focusing on in-vitro studies.

Methods

A systematic literature search was conducted, in accordance with the PRISMA guidelines. The following databases were systematically searched: PubMed/MEDLINE, Embase, Web of Science, Scopus, IEEE Xplore, Cochrane Library, CINAHL (Cumulative Index to Nursing and Allied Health Literature), and Google Scholar. The meta-analysis employed fixed-effects models to assess AI accuracy, calculating odds ratios (OR) for true positive rate (TPR), true negative rate (TNR), positive predictive value (PPV), and negative predictive value (NPV) with 95 % confidence intervals (CI). Heterogeneity and overall effect tests were applied to ensure the reliability of the findings.

Results

9 studies were selected that encompassed various objectives, such as tooth segmentation and classification, caries detection, maxillofacial bone segmentation, and 3D surface model creation. AI techniques included convolutional neural networks (CNNs), deep learning algorithms, and AI-driven tools. Imaging parameters assessed in these studies were specific to the respective dental tasks. The analysis of combined ORs indicated higher odds of accurate dental image assessments, highlighting the potential for AI to improve TPR, TNR, PPV, and NPV. The studies collectively revealed a statistically significant overall effect in favor of AI in dental imaging applications.

Conclusion

In summary, this systematic review and meta-analysis underscore the transformative impact of AI on dental imaging. AI has the potential to revolutionize the field by enhancing accuracy, efficiency, and time savings in various dental tasks. While further research in clinical settings is needed to validate these findings and address study limitations, the future implications of integrating AI into dental practice hold great promise for advancing patient care and the field of dentistry.

Volumetric Imitation Generative Adversarial Networks for Anatomical Human Body Modeling

Volumetric representation is a technique used to express 3D objects in various fields, such as medical applications. On the other hand, tomography images for reconstructing volumetric data have limited utilization because they contain personal information. Existing GAN-based medical image generation techniques can produce virtual tomographic images for volume reconstruction while preserving the patient's privacy. Nevertheless, these images often do not consider vertical correlations between the adjacent slices, leading to erroneous results in 3D reconstruction. Furthermore, while volume generation techniques have been introduced, they often focus on surface modeling, making it challenging to represent the internal anatomical features accurately. This paper proposes volumetric imitation GAN (VI-GAN), which imitates a human anatomical model to generate volumetric data. The primary goal of this model is to capture the attributes and 3D structure, including the external shape, internal slices, and the relationship between the vertical slices of the human anatomical model. The proposed network consists of a generator for feature extraction and up-sampling based on a 3D U-Net and ResNet structure and a 3D-convolution-based LFFB (local feature fusion block). In addition, a discriminator utilizes 3D convolution to evaluate the authenticity of the generated volume compared to the ground truth. VI-GAN also devises reconstruction loss, including feature and similarity losses, to converge the generated volumetric data into a human anatomical model. In this experiment, the CT data of 234 people were used to assess the reliability of the results. When using volume evaluation metrics to measure similarity, VI-GAN generated a volume that realistically represented the human anatomical model compared to existing volume generation methods.

The Integration of Artificial Intelligence into Clinical Practice

The purpose of this literature review is to provide a fundamental synopsis of current research pertaining to artificial intelligence (AI) within the domain of clinical practice. Artificial intelligence has revolutionized the field of medicine and healthcare by providing innovative solutions to complex problems. One of the most important benefits of AI in clinical practice is its ability to investigate extensive volumes of data with efficiency and precision. This has led to the development of various applications that have improved patient outcomes and reduced the workload of healthcare professionals. AI can support doctors in making more accurate diagnoses and developing personalized treatment plans. Successful examples of AI applications are outlined for a series of medical specialties like cardiology, surgery, gastroenterology, pneumology, nephrology, urology, dermatology, orthopedics, neurology, gynecology, ophthalmology, pediatrics, hematology, and critically ill patients, as well as diagnostic methods. Special reference is made to legal and ethical considerations like accuracy, informed consent, privacy issues, data security, regulatory framework, product liability, explainability, and transparency. Finally, this review closes by critically appraising AI use in clinical practice and its future perspectives. However, it is also important to approach its development and implementation cautiously to ensure ethical considerations are met.

YoloCurvSeg: You only label one noisy skeleton for vessel-style curvilinear structure segmentation

Weakly-supervised learning (WSL) has been proposed to alleviate the conflict between data annotation cost and model performance through employing sparsely-grained (i.e., point-, box-, scribble-wise) supervision and has shown promising performance, particularly in the image segmentation field. However, it is still a very challenging task due to the limited supervision, especially when only a small number of labeled samples are available. Additionally, almost all existing WSL segmentation methods are designed for star-convex structures which are very different from curvilinear structures such as vessels and nerves. In this paper, we propose a novel sparsely annotated segmentation framework for curvilinear structures, named YoloCurvSeg. A very essential component of YoloCurvSeg is image synthesis. Specifically, a background generator delivers image backgrounds that closely match the real distributions through inpainting dilated skeletons. The extracted backgrounds are then combined with randomly emulated curves generated by a Space Colonization Algorithm-based foreground generator and through a multilayer patch-wise contrastive learning synthesizer. In this way, a synthetic dataset with both images and curve segmentation labels is obtained, at the cost of only one or a few noisy skeleton annotations. Finally, a segmenter is trained with the generated dataset and possibly an unlabeled dataset. The proposed YoloCurvSeg is evaluated on four publicly available datasets (OCTA500, CORN, DRIVE and CHASEDB1) and the results show that YoloCurvSeg outperforms state-of-the-art WSL segmentation methods by large margins. With only one noisy skeleton annotation (respectively 0.14%, 0.03%, 1.40%, and 0.65% of the full annotation), YoloCurvSeg achieves more than 97% of the fully-supervised performance on each dataset. Code and datasets will be released at https://github.com/llmir/YoloCurvSeg.

Coupling synthetic and real-world data for a deep learning-based segmentation process of 4D flow MRI

BACKGROUND AND OBJECTIVE

Phase contrast magnetic resonance imaging (4D flow MRI) is an imaging technique able to provide blood velocity in vivo and morphological information. This capability has been used to study mainly the hemodynamics of large vessels, such as the thoracic aorta. However, the segmentation of 4D flow MRI data is a complex and time-consuming task. In recent years, neural networks have shown great accuracy in segmentation tasks if large datasets are provided. Unfortunately, in the context of 4D flow MRI, the availability of these data is limited due to its recent adoption in clinical settings. In this study, we propose a pipeline for generating synthetic thoracic aorta phase contrast magnetic resonance angiography (PCMRA) to expand the limited dataset of patient-specific PCMRA images, ultimately improving the accuracy of the neural network segmentation even with a small real dataset.

METHODS

The pipeline involves several steps. First, a statistical shape model is used to synthesize new artificial geometries to improve data numerosity and variability. Secondly, computational fluid dynamics simulations are employed to simulate the velocity fields and, finally, after a downsampling and a signal-to-noise and velocity limit adjustment in both frequency and spatial domains, volumes are obtained using the PCMRA formula. These synthesized volumes are used in combination with real-world data to train a 3D U-Net neural network. Different settings of real and synthetic data are tested.

RESULTS

Incorporating synthetic data into the training set significantly improved the segmentation performance compared to using only real data. The experiments with synthetic data achieved a DICE score (DS) value of 0.83 and a better target reconstruction with respect to the case with only real data (DS = 0.65).

CONCLUSION

The proposed pipeline demonstrated the ability to increase the dataset in terms of numerosity and variability and to improve the segmentation accuracy for the thoracic aorta using PCMRA.

Organ-aware CBCT enhancement via dual path learning for prostate cancer treatment

BACKGROUND

Cone-beam computed tomography (CBCT) plays a crucial role in the intensity modulated radiotherapy (IMRT) of prostate cancer. However, poor image contrast and fuzzy organ boundaries pose challenges to precise targeting for dose delivery and plan reoptimization for adaptive therapy.

PURPOSE

In this work, we aim to enhance pelvic CBCT images by translating them to high-quality CT images with a particular focus on the anatomical structures important for radiotherapy.

METHODS

We develop a novel dual-path learning framework, covering both global and local information, for organ-aware enhancement of the prostate, bladder and rectum. The global path learns coarse inter-modality translation at the image level. The local path learns organ-aware translation at the regional level. This dual-path learning architecture can serve as a plug-and-play module adaptable to other medical image-to-image translation frameworks.

RESULTS

We evaluated the performance of the proposed method both quantitatively and qualitatively. The training dataset consists of unpaired 40 CBCT and 40 CT scans, the validation dataset consists of 5 paired CBCT-CT scans, and the testing dataset consists of 10 paired CBCT-CT scans. The peak signal-to-noise ratio (PSNR) between enhanced CBCT and reference CT images is 27.22 ± 1.79, and the structural similarity (SSIM) between enhanced CBCT and the reference CT images is 0.71 ± 0.03. We also compared our method with state-of-the-art image-to-image translation methods, where our method achieves the best performance. Moreover, the statistical analysis confirms that the improvements achieved by our method are statistically significant.

CONCLUSIONS

The proposed method demonstrates its superiority in enhancing pelvic CBCT images, especially at the organ level, compared to relevant methods.

ROP-GAN: an image synthesis method for retinopathy of prematurity based on generative adversarial network

Objective. Training data with annotations are scarce in the intelligent diagnosis of retinopathy of prematurity (ROP), and existing typical data augmentation methods cannot generate data with a high degree of diversity. In order to increase the sample size and the generalization ability of the classification model, we propose a method called ROP-GAN for image synthesis of ROP based on a generative adversarial network.Approach. To generate a binary vascular network from color fundus images, we first design an image segmentation model based on U2-Net that can extract multi-scale features without reducing the resolution of the feature map. The vascular network is then fed into an adversarial autoencoder for reconstruction, which increases the diversity of the vascular network diagram. Then, we design an ROP image synthesis algorithm based on a generative adversarial network, in which paired color fundus images and binarized vascular networks are input into the image generation model to train the generator and discriminator, and attention mechanism modules are added to the generator to improve its detail synthesis ability.Main results. Qualitative and quantitative evaluation indicators are applied to evaluate the proposed method, and experiments demonstrate that the proposed method is superior to the existing ROP image synthesis methods, as it can synthesize realistic ROP fundus images.Significance. Our method effectively alleviates the problem of data imbalance in ROP intelligent diagnosis, contributes to the implementation of ROP staging tasks, and lays the foundation for further research. In addition to classification tasks, our synthesized images can facilitate tasks that require large amounts of medical data, such as detecting lesions and segmenting medical images.

Measurement of proximal contact of single crowns to assess interproximal relief: A pilot study

Background

It is common for dental technicians to adjust the proximal surface of adjacent teeth on casts when fabricating single crowns. However, whether the accuracy of the proximal contact is affected if this step is eliminated is unclear.

Objective

To evaluate the accuracy of the proximal contact of single crowns for mandibular first molars fabricated from four different restorative materials, without adjustment of the proximal surface of the adjacent teeth by the laboratory/dental technician.

Methods

This study was in vitro; all the clinical procedures were conducted on a dentoform. The mandibular first molar tooth on the dentoform was prepared using diamond burs and a high speed handpiece. Twenty single crowns were fabricated, five for each group (monolithic zirconia, lithium disilicate, metal ceramic, and cast gold). No proximal surface adjacent to the definitive crowns was adjusted for tight contact in the dental laboratory. Both the qualitative analyses, using dental floss and shimstock, and the quantitative analyses, using a stereo microscope, were performed to evaluate the accuracy of the proximal contact of the restoration with the adjacent teeth. In the quantitative analysis, one-way analysis of variance was used to compare mean values at a significance level of 0.05.

Results

In quantitative analysis, the differences between the proximal contact tightness of the four groups was not statistically significant (P = 0.802 for mesial contacts, P = 0.354 for distal contacts). In qualitative analysis, in most crowns, dental floss passed through the contact with tight resistance and only one film of shimstock could be inserted between the adjacent teeth and the restoration. However, one specimen from the cast gold crown had open contact.

Conclusions

Even without proximal surface adjustment of the adjacent teeth during the crown fabrication process, adequate proximal contact tightness between the restoration and adjacent teeth could be achieved.

Exploring patient medication adherence and data mining methods in clinical big data: A contemporary review

BACKGROUND

Increasingly, patient medication adherence data are being consolidated from claims databases and electronic health records (EHRs). Such databases offer an indirect avenue to gauge medication adherence in our data-rich healthcare milieu. The surge in data accessibility, coupled with the pressing need for its conversion to actionable insights, has spotlighted data mining, with machine learning (ML) emerging as a pivotal technique. Nonadherence poses heightened health risks and escalates medical costs. This paper elucidates the synergistic interaction between medical database mining for medication adherence and the role of ML in fostering knowledge discovery.

METHODS

We conducted a comprehensive review of EHR applications in the realm of medication adherence, leveraging ML techniques. We expounded on the evolution and structure of medical databases pertinent to medication adherence and harnessed both supervised and unsupervised ML paradigms to delve into adherence and its ramifications.

RESULTS

Our study underscores the applications of medical databases and ML, encompassing both supervised and unsupervised learning, for medication adherence in clinical big data. Databases like SEER and NHANES, often underutilized due to their intricacies, have gained prominence. Employing ML to excavate patient medication logs from these databases facilitates adherence analysis. Such findings are pivotal for clinical decision-making, risk stratification, and scholarly pursuits, aiming to elevate healthcare quality.

CONCLUSION

Advanced data mining in the era of big data has revolutionized medication adherence research, thereby enhancing patient care. Emphasizing bespoke interventions and research could herald transformative shifts in therapeutic modalities.

Fundus Image-Label Pairs Synthesis and Retinopathy Screening via GANs With Class-Imbalanced Semi-Supervised Learning

Retinopathy is the primary cause of irreversible yet preventable blindness. Numerous deep-learning algorithms have been developed for automatic retinal fundus image analysis. However, existing methods are usually data-driven, which rarely consider the costs associated with fundus image collection and annotation, along with the class-imbalanced distribution that arises from the relative scarcity of disease-positive individuals in the population. Semi-supervised learning on class-imbalanced data, despite a realistic problem, has been relatively little studied. To fill the existing research gap, we explore generative adversarial networks (GANs) as a potential answer to that problem. Specifically, we present a novel framework, named CISSL-GANs, for class-imbalanced semi-supervised learning (CISSL) by leveraging a dynamic class-rebalancing (DCR) sampler, which exploits the property that the classifier trained on class-imbalanced data produces high-precision pseudo-labels on minority classes to leverage the bias inherent in pseudo-labels. Also, given the well-known difficulty of training GANs on complex data, we investigate three practical techniques to improve the training dynamics without altering the global equilibrium. Experimental results demonstrate that our CISSL-GANs are capable of simultaneously improving fundus image class-conditional generation and classification performance under a typical label insufficient and imbalanced scenario. Our code is available at: https://github.com/Xyporz/CISSL-GANs.

Synthetic artificial intelligence using generative adversarial network for retinal imaging in detection of age-related macular degeneration

Introduction

Age-related macular degeneration (AMD) is one of the leading causes of vision impairment globally and early detection is crucial to prevent vision loss. However, the screening of AMD is resource dependent and demands experienced healthcare providers. Recently, deep learning (DL) systems have shown the potential for effective detection of various eye diseases from retinal fundus images, but the development of such robust systems requires a large amount of datasets, which could be limited by prevalence of the disease and privacy of patient. As in the case of AMD, the advanced phenotype is often scarce for conducting DL analysis, which may be tackled via generating synthetic images using Generative Adversarial Networks (GANs). This study aims to develop GAN-synthesized fundus photos with AMD lesions, and to assess the realness of these images with an objective scale.

Methods

To build our GAN models, a total of 125,012 fundus photos were used from a real-world non-AMD phenotypical dataset. StyleGAN2 and human-in-the-loop (HITL) method were then applied to synthesize fundus images with AMD features. To objectively assess the quality of the synthesized images, we proposed a novel realness scale based on the frequency of the broken vessels observed in the fundus photos. Four residents conducted two rounds of gradings on 300 images to distinguish real from synthetic images, based on their subjective impression and the objective scale respectively.

Results and discussion

The introduction of HITL training increased the percentage of synthetic images with AMD lesions, despite the limited number of AMD images in the initial training dataset. Qualitatively, the synthesized images have been proven to be robust in that our residents had limited ability to distinguish real from synthetic ones, as evidenced by an overall accuracy of 0.66 (95% CI: 0.61-0.66) and Cohen's kappa of 0.320. For the non-referable AMD classes (no or early AMD), the accuracy was only 0.51. With the objective scale, the overall accuracy improved to 0.72. In conclusion, GAN models built with HITL training are capable of producing realistic-looking fundus images that could fool human experts, while our objective realness scale based on broken vessels can help identifying the synthetic fundus photos.

Exploring the Possibility of Measuring Vertebrae Bone Structure Metrics Using MDCT Images: An Unpaired Image-to-Image Translation Method

Bone structure metrics are vital for the evaluation of vertebral bone strength. However, the gold standard for measuring bone structure metrics, micro-Computed Tomography (micro-CT), cannot be used in vivo, which hinders the early diagnosis of fragility fractures. This paper used an unpaired image-to-image translation method to capture the mapping between clinical multidetector computed tomography (MDCT) and micro-CT images and then generated micro-CT-like images to measure bone structure metrics. MDCT and micro-CT images were scanned from 75 human lumbar spine specimens and formed training and testing sets. The generator in the model focused on learning both the structure and detailed pattern of bone trabeculae and generating micro-CT-like images, and the discriminator determined whether the generated images were micro-CT images or not. Based on similarity metrics (i.e., SSIM and FID) and bone structure metrics (i.e., bone volume fraction, trabecular separation and trabecular thickness), a set of comparisons were performed. The results show that the proposed method can perform better in terms of both similarity metrics and bone structure metrics and the improvement is statistically significant. In particular, we compared the proposed method with the paired image-to-image method and analyzed the pros and cons of the method used.

Retinal Scans and Data Sharing: The Privacy and Scientific Development Equilibrium

In ophthalmology, extensive use of ancillary imaging has enabled the development of artificial intelligence models, for which data are crucial. A data-sharing environment promotes external validation, collaborative research, and bias assessment before implementation in the real world; however, legal and ethical concerns need to be addressed in this process. The proposed solutions for improving the security of ophthalmic data sharing are patient consent and data-sharing agreements with third parties. Federated learning enables decentralized algorithm development, however, with limited results and unknown risks. Deidentification techniques through image manipulations and synthetically generated images are possible alternatives to improve security. Still, there is no single solution available. The challenge is to determine the appropriate level of risk and ensure accountability for the use of data. Sharing data, including retinal scans, can and should be performed within a trusted research environment, where there are data use agreements and credentialing of researchers, including requirements for training in responsible conduct of data use. In this review, we discuss the challenges and consequences surrounding limited sharing of ophthalmic datasets in the development of digital innovations and explore potential solutions that will enable safer sharing of retinal scan data.

SynthEye: Investigating the Impact of Synthetic Data on Artificial Intelligence-assisted Gene Diagnosis of Inherited Retinal Disease

Purpose

Rare disease diagnosis is challenging in medical image-based artificial intelligence due to a natural class imbalance in datasets, leading to biased prediction models. Inherited retinal diseases (IRDs) are a research domain that particularly faces this issue. This study investigates the applicability of synthetic data in improving artificial intelligence-enabled diagnosis of IRDs using generative adversarial networks (GANs).

Design

Diagnostic study of gene-labeled fundus autofluorescence (FAF) IRD images using deep learning.

Participants

Moorfields Eye Hospital (MEH) dataset of 15 692 FAF images obtained from 1800 patients with confirmed genetic diagnosis of 1 of 36 IRD genes.

Methods

A StyleGAN2 model is trained on the IRD dataset to generate 512 × 512 resolution images. Convolutional neural networks are trained for classification using different synthetically augmented datasets, including real IRD images plus 1800 and 3600 synthetic images, and a fully rebalanced dataset. We also perform an experiment with only synthetic data. All models are compared against a baseline convolutional neural network trained only on real data.

Main Outcome Measures

We evaluated synthetic data quality using a Visual Turing Test conducted with 4 ophthalmologists from MEH. Synthetic and real images were compared using feature space visualization, similarity analysis to detect memorized images, and Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) score for no-reference-based quality evaluation. Convolutional neural network diagnostic performance was determined on a held-out test set using the area under the receiver operating characteristic curve (AUROC) and Cohen's Kappa (κ).

Results

An average true recognition rate of 63% and fake recognition rate of 47% was obtained from the Visual Turing Test. Thus, a considerable proportion of the synthetic images were classified as real by clinical experts. Similarity analysis showed that the synthetic images were not copies of the real images, indicating that copied real images, meaning the GAN was able to generalize. However, BRISQUE score analysis indicated that synthetic images were of significantly lower quality overall than real images (P < 0.05). Comparing the rebalanced model (RB) with the baseline (R), no significant change in the average AUROC and κ was found (R-AUROC = 0.86[0.85-88], RB-AUROC = 0.88[0.86-0.89], R-k = 0.51[0.49-0.53], and RB-k = 0.52[0.50-0.54]). The synthetic data trained model (S) achieved similar performance as the baseline (S-AUROC = 0.86[0.85-87], S-k = 0.48[0.46-0.50]).

Conclusions

Synthetic generation of realistic IRD FAF images is feasible. Synthetic data augmentation does not deliver improvements in classification performance. However, synthetic data alone deliver a similar performance as real data, and hence may be useful as a proxy to real data. Financial Disclosure(s): Proprietary or commercial disclosure may be found after the references.

Virtual high-resolution MR angiography from non-angiographic multi-contrast MRIs: synthetic vascular model populations for in-silico trials

Despite success on multi-contrast MR image synthesis, generating specific modalities remains challenging. Those include Magnetic Resonance Angiography (MRA) that highlights details of vascular anatomy using specialised imaging sequences for emphasising inflow effect. This work proposes an end-to-end generative adversarial network that can synthesise anatomically plausible, high-resolution 3D MRA images using commonly acquired multi-contrast MR images (e.g. T1/T2/PD-weighted MR images) for the same subject whilst preserving the continuity of vascular anatomy. A reliable technique for MRA synthesis would unleash the research potential of very few population databases with imaging modalities (such as MRA) that enable quantitative characterisation of whole-brain vasculature. Our work is motivated by the need to generate digital twins and virtual patients of cerebrovascular anatomy for in-silico studies and/or in-silico trials. We propose a dedicated generator and discriminator that leverage the shared and complementary features of multi-source images. We design a composite loss function for emphasising vascular properties by minimising the statistical difference between the feature representations of the target images and the synthesised outputs in both 3D volumetric and 2D projection domains. Experimental results show that the proposed method can synthesise high-quality MRA images and outperform the state-of-the-art generative models both qualitatively and quantitatively. The importance assessment reveals that T2 and PD-weighted images are better predictors of MRA images than T1; and PD-weighted images contribute to better visibility of small vessel branches towards the peripheral regions. In addition, the proposed approach can generalise to unseen data acquired at different imaging centres with different scanners, whilst synthesising MRAs and vascular geometries that maintain vessel continuity. The results show the potential for use of the proposed approach to generating digital twin cohorts of cerebrovascular anatomy at scale from structural MR images typically acquired in population imaging initiatives.

Use of artificial intelligence in forecasting glaucoma progression

Artificial intelligence (AI) has been widely used in ophthalmology for disease detection and monitoring progression. For glaucoma research, AI has been used to understand progression patterns and forecast disease trajectory based on analysis of clinical and imaging data. Techniques such as machine learning, natural language processing, and deep learning have been employed for this purpose. The results from studies using AI for forecasting glaucoma progression however vary considerably due to dataset constraints, lack of a standard progression definition and differences in methodology and approach. While glaucoma detection and screening have been the focus of most research that has been published in the last few years, in this narrative review we focus on studies that specifically address glaucoma progression. We also summarize the current evidence, highlight studies that have translational potential, and provide suggestions on how future research that addresses glaucoma progression can be improved.

ICAM-Reg: Interpretable Classification and Regression With Feature Attribution for Mapping Neurological Phenotypes in Individual Scans

An important goal of medical imaging is to be able to precisely detect patterns of disease specific to individual scans; however, this is challenged in brain imaging by the degree of heterogeneity of shape and appearance. Traditional methods, based on image registration, historically fail to detect variable features of disease, as they utilise population-based analyses, suited primarily to studying group-average effects. In this paper we therefore take advantage of recent developments in generative deep learning to develop a method for simultaneous classification, or regression, and feature attribution (FA). Specifically, we explore the use of a VAE-GAN (variational autoencoder - general adversarial network) for translation called ICAM, to explicitly disentangle class relevant features, from background confounds, for improved interpretability and regression of neurological phenotypes. We validate our method on the tasks of Mini-Mental State Examination (MMSE) cognitive test score prediction for the Alzheimer's Disease Neuroimaging Initiative (ADNI) cohort, as well as brain age prediction, for both neurodevelopment and neurodegeneration, using the developing Human Connectome Project (dHCP) and UK Biobank datasets. We show that the generated FA maps can be used to explain outlier predictions and demonstrate that the inclusion of a regression module improves the disentanglement of the latent space. Our code is freely available on GitHub https://github.com/CherBass/ICAM.

Observer-study-based approaches to quantitatively evaluate the realism of synthetic medical images

Objective.Synthetic images generated by simulation studies have a well-recognized role in developing and evaluating imaging systems and methods. However, for clinically relevant development and evaluation, the synthetic images must be clinically realistic and, ideally, have the same distribution as that of clinical images. Thus, mechanisms that can quantitatively evaluate this clinical realism and, ideally, the similarity in distributions of the real and synthetic images, are much needed.Approach.We investigated two observer-study-based approaches to quantitatively evaluate the clinical realism of synthetic images. In the first approach, we presented a theoretical formalism for the use of an ideal-observer study to quantitatively evaluate the similarity in distributions between the real and synthetic images. This theoretical formalism provides a direct relationship between the area under the receiver operating characteristic curve, AUC, for an ideal observer and the distributions of real and synthetic images. The second approach is based on the use of expert-human-observer studies to quantitatively evaluate the realism of synthetic images. In this approach, we developed a web-based software to conduct two-alternative forced-choice (2-AFC) experiments with expert human observers. The usability of this software was evaluated by conducting a system usability scale (SUS) survey with seven expert human readers and five observer-study designers. Further, we demonstrated the application of this software to evaluate a stochastic and physics-based image-synthesis technique for oncologic positron emission tomography (PET). In this evaluation, the 2-AFC study with our software was performed by six expert human readers, who were highly experienced in reading PET scans, with years of expertise ranging from 7 to 40 years (median: 12 years, average: 20.4 years).Main results.In the ideal-observer-study-based approach, we theoretically demonstrated that the AUC for an ideal observer can be expressed, to an excellent approximation, by the Bhattacharyya distance between the distributions of the real and synthetic images. This relationship shows that a decrease in the ideal-observer AUC indicates a decrease in the distance between the two image distributions. Moreover, a lower bound of ideal-observer AUC = 0.5 implies that the distributions of synthetic and real images exactly match. For the expert-human-observer-study-based approach, our software for performing the 2-AFC experiments is available athttps://apps.mir.wustl.edu/twoafc. Results from the SUS survey demonstrate that the web application is very user friendly and accessible. As a secondary finding, evaluation of a stochastic and physics-based PET image-synthesis technique using our software showed that expert human readers had limited ability to distinguish the real images from the synthetic images.Significance.This work addresses the important need for mechanisms to quantitatively evaluate the clinical realism of synthetic images. The mathematical treatment in this paper shows that quantifying the similarity in the distribution of real and synthetic images is theoretically possible by using an ideal-observer-study-based approach. Our developed software provides a platform for designing and performing 2-AFC experiments with human observers in a highly accessible, efficient, and secure manner. Additionally, our results on the evaluation of the stochastic and physics-based image-synthesis technique motivate the application of this technique to develop and evaluate a wide array of PET imaging methods.

Medical image data augmentation: techniques, comparisons and interpretations

Designing deep learning based methods with medical images has always been an attractive area of research to assist clinicians in rapid examination and accurate diagnosis. Those methods need a large number of datasets including all variations in their training stages. On the other hand, medical images are always scarce due to several reasons, such as not enough patients for some diseases, patients do not want to allow their images to be used, lack of medical equipment or equipment, inability to obtain images that meet the desired criteria. This issue leads to bias in datasets, overfitting, and inaccurate results. Data augmentation is a common solution to overcome this issue and various augmentation techniques have been applied to different types of images in the literature. However, it is not clear which data augmentation technique provides more efficient results for which image type since different diseases are handled, different network architectures are used, and these architectures are trained and tested with different numbers of data sets in the literature. Therefore, in this work, the augmentation techniques used to improve performances of deep learning based diagnosis of the diseases in different organs (brain, lung, breast, and eye) from different imaging modalities (MR, CT, mammography, and fundoscopy) have been examined. Also, the most commonly used augmentation methods have been implemented, and their effectiveness in classifications with a deep network has been discussed based on quantitative performance evaluations. Experiments indicated that augmentation techniques should be chosen carefully according to image types.

NeuRegenerate: A Framework for Visualizing Neurodegeneration

Recent advances in high-resolution microscopy have allowed scientists to better understand the underlying brain connectivity. However, due to the limitation that biological specimens can only be imaged at a single timepoint, studying changes to neural projections over time is limited to observations gathered using population analysis. In this article, we introduce NeuRegenerate, a novel end-to-end framework for the prediction and visualization of changes in neural fiber morphology within a subject across specified age-timepoints. To predict projections, we present neuReGANerator, a deep-learning network based on cycle-consistent generative adversarial network (GAN) that translates features of neuronal structures across age-timepoints for large brain microscopy volumes. We improve the reconstruction quality of the predicted neuronal structures by implementing a density multiplier and a new loss function, called the hallucination loss. Moreover, to alleviate artifacts that occur due to tiling of large input volumes, we introduce a spatial-consistency module in the training pipeline of neuReGANerator. Finally, to visualize the change in projections, predicted using neuReGANerator, NeuRegenerate offers two modes: (i) neuroCompare to simultaneously visualize the difference in the structures of the neuronal projections, from two age domains (using structural view and bounded view), and (ii) neuroMorph, a vesselness-based morphing technique to interactively visualize the transformation of the structures from one age-timepoint to the other. Our framework is designed specifically for volumes acquired using wide-field microscopy. We demonstrate our framework by visualizing the structural changes within the cholinergic system of the mouse brain between a young and old specimen.

Reciprocal assistance of intravascular imaging in three-dimensional stent reconstruction: Using cross-modal translation based on disentanglement representation

BACKGROUND

Accurate and efficient 3-dimension (3D) reconstruction of coronary stents in intravascular imaging of optical coherence tomography (OCT) or intravascular ultrasound (IVUS) is important for optimization of complex percutaneous coronary interventions (PCI). Deep learning has been used to address this technical challenge. However, manual annotation of stent is strenuous, especially for IVUS images. To this end, we aim to explore whether the OCT and IVUS images can assist each other in stent 3D reconstruction when one of them is lack of labeled dataset.

METHODS

We firstly performed cross-modal translation between OCT and IVUS images, where disentangled representation was employed to generate synthetic images with good stent consistency. The reciprocal assistance of OCT and IVUS in stent 3D reconstruction was then conducted by applying unsupervised and semi-supervised learning with the aid of synthetic images. Stent consistency in synthetic images and reciprocal effectiveness in stent 3D reconstruction were quantitatively assessed by F1-Score (FS) on two datasets: OCT-High Definition IVUS (HD IVUS) and OCT-Conventional IVUS (IVUS).

RESULTS

The employment of disentangled representation achieved higher stent consistency in synthetic images (OCT to HD IVUS: FS=0.789 vs 0.684; HD IVUS to OCT: FS=0.766 vs 0.682; OCT to IVUS: FS=0.806 vs 0.664; IVUS to OCT: FS=0.724 vs 0.673). For stent 3D reconstruction, the assistance from synthetic images significantly promoted unsupervised adaptation across modalities (OCT to HD IVUS: FS=0.776 vs 0.109; HD IVUS to OCT: FS=0.826 vs 0.125; OCT to IVUS: FS=0.782 vs 0.068; IVUS to OCT: FS=0.815 vs 0.123), and improved performance in semi-supervised learning, especially when only limited labeled data was available.

CONCLUSION

The intravascular images of OCT and IVUS can provide reciprocal assistance to each other in stent 3D reconstruction by cross-modal translation, where the stent consistency in synthetic images was maintained by disentangled representation.

Exploring healthy retinal aging with deep learning

Purpose

To study the individual course of retinal changes caused by healthy aging using deep learning.

Design

Retrospective analysis of a large data set of retinal OCT images.

Participants

A total of 85 709 adults between the age of 40 and 75 years of whom OCT images were acquired in the scope of the UK Biobank population study.

Methods

We created a counterfactual generative adversarial network (GAN), a type of neural network that learns from cross-sectional, retrospective data. It then synthesizes high-resolution counterfactual OCT images and longitudinal time series. These counterfactuals allow visualization and analysis of hypothetical scenarios in which certain characteristics of the imaged subject, such as age or sex, are altered, whereas other attributes, crucially the subject's identity and image acquisition settings, remain fixed.

Main Outcome Measures

Using our counterfactual GAN, we investigated subject-specific changes in the retinal layer structure as a function of age and sex. In particular, we measured changes in the retinal nerve fiber layer (RNFL), combined ganglion cell layer plus inner plexiform layer (GCIPL), inner nuclear layer to the inner boundary of the retinal pigment epithelium (INL-RPE), and retinal pigment epithelium (RPE).

Results

Our counterfactual GAN is able to smoothly visualize the individual course of retinal aging. Across all counterfactual images, the RNFL, GCIPL, INL-RPE, and RPE changed by -0.1 μm ± 0.1 μm, -0.5 μm ± 0.2 μm, -0.2 μm ± 0.1 μm, and 0.1 μm ± 0.1 μm, respectively, per decade of age. These results agree well with previous studies based on the same cohort from the UK Biobank population study. Beyond population-wide average measures, our counterfactual GAN allows us to explore whether the retinal layers of a given eye will increase in thickness, decrease in thickness, or stagnate as a subject ages.

Conclusion

This study demonstrates how counterfactual GANs can aid research into retinal aging by generating high-resolution, high-fidelity OCT images, and longitudinal time series. Ultimately, we envision that they will enable clinical experts to derive and explore hypotheses for potential imaging biomarkers for healthy and pathologic aging that can be refined and tested in prospective clinical trials.

Financial Disclosures

Proprietary or commercial disclosure may be found after the references.

Local contrastive loss with pseudo-label based self-training for semi-supervised medical image segmentation

Supervised deep learning-based methods yield accurate results for medical image segmentation. However, they require large labeled datasets for this, and obtaining them is a laborious task that requires clinical expertise. Semi/self-supervised learning-based approaches address this limitation by exploiting unlabeled data along with limited annotated data. Recent self-supervised learning methods use contrastive loss to learn good global level representations from unlabeled images and achieve high performance in classification tasks on popular natural image datasets like ImageNet. In pixel-level prediction tasks such as segmentation, it is crucial to also learn good local level representations along with global representations to achieve better accuracy. However, the impact of the existing local contrastive loss-based methods remains limited for learning good local representations because similar and dissimilar local regions are defined based on random augmentations and spatial proximity; not based on the semantic label of local regions due to lack of large-scale expert annotations in the semi/self-supervised setting. In this paper, we propose a local contrastive loss to learn good pixel level features useful for segmentation by exploiting semantic label information obtained from pseudo-labels of unlabeled images alongside limited annotated images with ground truth (GT) labels. In particular, we define the proposed contrastive loss to encourage similar representations for the pixels that have the same pseudo-label/GT label while being dissimilar to the representation of pixels with different pseudo-label/GT label in the dataset. We perform pseudo-label based self-training and train the network by jointly optimizing the proposed contrastive loss on both labeled and unlabeled sets and segmentation loss on only the limited labeled set. We evaluated the proposed approach on three public medical datasets of cardiac and prostate anatomies, and obtain high segmentation performance with a limited labeled set of one or two 3D volumes. Extensive comparisons with the state-of-the-art semi-supervised and data augmentation methods and concurrent contrastive learning methods demonstrate the substantial improvement achieved by the proposed method. The code is made publicly available at https://github.com/krishnabits001/pseudo_label_contrastive_training.

Synthesizing multi-frame high-resolution fluorescein angiography images from retinal fundus images using generative adversarial networks

BACKGROUND

Fundus fluorescein angiography (FA) can be used to diagnose fundus diseases by observing dynamic fluorescein changes that reflect vascular circulation in the fundus. As FA may pose a risk to patients, generative adversarial networks have been used to convert retinal fundus images into fluorescein angiography images. However, the available methods focus on generating FA images of a single phase, and the resolution of the generated FA images is low, being unsuitable for accurately diagnosing fundus diseases.

METHODS

We propose a network that generates multi-frame high-resolution FA images. This network consists of a low-resolution GAN (LrGAN) and a high-resolution GAN (HrGAN), where LrGAN generates low-resolution and full-size FA images with global intensity information, HrGAN takes the FA images generated by LrGAN as input to generate multi-frame high-resolution FA patches. Finally, the FA patches are merged into full-size FA images.

RESULTS

Our approach combines supervised and unsupervised learning methods and achieves better quantitative and qualitative results than using either method alone. Structural similarity (SSIM), normalized cross-correlation (NCC) and peak signal-to-noise ratio (PSNR) were used as quantitative metrics to evaluate the performance of the proposed method. The experimental results show that our method achieves better quantitative results with structural similarity of 0.7126, normalized cross-correlation of 0.6799, and peak signal-to-noise ratio of 15.77. In addition, ablation experiments also demonstrate that using a shared encoder and residual channel attention module in HrGAN is helpful for the generation of high-resolution images.

CONCLUSIONS

Overall, our method has higher performance for generating retinal vessel details and leaky structures in multiple critical phases, showing a promising clinical diagnostic value.

Unpaired mesh-to-image translation for 3D fluorescent microscopy images of neurons

While Generative Adversarial Networks (GANs) can now reliably produce realistic images in a multitude of imaging domains, they are ill-equipped to model thin, stochastic textures present in many large 3D fluorescent microscopy (FM) images acquired in biological research. This is especially problematic in neuroscience where the lack of ground truth data impedes the development of automated image analysis algorithms for neurons and neural populations. We therefore propose an unpaired mesh-to-image translation methodology for generating volumetric FM images of neurons from paired ground truths. We start by learning unique FM styles efficiently through a Gramian-based discriminator. Then, we stylize 3D voxelized meshes of previously reconstructed neurons by successively generating slices. As a result, we effectively create a synthetic microscope and can acquire realistic FM images of neurons with control over the image content and imaging configurations. We demonstrate the feasibility of our architecture and its superior performance compared to state-of-the-art image translation architectures through a variety of texture-based metrics, unsupervised segmentation accuracy, and an expert opinion test. In this study, we use 2 synthetic FM datasets and 2 newly acquired FM datasets of retinal neurons.

Lesion-aware generative adversarial networks for color fundus image to fundus fluorescein angiography translation

BACKGROUND AND OBJECTIVE

Fundus fluorescein angiography (FFA) is widely used in clinical ophthalmic diagnosis and treatment with the requirement of adverse fluorescent dyes injection. Recently, many deep Convolutional Neural Network(CNN)-based methods have been proposed to estimate FFA from color fundus (CF) images to eliminate the use of adverse fluorescent dyes. However, the robustness of these methods is affected by pathological changes.

METHOD

In this work, we present a CNN-based approach, lesion-aware generative adversarial networks (LA-GAN), to enhance the visual effect of lesion characteristics in the generated FFA images. First, we lead the generator notice lesion information by joint learning with lesion region segmentation. A new hierarchical correlation multi-task framework for high-resolution images is designed. Second, to enhance the visual contrast between normal regions and lesion regions, a newly designed region-level adversarial loss is used rather than the image-level adversarial loss. The code is publicly available at: https://github.com/nicetomeetu21/LA-GAN.

RESULTS

The effectiveness of LA-Net has been verified in data with branch retinal vein occlusion. The proposed model reported as measures of generation performance a mean structural similarity (SSIM) of 0.536, mean learned perceptual image patch similarity (LPIPS) 0.312, outperforming other FFA generation and general image generation methods. Further, due to the proposed multi-task learning framework, the lesion-region segmentation performance was further reported as the mean Dice increased from 0.714 to 0.797 and the mean accuracy increased from 0.873 to 0.905, outperforming general single-task image segmentation methods.

CONCLUSIONS

The results show that the visual effect of lesion characteristics can be improved by employing the region-level adversarial loss and the hierarchical correlation multi-task framework respectively. Based on the results of comparison with the state-of-the-art methods, LA-GAN is not only effective for CF-to-FFA translation, but also effective for lesion-region segmentation. Thus, it may be used for various image translation and lesion segmentation tasks in future research.

Synthetic MR image generation of macrotrabecular-massive hepatocellular carcinoma using generative adversarial networks

PURPOSE

The purpose of this study was to develop a method for generating synthetic MR images of macrotrabecular-massive hepatocellular carcinoma (MTM-HCC).

MATERIALS AND METHODS

A set of abdominal MR images including fat-saturated T1-weighted images obtained during the arterial and portal venous phases of enhancement and T2-weighted images of 91 patients with MTM-HCC, and another set of MR abdominal images from 67 other patients were used. Synthetic images were obtained using a 3-step pipeline that consisted in: (i), generating a synthetic MTM-HCC tumor on a neutral background; (ii), randomly selecting a background among the 67 patients and a position inside the liver; and (iii), merging the generated tumor in the background at the specified location. Synthetic images were qualitatively evaluated by three radiologists and quantitatively assessed using a mix of 1-nearest neighbor classifier metric and Fréchet inception distance.

RESULTS

A set of 1000 triplets of synthetic MTM-HCC images with consistent contrasts were successfully generated. Evaluation of selected synthetic images by three radiologists showed that the method gave realistic, consistent and diversified images. Qualitative and quantitative evaluation led to an overall score of 0.64.

CONCLUSION

This study shows the feasibility of generating realistic synthetic MR images with very few training data, by leveraging the wide availability of liver backgrounds. Further studies are needed to assess the added value of those synthetic images for automatic diagnosis of MTM-HCC.

Automated grading system of retinal arterio-venous crossing patterns: A deep learning approach replicating ophthalmologist's diagnostic process of arteriolosclerosis

The morphological feature of retinal arterio-venous crossing patterns is a valuable source of cardiovascular risk stratification as it directly captures vascular health. Although Scheie's classification, which was proposed in 1953, has been used to grade the severity of arteriolosclerosis as diagnostic criteria, it is not widely used in clinical settings as mastering this grading is challenging as it requires vast experience. In this paper, we propose a deep learning approach to replicate a diagnostic process of ophthalmologists while providing a checkpoint to secure explainability to understand the grading process. The proposed pipeline is three-fold to replicate a diagnostic process of ophthalmologists. First, we adopt segmentation and classification models to automatically obtain vessels in a retinal image with the corresponding artery/vein labels and find candidate arterio-venous crossing points. Second, we use a classification model to validate the true crossing point. At last, the grade of severity for the vessel crossings is classified. To better address the problem of label ambiguity and imbalanced label distribution, we propose a new model, named multi-diagnosis team network (MDTNet), in which the sub-models with different structures or different loss functions provide different decisions. MDTNet unifies these diverse theories to give the final decision with high accuracy. Our automated grading pipeline was able to validate crossing points with precision and recall of 96.3% and 96.3%, respectively. Among correctly detected crossing points, the kappa value for the agreement between the grading by a retina specialist and the estimated score was 0.85, with an accuracy of 0.92. The numerical results demonstrate that our method can achieve a good performance in both arterio-venous crossing validation and severity grading tasks following the diagnostic process of ophthalmologists. By the proposed models, we could build a pipeline reproducing ophthalmologists' diagnostic process without requiring subjective feature extractions. The code is available (https://github.com/conscienceli/MDTNet).

LAC-GAN: Lesion attention conditional GAN for Ultra-widefield image synthesis

Automatic detection of retinal diseases based on deep learning technology and Ultra-widefield (UWF) images plays an important role in clinical practices in recent years. However, due to small lesions and limited data samples, it is not easy to train a detection-accurate model with strong generalization ability. In this paper, we propose a lesion attention conditional generative adversarial network (LAC-GAN) to synthesize retinal images with realistic lesion details to improve the training of the disease detection model. Specifically, the generator takes the vessel mask and class label as the conditional inputs, and processes the random Gaussian noise by a series of residual block to generate the synthetic images. To focus on pathological information, we propose a lesion feature attention mechanism based on random forest (RF) method, which constructs its reverse activation network to activate the lesion features. For discriminator, a weight-sharing multi-discriminator is designed to improve the performance of model by affine transformations. Experimental results on multi-center UWF image datasets demonstrate that the proposed method can generate retinal images with reasonable details, which helps to enhance the performance of the disease detection model.