RetinaRegNet: A zero-shot approach for retinal image registration

Retinal image registration is essential for monitoring eye diseases and planning treatments, yet it remains challenging due to large deformations, minimal overlap, and varying image quality. To address these challenges, we propose RetinaRegNet, a multi-stage image registration model with zero-shot generalizability across multiple retinal imaging modalities. RetinaRegNet begins by extracting image features using a pretrained latent diffusion model. Feature points are sampled from the fixed image using a combination of the SIFT algorithm and random sampling. For each sampled point, its corresponding point in the moving image is estimated by cosine similarities between diffusion feature vectors of that point and all pixels in the moving image. Outliers in point correspondences are detected by an inverse consistency constraint, ensuring consistency in both forward and backward directions. Outliers with large distances between true and estimated points are further removed by a transformation-based outlier detector. The resulting point correspondences are then used to estimate a geometric transformation between the two images. We use a two-stage registration framework for robust and accurate alignment: the first stage estimates a homography for global alignment, and the second stage estimates a third-order polynomial transformation to capture local deformations. We evaluated RetinaRegNet on three imaging modalities: color fundus, fluorescein angiography, and laser speckle flowgraphy. Across these datasets, it consistently outperformed state-of-the-art methods, achieving AUC scores of 0.901, 0.868, and 0.861, respectively. RetinaRegNet's zero-shot performance highlights its potential as a valuable tool for tracking disease progression and evaluating treatment efficacy. Our code is publicly available at: https://github.com/mirthAI/RetinaRegNet.

User-Assisted Approach for Accurate Nonrigid Registration of Images and Traces

Fully automated registration algorithms are prone to getting trapped in solutions corresponding to local minima of their objective functions, leading to errors that are easy to detect but challenging to correct. Traditional solutions often involve iterative parameter tuning, data preprocessing and preregistering, and multiple algorithm reruns-an approach that is both time-consuming and does not guarantee satisfactory results. Therefore, for tasks where registration accuracy is more important than speed, it is appropriate to explore alternative, user-assisted registration strategies. In such tasks, finding and correcting errors in automated registration is often more time-consuming than directly integrating user input during the registration process. Therefore, this study evaluates a user-assisted approach for accurate nonrigid registration of images and traces. By leveraging the corresponding sets of fiducial points provided by the user to guide the registration, the algorithm computes an optimal nonrigid transformation that combines linear and nonlinear components. Our findings demonstrate that the registration accuracy of this approach improves consistently with the increased complexity of the linear transformation and as more fiducial points are provided. As a result, accuracy sufficient for many biomedical applications can be achieved within minutes, requiring only a small number of user-provided fiducial points.

COph100: A comprehensive fundus image registration dataset from infants constituting the "RIDIRP" database

Retinal image registration is vital for diagnostic therapeutic applications within the field of ophthalmology. Existing public datasets, focusing on adult retinal pathologies with high-quality images, have limited number of image pairs and neglect clinical challenges. To address this gap, we introduce COph100, a novel and challenging dataset known as the Comprehensive Ophthalmology Retinal Image Registration dataset for infants with a wide range of image quality issues constituting the public "RIDIRP" database. COph100 consists of 100 eyes, each with 2 to 9 examination sessions, amounting to a total of 491 image pairs carefully selected from the publicly available dataset. We manually labeled the corresponding ground truth image points and provided automatic vessel segmentation masks for each image. We have assessed COph100 in terms of image quality and registration outcomes using state-of-the-art algorithms. This resource enables a robust comparison of retinal registration methodologies and aids in the analysis of disease progression in infants, thereby deepening our understanding of pediatric ophthalmic conditions.

ConKeD: multiview contrastive descriptor learning for keypoint-based retinal image registration

Retinal image registration is of utmost importance due to its wide applications in medical practice. In this context, we propose ConKeD, a novel deep learning approach to learn descriptors for retinal image registration. In contrast to current registration methods, our approach employs a novel multi-positive multi-negative contrastive learning strategy that enables the utilization of additional information from the available training samples. This makes it possible to learn high-quality descriptors from limited training data. To train and evaluate ConKeD, we combine these descriptors with domain-specific keypoints, particularly blood vessel bifurcations and crossovers, that are detected using a deep neural network. Our experimental results demonstrate the benefits of the novel multi-positive multi-negative strategy, as it outperforms the widely used triplet loss technique (single-positive and single-negative) as well as the single-positive multi-negative alternative. Additionally, the combination of ConKeD with the domain-specific keypoints produces comparable results to the state-of-the-art methods for retinal image registration, while offering important advantages such as avoiding pre-processing, utilizing fewer training samples, and requiring fewer detected keypoints, among others. Therefore, ConKeD shows a promising potential towards facilitating the development and application of deep learning-based methods for retinal image registration.

Retinal image registration method for myopia development

OBJECTIVE

The development of myopia is usually accompanied by changes in retinal vessels, optic disc, optic cup, fovea, and other retinal structures as well as the length of the ocular axis. And the accurate registration of retinal images is very important for the extraction and analysis of retinal structural changes. However, the registration of retinal images with myopia development faces a series of challenges, due to the unique curved surface of the retina, as well as the changes in fundus curvature caused by ocular axis elongation. Therefore, our goal is to improve the registration accuracy of the retinal images with myopia development.

METHOD

In this study, we propose a 3D spatial model for the pair of retinal images with myopia development. In this model, we introduce a novel myopia development model that simulates the changes in the length of ocular axis and fundus curvature due to the development of myopia. We also consider the distortion model of the fundus camera during the imaging process. Based on the 3D spatial model, we further implement a registration framework, which utilizes corresponding points in the pair of retinal images to achieve registration in the way of 3D pose estimation.

RESULTS

The proposed method is quantitatively evaluated on the publicly available dataset without myopia development and our Fundus Image Myopia Development (FIMD) dataset. The proposed method is shown to perform more accurate and stable registration than state-of-the-art methods, especially for retinal images with myopia development.

SIGNIFICANCE

To the best of our knowledge, this is the first retinal image registration method for the study of myopia development. This method significantly improves the registration accuracy of retinal images which have myopia development. The FIMD dataset we constructed has been made publicly available to promote the study in related fields.

ACCURATE REGISTRATION BETWEEN ULTRA-WIDE-FIELD AND NARROW ANGLE RETINA IMAGES WITH 3D EYEBALL SHAPE OPTIMIZATION

The Ultra-Wide-Field (UWF) retina images have attracted wide attentions in recent years in the study of retina. However, accurate registration between the UWF images and the other types of retina images could be challenging due to the distortion in the peripheral areas of an UWF image, which a 2D warping can not handle. In this paper, we propose a novel 3D distortion correction method which sets up a 3D projection model and optimizes a dense 3D retina mesh to correct the distortion in the UWF image. The corrected UWF image can then be accurately aligned to the target image using 2D alignment methods. The experimental results show that our proposed method outperforms the state-of-the-art method by 30%.

A Straightforward Bifurcation Pattern-Based Fundus Image Registration Method

Fundus image registration is crucial in eye disease examination, as it enables the alignment of overlapping fundus images, facilitating a comprehensive assessment of conditions like diabetic retinopathy, where a single image's limited field of view might be insufficient. By combining multiple images, the field of view for retinal analysis is extended, and resolution is enhanced through super-resolution imaging. Moreover, this method facilitates patient follow-up through longitudinal studies. This paper proposes a straightforward method for fundus image registration based on bifurcations, which serve as prominent landmarks. The approach aims to establish a baseline for fundus image registration using these landmarks as feature points, addressing the current challenge of validation in this field. The proposed approach involves the use of a robust vascular tree segmentation method to detect feature points within a specified range. The method involves coarse vessel segmentation to analyze patterns in the skeleton of the segmentation foreground, followed by feature description based on the generation of a histogram of oriented gradients and determination of image relation through a transformation matrix. Image blending produces a seamless registered image. Evaluation on the FIRE dataset using registration error as the key parameter for accuracy demonstrates the method's effectiveness. The results show the superior performance of the proposed method compared to other techniques using vessel-based feature extraction or partially based on SURF, achieving an area under the curve of 0.526 for the entire FIRE dataset.

Automated inter-device 3D OCT image registration using deep learning and retinal layer segmentation

Optical coherence tomography (OCT) is the most widely used imaging modality in ophthalmology. There are multiple variations of OCT imaging capable of producing complementary information. Thus, registering these complementary volumes is desirable in order to combine their information. In this work, we propose a novel automated pipeline to register OCT images produced by different devices. This pipeline is based on two steps: a multi-modal 2D en-face registration based on deep learning, and a Z-axis (axial axis) registration based on the retinal layer segmentation. We evaluate our method using data from a Heidelberg Spectralis and an experimental PS-OCT device. The empirical results demonstrated high-quality registrations, with mean errors of approximately 46 µm for the 2D registration and 9.59 µm for the Z-axis registration. These registrations may help in multiple clinical applications such as the validation of layer segmentations among others.

Reliable and stable fundus image registration based on brain-inspired spatially-varying adaptive pyramid context aggregation network

The task of fundus image registration aims to find matching keypoints between an image pair. Traditional methods detect the keypoint by hand-designed features, which fail to cope with complex application scenarios. Due to the strong feature learning ability of deep neural network, current image registration methods based on deep learning directly learn to align the geometric transformation between the reference image and test image in an end-to-end manner. Another mainstream of this task aims to learn the displacement vector field between the image pair. In this way, the image registration has achieved significant advances. However, due to the complicated vascular morphology of retinal image, such as texture and shape, current widely used image registration methods based on deep learning fail to achieve reliable and stable keypoint detection and registration results. To this end, in this paper, we aim to bridge this gap. Concretely, since the vessel crossing and branching points can reliably and stably characterize the key components of fundus image, we propose to learn to detect and match all the crossing and branching points of the input images based on a single deep neural network. Moreover, in order to accurately locate the keypoints and learn discriminative feature embedding, a brain-inspired spatially-varying adaptive pyramid context aggregation network is proposed to incorporate the contextual cues under the supervision of structured triplet ranking loss. Experimental results show that the proposed method achieves more accurate registration results with significant speed advantage.

INSPIRE: Intensity and spatial information-based deformable image registration

We present INSPIRE, a top-performing general-purpose method for deformable image registration. INSPIRE brings distance measures which combine intensity and spatial information into an elastic B-splines-based transformation model and incorporates an inverse inconsistency penalization supporting symmetric registration performance. We introduce several theoretical and algorithmic solutions which provide high computational efficiency and thereby applicability of the proposed framework in a wide range of real scenarios. We show that INSPIRE delivers highly accurate, as well as stable and robust registration results. We evaluate the method on a 2D dataset created from retinal images, characterized by presence of networks of thin structures. Here INSPIRE exhibits excellent performance, substantially outperforming the widely used reference methods. We also evaluate INSPIRE on the Fundus Image Registration Dataset (FIRE), which consists of 134 pairs of separately acquired retinal images. INSPIRE exhibits excellent performance on the FIRE dataset, substantially outperforming several domain-specific methods. We also evaluate the method on four benchmark datasets of 3D magnetic resonance images of brains, for a total of 2088 pairwise registrations. A comparison with 17 other state-of-the-art methods reveals that INSPIRE provides the best overall performance. Code is available at github.com/MIDA-group/inspire.

A Fully Unsupervised Deep Learning Framework for Non-Rigid Fundus Image Registration

In ophthalmology, the registration problem consists of finding a geometric transformation that aligns a pair of images, supporting eye-care specialists who need to record and compare images of the same patient. Considering the registration methods for handling eye fundus images, the literature offers only a limited number of proposals based on deep learning (DL), whose implementations use the supervised learning paradigm to train a model. Additionally, ensuring high-quality registrations while still being flexible enough to tackle a broad range of fundus images is another drawback faced by most existing methods in the literature. Therefore, in this paper, we address the above-mentioned issues by introducing a new DL-based framework for eye fundus registration. Our methodology combines a U-shaped fully convolutional neural network with a spatial transformation learning scheme, where a reference-free similarity metric allows the registration without assuming any pre-annotated or artificially created data. Once trained, the model is able to accurately align pairs of images captured under several conditions, which include the presence of anatomical differences and low-quality photographs. Compared to other registration methods, our approach achieves better registration outcomes by just passing as input the desired pair of fundus images.

Robust Detection Model of Vascular Landmarks for Retinal Image Registration: A Two-Stage Convolutional Neural Network

Registration is useful for image processing in computer vision. It can be applied to retinal images and provide support for ophthalmologists in tracking disease progression and monitoring therapeutic responses. This study proposed a robust detection model of vascular landmarks to improve the performance of retinal image registration. The proposed model consists of a two-stage convolutional neural network, in which one segments the retinal vessels on a pair of images, and the other detects junction points from the vessel segmentation image. Information obtained from the model was utilized for the registration. The keypoints were extracted based on the acquired vascular landmark points, and the orientation features were calculated as descriptors. Then, the reference and sensed images were registered by matching keypoints using a homography matrix and random sample consensus algorithm. The proposed method was evaluated on five databases and seven evaluation metrics to verify both clinical effectiveness and robustness. The results established that the proposed method showed outstanding performance for registration compared with other state-of-the-art methods. In particular, the high and significantly improved registration results were identified on FIRE database with area under the curve (AUC) of 0.988, 0.511, and 0.803 in S, P, and A classes. Furthermore, the proposed method worked well on poor quality and multimodal datasets demonstrating an ability to achieve high AUC above 0.8.

Color fundus image registration using a learning-based domain-specific landmark detection methodology

Medical imaging, and particularly retinal imaging, allows to accurately diagnose many eye pathologies as well as some systemic diseases such as hypertension or diabetes. Registering these images is crucial to correctly compare key structures, not only within patients, but also to contrast data with a model or among a population. Currently, this field is dominated by complex classical methods because the novel deep learning methods cannot compete yet in terms of results and commonly used methods are difficult to adapt to the retinal domain. In this work, we propose a novel method to register color fundus images based on previous works which employed classical approaches to detect domain-specific landmarks. Instead, we propose to use deep learning methods for the detection of these highly-specific domain-related landmarks. Our method uses a neural network to detect the bifurcations and crossovers of the retinal blood vessels, whose arrangement and location are unique to each eye and person. This proposal is the first deep learning feature-based registration method in fundus imaging. These keypoints are matched using a method based on RANSAC (Random Sample Consensus) without the requirement to calculate complex descriptors. Our method was tested using the public FIRE dataset, although the landmark detection network was trained using the DRIVE dataset. Our method provides accurate results, a registration score of 0.657 for the whole FIRE dataset (0.908 for category S, 0.293 for category P and 0.660 for category A). Therefore, our proposal can compete with complex classical methods and beat the deep learning methods in the state of the art.

An O-shape Neural Network With Attention Modules to Detect Junctions in Biomedical Images Without Segmentation

Junction plays an important role in biomedical research such as retinal biometric identification, retinal image registration, eye-related disease diagnosis and neuron reconstruction. However, junction detection in original biomedical images is extremely challenging. For example, retinal images contain many tiny blood vessels with complicated structures and low contrast, which makes it challenging to detect junctions. In this paper, we propose an O-shape Network architecture with Attention modules (Attention O-Net), which includes Junction Detection Branch (JDB) and Local Enhancement Branch (LEB) to detect junctions in biomedical images without segmentation. In JDB, the heatmap indicating the probabilities of junctions is estimated and followed by choosing the positions with the local highest value as the junctions, whereas it is challenging to detect junctions when the images contain weak filament signals. Therefore, LEB is constructed to enhance the thin branch foreground and make the network pay more attention to the regions with low contrast, which is helpful to alleviate the imbalance of the foreground between thin and thick branches and to detect the junctions of the thin branch. Furthermore, attention modules are utilized to introduce the feature maps from LEB to JDB, which can establish a complementary relationship and further integrate local features and contextual information between the two branches. The proposed method achieves the highest average F1-scores of 0.82, 0.73 and 0.94 in two retinal datasets and one neuron dataset, respectively. The experimental results confirm that Attention O-Net outperforms other state-of-the-art detection methods, and is helpful for retinal biometric identification.

Retinal image registration as a tool for supporting clinical applications

BACKGROUND AND OBJECTIVE

The study of small vessels allows for the analysis and diagnosis of diseases with strong vasculopathy. This type of vessels can be observed non-invasively in the retina via fundoscopy. The analysis of these vessels can be facilitated by applications built upon Retinal Image Registration (RIR), such as mosaicing, Super Resolution (SR) or eye shape estimation. RIR is challenging due to possible changes in the retina across time, the utilization of diverse acquisition devices with varying properties, or the curved shape of the retina.

METHODS

We employ the Retinal Image Registration through Eye Modelling and Pose Estimation (REMPE) framework, which simultaneously estimates the cameras' relative poses, as well as eye shape and orientation to develop RIR applications and to study their effectiveness.

RESULTS

We assess quantitatively the suitability of the REMPE framework towards achieving SR and eye shape estimation. Additionally, we provide indicative results demonstrating qualitatively its usefulness in the context of longitudinal studies, mosaicing, and multiple image registration. Besides the improvement over registration accuracy, demonstrated via registration applications, the most important novelty presented in this work is the eye shape estimation and the generation of 3D point meshes. This has the potential for allowing clinicians to perform measurements on 3D representations of the eye, instead of doing so in 2D images that contain distortions induced because of the projection on the image space.

CONCLUSIONS

RIR is very effective in supporting applications such as SR, eye shape estimation, longitudinal studies, mosaicing and multiple image registration. Its improved registration accuracy compared to the state of the art translates directly in improved performance when supporting the aforementioned applications.