Real-time algorithm for retinal tracking

Fundus image mosaicking for information augmentation in computer-assisted slit-lamp imaging

Laser photocoagulation is currently the standard treatment for sight-threatening diseases worldwide, namely diabetic retinopathy and retinal vein occlusions. The slit lamp biomicroscope is the most commonly used device for this procedure, specially for the treatment of the eye periphery. However, only a small portion of the retina can be visualized through the biomicroscope, complicating the task of localizing and identifying surgical targets, increasing treatment duration and patient discomfort. In order to assist surgeons, we propose a method for creating intraoperative retina maps for view expansion using a slit-lamp device. Based on the mosaicking method described by Richa et al, 2012, the proposed method is a combination of direct and feature-based methods, suitable for the textured nature of the human retina. In this paper, we describe three major enhancements to the original formulation. The first is a visual tracking method using local illumination compensation to cope with the challenging visualization conditions. The second is an efficient pixel selection scheme for increased computational efficiency. The third is an entropy-based mosaic update method to dynamically improve the retina map during exploration. To evaluate the performance of the proposed method, we conducted several experiments on human subjects with a computer-assisted slit-lamp prototype. We also demonstrate the practical value of the system for photo documentation, diagnosis and intraoperative navigation.

Real-time multimodal retinal image registration for a computer-assisted laser photocoagulation system

An algorithm for the real-time registration of a retinal video sequence captured with a scanning digital ophthalmoscope (SDO) to a retinal composite image is presented. This method is designed for a computer-assisted retinal laser photocoagulation system to compensate for retinal motion and hence enhance the accuracy, speed, and patient safety of retinal laser treatments. The procedure combines intensity and feature-based registration techniques. For the registration of an individual frame, the translational frame-to-frame motion between preceding and current frame is detected by normalized cross correlation. Next, vessel points on the current video frame are identified and an initial transformation estimate is constructed from the calculated translation vector and the quadratic registration matrix of the previous frame. The vessel points are then iteratively matched to the segmented vessel centerline of the composite image to refine the initial transformation and register the video frame to the composite image. Criteria for image quality and algorithm convergence are introduced, which assess the exclusion of single frames from the registration process and enable a loss of tracking signal if necessary. The algorithm was successfully applied to ten different video sequences recorded from patients. It revealed an average accuracy of 2.47 ± 2.0 pixels (∼23.2 ± 18.8 μm) for 2764 evaluated video frames and demonstrated that it meets the clinical requirements.

Algorithm for registration of full Scanning Laser Ophthalmoscope video sequences

Fluorescein angiography is an established technique for examining the functional integrity of the retinal microcirculation for early detection of changes due to retinopathy. This paper describes a new method for the registration of large Scanning Laser Ophthalmoscope sequences (SLO), where the patient has been injected with a fluorescent dye. This allows the measurement of parameters such as the arteriovenous passage time. Due to the long time needed to acquire these sequences, there will inevitably be eye movement, which must be corrected prior to the application of quantitative analysis. The algorithm described here combines mutual information-based registration and landmark-based registration. The former will allow the alignment of the darkest frames of the sequence, where the dye has not still arrived to the retina, because of its ability to work with images without a preprocessing or segmentation, while the latter uses relevant features (the vessels) extracted by means of a robust creaseness operator, to get a very fast and accurate registration. The algorithm only detects rigid transformations but proves to be robust against the slight alterations derived from the eye location perspective during acquisition. Results were validated by expert clinicians.

Multimodal registration procedure for the initial spatial alignment of a retinal video sequence to a retinal composite image

Accurate placement of lesions is crucial for the effectiveness and safety of a retinal laser photocoagulation treatment. Computer assistance provides the capability for improvements to treatment accuracy and execution time. The idea is to use video frames acquired from a scanning digital ophthalmoscope (SDO) to compensate for retinal motion during laser treatment. This paper presents a method for the multimodal registration of the initial frame from an SDO retinal video sequence to a retinal composite image, which may contain a treatment plan. The retinal registration procedure comprises the following steps: 1) detection of vessel centerline points and identification of the optic disc; 2) prealignment of the video frame and the composite image based on optic disc parameters; and 3) iterative matching of the detected vessel centerline points in expanding matching regions. This registration algorithm was designed for the initialization of a real-time registration procedure that registers the subsequent video frames to the composite image. The algorithm demonstrated its capability to register various pairs of SDO video frames and composite images acquired from patients.

Design and development of a computer-assisted retinal laser surgery system

Since the mid-1980s, the development of a therapeutic, computer-assisted laser photocoagulation system to treat retinal disorders has progressed under the guidance of Dr. Welch, the Marion E. Forsman Centennial Professor of Engineering, Department of Biomedical Engineering, the University of Texas at Austin. This paper reviews the development of the system, related research in eye movement and laser-tissue interaction, and system implementation and testing. While subsets of these topics have been reported in prior publications, this paper brings the entire evolutionary design of the system together. We also discuss other recent "spinoff" uses of the system technology that have not been reported elsewhere and describe the impact of the latest technical advances on the overall system design.

Efficient Migration of Complex Off-Line Computer Vision Software to Real-Time System Implementation on Generic Computer Hardware

This paper addresses the problem of migrating large and complex computer vision code bases that have been developed off-line, into efficient real-time implementations avoiding the need for rewriting the software, and the associated costs. Creative linking strategies based on Linux loadable kernel modules are presented to create a simultaneous realization of real-time and off-line frame rate computer vision systems from a single code base. In this approach, systemic predictability is achieved by inserting time-critical components of a user-level executable directly into the kernel as a virtual device driver. This effectively emulates a single process space model that is nonpreemptable, nonpageable, and that has direct access to a powerful set of system-level services. This overall approach is shown to provide the basis for building a predictable frame-rate vision system using commercial off-the-shelf hardware and a standard uniprocessor Linux operating system. Experiments on a frame-rate vision system designed for computer-assisted laser retinal surgery show that this method reduces the variance of observed per-frame central processing unit cycle counts by two orders of magnitude. The conclusion is that when predictable application algorithms are used, it is possible to efficiently migrate to a predictable frame-rate computer vision system.

Predictive Scheduling Algorithms for Real-Time Feature Extraction and Spatial Referencing: Application to Retinal Image Sequences

Real-time spatial referencing is an important alternative to tracking for designing spatially aware ophthalmic instrumentation for procedures such as laser photocoagulation and perimetry. It requires independent, fast registration of each image frame from a digital video stream (1024 x 1024 pixels) to a spatial map of the retina. Recently, we have introduced a spatial referencing algorithm that works in three primary steps: 1) tracing the retinal vasculature to extract image feature (landmarks); 2) invariant indexing to generate hypothesized landmark correspondences and initial transformations; and 3) alignment and verification steps to robustly estimate a 12-parameter quadratic spatial transformation between the image frame and the map. The goal of this paper is to introduce techniques to minimize the amount of computation for successful spatial referencing. The fundamental driving idea is to make feature extraction subservient to registration and, therefore, only produce the information needed for verified, accurate transformations. To this end, the image is analyzed along one-dimensional, vertical and horizontal grid lines to produce a regular sampling of the vasculature, needed for step 3) and to initiate step 1). Tracing of the vascular is then prioritized hierarchically to quickly extract landmarks and groups (constellations) of landmarks for indexing. Finally, the tracing and spatial referencing computations are integrated so that landmark constellations found by tracing are tested immediately. The resulting implementation is an order-of-magnitude faster with the same success rate. The average total computation time is 31.2 ms per image on a 2.2-GHz Pentium Xeon processor.

A new real-time retinal tracking system for image-guided laser treatment

A new system is proposed for tracking sensitive areas in the retina for computer-assisted laser treatment of choroidal neovascularization (CNV). The system consists of a fundus camera using red-free illumination mode interfaced to a computer that allows real-time capturing of video input. The first image acquired is used as the reference image and utilized by the treatment physician for treatment planning. A grid of seed contours over the whole image is initiated and allowed to deform by splitting and/or merging according to preset criteria until the whole vessel tree is demarcated. Then, the image is filtered using a one-dimensional Gaussian filter in two perpendicular directions to extract the core areas of such vessels. Faster segmentation can be obtained for subsequent images by automatic registration to compensate for eye movement and saccades. An efficient registration technique is developed whereby some landmarks are detected in the reference frame then tracked in the subsequent frames. Using the relation between these two sets of corresponding points, an optimal transformation can be obtained. The implementation details of proposed strategy are presented and the obtained results indicate that it is suitable for real-time location determination and tracking of treatment positions.

Initial in vivo results of a hybrid retinal photocoagulation system

We describe initial in vivo experimental results of a new hybrid digital and analog design for retinal tracking and laser beam control. An overview of the design is given. The results show in vivo tracking rates which exceed the equivalent of 38 degrees/s in the eye. A robotically assisted lesion pattern is created for laser surgery to treat conditions such as diabetic retinopathy and retinal breaks.

Computer-vision-enabled augmented reality fundus biomicroscopy

PURPOSE

To guide treatment for macular diseases and to facilitate real-time image measurement and comparison, investigations were initiated to permit overlay of previously stored photographic and angiographic images directly onto the real-time slit-lamp biomicroscopic fundus image.

DESIGN

Experimental study in model eyes, and preliminary observations in human subjects.

METHODS

A modified, binocular video slit lamp interfaced to a personal computer and framegrabber allows for image acquisition and rendering of stored images overlaid onto the real-time slit-lamp biomicroscopic fundus image. Development proceeds with rendering on a computer monitor, while construction is completed on a miniature display interfaced directly with one of the slit-lamp oculars. Registration and tracking are performed with in-house-developed software.

MAIN OUTCOME MEASURES

Tracking speed and accuracy, ergonomic acceptability.

RESULTS

Computer-vision algorithms permit robust montaging, tracking, registration, and rendering of previously stored photographic and angiographic images onto the real-time slit-lamp fundus biomicroscopic image. In model eyes and in preliminary studies in a human eye, optimized registration permits near-video-rate image overlay with updates at 3 to 10 Hz and misregistration errors on the order of 1 to 5 pixels.

CONCLUSIONS

A prototype for ophthalmic augmented reality (image overlay) is presented. The current hardware/software implementation allows for robust performance.

Rapid automated tracing and feature extraction from retinal fundus images using direct exploratory algorithms

Algorithms are presented for rapid, automatic, robust, adaptive, and accurate tracing of retinal vasculature and analysis of intersections and crossovers. This method improves upon prior work in several ways: 1) automatic adaptation from frame to frame without manual initialization/adjustment, with few tunable parameters; 2) robust operation on image sequences exhibiting natural variability, poor and varying imaging conditions, including over/under-exposure, low contrast, and artifacts such as glare; 3) does not require the vasculature to be connected, so it can handle partial views; and 4) operation is efficient enough for use on unspecialized hardware, and amenable to deadline-driven computing, being able to produce a rapidly and monotonically improving sequence of usable partial results. Increased computation can be traded for superior tracing performance. Its efficiency comes from direct processing on gray-level data without any preprocessing, and from processing only a minimally necessary fraction of pixels in an exploratory manner, avoiding low-level image-wide operations such as thresholding, edge detection, and morphological processing. These properties make the algorithm suited to real-time, on-line (live) processing and is being applied to computer-assisted laser retinal surgery.

Automatic retinal image registration scheme using global optimization techniques

Retinal image registration is commonly required in order to combine the complementary information in different retinal modalities. In this paper, a new automatic scheme to register retinal images is presented and is currently tested in a clinical environment. The scheme considers the suitability and efficiency of different image transformation models and function optimization techniques, following an initial preprocessing stage. Three different transformation models--affine, bilinear and projective--as well as three optimization techniques--downhill simplex method, simulated annealing and genetic algorithms--are investigated and compared in terms of accuracy and efficiency. The registration of 26 pairs of Fluoroscein Angiography and Indocyanine Green Chorioangiography images with the corresponding Red-Free retinal images, showed the superiority of combining genetic algorithms with the affine and bilinear transformation models. A comparative study of the proposed automatic registration scheme against the manual method, commonly used in the clinical practice, is finally presented showing the advantage of the proposed automatic scheme in terms of accuracy and consistency.

Image processing algorithms for retinal montage synthesis, mapping, and real-time location determination

Although laser retinal surgery is the best available treatment for choridal neovascularization, the current procedure has a low success rate (50%). Challenges, such as motion-compensated beam steering, ensuring complete coverage and minimizing incidental photodamage, can be overcome with improved instrumentation. This paper presents core image processing algorithms for 1) rapid identification of branching and crossover points of the retinal vasculature; 2) automatic montaging of video retinal angiograms; 3) real-time location determination and tracking using a combination of feature-tagged point-matching and dynamic-pixel templates. These algorithms tradeoff conflicting needs for accuracy, robustness to image variations (due to movements and the difficulty of providing steady illumination) and noise, and operational speed in the context of available hardware. The algorithm for locating vasculature landmarks performed robustly at a speed of 16-30 video image frames/s depending upon the field on a Silicon Graphics workstation. The montaging algorithm performed at a speed of 1.6-4 s for merging 5-12 frames. The tracking algorithm was validated by manually locating six landmark points on an image sequence with 180 frames, demonstrating a mean-squared error of 1.35 pixels. It successfully detected and rejected instances when the image dimmed, faded, lost contrast, or lost focus.