Detecting Glaucoma from Fundus Photographs Using Deep Learning without Convolutions: Transformer for Improved Generalization

Purpose

To compare the diagnostic accuracy and explainability of a Vision Transformer deep learning technique, Data-efficient image Transformer (DeiT), and ResNet-50, trained on fundus photographs from the Ocular Hypertension Treatment Study (OHTS) to detect primary open-angle glaucoma (POAG) and identify the salient areas of the photographs most important for each model's decision-making process.

Design

Evaluation of a diagnostic technology.

Subjects Participants and Controls

Overall 66 715 photographs from 1636 OHTS participants and an additional 5 external datasets of 16 137 photographs of healthy and glaucoma eyes.

Methods

Data-efficient image Transformer models were trained to detect 5 ground-truth OHTS POAG classifications: OHTS end point committee POAG determinations because of disc changes (model 1), visual field (VF) changes (model 2), or either disc or VF changes (model 3) and Reading Center determinations based on disc (model 4) and VFs (model 5). The best-performing DeiT models were compared with ResNet-50 models on OHTS and 5 external datasets.

Main Outcome Measures

Diagnostic performance was compared using areas under the receiver operating characteristic curve (AUROC) and sensitivities at fixed specificities. The explainability of the DeiT and ResNet-50 models was compared by evaluating the attention maps derived directly from DeiT to 3 gradient-weighted class activation map strategies.

Results

Compared with our best-performing ResNet-50 models, the DeiT models demonstrated similar performance on the OHTS test sets for all 5 ground-truth POAG labels; AUROC ranged from 0.82 (model 5) to 0.91 (model 1). Data-efficient image Transformer AUROC was consistently higher than ResNet-50 on the 5 external datasets. For example, AUROC for the main OHTS end point (model 3) was between 0.08 and 0.20 higher in the DeiT than ResNet-50 models. The saliency maps from the DeiT highlight localized areas of the neuroretinal rim, suggesting important rim features for classification. The same maps in the ResNet-50 models show a more diffuse, generalized distribution around the optic disc.

Conclusions

Vision Transformers have the potential to improve generalizability and explainability in deep learning models, detecting eye disease and possibly other medical conditions that rely on imaging for clinical diagnosis and management.

Detecting Glaucoma in the Ocular Hypertension Study Using Deep Learning

IMPORTANCE

Automated deep learning (DL) analyses of fundus photographs potentially can reduce the cost and improve the efficiency of reading center assessment of end points in clinical trials.

OBJECTIVE

To investigate the diagnostic accuracy of DL algorithms trained on fundus photographs from the Ocular Hypertension Treatment Study (OHTS) to detect primary open-angle glaucoma (POAG).

DESIGN, SETTING, AND PARTICIPANTS

In this diagnostic study, 1636 OHTS participants from 22 sites with a mean (range) follow-up of 10.7 (0-14.3) years. A total of 66 715 photographs from 3272 eyes were used to train and test a ResNet-50 model to detect the OHTS Endpoint Committee POAG determination based on optic disc (287 eyes, 3502 photographs) and/or visual field (198 eyes, 2300 visual fields) changes. Three independent test sets were used to evaluate the generalizability of the model.

MAIN OUTCOMES AND MEASURES

Areas under the receiver operating characteristic curve (AUROC) and sensitivities at fixed specificities were calculated to compare model performance. Evaluation of false-positive rates was used to determine whether the DL model detected POAG before the OHTS Endpoint Committee POAG determination.

RESULTS

A total of 1147 participants were included in the training set (661 [57.6%] female; mean age, 57.2 years; 95% CI, 56.6-57.8), 167 in the validation set (97 [58.1%] female; mean age, 57.1 years; 95% CI, 55.6-58.7), and 322 in the test set (173 [53.7%] female; mean age, 57.2 years; 95% CI, 56.1-58.2). The DL model achieved an AUROC of 0.88 (95% CI, 0.82-0.92) for the OHTS Endpoint Committee determination of optic disc or VF changes. For the OHTS end points based on optic disc changes or visual field changes, AUROCs were 0.91 (95% CI, 0.88-0.94) and 0.86 (95% CI, 0.76-0.93), respectively. False-positive rates (at 90% specificity) were higher in photographs of eyes that later developed POAG by disc or visual field (27.5% [56 of 204]) compared with eyes that did not develop POAG (11.4% [50 of 440]) during follow-up. The diagnostic accuracy of the DL model developed on the optic disc end point applied to 3 independent data sets was lower, with AUROCs ranging from 0.74 (95% CI, 0.70-0.77) to 0.79 (95% CI, 0.78-0.81).

CONCLUSIONS AND RELEVANCE

The model's high diagnostic accuracy using OHTS photographs suggests that DL has the potential to standardize and automate POAG determination for clinical trials and management. In addition, the higher false-positive rate in early photographs of eyes that later developed POAG suggests that DL models detected POAG in some eyes earlier than the OHTS Endpoint Committee, reflecting the OHTS design that emphasized a high specificity for POAG determination by requiring a clinically significant change from baseline.

Caribbean massive corals not recovering from repeated thermal stress events during 2005-2013

Massive coral bleaching events associated with high sea surface temperatures are forecast to become more frequent and severe in the future due to climate change. Monitoring colony recovery from bleaching disturbances over multiyear time frames is important for improving predictions of future coral community changes. However, there are currently few multiyear studies describing long‐term outcomes for coral colonies following acute bleaching events. We recorded colony pigmentation and size for bleached and unbleached groups of co‐located conspecifics of three major reef‐building scleractinian corals (Orbicella franksi, Siderastrea siderea, and Stephanocoenia michelini; n = 198 total) in Bocas del Toro, Panama, during the major 2005 bleaching event and then monitored pigmentation status and changes live tissue colony size for 8 years (2005–2013). Corals that were bleached in 2005 demonstrated markedly different response trajectories compared to unbleached colony groups, with extensive live tissue loss for bleached corals of all species following bleaching, with mean live tissue losses per colony 9 months postbleaching of 26.2% (±5.4 SE) for O. franksi, 35.7% (±4.7 SE) for S. michelini, and 11.2% (±3.9 SE) for S. siderea. Two species, O. franksi and S. michelini, later recovered to net positive growth, which continued until a second thermal stress event in 2010. Following this event, all species again lost tissue, with previously unbleached colony species groups experiencing greater declines than conspecific sample groups, which were previously bleached, indicating a possible positive acclimative response. However, despite this beneficial effect for previously bleached corals, all groups experienced substantial net tissue loss between 2005 and 2013, indicating that many important Caribbean reef‐building corals will likely suffer continued tissue loss and may be unable to maintain current benthic coverage when faced with future thermal stress forecast for the region, even with potential benefits from bleaching‐related acclimation.

Methods and measurement variance for field estimations of coral colony planar area using underwater photographs and semi-automated image segmentation

Size and growth rates for individual colonies are some of the most essential descriptive parameters for understanding coral communities, which are currently experiencing worldwide declines in health and extent. Accurately measuring coral colony size and changes over multiple years can reveal demographic, growth, or mortality patterns often not apparent from short-term observations and can expose environmental stress responses that may take years to manifest. Describing community size structure can reveal population dynamics patterns, such as periods of failed recruitment or patterns of colony fission, which have implications for the future sustainability of these ecosystems. However, rapidly and non-invasively measuring coral colony sizes in situ remains a difficult task, as three-dimensional underwater digital reconstruction methods are currently not practical for large numbers of colonies. Two-dimensional (2D) planar area measurements from projection of underwater photographs are a practical size proxy, although this method presents operational difficulties in obtaining well-controlled photographs in the highly rugose environment of the coral reef, and requires extensive time for image processing. Here, we present and test the measurement variance for a method of making rapid planar area estimates of small to medium-sized coral colonies using a lightweight monopod image-framing system and a custom semi-automated image segmentation analysis program. This method demonstrated a coefficient of variation of 2.26% for repeated measurements in realistic ocean conditions, a level of error appropriate for rapid, inexpensive field studies of coral size structure, inferring change in colony size over time, or measuring bleaching or disease extent of large numbers of individual colonies.

Towards Automated Annotation of Benthic Survey Images: Variability of Human Experts and Operational Modes of Automation

Global climate change and other anthropogenic stressors have heightened the need to rapidly characterize ecological changes in marine benthic communities across large scales. Digital photography enables rapid collection of survey images to meet this need, but the subsequent image annotation is typically a time consuming, manual task. We investigated the feasibility of using automated point-annotation to expedite cover estimation of the 17 dominant benthic categories from survey-images captured at four Pacific coral reefs. Inter- and intra- annotator variability among six human experts was quantified and compared to semi- and fully- automated annotation methods, which are made available at coralnet.ucsd.edu. Our results indicate high expert agreement for identification of coral genera, but lower agreement for algal functional groups, in particular between turf algae and crustose coralline algae. This indicates the need for unequivocal definitions of algal groups, careful training of multiple annotators, and enhanced imaging technology. Semi-automated annotation, where 50% of the annotation decisions were performed automatically, yielded cover estimate errors comparable to those of the human experts. Furthermore, fully-automated annotation yielded rapid, unbiased cover estimates but with increased variance. These results show that automated annotation can increase spatial coverage and decrease time and financial outlay for image-based reef surveys.

Wide field-of-view fluorescence imaging of coral reefs

Coral reefs globally are declining rapidly because of both local and global stressors. Improved monitoring tools are urgently needed to understand the changes that are occurring at appropriate temporal and spatial scales. Coral fluorescence imaging tools have the potential to improve both ecological and physiological assessments. Although fluorescence imaging is regularly used for laboratory studies of corals, it has not yet been used for large-scale in situ assessments. Current obstacles to effective underwater fluorescence surveying include limited field-of-view due to low camera sensitivity, the need for nighttime deployment because of ambient light contamination, and the need for custom multispectral narrow band imaging systems to separate the signal into meaningful fluorescence bands. Here we describe the Fluorescence Imaging System (FluorIS), based on a consumer camera modified for greatly increased sensitivity to chlorophyll-a fluorescence, and we show high spectral correlation between acquired images and in situ spectrometer measurements. This system greatly facilitates underwater wide field-of-view fluorophore surveying during both night and day, and potentially enables improvements in semi-automated segmentation of live corals in coral reef photographs and juvenile coral surveys.

Detecting particles in cryo-EM micrographs using learned features

A new learning-based approach is presented for particle detection in cryo-electron micrographs using the Adaboost learning algorithm. The approach builds directly on the successful detectors developed for the domain of face detection. It is a discriminative algorithm which learns important features of the particle's appearance using a set of training examples of the particles and a set of images that do not contain particles. The algorithm is fast (10 s on a 1.3 GHz Pentium M processor), is generic, and is not limited to any particular shape or size of the particle to be detected. The method has been evaluated on a publicly available dataset of 82 cryoEM images of keyhole lympet hemocyanin (KLH). From 998 automatically extracted particle images, the 3-D structure of KLH has been reconstructed at a resolution of 23.2 A which is the same resolution as obtained using particles manually selected by a trained user.

Leginon: an automated system for acquisition of images from vitreous ice specimens

We have developed a system to automatically acquire cryo-electron micrographs. The system is designed to emulate all of the decisions and actions of a highly trained microscopist in collecting data from a vitreous ice specimen. These include identifying suitable areas of vitreous ice at low magnification, determining the presence and location of specimen on the grid, automatically adjusting imaging parameters (focus, astigmatism) under low-dose conditions, and acquiring images at high magnification to either film or a digital camera. This system is responsible for every aspect of image acquisition and can run unattended, other than requiring periodic refilling of the cryogens, for over 24 h. The system has been tested out on a variety of specimens that represent typical challenges in the field of cryo-electron microscopy. The results show that the overall performance of the system is equivalent to that of an experienced microscopist.

Compassion and altruism in psychoanalytic theory: an evolutionary analysis of self psychology

Freud's creation of psychoanalysis was, in part, a reaction to the societal, religious morality that denied the ubiquitous drivenness that repeatedly confronted him, the essential animal nature of homo sapiens as had been recently made clear by the theory of evolution. For example, Freud (1933) wrote an aggression, It is a general principle. . .that conflicts of interest between men are settled by the use of violence. This is true of the whole animal kingdom, from which men have no business to exclude themselves. Though evolutionary theory was in its infancy, incompletely understood even by its creator, Freud's commitment to facing its truths led to an unswerving stance in reaction to attempts to deny the narcissistic injury inherent in his psychoanalytic discoveries. He insisted on trying to reinterpret virtually all social behaviors in the light of his new theory, and he and his followers have stretched his drive/structure model to its limits. Yet, as we have seen, this evolutionary creation--the human psyche--cannot be fully accounted for utilizing the vicissitudes of Freud's two instincts. What we come to appreciate when we bring the perspective of the theory of evolution to the relational/structure versus drive/structure debate, is that the debate is about the two sides of the same coin. Like this metaphor, in the case of the selfish, yet social, human animal, you cannot have a one-sided coin. Both drives and relationships are biologically inherent and have their structuralizing effect upon the supraordinate self. A modern evolutionary biological, psychoanalytic conception of conflicts and drives may actually be closer to the adaptive/functional tone of aspects of the self psychological paradigm than to the traditional perspective. As human animals we are inherently in conflict over our irreducible biologically based driven, asocial needs (i.e., self-enhancing pleasure seeking and avoidance of unpleasure) and our irreducible biologically based needs for a self-selfobject milieu. A dynamic tension between these two motivational pulls is adaptive (has been selected for) due to the great flexibility it provides in enabling this large brained, nonreflexively driven, social organism fully to exploit all aspects of its environment in pursuit of its own best interest. In its clinical application this viewpoint sees our patients caught between their basic self-enhancing instinctual drivenness for sensual pleasure and power, and their irreducible self-enhancing need for a self-selfobject milieu.(ABSTRACT TRUNCATED AT 400 WORDS)