What do blind people "see" with retinal prostheses? Observations and qualitative reports of epiretinal implant users

Machine Learning Techniques for Simulating Human Psychophysical Testing of Low-Resolution Phosphene Face Images in Artificial Vision

To evaluate the quality of artificial visual percepts generated by emerging methodologies, researchers often rely on labor-intensive and tedious human psychophysical experiments. These experiments necessitate repeated iterations upon any major/minor modifications in the hardware/software configurations. Here, the capacity of standard machine learning (ML) models is investigated to accurately replicate quaternary match-to-sample tasks using low-resolution facial images represented by arrays of phosphenes as input stimuli. Initially, the performance of the ML models trained to approximate innate human facial recognition abilities across a dataset comprising 3600 phosphene images of human faces is analyzed. Subsequently, due to the time constraints and the potential for subject fatigue, the psychophysical test is limited to presenting only 720 low-resolution phosphene images to 36 human subjects. Notably, the superior model adeptly mirrors the behavioral trend of human subjects, offering precise predictions for 8 out of 9 phosphene quality levels on the overlapping test queries. Subsequently, human recognition performances for untested phosphene images are predicted, streamlining the process and minimizing the need for additional psychophysical tests. The findings underscore the transformative potential of ML in reshaping the research paradigm of visual prosthetics, facilitating the expedited advancement of prostheses.

Short pulse epiretinal stimulation allows focal activation of retinal ganglion cells

Epiretinal implants suffer from a lack of spatial resolution, which is greatly influenced by the undesired co-activation of distal cells with their axons passing close to targeted somas. Short current pulses in the range of 50μs have been shown to preferentially activate somas, but the low specificity may limit practical applications. In this paper, we explored decreasing pulse durations down to 10μs for achieving focal activation, i.e., a large differentiation between axonal and somatic activation in epiretinal configuration. We determined thresholds for pulses ranging between 10 and 500μs in retinal ganglion cells of both wild-type and photoreceptor-degenerated mouse retina. Ex-vivo stimulation using biphasic rectangular pulses was performed using a custom-built modified Howland-type current-controlled stimulator and a microelectrode. We demonstrate reliable direct activation of retinal ganglion cells using 10μs pulses for both somatic and axonal electrode positions. Cells from wild-type and photoreceptor-degenerated retinas exhibited similar thresholds. Axonal thresholds were significantly higher for all pulse durations, with the ratio between axonal and somatic thresholds strongly increasing with decreasing pulse duration (1.32 and 4.39 for pulse durations of 500 and 10μs, respectively). Computational modeling points to somatic polarization as the underlying mechanism for lower somatic thresholds. Our results demonstrate focal activation with pulses in the range of 10μs as a potential strategy to avoid the long-standing problem of axonal co-activation in epiretinal implants.

Wide-angle simulated artificial vision enhances spatial navigation and object interaction in a naturalistic environment

Objective. Vision restoration approaches, such as prosthetics and optogenetics, provide visual perception to blind individuals in clinical settings. Yet their effectiveness in daily life remains a challenge. Stereotyped quantitative tests used in clinical trials often fail to translate into practical, everyday applications. On the one hand, assessing real-life benefits during clinical trials is complicated by environmental complexity, reproducibility issues, and safety concerns. On the other hand, predicting behavioral benefits of restorative therapies in naturalistic environments may be a crucial step before starting clinical trials to minimize patient discomfort and unmet expectations.Approach. To address this, we leverage advancements in virtual reality technology to conduct a fully immersive and ecologically valid task within a physical artificial street environment. As a case study, we assess the impact of the visual field size in simulated artificial vision for common outdoor tasks.Main results. We show that a wide visual angle (45°) enhances participants' ability to navigate and solve tasks more effectively, safely, and efficiently. Moreover, it promotes their learning and generalization capability. Concurrently, it changes the visual exploration behavior and facilitates a more accurate mental representation of the environment. Further increasing the visual angle beyond this value does not yield significant additional improvements in most metrics.Significance. We present a methodology combining augmented reality with a naturalistic environment, enabling participants to perceive the world as patients with retinal implants would and to interact physically with it. Combining augmented reality in naturalistic environments is a valuable framework for low vision and vision restoration research.

Aligning Visual Prosthetic Development With Implantee Needs

Purpose

Visual prosthetics are a promising assistive technology for vision loss, yet research often overlooks the human aspects of this technology. While previous studies focus on the perceptual experiences or attitudes of implant recipients (implantees), a systematic account of how current implants are being used in everyday life is still lacking.

Methods

We interviewed six recipients of the most widely used visual implants (Argus II and Orion) and six leading researchers in the field. Through thematic analyses, we explored the daily usage of these implants by implantees and compared their responses to the expectations of researchers. We also sought implantees' input on desired features for future versions, aiming to inform the development of the next generation of implants.

Results

Although implants are designed to facilitate various daily activities, we found that implantees use them less frequently than researchers expect. This discrepancy primarily stems from issues with usability and reliability, with implantees finding alternative methods to accomplish tasks, reducing the need to rely on the implant. For future implants, implantees emphasized the desire for improved vision, smart integration, and increased independence.

Conclusions

Our study reveals a significant gap between researcher expectations and implantee experiences with visual prostheses. Although limited by access to a small population of implantees, this study highlights the importance of focusing future research on usability and real-world applications.

Translational Relevance

This retrospective qualitative study advocates for a better alignment between technology development and implantee needs to enhance clinical relevance and practical utility of visual prosthetics.

Pre-processing visual scenes for retinal prosthesis systems: A comprehensive review

BACKGROUND

Retinal prostheses offer hope for individuals with degenerative retinal diseases by stimulating the remaining retinal cells to partially restore their vision. This review delves into the current advancements in retinal prosthesis technology, with a special emphasis on the pivotal role that image processing and machine learning techniques play in this evolution.

METHODS

We provide a comprehensive analysis of the existing implantable devices and optogenetic strategies, delineating their advantages, limitations, and challenges in addressing complex visual tasks. The review extends to various image processing algorithms and deep learning architectures that have been implemented to enhance the functionality of retinal prosthetic devices. We also illustrate the testing results by demonstrating the clinical trials or using Simulated Prosthetic Vision (SPV) through phosphene simulations, which is a critical aspect of simulating visual perception for retinal prosthesis users.

RESULTS

Our review highlights the significant progress in retinal prosthesis technology, particularly its capacity to augment visual perception among the visually impaired. It discusses the integration between image processing and deep learning, illustrating their impact on individual interactions and navigations within the environment through applying clinical trials and also illustrating the limitations of some techniques to be used with current devices, as some approaches only use simulation even on sighted-normal individuals or rely on qualitative analysis, where some consider realistic perception models and others do not.

CONCLUSION

This interdisciplinary field holds promise for the future of retinal prostheses, with the potential to significantly enhance the quality of life for individuals with retinal prostheses. Future research directions should pivot towards optimizing phosphene simulations for SPV approaches, considering the distorted and confusing nature of phosphene perception, thereby enriching the visual perception provided by these prosthetic devices. This endeavor will not only improve navigational independence but also facilitate a more immersive interaction with the environment.

Aligning visual prosthetic development with implantee needs

Purpose

Visual prosthetics are a promising assistive technology for vision loss, yet research often overlooks the human aspects of this technology. While previous studies focus on the perceptual experiences or attitudes of implant recipients (implantees) , a systematic account of how current implants are being used in everyday life is still lacking.

Methods

We interviewed six recipients of the most widely used visual implants (Argus II and Orion) and six leading researchers in the field. Through thematic analyses, we explored the daily usage of these implants by implantees and compared their responses to the expectations of researchers. We also sought implantees' input on desired features for future versions, aiming to inform the development of the next generation of implants.

Results

Although implants are designed to facilitate various daily activities, we found that implantees use them less frequently than researchers expect. This discrepancy primarily stems from issues with usability and reliability, with implantees finding alternative methods to accomplish tasks, reducing the need to rely on the implant. For future implants, implantees emphasized the desire for improved vision, smart integration, and increased independence.

Conclusions

Our study reveals a significant gap between researcher expectations and implantee experiences with visual prostheses. Although limited by access to a small population of implantees, this study highlights the importance of focusing future research on usability and real-world applications.

Translational relevance

This retrospective qualitative study advocates for a better alignment between technology development and implantee needs to enhance clinical relevance and practical utility of visual prosthetics.

Assessment of Accuracy of Spatial Object Localization by Means of Mono and Stereo Modes of Visual-to-Auditory Sensory Substitution in People with Visual Impairments (a Pilot Study)

The aim of the study is to assess the accuracy of spatial object localization in mono and stereo modes of visual-to-auditory sensory substitution by means of the developed system tested on persons with normal or corrected-to-normal vision.

Materials and Methods

A prototype of a visual-to-auditory sensory substitution device based on a video camera with two lenses was prepared. Software to convert the signal from a video camera into an audio signal in mono and stereo modes was developed.To assess the developed system, an experimental study with 30 blindfolded sighted participants was conducted. 15 persons were tested in mono mode, 15 - in stereo mode. All persons were trained to use the visual-to-auditory sensory substitution system. During the experiment, participants were to locate a white plastic cube with dimensions of 4×4×4 cm3 on a working surface. The researcher placed the cube in one of 20 positions on the working surface in a pseudo-random order.

Results

To assess the accuracy of the cube localization, deviations along the X- and Y-axes and absolute deviations were calculated. The general dynamics of localization accuracy was positive both in mono and stereo modes. Absolute deviation and X-axis deviation were significantly higher in stereo mode; there was no significant difference in Y-axis deviation between modes. On average, participants tended to underestimate the distance to the cube when it was on the left, right, or far side of the working surface, and overestimate the distance to the cube when it was on the near side of the working surface.

Conclusion

Tests demonstrated that the accuracy of object localization in stereo mode can be improved by increasing the time for training the participants and by showing them more presentations. The results of the study can be used to develop assistive techniques for people with visual impairments, to manufacture medical equipment, and create brain-computer interfaces.

Disentangling Function from Benefit: Participant Perspectives from an Early Feasibility Trial for a Novel Visual Cortical Prosthesis

Visual cortical prostheses (VCPs) have the potential to provide artificial vision for visually impaired persons. However, the nature and utility of this form of vision is not yet fully understood. Participants in the early feasibility trial for the Orion VCP were interviewed to gain insight into their experiences using artificial vision, their motivations for participation, as well as their expectations and assessments of risks and benefits. Analyzed using principles of grounded theory and an interpretive description approach, these interviews yielded six themes, including: the irreducibility of benefit to device functionality, mixed expectations for short-term device functionality and long-term technological advancement of visual prostheses, and a broad range of risks, concerns, and fears related to trial participation. We argue that these narratives motivate a nuanced set of ethical considerations related to the complex relationship between functionality and benefit, the intersection of user experience with disability justice, and the import of expectations and indirect risks on consent.

Predicting the Temporal Dynamics of Prosthetic Vision

Retinal implants are a promising treatment option for degenerative retinal disease. While numerous models have been developed to simulate the appearance of elicited visual percepts ("phosphenes"), these models often either focus solely on spatial characteristics or inadequately capture the complex temporal dynamics observed in clinical trials, which vary heavily across implant technologies, subjects, and stimulus conditions. Here we introduce two computational models designed to accurately predict phosphene fading and persistence under varying stimulus conditions, cross-validated on behavioral data reported by nine users of the Argus II Retinal Prosthesis System. Both models segment the time course of phosphene perception into discrete intervals, decomposing phosphene fading and persistence into either sinusoidal or exponential components. Our spectral model demonstrates state-of-the-art predictions of phosphene intensity over time (r = 0.7 across all participants). Overall, this study lays the groundwork for enhancing prosthetic vision by improving our understanding of phosphene temporal dynamics.

Photovoltaic, wireless wide-field epiretinal prosthesis to treat retinitis pigmentosa

PURPOSE

To develop and evaluate a photovoltaic, wireless wide-field epiretinal prosthesis for the treatment of retinitis pigmentosa.

METHODS

A mosaic array of thinned silicon-based photodiodes with integrated thin-film stimulation electrodes was fabricated with a flexible polyimide substrate film to form a film-based miniaturized electronic system with wireless optical power and signal transmission and integrated electrostimulation. Manufactured implants were characterized with respect to their optoelectronic performance and biocompatibility following DIN EN ISO 10993.

RESULTS

A 14 mm diameter prosthesis containing 1276 pixels with a maximum sensitivity at a near infrared wavelength of 905 nm and maximized stimulation current density 30-50 μm below the electrodes was developed for direct activation of retinal ganglion cells during epiretinal stimulation. Fabricated prostheses demonstrated mucosal tolerance and the preservation of both metabolic activity, proliferation and membrane integrity of human fibroblasts as well as the retinal functions of bovine retinas. Illumination of the prosthesis, which was placed epiretinally on an isolated perfused bovine retina, with infrared light resulted in electrophysiological recordings reminiscent of an a-wave (hyperpolarization) and b-wave (depolarization).

CONCLUSIONS

A photovoltaic, wireless wide-field epiretinal prosthesis for the treatment of retinitis pigmentosa using near infrared light for signal transmission was designed, manufactured and its biocompatibility and functionality demonstrated in vitro and ex vivo.

The Influence of Phosphene Synchrony in Driving Object Binding in a Simulation of Artificial Vision

Purpose

Electrical microstimulation techniques used in visual prostheses are designed to restore visual function following acquired blindness. Patterns of induced focal percepts, known as phosphenes, are achieved by applying localized electrical pulses to the visual pathway to bypass the impaired site in order to convey images from the external world. Here, we use a simulation of artificial vision to manipulate relationships between individual phosphenes to observe the effects on object binding and perception. We hypothesize that synchronous phosphene presentation will facilitate object perception as compared to asynchronous presentation.

Methods

A model system that tracks gaze position of normal, sighted participants to present patterns of phosphenes on a computer screen was used to simulate prosthetic vision. Participants performed a reading task at varying font sizes (1.1-1.4 logMAR) and under varying levels of phosphene temporal noise while reading accuracy and speed were measured.

Results

Reading performance was significantly affected by temporal noise in phosphene presentation, with increasing desynchronization leading to lower reading scores. A drop in performance was also observed when the total latency between the gaze position and phosphene update was increased without adding temporal noise.

Conclusions

Object perception (here, text perception) is enhanced with synchronously presented phosphenes as compared to asynchronously presented ones. These results are fundamental for developing an efficient temporal pattern of stimulation and for the creation of high-fidelity prosthetic vision.

Human-in-the-Loop Optimization for Deep Stimulus Encoding in Visual Prostheses

Neuroprostheses show potential in restoring lost sensory function and enhancing human capabilities, but the sensations produced by current devices often seem unnatural or distorted. Exact placement of implants and differences in individual perception lead to significant variations in stimulus response, making personalized stimulus optimization a key challenge. Bayesian optimization could be used to optimize patient-specific stimulation parameters with limited noisy observations, but is not feasible for high-dimensional stimuli. Alternatively, deep learning models can optimize stimulus encoding strategies, but typically assume perfect knowledge of patient-specific variations. Here we propose a novel, practically feasible approach that overcomes both of these fundamental limitations. First, a deep encoder network is trained to produce optimal stimuli for any individual patient by inverting a forward model mapping electrical stimuli to visual percepts. Second, a preferential Bayesian optimization strategy utilizes this encoder to optimize patient-specific parameters for a new patient, using a minimal number of pairwise comparisons between candidate stimuli. We demonstrate the viability of this approach on a novel, state-of-the-art visual prosthesis model. We show that our approach quickly learns a personalized stimulus encoder, leads to dramatic improvements in the quality of restored vision, and is robust to noisy patient feedback and misspecifications in the underlying forward model. Overall, our results suggest that combining the strengths of deep learning and Bayesian optimization could significantly improve the perceptual experience of patients fitted with visual prostheses and may prove a viable solution for a range of neuroprosthetic technologies.

Optogenetic activation of visual thalamus generates artificial visual percepts

The lateral geniculate nucleus (LGN), a retinotopic relay center where visual inputs from the retina are processed and relayed to the visual cortex, has been proposed as a potential target for artificial vision. At present, it is unknown whether optogenetic LGN stimulation is sufficient to elicit behaviorally relevant percepts, and the properties of LGN neural responses relevant for artificial vision have not been thoroughly characterized. Here, we demonstrate that tree shrews pretrained on a visual detection task can detect optogenetic LGN activation using an AAV2-CamKIIα-ChR2 construct and readily generalize from visual to optogenetic detection. Simultaneous recordings of LGN spiking activity and primary visual cortex (V1) local field potentials (LFPs) during optogenetic LGN stimulation show that LGN neurons reliably follow optogenetic stimulation at frequencies up to 60 Hz and uncovered a striking phase locking between the V1 LFP and the evoked spiking activity in LGN. These phase relationships were maintained over a broad range of LGN stimulation frequencies, up to 80 Hz, with spike field coherence values favoring higher frequencies, indicating the ability to relay temporally precise information to V1 using light activation of the LGN. Finally, V1 LFP responses showed sensitivity values to LGN optogenetic activation that were similar to the animal's behavioral performance. Taken together, our findings confirm the LGN as a potential target for visual prosthetics in a highly visual mammal closely related to primates.

Attitudes of potential recipients toward emerging visual prosthesis technologies

With the advent of multiple visual prosthesis devices to treat blindness, the question of how potential patients view such interventions becomes important in order to understand the levels of expectation and acceptance, and the perceived risk-reward balance across the different device approaches. Building on previous work on single device approaches done with blind individuals in Chicago and Detroit, USA, Melbourne, Australia, and Bejing, China, we investigated attitudes in blind individuals in Athens, Greece with coverage expanded to three of the contemporary approaches, Retinal, Thalamic, and Cortical. We presented an informational lecture on the approaches, had potential participants fill out a preliminary Questionnaire 1, then organized selected subjects into focus groups for guided discussion on visual prostheses, and finally had these subjects fill out a more detailed Questionnaire 2. We report here the first quantitative data that compares multiple prosthesis approaches. Our primary findings are that for these potential patients, perceived risk continues to outweigh perceived benefits, with the Retinal approach having the least negative overall impression and the Cortical approach the most negative. Concerns about the quality of restored vision were primary. Factors that drove the choice of hypothetical participation in a clinical trial were age and years of blindness. Secondary factors focused on positive clinical outcomes. The focus groups served to swing the impressions of each approach from neutrality toward the extremes of a Likert scale, and shifted the overall willingness to participate in a clinical trial from neutral to negative. These results, coupled with informal assessment of audience questions after the informational lecture, suggest that a substantial improvement in performance over currently available devices will be necessary before visual prostheses gain wide acceptance.

A Deep Learning-based in silico Framework for Optimization on Retinal Prosthetic Stimulation

We propose a neural network-based framework to optimize the perceptions simulated by the in silico retinal implant model pulse2percept. The overall pipeline consists of a trainable encoder, a pre-trained retinal implant model and a pre-trained evaluator. The encoder is a U-Net, which takes the original image and outputs the stimulus. The pre-trained retinal implant model is also a U-Net, which is trained to mimic the biomimetic perceptual model implemented in pulse2percept. The evaluator is a shallow VGG classifier, which is trained with original images. Based on 10,000 test images from the MNIST dataset, we show that the convolutional neural network-based encoder performs significantly better than the trivial downsampling approach, yielding a boost in the weighted F1-Score by 36.17% in the pre-trained classifier with 6×10 electrodes. With this fully neural network-based encoder, the quality of the downstream perceptions can be fine-tuned using gradient descent in an end-to-end fashion.

Retinal Prostheses: Engineering and Clinical Perspectives for Vision Restoration

A retinal prosthesis, also known as a bionic eye, is a device that can be implanted to partially restore vision in patients with retinal diseases that have resulted in the loss of photoreceptors (e.g., age-related macular degeneration and retinitis pigmentosa). Recently, there have been major breakthroughs in retinal prosthesis technology, with the creation of numerous types of implants, including epiretinal, subretinal, and suprachoroidal sensors. These devices can stimulate the remaining cells in the retina with electric signals to create a visual sensation. A literature review of the pre-clinical and clinical studies published between 2017 and 2023 is conducted. This narrative review delves into the retinal anatomy, physiology, pathology, and principles underlying electronic retinal prostheses. Engineering aspects are explored, including electrode-retina alignment, electrode size and material, charge density, resolution limits, spatial selectivity, and bidirectional closed-loop systems. This article also discusses clinical aspects, focusing on safety, adverse events, visual function, outcomes, and the importance of rehabilitation programs. Moreover, there is ongoing debate over whether implantable retinal devices still offer a promising approach for the treatment of retinal diseases, considering the recent emergence of cell-based and gene-based therapies as well as optogenetics. This review compares retinal prostheses with these alternative therapies, providing a balanced perspective on their advantages and limitations. The recent advancements in retinal prosthesis technology are also outlined, emphasizing progress in engineering and the outlook of retinal prostheses. While acknowledging the challenges and complexities of the technology, this article highlights the significant potential of retinal prostheses for vision restoration in individuals with retinal diseases and calls for continued research and development to refine and enhance their performance, ultimately improving patient outcomes and quality of life.

A systematic review of extended reality (XR) for understanding and augmenting vision loss

Over the past decade, extended reality (XR) has emerged as an assistive technology not only to augment residual vision of people losing their sight but also to study the rudimentary vision restored to blind people by a visual neuroprosthesis. A defining quality of these XR technologies is their ability to update the stimulus based on the user's eye, head, or body movements. To make the best use of these emerging technologies, it is valuable and timely to understand the state of this research and identify any shortcomings that are present. Here we present a systematic literature review of 227 publications from 106 different venues assessing the potential of XR technology to further visual accessibility. In contrast to other reviews, we sample studies from multiple scientific disciplines, focus on technology that augments a person's residual vision, and require studies to feature a quantitative evaluation with appropriate end users. We summarize prominent findings from different XR research areas, show how the landscape has changed over the past decade, and identify scientific gaps in the literature. Specifically, we highlight the need for real-world validation, the broadening of end-user participation, and a more nuanced understanding of the usability of different XR-based accessibility aids.

Towards aSmart Bionic Eye: AI-powered artificial vision for the treatment of incurable blindness

Objective.How can we return a functional form of sight to people who are living with incurable blindness? Despite recent advances in the development of visual neuroprostheses, the quality of current prosthetic vision is still rudimentary and does not differ much across different device technologies.Approach.Rather than aiming to represent the visual scene as naturally as possible, aSmart Bionic Eyecould provide visual augmentations through the means of artificial intelligence-based scene understanding, tailored to specific real-world tasks that are known to affect the quality of life of people who are blind, such as face recognition, outdoor navigation, and self-care.Main results.Complementary to existing research aiming to restore natural vision, we propose a patient-centered approach to incorporate deep learning-based visual augmentations into the next generation of devices.Significance.The ability of a visual prosthesis to support everyday tasks might make the difference between abandoned technology and a widely adopted next-generation neuroprosthetic device.

Potential therapeutic strategies for photoreceptor degeneration: the path to restore vision

Photoreceptors (PRs), as the most abundant and light-sensing cells of the neuroretina, are responsible for converting light into electrical signals that can be interpreted by the brain. PR degeneration, including morphological and functional impairment of these cells, causes significant diminution of the retina's ability to detect light, with consequent loss of vision. Recent findings in ocular regenerative medicine have opened promising avenues to apply neuroprotective therapy, gene therapy, cell replacement therapy, and visual prostheses to the challenge of restoring vision. However, successful visual restoration in the clinical setting requires application of these therapeutic approaches at the appropriate stage of the retinal degeneration. In this review, firstly, we discuss the mechanisms of PR degeneration by focusing on the molecular mechanisms underlying cell death. Subsequently, innovations, recent developments, and promising treatments based on the stage of disorder progression are further explored. Then, the challenges to be addressed before implementation of these therapies in clinical practice are considered. Finally, potential solutions to overcome the current limitations of this growing research area are suggested. Overall, the majority of current treatment modalities are still at an early stage of development and require extensive additional studies, both pre-clinical and clinical, before full restoration of visual function in PR degeneration diseases can be realized.

Hybrid Neural Autoencoders for Stimulus Encoding in Visual and Other Sensory Neuroprostheses

Sensory neuroprostheses are emerging as a promising technology to restore lost sensory function or augment human capabilities. However, sensations elicited by current devices often appear artificial and distorted. Although current models can predict the neural or perceptual response to an electrical stimulus, an optimal stimulation strategy solves the inverse problem: what is the required stimulus to produce a desired response? Here, we frame this as an end-to-end optimization problem, where a deep neural network stimulus encoder is trained to invert a known and fixed forward model that approximates the underlying biological system. As a proof of concept, we demonstrate the effectiveness of this hybrid neural autoencoder (HNA) in visual neuroprostheses. We find that HNA produces high-fidelity patient-specific stimuli representing handwritten digits and segmented images of everyday objects, and significantly outperforms conventional encoding strategies across all simulated patients. Overall this is an important step towards the long-standing challenge of restoring high-quality vision to people living with incurable blindness and may prove a promising solution for a variety of neuroprosthetic technologies.

The Relative Importance of Depth Cues and Semantic Edges for Indoor Mobility Using Simulated Prosthetic Vision in Immersive Virtual Reality

Visual neuroprostheses (bionic eyes) have the potential to treat degenerative eye diseases that often result in low vision or complete blindness. These devices rely on an external camera to capture the visual scene, which is then translated frame-by-frame into an electrical stimulation pattern that is sent to the implant in the eye. To highlight more meaningful information in the scene, recent studies have tested the effectiveness of deep-learning based computer vision techniques, such as depth estimation to highlight nearby obstacles (DepthOnly mode) and semantic edge detection to outline important objects in the scene (EdgesOnly mode). However, nobody has yet attempted to combine the two, either by presenting them together (EdgesAndDepth) or by giving the user the ability to flexibly switch between them (EdgesOrDepth). Here, we used a neurobiologically inspired model of simulated prosthetic vision (SPV) in an immersive virtual reality (VR) environment to test the relative importance of semantic edges and relative depth cues to support the ability to avoid obstacles and identify objects. We found that participants were significantly better at avoiding obstacles using depth-based cues as opposed to relying on edge information alone, and that roughly half the participants preferred the flexibility to switch between modes (EdgesOrDepth). This study highlights the relative importance of depth cues for SPV mobility and is an important first step towards a visual neuroprosthesis that uses computer vision to improve a user's scene understanding.

Assessing people with visual impairments’ access to information, awareness and satisfaction with high-tech assistive technology

Assistive technology (AT) devices are designed to help people with visual impairments (PVIs) perform activities that would otherwise be difficult or impossible. Devices specifically designed to assist PVIs by attempting to restore sight or substitute it for another sense have a very low uptake rate. This study, conducted in England, aimed to investigate why this is the case by assessing accessibility to knowledge, awareness, and satisfaction with AT in general and with sensory restoration and substitution devices in particular. From a sample of 25 PVIs, ranging from 21 to 68 years old, results showed that participants knew where to find AT information; however, health care providers were not the main source of this information. Participants reported good awareness of different ATs, and of technologies they would not use, but reported poor awareness of specific sensory substitution and restoration devices. Only three participants reported using AT, each with different devices and varying levels of satisfaction. The results from this study suggest a possible breakdown in communication between health care providers and PVIs, and dissociation between reported AT awareness and reported access to AT information. Moreover, awareness of sensory restoration and substitution devices is poor, which may explain the limited use of such technology.

Clinical Progress and Optimization of Information Processing in Artificial Visual Prostheses

Visual prostheses, used to assist in restoring functional vision to the visually impaired, convert captured external images into corresponding electrical stimulation patterns that are stimulated by implanted microelectrodes to induce phosphenes and eventually visual perception. Detecting and providing useful visual information to the prosthesis wearer under limited artificial vision has been an important concern in the field of visual prosthesis. Along with the development of prosthetic device design and stimulus encoding methods, researchers have explored the possibility of the application of computer vision by simulating visual perception under prosthetic vision. Effective image processing in computer vision is performed to optimize artificial visual information and improve the ability to restore various important visual functions in implant recipients, allowing them to better achieve their daily demands. This paper first reviews the recent clinical implantation of different types of visual prostheses, summarizes the artificial visual perception of implant recipients, and especially focuses on its irregularities, such as dropout and distorted phosphenes. Then, the important aspects of computer vision in the optimization of visual information processing are reviewed, and the possibilities and shortcomings of these solutions are discussed. Ultimately, the development direction and emphasis issues for improving the performance of visual prosthesis devices are summarized.

Factors affecting two-point discrimination in Argus II patients

Two of the main obstacles to the development of epiretinal prosthesis technology are electrodes that require current amplitudes above safety limits to reliably elicit percepts, and a failure to consistently elicit pattern vision. Here, we explored the causes of high current amplitude thresholds and poor spatial resolution within the Argus II epiretinal implant. We measured current amplitude thresholds and two-point discrimination (the ability to determine whether one or two electrodes had been stimulated) in 3 blind participants implanted with Argus II devices. Our data and simulations show that axonal stimulation, lift and retinal damage all play a role in reducing performance in the Argus 2, by either limiting sensitivity and/or reducing spatial resolution. Understanding the relative role of these various factors will be critical for developing and surgically implanting devices that can successfully subserve pattern vision.

Retinal ganglion cell desensitization is mitigated by varying parameter constant excitation pulse trains

Retinal prostheses partially restore vision in patients blinded by retinitis pigmentosa (RP) and age-related macular degeneration (AMD). One issue that limits the effectiveness of retinal stimulation is the desensitization of the retina response to repeated pulses. Rapid fading of percepts is reported in clinical studies. We studied the retinal output evoked by fixed pulse trains vs. pulse trains that have variable parameters pulse-to-pulse. We used the current clamp to record RGC spiking in the isolated mouse retina. Trains of biphasic current pulses at different frequencies and amplitudes were applied. The main results we report are: (1) RGC desensitization was induced by increasing stimulus frequency, but was unrelated to stimulus amplitude. Desensitization persisted when the 20 Hz stimulation pulses were applied to the retinal ganglion cells at 65 μA, 85 μA, and 105 μA. Subsequent pulses in the train evoked fewer spikes. There was no obvious desensitization when 2 Hz stimulation pulse trains were applied. (2) Blocking inhibitory GABA_A receptor increased spontaneous activity but did not reduce desensitization. (3) Pulse trains with constant charge or excitation (based on strength-duration curves) but varying pulse width, amplitude, and shape increased the number of evoked spikes/pulse throughout the pulse train. This suggests that retinal desensitization can be partially overcome by introducing variability into each pulse.

Reading text works better than watching videos to improve acuity in a simulation of artificial vision

Simulated artificial vision is used in visual prosthesis design to answer questions about device usability. We previously reported a striking increase in equivalent visual acuity with daily use of a simulation of artificial vision in an active task, reading sentences, that required high levels of subject engagement, but passive activities are more likely to dominate post-implant experience. Here, we investigated the longitudinal effects of a passive task, watching videos. Eight subjects used a simulation of a thalamic visual prosthesis with 1000 phosphenes to watch 23 episodes of classic American television in daily, 25-min sessions, for a period of 1 month with interspersed reading tests that quantified reading accuracy and reading speed. For reading accuracy, we found similar dynamics to the early part of the learning process in our previous report, here leading to an improvement in visual acuity of 0.15 ± 0.05 logMAR. For reading speed, however, no change was apparent by the end of training. We found that single reading sessions drove about twice the improvement in acuity of single video sessions despite being only half as long. We conclude that while passive viewing tasks may prove useful for post-implant rehabilitation, active tasks are likely to be preferable.

Improving the spatial resolution of artificial vision using midget retinal ganglion cell populations modelled at the human fovea

OBJECTIVE

Retinal prostheses seek to create artificial vision by stimulating surviving retinal neurons of patients with profound vision impairment. Notwithstanding tremendous research efforts, the performance of all implants tested to date has remained rudimentary, incapable of overcoming the threshold for legal blindness. To maximize the perceptual efficacy of retinal prostheses, a device must be capable of controlling retinal neurons with greater spatiotemporal precision. Most studies of retinal stimulation were derived from either non-primate species or the peripheral primate retina. We investigated if artificial stimulation could leverage the high spatial resolution afforded by the neural substrates at the primate fovea and surrounding regions to achieve improved percept qualities.

APPROACH

We began by developing a new computational model capable of generating anatomically accurate retinal ganglion cell (RGC) populations within the human central retina. Next, multiple RGC populations across the central retina were stimulated in-silico to compare clinical and recently proposed neurostimulation configurations based on their ability to improve perceptual efficacy and reduce activation thresholds.

MAIN RESULTS

Our model uniquely upholds eccentricity-dependent characteristics such as RGC density and dendritic field diameter, whilst incorporating anatomically accurate features such as axon projection and three-dimensional RGC layering, features often forgone in favor of reduced computational complexity. Following epiretinal stimulation, the RGCs in our model produced response patterns in shapes akin to the complex percepts reported in clinical trials. Our results also demonstrated that even within the neuron-dense central retina, epiretinal stimulation using a multi-return hexapolar electrode arrangement could reliably achieve spatially focused RGC activation and could achieve single-cell excitation in 74% of all tested locations.

SIGNIFICANCE

This study establishes an anatomically accurate three-dimensional model of the human central retina and demonstrates the potential for an epiretinal hexapolar configuration to achieve consistent, spatially confined retinal responses, even within the neuron-dense foveal region. Our results promote the prospect and optimization of higher spatial resolution in future epiretinal implants.

Simulating the perceptual effects of electrode-retina distance in prosthetic vision

OBJECTIVE

Retinal prostheses aim to restore some vision in retinitis pigmentosa and age-related macular degeneration blind patients. Many spatial and temporal aspects have been found to affect prosthetic vision. Our objective is to study the impact of the space-variant distance between the stimulating electrodes and the surface of the retina on prosthetic vision and how to mitigate this impact.

APPROACH

A prosthetic vision simulation was built to demonstrate the perceptual effects of the electrode-retina distance (ERD) with different random spatial variations, such as size, brightness, shape, dropout, and spatial shifts. Three approaches for reducing the ERD effects are demonstrated: electrode grouping (quads), ERD-based input-image enhancement, and object scanning with and without phosphene persistence. A quantitative assessment for the first two approaches was done based on experiments with 20 subjects and three vision-based computational image similarity metrics.

MAIN RESULTS

The effects of various ERDs on phosphenes' size, brightness, and shape were simulated. Quads, chosen according to the ERDs, effectively elicit phosphenes without exceeding the safe charge density limit, whereas single electrodes with large ERD cannot do so. Input-image enhancement reduced the ERD effects effectively. These two approaches significantly improved ERD-affected prosthetic vision according to the experiment and image similarity metrics. A further reduction of the ERD effects was achieved by scanning an object while moving the head.

SIGNIFICANCE

ERD has multiple effects on perception with retinal prostheses. One of them is vision loss caused by the incapability of electrodes with large ERD to evoke phosphenes. The three approaches presented in this study can be used separately or together to mitigate the impact of ERD. A consideration of our approaches in reducing the perceptual effects of the ERD may help improve the perception with current prosthetic technology and influence the design of future prostheses.

Electrical devices for visual restoration

Given the rising number of patients with blindness from macular, optic nerve, and visual pathway disease, there is considerable interest in the potential of electrical stimulation devices to restore vision. Electrical devices for restoration of visual function can be grouped into three categories: (1) visual prostheses whose goal is to bypass damaged areas and directly activate downstream intact portions of the visual pathway; (2) electric field stimulation whose goal is to activate endogenous transcriptional and molecular signaling pathways to promote neuroprotection and neuro-regeneration; and (3) neuromodulation whose stimulation would resuscitate neural circuits vital to coordinating responses to visual input.  In this review, we discuss these three approaches, describe advances made in the different fields, and comment on limitations and potential future directions.

Controlled assembly of retinal cells on fractal and Euclidean electrodes

Controlled assembly of retinal cells on artificial surfaces is important for fundamental cell research and medical applications. We investigate fractal electrodes with branches of vertically-aligned carbon nanotubes and silicon dioxide gaps between the branches that form repeating patterns spanning from micro- to milli-meters, along with single-scaled Euclidean electrodes. Fluorescence and electron microscopy show neurons adhere in large numbers to branches while glial cells cover the gaps. This ensures neurons will be close to the electrodes’ stimulating electric fields in applications. Furthermore, glia won’t hinder neuron-branch interactions but will be sufficiently close for neurons to benefit from the glia’s life-supporting functions. This cell ‘herding’ is adjusted using the fractal electrode’s dimension and number of repeating levels. We explain how this tuning facilitates substantial glial coverage in the gaps which fuels neural networks with small-world structural characteristics. The large branch-gap interface then allows these networks to connect to the neuron-rich branches.

An in-silico analysis of electrically-evoked responses of midget and parasol retinal ganglion cells in different retinal regions

BACKGROUND

Visual outcomes provided by present retinal prostheses that primarily target retinal ganglion cells (RGCs) through epiretinal stimulation remain rudimentary, partly due to the limited knowledge of retinal responses under electrical stimulation. Better understanding of how different retinal regions can be quantitatively controlled with high spatial accuracy, will be beneficial to the design of micro-electrode arrays (MEAs) and stimulation strategies for next-generation wide-view, high-resolution epiretinal implants.

METHODS

A computational model was developed to assess neural activity at different eccentricities (2 mm and 5 mm) within the human retina. This model included midget and parasol RGCs with anatomically accurate cell distribution and cell-specific morphological information. We then performed in silico investigations of region-specific RGC responses to epiretinal electrical stimulation using varied electrode sizes (5 µm - 210 µm diameter), emulating both commercialized retinal implants and recently-developed prototype devices.

RESULTS

Our model of epiretinal stimulation predicted RGC population excitation analogous to the complex percepts reported in human subjects. Following this, our simulations suggest that midget and parasol RGCs have characteristic regional differences in excitation under preferred electrode sizes. Relatively central (2 mm) regions demonstrated higher number of excited RGCs but lower overall activated receptive field (RF) areas under the same stimulus amplitudes (two-way ANOVA, p < 0.05). Furthermore, the activated RGC numbers per unit active RF area (number-RF ratio) were significantly higher in central than in peripheral regions, and higher in the midget than in the parasol population under all tested electrode sizes (two-way ANOVA, p < 0.05). Our simulations also suggested that smaller electrodes exhibit a higher range of controllable stimulation parameters to achieve pre-defined performance of RGC excitation. ..

Virtual reality validation of naturalistic modulation strategies to counteract fading in retinal stimulation

OBJECTIVE

Temporal resolution is a key challenge in artificial vision. Several prosthetic approaches are limited by the perceptual fading of evoked phosphenes upon repeated stimulation from the same electrode. Therefore, implanted patients are forced to perform active scanning, via head movements, to refresh the visual field viewed by the camera. However, active scanning is a draining task, and it is crucial to find compensatory strategies to reduce it.

APPROACH

To address this question, we implemented perceptual fading in simulated prosthetic vision using virtual reality. Then, we quantified the effect of fading on two indicators: the time to complete a reading task and the head rotation during the task. We also tested if stimulation strategies previously proposed to increase the persistence of responses in retinal ganglion cells to electrical stimulation could improve these indicators.

MAIN RESULTS

This study shows that stimulation strategies based on interrupted pulse trains and randomisation of the pulse duration allows significant reduction of both the time to complete the task and the head rotation during the task.

SIGNIFICANCE

The stimulation strategy used in retinal implants is crucial to counteract perceptual fading and to reduce active head scanning during prosthetic vision. In turn, less active scanning might improve the patient's comfort in artificial vision.

Immersive Virtual Reality Simulations of Bionic Vision

Bionic vision uses neuroprostheses to restore useful vision to people living with incurable blindness. However, a major outstanding challenge is predicting what people "see" when they use their devices. The limited field of view of current devices necessitates head movements to scan the scene, which is difficult to simulate on a computer screen. In addition, many computational models of bionic vision lack biological realism. To address these challenges, we present VR-SPV, an open-source virtual reality toolbox for simulated prosthetic vision that uses a psychophysically validated computational model to allow sighted participants to "see through the eyes" of a bionic eye user. To demonstrate its utility, we systematically evaluated how clinically reported visual distortions affect performance in a letter recognition and an immersive obstacle avoidance task. Our results highlight the importance of using an appropriate phosphene model when predicting visual outcomes for bionic vision.

Learning to see again: Perceptual learning of simulated abnormal on- off-cell population responses in sighted individuals

Many forms of artificial sight recovery, such as electronic implants and optogenetic proteins, generally cause simultaneous, rather than complementary firing of on- and off-center retinal cells. Here, using virtual patients—sighted individuals viewing distorted input—we examine whether plasticity might compensate for abnormal neuronal population responses. Five participants were dichoptically presented with a combination of original and contrast-reversed images. Each image (I) and its contrast-reverse (Iʹ) was filtered using a radial checkerboard (F) in Fourier space and its inverse (Fʹ). [I * F′] + [Iʹ * F] was presented to one eye, and [I * F] + [Iʹ * F′] was presented to the other, such that regions of the image that produced on-center responses in one eye produced off-center responses in the other eye, and vice versa. Participants continuously improved in a naturalistic object discrimination task over 20 one-hour sessions. Pre-training and post-training tests suggest that performance improvements were due to two learning processes: learning to recognize objects with reduced visual information and learning to suppress contrast-reversed image information in a non–eye-selective manner. These results suggest that, with training, it may be possible to adapt to the unnatural on- and off-cell population responses produced by electronic and optogenetic sight recovery technologies.

A Computational Model of Phosphene Appearance for Epiretinal Prostheses

Retinal neuroprostheses are the only FDA-approved treatment option for blinding degenerative diseases. A major outstanding challenge is to develop a computational model that can accurately predict the elicited visual percepts (phosphenes) across a wide range of electrical stimuli. Here we present a phenomenological model that predicts phosphene appearance as a function of stimulus amplitude, frequency, and pulse duration. The model uses a simulated map of nerve fiber bundles in the retina to produce phosphenes with accurate brightness, size, orientation, and elongation. We validate the model on psychophysical data from two independent studies, showing that it generalizes well to new data, even with different stimuli and on different electrodes. Whereas previous models focused on either spatial or temporal aspects of the elicited phosphenes in isolation, we describe a more comprehensive approach that is able to account for many reported visual effects. The model is designed to be flexible and extensible, and can be fit to data from a specific user. Overall this work is an important first step towards predicting visual outcomes in retinal prosthesis users across a wide range of stimuli.

Retinal prosthetic vision simulation: temporal aspects

Objective. The perception of individuals fitted with retinal prostheses is not fully understood, although several retinal implants have been tested and commercialized. Realistic simulations of perception with retinal implants would be useful for future development and evaluation of such systems.Approach.We implemented a retinal prosthetic vision simulation, including temporal features, which have not been previously simulated. In particular, the simulation included temporal aspects such as persistence and perceptual fading of phosphenes and the electrode activation rate.Main results.The simulated phosphene persistence showed an effective reduction in flickering at low electrode activation rates. Although persistence has a positive effect on static scenes, it smears dynamic scenes. Perceptual fading following continuous stimulation affects prosthetic vision of both static and dynamic scenes by making them disappear completely or partially. However, we showed that perceptual fading of a static stimulus might be countered by head-scanning motions, which together with the persistence revealed the contours of the faded object. We also showed that changing the image polarity may improve simulated prosthetic vision in the presence of persistence and perceptual fading.Significance.Temporal aspects have important roles in prosthetic vision, as illustrated by the simulations. Considering these aspects may improve the future design, the training with, and evaluation of retinal prostheses.

Full gaze contingency provides better reading performance than head steering alone in a simulation of prosthetic vision

The visual pathway is retinotopically organized and sensitive to gaze position, leading us to hypothesize that subjects using visual prostheses incorporating eye position would perform better on perceptual tasks than with devices that are merely head-steered. We had sighted subjects read sentences from the MNREAD corpus through a simulation of artificial vision under conditions of full gaze compensation, and head-steered viewing. With 2000 simulated phosphenes, subjects (n = 23) were immediately able to read under full gaze compensation and were assessed at an equivalent visual acuity of 1.0 logMAR, but were nearly unable to perform the task under head-steered viewing. At the largest font size tested, 1.4 logMAR, subjects read at 59 WPM (50% of normal speed) with 100% accuracy under the full-gaze condition, but at 0.7 WPM (under 1% of normal) with below 15% accuracy under head-steering. We conclude that gaze-compensated prostheses are likely to produce considerably better patient outcomes than those not incorporating eye movements.

Multisensory perception in Argus II retinal prosthesis patients: Leveraging auditory-visual mappings to enhance prosthesis outcomes

Crossmodal mappings associate features (such as spatial location) between audition and vision, thereby aiding sensory binding and perceptual accuracy. Previously, it has been unclear whether patients with artificial vision will develop crossmodal mappings despite the low spatial and temporal resolution of their visual perception (particularly in light of the remodeling of the retina and visual cortex that takes place during decades of vision loss). To address this question, we studied crossmodal mappings psychophysically in Retinitis Pigmentosa patients with partial visual restoration by means of Argus II retinal prostheses, which incorporate an electrode array implanted on the retinal surface that stimulates still-viable ganglion cells with a video stream from a head-mounted camera. We found that Argus II patients (N = 10) exhibit significant crossmodal mappings between auditory location and visual location, and between auditory pitch and visual elevation, equivalent to those of age-matched sighted controls (N = 10). Furthermore, Argus II patients (N = 6) were able to use crossmodal mappings to locate a visual target more quickly with auditory cueing than without. Overall, restored artificial vision was shown to interact with audition via crossmodal mappings, which implies that the reorganization during blindness and the limitations of artificial vision did not prevent the relearning of crossmodal mappings. In particular, cueing based on crossmodal mappings was shown to improve visual search with a retinal prosthesis. This result represents a key first step toward leveraging crossmodal interactions for improved patient visual functionality.