High-speed switchable lens enables the development of a volumetric stereoscopic display

Free-space combiner based on holographic mirrors for head-mounted displays

The paper presents an augmented reality display based on the free-space combiner, consisting of two holographic mirrors: flat and spherical. A feature of such a system is a high-reflection coefficient of mirrors up to 90% for one wavelength and a high transmittance to the lumen up to 67% throughout the rest of the visible spectrum, unlike similar solutions implementing principles of polarization or dichroic beam splitting. The recording scheme and calculation of a separate flat holographic mirror and spherical holographic mirror are described. The proposed augmented reality display can provide multifocality, i.e., reproduction of virtual images at several distances, which demonstrates a certain perspective of this implementation in terms of compensation for the vergence accommodation conflict.

Impact of focus cue presentation on perceived realism of 3-D scene structure: Implications for scene perception and for display technology

Stereoscopic imagery often aims to evoke three-dimensional (3-D) percepts that are accurate and realistic-looking. The "gap" between 3-D imagery and real scenes is small, but focus cues typically remain incorrect because images are displayed on a single focal plane. Research has concentrated on the resulting vergence-accommodation conflicts. Yet, incorrect focus cues may also affect the appearance of 3-D imagery. We investigated whether incorrect focus cues reduce perceived realism of 3-D structure ("depth realism"). Experiment 1 used a multiple-focal-planes display to compare depth realism with correct focus cues vs. conventional stereo presentation. The stimuli were random-dot stereograms, which isolated the role of focus cues. Depth realism was consistently lower with incorrect focus cues, providing proof-of-principle evidence that they contribute to perceptual realism. Experiments 2 and 3 examined whether focus cues play a similar role with realistic objects, presented with an almost complete set of visual cues using a high-resolution, high-dynamic-range multiple-focal-planes display. We also examined the efficacy of approximating correct focus cues via gaze-contingent depth-of-field rendering. Improvements in depth realism with correct focus cues were less clear in more realistic scenes, indicating that the role of focus cues in depth realism depends on scene content. Rendering-based approaches, if anything, reduced depth realism, which we attribute to their inability to present higher-order aspects of blur correctly. Our findings suggest future general 3-D display solutions may need to present focus cues correctly to maximise perceptual realism.

The Perceptual Science of Augmented Reality

Augmented reality (AR) systems aim to alter our view of the world and enable us to see things that are not actually there. The resulting discrepancy between perception and reality can create compelling entertainment and can support innovative approaches to education, guidance, and assistive tools. However, building an AR system that effectively integrates with our natural visual experience is hard. AR systems often suffer from visual limitations and artifacts, and addressing these flaws requires basic knowledge of perception. At the same time, AR system development can serve as a catalyst that drives innovative new research in perceptual science. This review describes recent perceptual research pertinent to and driven by modern AR systems, with the goal of highlighting thought-provoking areas of inquiry and open questions.

Design of a dual focal-plane near-eye display using diffractive waveguides and multiple lenses

We propose a method to construct a compact dual focal-plane optical see-through near-eye display using diffractive waveguides and multiple lenses. A virtual image from a display device is projected into a three-grating waveguide using an objective lens, and a virtual image can be shown at a far distance with an extended eye box. One negative lens is employed to reduce the focus distance of the virtual image, and a corresponding positive lens is used to compensate for the distortion and accommodation errors. Thus, not only can a virtual image with a near distance be achieved, but also a virtual plane with a further distance can be generated by introducing another projection module and waveguide. Only two waveguides and two pieces of lenses are used in front of one eye to obtain a lightweight outlook. To verify the proposed method, a proof-of-concept prototype was developed to provide vivid virtual images at different depths in front of the human eye.

Liquid crystal lens set in augmented reality systems and virtual reality systems for rapidly varifocal images and vision correction

The major challenges of augmented reality (AR) systems and virtual reality (VR) systems are varifocal images for vergence-accommodation conflict (VAC) and vision corrections. In this paper, we design a liquid crystal (LC) lens set consisting of three LC lenses for varifocal images and vision corrections in AR and VR. Four operating modes of such a LC lens set present three electrically tunable lens powers: 0, -0.79 diopters, -2 diopters, and -3.06 diopters by means of manipulation of polarization of incident light using electrically tunable half-wave-plates. The response time is fast(< 50 ms). We also demonstrate AR and VR systems by adopting the LC lens set to exhibit functions of varifocal images and vision corrections which enable to solve VAC as well as vision problem in AR and VR.

The Effect of Context Switching, Focal Switching Distance, Binocular and Monocular Viewing, and Transient Focal Blur on Human Performance in Optical See-Through Augmented Reality

In optical see-through augmented reality (AR), information is often distributed between real and virtual contexts, and often appears at different distances from the user. To integrate information, users must repeatedly switch context and change focal distance. If the user's task is conducted under time pressure, they may attempt to integrate information while their eye is still changing focal distance, a phenomenon we term transient focal blur. Previously, Gabbard, Mehra, and Swan (2018) examined these issues, using a text-based visual search task on a one-eye optical see-through AR display. This paper reports an experiment that partially replicates and extends this task on a custom-built AR Haploscope. The experiment examined the effects of context switching, focal switching distance, binocular and monocular viewing, and transient focal blur on task performance and eye fatigue. Context switching increased eye fatigue but did not decrease performance. Increasing focal switching distance increased eye fatigue and decreased performance. Monocular viewing also increased eye fatigue and decreased performance. The transient focal blur effect resulted in additional performance decrements, and is an addition to knowledge about AR user interface design issues.

Depth from blur and grouping under inattention

Previous studies provided evidence in support of attention operating in three-dimensional space, and the iterative and multistage nature of organizational processes in relation to attention and depth. We investigated depth perception and attentional demands in grouping organizations that contain blur as a depth cue. Contrary to previous studies, in our displays, no depth from occlusion could be implied from a shared border between groups or surfaces. To evaluate depth perception, subjective reports were collected where participants indicated which elements, blurry or sharp, they perceived as closer. To examine whether depth perception from blur can alleviate attentional demands, we used an inattention paradigm. We presented displays of grouping organizations by collinearity or color similarity that were previously found to require attention and added blur to the figure or the background elements to generate depth perception. In addition, we presented similar displays containing grouping by blur similarity as a single cue. We hypothesized that adding blur would facilitate the segmentation of element groups due to their perceived depth, which might lead to a diminished demand for attention. Our results confirmed that blur led to depth perception, and that sharp elements were perceived as closer more frequently than blurry elements. Thus, these results provide novel evidence for depth from blur in grouping where no inference of occlusion can be derived from a border. However, although the results suggest that blur information was processed under inattention, little evidence was found for decreased attentional demands for grouping processes in the presence of blur.

Multifocal Stereoscopic Projection Mapping

Stereoscopic projection mapping (PM) allows a user to see a three-dimensional (3D) computer-generated (CG) object floating over physical surfaces of arbitrary shapes around us using projected imagery. However, the current stereoscopic PM technology only satisfies binocular cues and is not capable of providing correct focus cues, which causes a vergence-accommodation conflict (VAC). Therefore, we propose a multifocal approach to mitigate VAC in stereoscopic PM. Our primary technical contribution is to attach electrically focus-tunable lenses (ETLs) to active shutter glasses to control both vergence and accommodation. Specifically, we apply fast and periodical focal sweeps to the ETLs, which causes the "virtual image" (as an optical term) of a scene observed through the ETLs to move back and forth during each sweep period. A 3D CG object is projected from a synchronized high-speed projector only when the virtual image of the projected imagery is located at a desired distance. This provides an observer with the correct focus cues required. In this study, we solve three technical issues that are unique to stereoscopic PM: (1) The 3D CG object is displayed on non-planar and even moving surfaces; (2) the physical surfaces need to be shown without the focus modulation; (3) the shutter glasses additionally need to be synchronized with the ETLs and the projector. We also develop a novel compensation technique to deal with the "lens breathing" artifact that varies the retinal size of the virtual image through focal length modulation. Further, using a proof-of-concept prototype, we demonstrate that our technique can present the virtual image of a target 3D CG object at the correct depth. Finally, we validate the advantage provided by our technique by comparing it with conventional stereoscopic PM using a user study on a depth-matching task.

Distortion corrected tomographic near-eye displays using light field optimization

Several multifocal displays have been proposed to provide accurate accommodation cues. However, multifocal displays have an undesirable feature, which is especially emphasized in near-eye displays configuration, that the field of views (FOVs) of the virtual planes change over depth. We demonstrate that this change in FOV causes image distortions, which reduces overall image quality, and depth perception error due to the variation of image sizes according to depths. Here, we introduce a light field optimization technique to compensate for magnification variations among the focal planes. Our approach alleviates image distortions, especially noticeable in the contents with large depth discontinuity, and reconstructs the image size to precise depths, while maintaining a specific tolerance length for the target eye relief. To verify the feasibility of the algorithm, we employ this optimization method for the tomographic near-eye display system to acquire the optimal image and backlight sequences for a volumetric scene. In general, we confirm that the structural similarity index measure of reconstructed images against ground truth increases by 20% when the eye relief is 15 mm, and the accommodation cue is appropriately stimulated at the target depth with our proposed method.

When virtual and real worlds coexist: Visualization and visual system affect spatial performance in augmented reality

New visualization approaches are being actively developed aiming to mitigate the effect of vergence-accommodation conflict in stereoscopic augmented reality; however, high interindividual variability in spatial performance makes it difficult to predict user gain. To address this issue, we investigated the effects of consistent and inconsistent binocular and focus cues on perceptual matching in the stereoscopic environment of augmented reality using a head-mounted display that was driven in multifocal and single focal plane modes. Participants matched the distance of a real object with images projected at three viewing distances, concordant with the display focal planes when driven in the multifocal mode. As a result, consistency of depth cues facilitated faster perceptual judgments on spatial relations. Moreover, the individuals with mild binocular and accommodative disorders benefited from the visualization of information on the focal planes corresponding to image planes more than individuals with normal vision, which was reflected in performance accuracy. Because symptoms and complaints may be absent when the functionality of the sensorimotor system is reduced, the results indicate the need for a detailed assessment of visual functions in research on spatial performance. This study highlights that the development of a visualization system that reduces visual stress and improves user performance should be a priority for the successful implementation of augmented reality displays.

Electrically Actuated Varifocal Lens Based on Liquid-Crystal-Embedded Dielectric Metasurfaces

Compact varifocal lenses are essential to various imaging and vision technologies. However, existing varifocal elements typically rely on mechanically actuated systems with limited tuning speeds and scalability. Here, an ultrathin electrically controlled varifocal lens based on a liquid crystal (LC) encapsulated dielectric metasurface is demonstrated. Enabled by the field-dependent LC anisotropy, applying a voltage bias across the LC cell modifies the local phase response of the silicon meta-atoms, in turn modifying the metalens focal length. In a numerical implementation, a voltage-actuated metalens with continuous zoom and up to 20% total focal shift is demonstrated. The LC-based metalens concept is experimentally verified through the design and fabrication of a bifocal metalens that facilitates high-contrast switching between two discrete focal lengths upon application of a 9.8 V_pp voltage bias. Owing to their ultrathin thickness and adaptable design, LC-driven dielectric metasurfaces open new opportunities for compact varifocal lensing in a diversity of modern imaging applications.

Scaffolding depth cues and perceptual learning in VR to train stereovision: a proof of concept pilot study

Stereopsis is a valuable feature of human visual perception, which may be impaired or absent in amblyopia and/or strabismus but can be improved through perceptual learning (PL) and videogames. The development of consumer virtual reality (VR) may provide a useful tool for improving stereovision. We report a proof of concept study, especially useful for strabismic patients and/or those with reduced or null stereoacuity. Our novel VR PL strategy is based on a principled approach which included aligning and balancing the perceptual input to the two eyes, dichoptic tasks, exposure to large disparities, scaffolding depth cues and perception for action. We recruited ten adults with normal vision and ten with binocular impairments. Participants played two novel PL games (DartBoard and Halloween) using a VR-HMD. Each game consisted of three depth cue scaffolding conditions, starting with non-binocular and binocular cues to depth and ending with only binocular disparity. All stereo-anomalous participants improved in the game and most (9/10) showed transfer to clinical and psychophysical stereoacuity tests (mean stereoacuity changed from 569 to 296 arc seconds, P < 0.0001). Stereo-normal participants also showed in-game improvement, which transferred to psychophysical tests (mean stereoacuity changed from 23 to a ceiling value of 20 arc seconds, P = 0.001). We conclude that a VR PL approach based on depth cue scaffolding may provide a useful method for improving stereoacuity, and the in-game performance metrics may provide useful insights into principles for effective treatment of stereo anomalies.This study was registered as a clinical trial on 04/05/2010 with the identifier NCT01115283 at ClinicalTrials.gov.

Design of a near-eye display measurement system using an anthropomorphic vision imaging method

We developed a new near-eye display measurement system using anthropomorphic vision imaging to measure the key parameters of near-eye displays, including field-of-view (FOV), angular resolution, eye box, and virtual image depth. The characteristics of the human eye, such as pupil position, pupil size variation, accommodation function, and the high resolution of the fovea, are imitated by the proposed measurement system. A FOV scanning structure, together with a non-vignetting image-telecentric lens system, captures the virtual image from the near-eye display by imitating human eye function. As a proof-of-concept, a prototype device was used to obtain large-range, high-resolution measurements for key parameters of near-eye displays.

Fast refocusing lens based on ferroelectric liquid crystals

Optical devices like virtual reality (VR) headsets present challenges in terms of vergence-accommodation conflict that leads to visual fatigue for the user over time. Lenses available to meet these challenges include liquid crystal (LC) lenses, which possess a response time in the millisecond range. This response time is slow, while accessing multiple focal lengths. A ferroelectric liquid crystal (FLC) has a response time in the microsecond range. In this article, we disclose a switchable lens device having a combination of the fast FLC-based polarization rotation unit and a passive polarization-dependent LC lens. A cascaded combination of three such lens units allows access to eight different focal points quite rapidly and can be a convenient device for VR applications.

Full-color retinal-projection near-eye display using a multiplexing-encoding holographic method

We propose a novel method to construct an optical see-through retinal-projection near-eye display using the Maxwellian view and a holographic method. To provide a dynamic full-color virtual image, a single phase-only spatial light modulator (SLM) was employed in conjunction with a multiplexing-encoding holographic method. Holographic virtual images can be directly projected onto the retina using an optical see-through eyepiece. The virtual image is sufficiently clear when the crystal lens can focus at different depths; the presented method can resolve convergence and accommodation conflict during the use of near-eye displays. To verify the proposed method, a proof-of-concept prototype was developed to provide vivid virtual images alongside real-world ones.

Augmented reality display system using modulated moiré imaging technique

To enhance the depth rendering ability of augmented reality (AR) display systems, a modulated moiré imaging technique is used to render the true three-dimensional (3D) images for AR display systems. 3D images with continuous depth information and large depth of field are rendered and superimposed on the real scene. The proposed AR system consists of a modulated moiré imaging subsystem and an optical combiner. The modulated moiré imaging subsystem employs modulated point light sources, a display device, and a microlens array to generate 3D images. A defocussing equal period moiré imaging structure is used, which gives a chance for the point light sources to modulate the depth position of 3D images continuously. The principles of the imaging system are deduced analytically. A custom-designed transparent off-axis spherical reflective lens is used as an optical combiner to project the 3D images into the real world. An experimental AR system that provides continuous 3D images with depth information ranging from 0.5 to 2.5 m is made to verify the feasibility of the proposed technique.

Toward the next-generation VR/AR optics: a review of holographic near-eye displays from a human-centric perspective

Wearable near-eye displays for virtual and augmented reality (VR/AR) have seen enormous growth in recent years. While researchers are exploiting a plethora of techniques to create life-like three-dimensional (3D) objects, there is a lack of awareness of the role of human perception in guiding the hardware development. An ultimate VR/AR headset must integrate the display, sensors, and processors in a compact enclosure that people can comfortably wear for a long time while allowing a superior immersion experience and user-friendly human-computer interaction. Compared with other 3D displays, the holographic display has unique advantages in providing natural depth cues and correcting eye aberrations. Therefore, it holds great promise to be the enabling technology for next-generation VR/AR devices. In this review, we survey the recent progress in holographic near-eye displays from the human-centric perspective.

Light source optimization for partially coherent holographic displays with consideration of speckle contrast, resolution, and depth of field

Speckle reduction is an important topic in holographic displays as speckles not only reduce signal-to-noise ratio but also possess an eye-safety issue. Despite thorough exploration of speckle reduction methods using partially coherent light sources, the trade-off involved by the partial coherence has not been thoroughly discussed. Here, we introduce theoretical models that quantify the effects of partial coherence on the resolution and the speckle contrast. The theoretical models allow us to find an optimal light source that maximizes the speckle reduction while minimizing the decline of the other terms. We implement benchtop prototypes of partially coherent holographic displays using the optimal light source, and verify the theoretical models via simulation and experiment. We also present a criterion to evaluate the depth of field in partially coherent holographic displays. We conclude with a discussion about approximations and limitations inherent in the theoretical models.

Varifocal augmented reality adopting electrically tunable uniaxial plane-parallel plates

Vergence-accommodation conflict (VAC) is a major challenge in optical-see through augmented reality (AR) system. To resolve this conflict, many approaches are proposed, for instance, by means of adjustment of the projected virtual image to coincide with the surroundings, called image registration, which is more often referred to as varifocal function. In this paper, a varifocal AR system is demonstrated by adopting electrically tunable liquid crystal (LC) plane-parallel plates to solve VAC problem. The LC plates provide electrically tunable optical paths when the directors of LC molecules are re-orientated with applied voltages, which leads to a corresponding change of light speed for an extraordinary wave. To provide a sufficient tunable optical path, three pieces of multiple-layered LC structures are used with the total thickness of the active LC layers (∼510 μm). In experiments, the projected virtual image can be adjusted from 1.4 m to 2.1 m away from the AR system, while the thickness of LC plane-parallel plates are only less than 3 mm without any mechanical moving part. When light propagates in the uniaxial LC layers, the wave vector and the Poynting vector are different. The longitudinal displacement of the image plane is determined by Poynting vectors rather than wave vectors. As a result, the analysis of the AR system should be based on Poynting vectors during geometrical optical analysis. Surprisingly, the tunable range of the longitudinal displacement of Poynting vectors is 2-fold larger than the tunable range of the wave vectors. Moreover, the virtual image shifts in opposite directions with respect to the Poynting vectors and wave vectors. The proposed AR system is not only simple but also thin, and it exhibits a large clear aperture. The investigation here paves the way to a simple solution of the VAC problem for augmented reality systems.

Factored Occlusion: Single Spatial Light Modulator Occlusion-capable Optical See-through Augmented Reality Display

Occlusion is a powerful visual cue that is crucial for depth perception and realism in optical see-through augmented reality (OST-AR). However, existing OST-AR systems additively overlay physical and digital content with beam combiners - an approach that does not easily support mutual occlusion, resulting in virtual objects that appear semi-transparent and unrealistic. In this work, we propose a new type of occlusion-capable OST-AR system. Rather than additively combining the real and virtual worlds, we employ a single digital micromirror device (DMD) to merge the respective light paths in a multiplicative manner. This unique approach allows us to simultaneously block light incident from the physical scene on a pixel-by-pixel basis while also modulating the light emitted by a light-emitting diode (LED) to display digital content. Our technique builds on mixed binary/continuous factorization algorithms to optimize time-multiplexed binary DMD patterns and their corresponding LED colors to approximate a target augmented reality (AR) scene. In simulations and with a prototype benchtop display, we demonstrate hard-edge occlusions, plausible shadows, and also gaze-contingent optimization of this novel display mode, which only requires a single spatial light modulator.

Towards a Switchable AR/VR Near-eye Display with Accommodation-Vergence and Eyeglass Prescription Support

In this paper, we present our novel design for switchable AR/VR near-eye displays which can help solve the vergence-accommodation-conflict issue. The principal idea is to time-multiplex virtual imagery and real-world imagery and use a tunable lens to adjust focus for the virtual display and the see-through scene separately. With this novel design, prescription eyeglasses for near- and far-sighted users become unnecessary. This is achieved by integrating the wearer's corrective optical prescription into the tunable lens for both virtual display and see-through environment. We built a prototype based on the design, comprised of micro-display, optical systems, a tunable lens, and active shutters. The experimental results confirm that the proposed near-eye display design can switch between AR and VR and can provide correct accommodation for both.

Holographic multiplane near-eye display based on amplitude-only wavefront modulation

We present a holographic multiplane near-eye display method based on Fresnel holography and amplitude-only wavefront modulation. Our method can create multiple focal images across a wide depth range while maintaining a high resolution (1080P) and refresh rate (60 Hz). To suppress the DC and conjugation signals inherent in amplitude-only wavefront modulation, we develop an optimization algorithm which completely separates primary diffracted light from DC and conjugation at a pre-defined intermediate plane. Spatial filtering at this plane leads to a dramatic increase in the image contrast. The experimental results demonstrate our approach can create continuous focus cues in complex 3D scenes.

Perception of Looming Motion in Virtual Reality Egocentric Interception Tasks

Motion in depth is commonly misperceived in Virtual Reality (VR), making it difficult to intercept moving objects, for example, in games. We investigate whether motion cues could be modified to improve these interactions in VR. We developed a time-to-contact estimation task, in which observers ($n=18$n=18) had to indicate by button press when a looming virtual object would collide with their head. We show that users consistently underestimate speed. We construct a user-specific model of motion-in-depth perception, and use this model to propose a novel method to modify monocular depth cues tailored to the specific user, correcting individual response errors in speed estimation. A user study was conducted in a simulated baseball environment and observers were asked to hit a looming baseball back in the direction of the pitcher. The study was conducted with and without intervention and demonstrates the effectiveness of the method in reducing interception errors following cue modifications. The intervention was particularly effective at fast ball speeds where performance is most limited by the user's sensorimotor constraints. The proposed approach is easy to implement and could improve the user experience of interacting with dynamic virtual environments.

Wavelength-multiplexed multi-focal-plane seethrough near-eye displays

We demonstrate a multi-focal-plane see-through near-eye display with effective focus cues enabled by wavelength multiplexing. A spectral notch filter is implemented as the wavelength-sensitive depth separation element. The vergence-accommodation conflict can be mitigated with the proposed design without space- or time-multiplexing. Another design of a dual-focus projection module for the waveguide-type augmented reality devices using wavelength-multiplexing is also presented.

Accommodation-Free Head Mounted Display with Comfortable 3D Perception and an Enlarged Eye-box

An accommodation-free displays, also known as Maxwellian displays, keep the displayed image sharp regardless of the viewer's focal distance. However, they typically suffer from a small eye-box and limited effective field of view (FOV) which requires careful alignment before a viewer can see the image. This paper presents a high-quality accommodation-free head mounted display (aHMD) based on pixel beam scanning for direct image forming on retina. It has an enlarged eye-box and FOV for easy viewing by replicating the viewing points with an array of beam splitters. A prototype aHMD is built using this concept, which shows high definition, low colour aberration 3D augmented reality (AR) images with an FOV of 36°. The advantage of the proposed design over other head mounted display (HMD) architectures is that, due to the narrow, collimated pixel beams, the high image quality is unaffected by changes in eye accommodation, and the approach to enlarge the eye-box is scalable. Most importantly, such an aHMD can deliver realistic three-dimensional (3D) viewing perception with no vergence-accommodation conflict (VAC). It is found that viewing the accommodation-free 3D images with the aHMD presented in this work is comfortable for viewers and does not cause the nausea or eyestrain side effects commonly associated with conventional stereoscopic 3D or HMD displays, even for all day use.

The Science Behind Virtual Reality Displays

Virtual reality (VR) is becoming an increasingly important way to investigate sensory processing. The converse is also true: in order to build good VR technologies, one needs an intimate understanding of how our brain processes sensory information. One of the key advantages of studying perception with VR is that it allows an experimenter to probe perceptual processing in a more naturalistic way than has been possible previously. In VR, one is able to actively explore and interact with the environment, just as one would do in real life. In this article, we review the history of VR displays, including the philosophical origins of VR, before discussing some key challenges involved in generating good VR and how a sense of presence in a virtual environment can be measured. We discuss the importance of multisensory VR and evaluate the experimental tension that exists between artifice and realism when investigating sensory processing.

Glasses-free 3-D visualization with multi-layered transparent cholesteric films

Fabrication of a three-dimensional (3D) device with volume-filling autostereoscopic imagery provides a unique solution to the accommodation-convergence mismatch problem in a conventional stereoscopic 3D display. We demonstrate a volumetric 3D display consisting of a stack of sequentially-driven transparent cholesteric films that illuminate every point in the display volume to generate a synthetic 3D image. Transparent cholesteric films comprise of the reverse-mode polymer-stabilized cholesteric texture (R-PSCT) that exhibits > 85% transmittance in its off-state (Grandjean texture) and <15% transmittance in its on-state (focal-conic texture). The polymer network in the R-PSCT is formed with the mixture of reactive monomers: RM6 and RM257. The electro-optical measurements show a strongly influence of concentration of each reactive monomer in the total mixture on the optical contrast and driving voltage of R-PSCT films. Hence, we optimize the composition of reactive monomers in R-PSCT to achieve a low driving voltage, high optical contrast and high electro-mechanical stability. Furthermore, we fabricate a multi-surface and a see-through volumetric 3D display prototype that consists an optical element with 15 R-PSCT films that are sequentially-switched with a microcontroller and operated at 250 Hz. A high-speed DLP projector is used to project a time-multiplexed series of two-dimensional images that are used to build the multi-planar volumetric images.

Tomographic near-eye displays

The ultimate 3D displays should provide both psychological and physiological cues for depth recognition. However, it has been challenging to satisfy the essential features without making sacrifices in the resolution, frame rate, and eye box. Here, we present a tomographic near-eye display that supports a wide depth of field, quasi-continuous accommodation, omni-directional motion parallax, preserved resolution, full frame, and moderate field of view within a sufficient eye box. The tomographic display consists of focus-tunable optics, a display panel, and a fast spatially adjustable backlight. The synchronization of the focus-tunable optics and the backlight enables the display panel to express the depth information. We implement a benchtop prototype near-eye display, which is the most promising application of tomographic displays. We conclude with a detailed analysis and thorough discussion of the display's optimal volumetric reconstruction. of tomographic displays.

A Perception-driven Hybrid Decomposition for Multi-layer Accommodative Displays

Multi-focal plane and multi-layered light-field displays are promising solutions for addressing all visual cues observed in the real world. Unfortunately, these devices usually require expensive optimizations to compute a suitable decomposition of the input light field or focal stack to drive individual display layers. Although these methods provide near-correct image reconstruction, a significant computational cost prevents real-time applications. A simple alternative is a linear blending strategy which decomposes a single 2D image using depth information. This method provides real-time performance, but it generates inaccurate results at occlusion boundaries and on glossy surfaces. This paper proposes a perception-based hybrid decomposition technique which combines the advantages of the above strategies and achieves both real-time performance and high-fidelity results. The fundamental idea is to apply expensive optimizations only in regions where it is perceptually superior, e.g., depth discontinuities at the fovea, and fall back to less costly linear blending otherwise. We present a complete, perception-informed analysis and model that locally determine which of the two strategies should be applied. The prediction is later utilized by our new synthesis method which performs the image decomposition. The results are analyzed and validated in user experiments on a custom multi-plane display.

Multi-plane augmented reality display based on cholesteric liquid crystal reflective films

To address the accommodation-convergence conflict problem in conventional augmented reality (AR) head-mounted displays, we propose a compact multi-plane display design based on cholesteric liquid crystal (CLC) reflective films and a polarization switch. Because of the polarization selectivity of CLC films, circularly-polarized light with different handedness is reflected by different CLC films, resulting in different optical path lengths and different image depths by the lens. A flicker-free dual-plane prototype with correct focus cues and relatively low operating voltage has been implemented. Moreover, a multi-plane AR display scheme with more than 2 depth planes is proposed by stacking multiple CLC films and polarization switches together.

All-passive transformable optical mapping near-eye display

We present an all-passive, transformable optical mapping (ATOM) near-eye display based on the “human-centric design” principle. By employing a diffractive optical element, a distorted grating, the ATOM display can project different portions of a two-dimensional display screen to various depths, rendering a real three-dimensional image with correct focus cues. Thanks to its all-passive optical mapping architecture, the ATOM display features a reduced form factor and low power consumption. Moreover, the system can readily switch between a real-three-dimensional and a high-resolution two-dimensional display mode, providing task-tailored viewing experience for a variety of VR/AR applications.

A High-Efficiency Low-Power Chip-Based CMOS Liquid Crystal Driver for Tunable Electro-Optic Eyewear

A high-efficiency low-power chip-based liquid crystal (LC) driver has been successfully designed and implemented for adaptive electro-optic eyewear including tunable vision correction devices (eyeglass, contact lens, intraocular lens, occluder, and prism), phoropter, iris, head-mounted display, and 3D imaging. The driver can generate a 1 kHz bipolar square wave with magnitude tunable from 0 V to 15 V to change the lens focus adaptively. The LC driver output magnitude is controlled by a reference DC voltage that is manually tunable between 0 and 3 V. A multi-mode 1×/2×/3×/4×/5× charge pump is developed for DC-DC conversion to expand the output range with a fast-sink function implemented to regulate the charge pump output. In addition, a new four-phase H-bridge driving scheme is employed to improve the DC/AC inverter efficiency. The LC driver has been successfully implemented and tested as an IC chip (8.6 mm × 8.6 mm) using AMS 0.18 μm High-Voltage CMOS technology.

Polarization-multiplexed multiplane display

We demonstrate a polarization-multiplexed multiplane display system for near-eye applications. A polarization-sensitive Pancharatnam-Berry phase lens is implemented to generate two focal depths simultaneously. A spatial polarization modulator is utilized to direct the two images to designated focal planes. Based on this design, a dual-focal-plane display system is constructed without space- or time-multiplexing operations, to suppress the vergence-accommodation conflict successfully.

An Extended Depth-at-Field Volumetric Near-Eye Augmented Reality Display

We introduce an optical design and a rendering pipeline for a full-color volumetric near-eye display which simultaneously presents imagery with near-accurate per-pixel focus across an extended volume ranging from 15cm (6.7 diopters) to 4M (0.25 diopters), allowing the viewer to accommodate freely across this entire depth range. This is achieved using a focus-tunable lens that continuously sweeps a sequence of 280 synchronized binary images from a high-speed, Digital Micromirror Device (DMD) projector and a high-speed, high dynamic range (HDR) light source that illuminates the DMD images with a distinct color and brightness at each binary frame. Our rendering pipeline converts 3-D scene information into a 2-D surface of color voxels, which are decomposed into 280 binary images in a voxel-oriented manner, such that 280 distinct depth positions for full-color voxels can be displayed.

Compact design for optical-see-through holographic displays employing holographic optical elements

Holographic AR display is a promising technology for head-mounted display devices. However, it usually has a complicated optical system and a large form factor, preventing it from widespread applications. In this work, we propose a flat-panel design to produce a compact holographic AR display, where traditional optical elements are replaced by two holographic optical elements (HOEs). Here, these two thin HOEs together perform the optical functions of a beam expander, an ocular lens, and an optical combiner. Without any bulky traditional optics, our design could achieve a compact form factor that is similar to a pair of glasses. We also implemented a proof-of-concept prototype to verify its feasibility. Being compact, lightweight and free from accommodation-convergence discrepancy, our design is promising for fatigue-free AR displays.

Hybrid multi-layer displays providing accommodation cues

Hybrid multi-layer displays are proposed as the system combines additive light field (LF) displays and multiplicative LF displays. The system is implemented by integrating the multiplicative LF displays with a half mirror to expand the overall depth of field. The hybrid displays are advantageous in that the form factor is competitive with existing additive LF displays with 2 layers implemented by a half mirror and two panels, only half of brightness loss is experienced compared to multiplicative LF displays with 2 layers, and no time-division is required to provide images for multi-layer displays. The images for presentation planes are processed by light field factorization and optimized with the presented algorithm. Retinal images are reconstructed based on various accommodation states and display types to check the accommodation response and utilized to compare the proposed displays with existing displays. With ray tracing method, retinal images generated by the proposed displays can be obtained. To verify the feasibility of the system, a prototype of hybrid multi-layer displays was implemented and display photographs were captured with different accommodation states of camera. With the simulation results and experimental results, this system was confirmed to support accommodation cues in a range of 1.8 diopters.

Digitally switchable multi-focal lens using freeform optics

Optical technologies offering electrically tunable optical power have found a broad range of applications, from head-mounted displays for virtual and augmented reality applications to microscopy. In this paper, we present a novel design and prototype of a digitally switchable multi-focal lens (MFL) that offers the capability of rapidly switching the optical power of the system among multiple foci. It consists of a freeform singlet and a customized programmable optical shutter array (POSA). Time-multiplexed multiple foci can be obtained by electrically controlling the POSA to switch the light path through different segments of the freeform singlet rapidly. While this method can be applied to a broad range of imaging and display systems, we experimentally demonstrate a proof-of-concept prototype for a multi-foci imaging system.

Bifocal computational near eye light field displays and Structure parameters determination scheme for bifocal computational display

We propose a bifocal computational near eye light field display (bifocal computational display) and structure parameters determination scheme (SPDS) for bifocal computational display that achieves greater depth of field (DOF), high resolution, accommodation and compact form factor. Using a liquid varifocal lens, two single-focal computational light fields are superimposed to reconstruct a virtual object's light field by time multiplex and avoid the limitation on high refresh rate. By minimizing the deviation between reconstructed light field and original light field, we propose a determination framework to determine the structure parameters of bifocal computational light field display. When applied to different objective to SPDS, it can achieve high average resolution or uniform resolution display over scene depth range. To analyze the advantages and limitation of our proposed method, we have conducted simulations and constructed a simple prototype which comprises a liquid varifocal lens, dual-layer LCDs and a uniform backlight. The results of simulation and experiments with our method show that the proposed system can achieve expected performance well. Owing to the excellent performance of our system, we motivate bifocal computational display and SPDS to contribute to a daily-use and commercial virtual reality display.

High-resolution additive light field near-eye display by switchable Pancharatnam-Berry phase lenses

Conventional head-mounted displays present different images to each eye, and thereby create three-dimensional (3D) sensation for viewers. This method can only control the stimulus to vergence but not accommodation, which is located at the apparent location of the physical displays. The disrupted coupling between vergence and accommodation could cause considerable visual discomfort. To address this problem, in this paper a novel multi-focal plane 3D display system is proposed. A stack of switchable liquid crystal Pancharatnam-Berry phase lenses is implemented to create real depths for each eye, which is able to provide approximate focus cue and relieve the discomfort from vergence-accommodation conflict. The proposed multi-focal plane generation method has great potential for both virtual reality and augmented reality applications, where correct focus cue is highly desirable.

Reverse-mode PSLC multi-plane optical see-through display for AR applications

In this paper we propose an optical see-through multi-plane display with reverse-mode polymer-stabilized liquid crystal (PSLC). Our design solves the problem of accommodation-vergence conflict with correct focus cues. In the reverse mode PSLC system, power consumption could be reduced to ~1/(N-1) of that in a normal mode system if N planes are displayed. The PSLC films fabricated in our experiment exhibit a low saturation voltage ~20 V_rms, a high transparent-state transmittance (92%), and a fast switching time within 2 ms and polarization insensitivity. A proof-of-concept two-plane color display prototype and a four-plane monocolor display prototype were implemented.

Vergence driven accommodation with simulated disparity in myopia and emmetropia

The formation of focused and corresponding foveal images requires a close synergy between the accommodation and vergence systems. This linkage is usually decoupled in virtual reality systems and may be dysfunctional in people who are at risk of developing myopia. We study how refractive error affects vergence-accommodation interactions in stereoscopic displays. Vergence and accommodative responses were measured in 21 young healthy adults (n=9 myopes, 22-31 years) while subjects viewed naturalistic stimuli on a 3D display. In Step 1, vergence was driven behind the monitor using a blurred, non-accommodative, uncrossed disparity target. In Step 2, vergence and accommodation were driven back to the monitor plane using naturalistic images that contained structured depth and focus information from size, blur and/or disparity. In Step 1, both refractive groups converged towards the stereoscopic target depth plane, but the vergence-driven accommodative change was smaller in emmetropes than in myopes (F_1,19=5.13, p=0.036). In Step 2, there was little effect of peripheral depth cues on accommodation or vergence in either refractive group. However, vergence responses were significantly slower (F_1,19=4.55, p=0.046) and accommodation variability was higher (F_1,19=12.9, p=0.0019) in myopes. Vergence and accommodation responses are disrupted in virtual reality displays in both refractive groups. Accommodation responses are less stable in myopes, perhaps due to a lower sensitivity to dioptric blur. Such inaccuracies of accommodation may cause long-term blur on the retina, which has been associated with a failure of emmetropization.

Stereoscopic depth constancy

Depth constancy is the ability to perceive a fixed depth interval in the world as constant despite changes in viewing distance and the spatial scale of depth variation. It is well known that the spatial frequency of depth variation has a large effect on threshold. In the first experiment, we determined that the visual system compensates for this differential sensitivity when the change in disparity is suprathreshold, thereby attaining constancy similar to contrast constancy in the luminance domain. In a second experiment, we examined the ability to perceive constant depth when the spatial frequency and viewing distance both changed. To attain constancy in this situation, the visual system has to estimate distance. We investigated this ability when vergence, accommodation and vertical disparity are all presented accurately and therefore provided veridical information about viewing distance. We found that constancy is nearly complete across changes in viewing distance. Depth constancy is most complete when the scale of the depth relief is constant in the world rather than when it is constant in angular units at the retina. These results bear on the efficacy of algorithms for creating stereo content.

This article is part of the themed issue ‘Vision in our three-dimensional world’.

See-through optical combiner for augmented reality head-mounted display: index-matched anisotropic crystal lens

A novel see-through optical device to combine the real world and the virtual image is proposed which is called an index-matched anisotropic crystal lens (IMACL). The convex lens made of anisotropic crystal is enveloped with the isotropic material having same refractive index with the extraordinary refractive index of the anisotropic crystal. This optical device functions as the transparent glass or lens according to the polarization state of the incident light. With the novel optical property, IMACL can be utilized in the see-through near eye display, or head-mounted display for augmented reality. The optical property of the proposed optical device is analyzed and aberration by the anisotropic property of the index-matched anisotropic crystal lens is described with the simulation. The concept of the head-mounted display using IMACL is introduced and various optical performances such as field of view, form factor and transmittance are analyzed. The prototype is implemented to verify the proposed system and experimental results show the mixture between the virtual image and real world scene.

Blur perception throughout the visual field in myopia and emmetropia

We evaluated the ability of emmetropic and myopic observers to detect and discriminate blur across the retina under monocular or binocular viewing conditions. We recruited 39 young (23-30 years) healthy adults (n = 19 myopes) with best-corrected visual acuity 0.0 LogMAR (20/20) or better in each eye and no binocular or accommodative dysfunction. Monocular and binocular blur discrimination thresholds were measured as a function of pedestal blur using naturalistic stimuli with an adaptive 4AFC procedure. Stimuli were presented in a 46° diameter window at 40 cm. Gaussian blur pedestals were confined to an annulus at either 0°, 4°, 8°, or 12° eccentricity, with a blur increment applied to only one quadrant of the image. The adaptive procedure efficiently estimated a dipper shaped blur discrimination threshold function with two parameters: intrinsic blur and blur sensitivity. The amount of intrinsic blur increased for retinal eccentricities beyond 4° (p < 0.001) and was lower in binocular than monocular conditions (p < 0.001), but was similar across refractive groups (p = 0.47). Blur sensitivity decreased with retinal eccentricity (p < 0.001) and was highest for binocular viewing, but only for central vision (p < 0.05). Myopes showed worse blur sensitivity than emmetropes monocularly (p < 0.05) but not binocularly (p = 0.66). As expected, blur perception worsens in the visual periphery and binocular summation is most evident in central vision. Furthermore, myopes exhibit a monocular impairment in blur sensitivity that improves under binocular conditions. Implications for the development of myopia are discussed.

Focus and perspective adaptive digital surgical microscope: optomechanical design and experimental implementation

This paper relates to the improvement of conventional surgical stereo microscopy via the application of digital recording devices and adaptive optics. The research is aimed at improving the working conditions of the surgeon during the operation, such that free head movement is possible. The depth clues known from conventional stereo microscopy in interaction with the human eye’s functionality, such as convergence, disparity, angular elevation, parallax, and accommodation, are implemented in a digital recording system via adaptive optomechanical components. Two laterally moving pupil apertures have been used mimicking the digital implementation of the eye’s vergence and head motion. The natural eye’s accommodation is mimicked via the application of a tunable lens. Additionally, another system has been built, which enables tracking the surgeon’s eye pupil through a digital displaying stereoscopic microscope to supply the necessary information for steering the recording system. The optomechanical design and experimental results for both systems, digital recording stereoscopic microscope and pupil tracking system, are shown.

Augmented reality with image registration, vision correction and sunlight readability via liquid crystal devices

Augmented reality (AR), which use computer-aided projected information to augment our sense, has important impact on human life, especially for the elder people. However, there are three major challenges regarding the optical system in the AR system, which are registration, vision correction, and readability under strong ambient light. Here, we solve three challenges simultaneously for the first time using two liquid crystal (LC) lenses and polarizer-free attenuator integrated in optical-see-through AR system. One of the LC lens is used to electrically adjust the position of the projected virtual image which is so-called registration. The other LC lens with larger aperture and polarization independent characteristic is in charge of vision correction, such as myopia and presbyopia. The linearity of lens powers of two LC lenses is also discussed. The readability of virtual images under strong ambient light is solved by electrically switchable transmittance of the LC attenuator originating from light scattering and light absorption. The concept demonstrated in this paper could be further extended to other electro-optical devices as long as the devices exhibit the capability of phase modulations and amplitude modulations.

Effect of apparent depth cues on accommodation in a Badal optometer

BACKGROUND

The aim was to analyse the effect of peripheral depth cues on accommodation in Badal optometers.

METHODS

Monocular refractions at 0.17 and 5.00 D of accommodative stimulus were measured with the PowerRef II autorefractor (Plusoptix Inc., Atlanta, Georgia, USA). Subjects looked (randomly) at four different scenes, one real scene comprising familiar objects at different depth planes (Real) and three virtual scenes comprising different two-dimensional pictures seen through a Badal lens. The first image consisted of a photograph of the real scene taken in conditions that closely mimic a healthy standard human eye performance (out-of-focus [OoF] blur); the second image was the same photograph rendered with a depth of focus to infinity (OoF sharpness); and finally the third image consisted of a fixation target and a even white surrounding (White). In all cases the field of view was 25.0° and the fixation target was a Maltese cross subtending to two degrees.

RESULTS

Twenty-eight right eyes from healthy young subjects were measured. The achieved statistical power was 0.9. At 5.00 D of accommodative stimulus, the repeated measures analysis of variance was statistically significant (p < 0.05) and the corresponding Bonferroni post hoc tests showed the following mean accommodative response differences and standard deviation (p-value) between the real and the virtual scenes: real-white =-0.66 ± 0.92 D (p < 0.01); real-OoF sharpness = -0.43 ± 0.88 D (p = 0.07); real-OoF blur =-0.25 ± 0.93 D (p = 0.89).

CONCLUSIONS

A stimulus poor in depth cues inaccurately stimulates accommodation in Badal optometers; however, accommodation can be significantly improved in the same Badal optometer, when displaying a realistic image rich in peripheral depth cues, even though these peripheral cues (also referred to as retinal blur cues) are shown in the same plane as the fixation target. These results have important implications in stereoscopic virtual reality systems that fail to represent appropriately retinal blur.

Optimizing virtual reality for all users through gaze-contingent and adaptive focus displays

From the desktop to the laptop to the mobile device, personal computing platforms evolve over time. Moving forward, wearable computing is widely expected to be integral to consumer electronics and beyond. The primary interface between a wearable computer and a user is often a near-eye display. However, current generation near-eye displays suffer from multiple limitations: they are unable to provide fully natural visual cues and comfortable viewing experiences for all users. At their core, many of the issues with near-eye displays are caused by limitations in conventional optics. Current displays cannot reproduce the changes in focus that accompany natural vision, and they cannot support users with uncorrected refractive errors. With two prototype near-eye displays, we show how these issues can be overcome using display modes that adapt to the user via computational optics. By using focus-tunable lenses, mechanically actuated displays, and mobile gaze-tracking technology, these displays can be tailored to correct common refractive errors and provide natural focus cues by dynamically updating the system based on where a user looks in a virtual scene. Indeed, the opportunities afforded by recent advances in computational optics open up the possibility of creating a computing platform in which some users may experience better quality vision in the virtual world than in the real one.

Compact three-dimensional head-mounted display system with Savart plate

We propose three-dimensional (3D) head-mounted display (HMD) providing multi-focal and wearable functions by using polarization-dependent optical path switching in Savart plate. The multi-focal function is implemented as micro display with high pixel density of 1666 pixels per inches is optically duplicated in longitudinal direction according to the polarization state. The combination of micro display, fast switching polarization rotator and Savart plate retains small form factor suitable for wearable function. The optical aberrations of duplicated panels are investigated by ray tracing according to both wavelength and polarization state. Astigmatism and lateral chromatic aberration of extraordinary wave are compensated by modification of the Savart plate and sub-pixel shifting method, respectively. To verify the feasibility of the proposed system, a prototype of the HMD module for monocular eye is implemented. The module has the compact size of 40 mm by 90 mm by 40 mm and the weight of 131 g with wearable function. The micro display and polarization rotator are synchronized in real-time as 30 Hz and two focal planes are formed at 640 and 900 mm away from eye box, respectively. In experiments, the prototype also provides augmented reality function by combining the optically duplicated panels with a beam splitter. The multi-focal function of the optically duplicated panels without astigmatism and color dispersion compensation is verified. When light field optimization for two additive layers is performed, perspective images are observed, and the integration of real world scene and high quality 3D images is confirmed.

3D Displays

Creating realistic three-dimensional (3D) experiences has been a very active area of research and development, and this article describes progress and what remains to be solved. A very active area of technical development has been to build displays that create the correct relationship between viewing parameters and triangulation depth cues: stereo, motion, and focus. Several disciplines are involved in the design, construction, evaluation, and use of 3D displays, but an understanding of human vision is crucial to this enterprise because in the end, the goal is to provide the desired perceptual experience for the viewer. In this article, we review research and development concerning displays that create 3D experiences. And we highlight areas in which further research and development is needed.