Foveated imaging for near-eye displays

Pupil-Adaptive Retina Projection Augment Reality Displays With Switchable Ultra-Dense Viewpoints

Multi-viewpoint retina projection displays (RPD) with an expanded visible range have been utilized in recent augmented reality (AR) systems to address the vergence-accommodation conflict (VAC) by providing a long depth of field (DOF). However, these fixed multi-viewpoint RPD systems still face a common critical challenge of imaging overlap or discontinuity when eyes rotate or under varying ambient light. To address this, an RPD AR system featuring switchable ultra-dense viewpoints is presented, enabled by a photo-alignment liquid crystal Dammann grating (p-LCDG). The number of viewpoints reaches 49, which forms an ultra-high density of diffraction lattice in front of the pupil with a record high rotation precision of 1.28°/viewpoint, allowing for a substantial range of 36 mm2. More importantly, the spacing of adjacent viewpoints is 0.532 mm, much smaller than the minimum radius of the pupil (≈1 mm). To facilitate viewpoint switching, a light selector is implemented, ensuring that only the light from a specific viewpoint reaches the eye, which effectively eliminates the image missing or discontinuity. By combining eye tracking technology, the viewer can consistently perceive a singular and clear image from the proposed RPD system, achieving seamless switching of viewpoints. This innovative design paves the way for high-performance RPDs in AR applications.

The Perceptual Science of Augmented Reality

Augmented reality (AR) systems aim to alter our view of the world and enable us to see things that are not actually there. The resulting discrepancy between perception and reality can create compelling entertainment and can support innovative approaches to education, guidance, and assistive tools. However, building an AR system that effectively integrates with our natural visual experience is hard. AR systems often suffer from visual limitations and artifacts, and addressing these flaws requires basic knowledge of perception. At the same time, AR system development can serve as a catalyst that drives innovative new research in perceptual science. This review describes recent perceptual research pertinent to and driven by modern AR systems, with the goal of highlighting thought-provoking areas of inquiry and open questions.

Achromatic diffractive liquid-crystal optics for virtual reality displays

Diffractive liquid-crystal optics is a promising optical element for virtual reality (VR) and mixed reality as it provides an ultrathin formfactor and lightweight for human factors and ergonomics. However, its severe chromatic aberrations impose a big challenge for full-color display applications. In this study, we demonstrate an achromatic diffractive liquid-crystal device to overcome this longstanding chromatic aberration issue. The proposed device consists of three stacked diffractive liquid crystal optical elements with specifically designed spectral response and polarization selectivity. The concept is validated by both simulations and experiments. Our experimental results show a significant improvement in imaging performance with two types of light engines: a laser projector and an organic light-emitting diode display panel. In addition, our simulation results indicate that the lateral color shift is reduced by ~100 times in comparison with conventional broadband diffractive liquid-crystal lens. Potential applications for VR-enabled metaverse, spatial computing, and digital twins that have found widespread applications in smart tourism, smart education, smart healthcare, smart manufacturing, and smart construction are foreseeable.

Perceptually-guided Dual-mode Virtual Reality System For Motion-adaptive Display

An ideal Virtual reality (VR) device should simultaneously provide retina-level resolution, wide field-of-view (FOV), and high refresh rate display, thereby bringing users into a deeply immersive virtual world. However, directly providing such high-quality display poses great challenges for display panel fabrication, real-time rendering, and data transfer. To address this issue, we introduce a dual-mode virtual reality system based on the spatio-temporal perception characteristics of human vision. The proposed VR system has a novel optical architecture. It can switch display modes according to the user's perceptual requirements for different display scenes to adaptively adjust the display spatial and temporal resolution based on a given display budget, thus providing users with the optimal visual perception quality. In this work, a complete design pipeline for the dual-mode VR optical system is proposed, and a bench-top prototype is built with only off-the-shelf hardware and components to verify its capability. Compared to the conventional VR system, our proposed scheme is more efficient and flexible in utilizing the display budget, and this work is expected to facilitate the development of the VR device based on the human visual system.

Design of a statically foveated display based on a perceptual-driven approach

Foveated display technology has the potential to offer both a large field of view (FOV) and high spatial resolution for head-mounted display (HMD) systems, through allocating the limited resources differently between the region of interest (ROI) and the peripheral region. However, the common method used in the prior studies is based on a dual-resolution dynamic foveation scheme, which is inevitably complex and high cost due to the requirements for multiple display sources, a 2D steering mechanism, and an eye tracking device. We recently proposed a new perceptual-driven approach to design a statically foveated HMD with the goal of offering a wide FOV with nearly imperceptible or minimal degradation of the perceived image resolution within regions where frequent eye movement occurs. Compared to a dynamical dual-resolution foveation approach, it not only minimizes the hardware complexity by eliminating the need for an eyetracker, a scanning mechanism, and multiple display sources, but also offer continuous degradation in resolution to avoid visual artifacts. In this paper, a statically foveated display is designed by carefully controlling the spatial variation of optical magnification of the eyepiece optics, which covers an 80° FOV and achieves a peak resolution of 1.5 arcminutes per pixel. The angular resolution distribution of the prototype design closely matches the theoretical statically foveated scheme described in our previous work with excellent perceived performance. Finally, a foveated display prototype based on the design was experimentally demonstrated with excellent perceived performance matching the designed resolution distribution.

Large-scale metagrating complex-based light field 3D display with space-variant resolution for non-uniform distribution of information and energy

Glasses-free three-dimensional (3D) display has attracted wide interest for providing stereoscopic virtual contents with depth cues. However, how to achieve high spatial and angular resolution while keeping ultrawide field of view (FOV) remains a significant challenge in 3D display. Here, we propose a light field 3D display with space-variant resolution for non-uniform distribution of information and energy. The spatial resolution of each view is modulated according to watching habit. A large-scale combination of pixelated 1D and 2D metagratings is used to manipulate dot and horizontal line views. With the joint modulation of pixel density and view arrangement, the information density and illuminance of high-demand views are at most 5.6 times and 16 times that of low-demand views, respectively. Furthermore, a full-color and video rate light field 3D display with non-uniform information distribution is demonstrated. The prototype provides 3D images with a high spatial resolution of 119.6 pixels per inch and a high angular resolution of 0.25 views per degree in the high-demand views. An ultrawide viewing angle of 140° is also provided. The proposed light field 3D display does not require ultrahigh-resolution display panels and has form factors of thin and light. Thus, it has the potential to be used in portable electronics, window display, exhibition display, as well as tabletop display.

Bifocal flat lens with different imaging characteristics for a dual-sensor imaging system

Wide field of view (FOV) images and magnified images can be taken simultaneously by dual-sensor imaging systems. Here, we propose an approach for creating a bifocal flat lens with different imaging characteristics of its two foci, which makes dual-sensor imaging systems more integrated and miniaturized. That is, two special parts of two different conventional ZP are extracted and then combine the two elements in a specific way. So that there are two foci with different characteristics along the optical axis, one is long focus with higher resolution, the other is short focus with long depth of focus (DOF). Under the proposed approach, a thin and light bifocal diffractive lens (BDL) with thickness of 0.6 μm is developed. The long and short focal lengths of the BDL are ~ 81 mm and ~ 27 mm, respectively, with a diameter of 6 mm. We experimentally demonstrate that the long focus of the BDL is capable of taking high-resolution magnified images, and its resolution is up to 21.90″. The short focus is able to take wide FOV with long DOF images, and two objects spread 2880 mm apart can be imaged clearly. The experiment results demonstrate that all of these metrics are better than those of a conventional refractive lens.

Creating the ultimate virtual reality display

Scientists are exploring new material designs to make smaller and denser pixel displays.

Advanced liquid crystal devices for augmented reality and virtual reality displays: principles and applications

Liquid crystal displays (LCDs) and photonic devices play a pivotal role to augmented reality (AR) and virtual reality (VR). The recently emerging high-dynamic-range (HDR) mini-LED backlit LCDs significantly boost the image quality and brightness and reduce the power consumption for VR displays. Such a light engine is particularly attractive for compensating the optical loss of pancake structure to achieve compact and lightweight VR headsets. On the other hand, high-resolution-density, and high-brightness liquid-crystal-on-silicon (LCoS) is a promising image source for the see-through AR displays, especially under high ambient lighting conditions. Meanwhile, the high-speed LCoS spatial light modulators open a new door for holographic displays and focal surface displays. Finally, the ultrathin planar diffractive LC optical elements, such as geometric phase LC grating and lens, have found useful applications in AR and VR for enhancing resolution, widening field-of-view, suppressing chromatic aberrations, creating multiplanes to overcome the vergence-accommodation conflict, and dynamic pupil steering to achieve gaze-matched Maxwellian displays, just to name a few. The operation principles, potential applications, and future challenges of these advanced LC devices will be discussed.

Cross-talk elimination for lenslet array near eye display based on eye-gaze tracking

Lenslet array (LA) near-eye displays (NEDs) are a recent technical development that creates a virtual image in the field of view of one or both eyes. A problem occurs when the user's pupil moves out of the LA-NED eye box (i.e., cross-talk) making the image look doubled or ghosted. It negatively impacts the user experience. Although eye-gaze tracking can mitigate this problem, the effect of the solution has not been studied to understand the impact of pupil size and human perception. In this paper, we redefine the cross-talk region as the practical pupil movable region (PPMR₅₀), which differs from eye box size because it considers pupil size and human visual perception. To evaluate the effect of eye-gaze tracking on subjective image quality, three user studies were conducted. From the results, PPMR₅₀ was found to be consistent with human perception, and cross-talk elimination via eye-gaze tracking was better understood in a static gaze scenario. Although the system latency prevented the complete elimination of cross-talk for fast movements or large pupil changes, the problem was greatly alleviated. We also analyzed system delays based on PPMR₅₀, which we newly defined in this paper and provided an optimization scheme to meet the maximum eyeball rotation speed.

Recent Advances in Planar Optics-Based Glasses-Free 3D Displays

Glasses-free three-dimensional (3D) displays are one of the technologies that will redefine human-computer interfaces. However, many geometric optics-based 3D displays suffer from a limited field of view (FOV), severe resolution degradation, and visual fatigue. Recently, planar optical elements (e.g., diffraction gratings, diffractive lenses and metasurfaces) have shown superior light manipulating capability in terms of light intensity, phase, and polarization. As a result, planar optics hold great promise to tackle the critical challenges for glasses-free 3D displays, especially for portable electronics and transparent display applications. In this review, the limitations of geometric optics-based glasses-free 3D displays are analyzed. The promising solutions offered by planar optics for glasses-free 3D displays are introduced in detail. As a specific application and an appealing feature, augmented reality (AR) 3D displays enabled by planar optics are comprehensively discussed. Fabrication technologies are important challenges that hinder the development of 3D displays. Therefore, multiple micro/nanofabrication methods used in 3D displays are highlighted. Finally, the current status, future direction and potential applications for glasses-free 3D displays and glasses-free AR 3D displays are summarized.

Towards retina-quality VR video streaming

Virtual reality systems today cannot yet stream immersive, retina-quality virtual reality video over a network. One of the greatest challenges to this goal is the sheer data rates required to transmit retina-quality video frames at high resolutions and frame rates. Recent work has leveraged the decay of visual acuity in human perception in novel gaze-contingent video compression techniques. In this paper, we show that reducing the motion-to-photon latency of a system itself is a key method for improving the compression ratio of gaze-contingent compression. Our key finding is that a client and streaming server system with sub-15ms latency can achieve 5x better compression than traditional techniques while also using simpler software algorithms than previous work.

Achromatic doublet electrowetting prism array for beam steering device in foveated display

A foveated display is a technology that can solve the problem of insufficient angular resolution (relative to the human eye) for near-eye display. In a high-resolution foveated display, a beam steering element is required to track the human gaze. An electrowetting prism array is a transmissive non-mechanical beam steering device, that allows a light and compact optical system to be configured and a large aperture possible. However, the view is obstructed by the sidewall of the prism array. When the size of the cell prism is 7mm, the prism array has an 87% fill-factor. To push the fill-factor to 100%, the cell prisms were magnified using a lens array. Image processing was performed such that the image produced by the lens array was identical to the original. Beam steering by refraction is accompanied by chromatic dispersion, which causes chromatic aberration, making colors appear blurry. The refractive index condition to reduce chromatic dispersion was obtained using the doublet structure of the electrowetting prism. The chromatic dispersion was reduced by 70% on average.

Augmented reality and virtual reality displays: emerging technologies and future perspectives

With rapid advances in high-speed communication and computation, augmented reality (AR) and virtual reality (VR) are emerging as next-generation display platforms for deeper human-digital interactions. Nonetheless, to simultaneously match the exceptional performance of human vision and keep the near-eye display module compact and lightweight imposes unprecedented challenges on optical engineering. Fortunately, recent progress in holographic optical elements (HOEs) and lithography-enabled devices provide innovative ways to tackle these obstacles in AR and VR that are otherwise difficult with traditional optics. In this review, we begin with introducing the basic structures of AR and VR headsets, and then describing the operation principles of various HOEs and lithography-enabled devices. Their properties are analyzed in detail, including strong selectivity on wavelength and incident angle, and multiplexing ability of volume HOEs, polarization dependency and active switching of liquid crystal HOEs, device fabrication, and properties of micro-LEDs (light-emitting diodes), and large design freedoms of metasurfaces. Afterwards, we discuss how these devices help enhance the AR and VR performance, with detailed description and analysis of some state-of-the-art architectures. Finally, we cast a perspective on potential developments and research directions of these photonic devices for future AR and VR displays.

Perceptual-driven approach to statically foveated head-mounted displays

A foveated display is a promising technique to realize displays offering both a large field of view (FOV) and high spatial resolution. Although several prior works have attempted to apply a foveation method to the design of a head-mounted display (HMD) system, the common method is based on a dual-resolution dynamic foveation scheme which is inevitably complex and has a high cost due to the requirements for multiple display sources, a 2D steering mechanism, and eye tracker. In this paper, a new perceptual-driven approach to the design of a statically foveated HMD is proposed with the goal of offering a wide FOV across which the degradation of the perceived image resolution is nearly imperceptible or minimal within regions of frequent eye movements. Compared to a dual-resolution discrete and dynamic foveation approach in the prior art, the static foveation approach will not only maintain resolution continuity but also eliminate the need for a scanning mechanism, multiple display sources, and an eyetracker, and therefore minimize hardware complexity. We present the general approach for creating a static foveation scheme, performance metrics for evaluating the perceived image quality, and the process of optimizing a foveation scheme to meet different requirements. Finally, we experimentally demonstrate and validate the proposed foveation scheme using a testbed system. Overall, we demonstrate a statically foveated scheme is capable of offering a display with a total 160° FOV, a constant resolution of 0.5 or 1 arcminutes per pixel within the ±10° region where frequent eye movements occur, an adequate resolution no less than 45% of peak resolution within the parafovea region of ±30°, and a data sampling efficiency as high as 90%.

Foveated light-field display and real-time rendering for virtual reality

Glasses-free light field displays have significantly progressed due to advances in high-resolution microdisplays and high-end graphics processing units (GPUs). However, for near-eye light-field displays requiring portability, the fundamental trade-off regarding achieved spatial resolution remains: retinal blur quality must be degraded; otherwise, computational consumption increases. This has prevented synthesizing the high-quality light field from being fast. By integrating off-the-shelf gaze tracking modules into near-eye light-field displays, we present wearable virtual reality prototypes supporting human visual system-oriented focus cues. An optimized, foveated light field is delivered to each eye subject to the gaze point, providing more natural visual experiences than state-of-the-art solutions. Importantly, the factorization runtime can be immensely reduced, since the image resolution is only high within the gaze cone. In addition, we demonstrate significant improvements in computation and retinal blur quality over counterpart near-eye displays.

Human gaze is systematically offset from the center of cone topography

The small physical depression of the human retina, the fovea, is the retinal locus of prime visual resolution, achieved by a peaking topography of the light-sensitive cone photoreceptor outer segments^1-3 and a post-receptor wiring scheme preserving high-density sampling.^4,5 Humans dynamically direct their gaze such that the retinal images of objects of interest fall onto the foveola, the central one-degree diameter of the fovea,^6-8 but it is yet unclear whether a relationship between the individual photoreceptor topography at this location and visual fixation behavior exists.^9,10 By employing adaptive optics in vivo imaging and micro-stimulation,^11-13 we created topographical maps of the complete foveolar cone mosaics in both eyes of 20 healthy participants while simultaneously recording the retinal location of a fixated visual object in a psychophysical experiment with cellular resolution. We found that the locus of fixation was systematically shifted away from the topographical center toward a naso-superior quadrant on the retina, about 5 min of arc of visual angle on average, with a mirror symmetrical trend between fellow eyes. In cyclopean view, the topographical centers were superior to the fixated target, corresponding to areas in the visual field usually more distant^14,15 and thus containing higher spatial frequencies. Given the large variability in foveal topography between individuals, and the surprising precision with which fixation is repeatedly directed to just a small bouquet of cones in the foveola, these findings demonstrate a finely tuned, functionally relevant link between the development of the cellular mosaic of photoreceptors and visual behavior.

LensIet VR: Thin, Flat and Wide-FOV Virtual Reality Display Using Fresnel Lens and LensIet Array

We propose a new thin and flat virtual reality (VR) display design using a Fresnel lenslet array, a Fresnel lens, and a polarization-based optical folding technique. The proposed optical system has a wide field of view (FOV) of 102°x102°, a wide eye-box of 8.8 mm, and an ergonomic eye-relief of 20 mm. Simultaneously, only 3.3 mm of physical distance is required between the display panel and the lens, so that the integrated VR display can have a compact form factor like sunglasses. Moreover, since all lenslet of the lenslet array is designed to operate under on-axis condition with low aberration, the discontinuous pupil swim distortion between the lenslets is hardly observed. In addition, all on-axis lenslets can be designed identically, reducing production cost, and even off-the-shelf Fresnel optics can be used. In this paper, we introduce how we design system parameters and analyze system performance. Finally, we demonstrate two prototypes and experimentally verify that the proposed VR display system has the expected performance while having a glasses-like form factor.

Design of a near-eye display measurement system using an anthropomorphic vision imaging method

We developed a new near-eye display measurement system using anthropomorphic vision imaging to measure the key parameters of near-eye displays, including field-of-view (FOV), angular resolution, eye box, and virtual image depth. The characteristics of the human eye, such as pupil position, pupil size variation, accommodation function, and the high resolution of the fovea, are imitated by the proposed measurement system. A FOV scanning structure, together with a non-vignetting image-telecentric lens system, captures the virtual image from the near-eye display by imitating human eye function. As a proof-of-concept, a prototype device was used to obtain large-range, high-resolution measurements for key parameters of near-eye displays.

Interaction between sampled rays' defocusing and number on accommodative response in integral imaging near-eye light field displays

In an integral imaging near-eye light field display using a microlens array, a point on a reconstructed depth plane (RDP) is reconstructed by sampled rays. Previous studies respectively suggested the accommodative response may shift from the RDP under two circumstances: (i) the RDP is away from the central depth plane (CDP) to introduce defocusing in sampled rays; (ii) the sampled ray number is too low. However, sampled rays' defocusing and number may interact, and the interaction's influence on the accommodative response has been little revealed. Therefore, this study adopts a proven imaging model providing retinal images to analyze the accommodative response. As a result, when the RDP and the CDP coincide, the accommodative response matches the RDP. When the RDP deviates from the CDP, defocusing is introduced in sampled rays, causing the accommodative response to shift from the RDP towards the CDP. For example, in a system with a CDP of 4 diopters (D) and 45 sampled rays, when the RDP is at 3, 2, 1, and 0 D, the accommodative response shifts to 3.25, 2.75, 2, and 1.75 D, respectively. With fewer rays, the accommodative response tends to further shift to the CDP. Eventually, with fewer than five rays, the eye accommodates to the CDP and loses the 3D display capacity. Moreover, under different RDPs, the ray number influences differently, and vice versa. An x-y polynomial equation containing three interactive terms is finally provided to reveal the interaction between RDP position and ray number. In comparison, in a pinhole-based system with no CDP, the accommodative response always matches the RDP when the sampled ray number is greater than five.

Maxwellian near-eye display with an expanded eyebox

Maxwellian view systems can be employed to circumvent the vergence-accommodation conflict in near-eye displays (NEDs), which directly project images onto the retina regardless of the human eye's depth of focus. However, Maxwellian view optics typically have a limited eyebox, which prevents broader applications of this architecture in NEDs. Here, we demonstrate a thin-film two-dimensional beam deflector composed of multi-twist broad-band Pancharatnam-Berry deflectors to mitigate this limitation via eyebox replication. Based on experimental validation, our proposed design can display always-focused full-color images within a 9 mm × 9 mm eyebox and thus mitigate the limitation of conventional Maxwellian displays while adding negligible weight and volume.

Toward the next-generation VR/AR optics: a review of holographic near-eye displays from a human-centric perspective

Wearable near-eye displays for virtual and augmented reality (VR/AR) have seen enormous growth in recent years. While researchers are exploiting a plethora of techniques to create life-like three-dimensional (3D) objects, there is a lack of awareness of the role of human perception in guiding the hardware development. An ultimate VR/AR headset must integrate the display, sensors, and processors in a compact enclosure that people can comfortably wear for a long time while allowing a superior immersion experience and user-friendly human-computer interaction. Compared with other 3D displays, the holographic display has unique advantages in providing natural depth cues and correcting eye aberrations. Therefore, it holds great promise to be the enabling technology for next-generation VR/AR devices. In this review, we survey the recent progress in holographic near-eye displays from the human-centric perspective.

Foveated display system based on a doublet geometric phase lens

We propose a new concept of a foveated display with a single display module. A multi-resolution and wide field of view (FOV) can be simultaneously achieved using only a single display, based on temporal polarization-multiplexing. The polarization-dependent lens set functions as an optical window or beam expander system depending on the polarization state, which can provide two operating modes: fovea mode for a high-resolution and peripheral mode for a wide viewing angle. By superimposing two-mode images, the proposed system supports a foveated and wide FOV image without an ultra-high-resolution display. We demonstrate the feasibility of the proposed configuration through the proof-of-concept system.

Augmented Reality and Virtual Reality Displays: Perspectives and Challenges

As one of the most promising candidates for next-generation mobile platform, augmented reality (AR) and virtual reality (VR) have potential to revolutionize the ways we perceive and interact with various digital information. In the meantime, recent advances in display and optical technologies, together with the rapidly developing digital processers, offer new development directions to advancing the near-eye display systems further. In this perspective paper, we start by analyzing the optical requirements in near-eye displays poised by the human visual system and then compare it against the specifications of state-of-the-art devices, which reasonably shows the main challenges in near-eye displays at the present stage. Afterward, potential solutions to address these challenges in both AR and VR displays are presented case by case, including the most recent optical research and development, which are already or have the potential to be industrialized for extended reality displays.

Standing wave polarization holography for realizing liquid crystal Pancharatnum-Berry phase lenses

A standing wave polarization holography setup is proposed to generate the desired polarization field for fabricating both on-axis and off-axis liquid crystal Pancharatnum-Berry phase lenses. Compared to other interference exposure setups, standing wave interferometry can double the polarization field amplitude because it does not require a beam splitter. Moreover, the optical axis angle of the lenses can be easily adjusted without realigning the optical setup. Based on the design, we first theoretically derive the polarization field distribution. In the experiment, we build the recording optical system and fabricate a series of on-axis and off-axis lenses. Further optical characterization proves the high diffraction efficiency of the fabricated lenses.

Photo-Aligned Ferroelectric Liquid Crystal Devices with Novel Electro-Optic Characteristics

This paper examines different applications of ferroelectric liquid crystal devices based on photo-alignment. Successful application of the photo-alignment technique is considered to be a critical breakthrough. A variety of display and photonic devices with azo dye aligned ferroelectric liquid crystals is presented: smart glasses, liquid crystal Pancharatnam–Berry phase optical elements, 2D/3D switchable lenses, and laser therapy devices. Comparison of electro-optical behavior of ferroelectric liquid crystals is described considering the performance of devices. This paper facilitates the optimization of device design, and broadens the possible applications in the display and photonic area.

Mini-LED, Micro-LED and OLED displays: present status and future perspectives

Presently, liquid crystal displays (LCDs) and organic light-emitting diode (OLED) displays are two dominant flat panel display technologies. Recently, inorganic mini-LEDs (mLEDs) and micro-LEDs (μLEDs) have emerged by significantly enhancing the dynamic range of LCDs or as sunlight readable emissive displays. "mLED, OLED, or μLED: who wins?" is a heated debatable question. In this review, we conduct a comprehensive analysis on the material properties, device structures, and performance of mLED/μLED/OLED emissive displays and mLED backlit LCDs. We evaluate the power consumption and ambient contrast ratio of each display in depth and systematically compare the motion picture response time, dynamic range, and adaptability to flexible/transparent displays. The pros and cons of mLED, OLED, and μLED displays are analysed, and their future perspectives are discussed.

Eccentricity effects on blur and depth perception

Foveation and (de)focus are two important visual factors in designing near eye displays. Foveation can reduce computational load by lowering display details towards the visual periphery, while focal cues can reduce vergence-accommodation conflict thereby lessening visual discomfort in using near eye displays. We performed two psychophysical experiments to investigate the relationship between foveation and focus cues. The first study measured blur discrimination sensitivity as a function of visual eccentricity, where we found discrimination thresholds significantly lower than previously reported. The second study measured depth discrimination threshold where we found a clear dependency on visual eccentricity. We discuss the study results and suggest further investigation.

Broadband wide-view Pancharatnam-Berry phase deflector

We demonstrate a high-efficiency achromatic, wide-view Pancharatnam-Berry phase deflector (PBD) based on a three-layer multi-twist structure. A practical method to measure the thickness and twist angle of liquid crystal (LC) polymer films is developed based on Jones matrix of twist-nematic liquid crystals. With the help of this new measurement method, we fabricated a three-layer multi-twist PBD. The imaging performance and the angular response of the achromatic wide-view PBD are also characterized. Potential application of PBD for near-eye displays is foreseeable.

Foveated holographic near-eye 3D display

We present a foveated rendering method to accelerate the amplitude-only computer-generated hologram (AO-CGH) calculation in a holographic near-eye 3D display. For a given target image, we compute a high-resolution foveal region and a low-resolution peripheral region with dramatically reduced pixel numbers. Our technique significantly improves the computation speed of the AO-CGH while maintaining the perceived image quality in the fovea. Moreover, to accommodate the eye gaze angle change, we develop an algorithm to laterally shift the foveal image with negligible extra computational cost. Our technique holds great promise in advancing the holographic 3D display in real-time use.

Patch scanning displays: spatiotemporal enhancement for displays

Emerging fields of mixed reality and electronic sports necessitate greater spatial and temporal resolutions in displays. We introduce a novel scanning display method that enhances spatiotemporal qualities of displays. Specifically, we demonstrate that scanning multiple image patches that are representing basis functions of each block in a target image can help to synthesize spatiotemporally enhanced visuals. To discover the right image patches, we introduce an optimization framework tailored to our hardware. In our method, spatiotemporally enhanced visuals are synthesized using an optical scanner scanning image patches from an image generator illuminated by a locally addressable backlight. As a validation of our method, we demonstrate a prototype using commodity equipment. Our method improves pixel fill factor to hundred percent and enhances spatial resolution of a display up to four times. An inherent constrain regarding to spatiotemporal qualities of displays could be solved using our method.

Design of foveated contact lens display for augmented reality

We present a design of a contact lens display, which features an array of collimated light-emitting diodes and a contact lens, for the augmented reality. By building the infrastructure directly on top of the eye, eye is allowed to move or rotate freely without the need of exit pupil expansion nor eye tracking. The resolution of light-emitting diodes is foveated to match with the density of cones on the retina. In this manner, the total number of pixels as well as the latency of image processing can be significantly reduced. Based on the simulation, the device performance is quantitatively analyzed. For the real image, modulation transfer functions is 0.669757 at 30 cycle/degree, contrast ratio is 5, and distortion is 10%. For the virtual image, the field of view is 82°, best angular resolution is 0.38', modulation transfer function is above 0.999999 at 30 cycle/degree, contrast ratio is 4988, and distortion is 6%.

Spatial loss factor for the analysis of accommodation depth cue on near-eye light field displays

In order to address the vergence-accommodation conflict problem, a generalized model consisting of a human visual system for the 4D light field display system was proposed. This model includes the key factors, such as retinal resolution, the central depth plane (CDP), and the proposed spatial loss factor, for the light field display system. The spatial resolution of the target plane in the depth of field (DOF) were quantitatively evaluated based on the proposed model. The results showed that the inconsistency of spatial resolution in DOF results in unstable eye accommodation response. Based on the fovea resolution-limit, we evaluated and simulated the resolution of perceived images on retina and the accommodation response based on spatial loss factor. The simulation results verified that a near-eye light field display (NE-LFD) configuration with a spatial loss factor greater than 0.8, corresponding to 2 by 2 views, or a minimum spatial loss factor 0.6, corresponding to 3 by 3 views, has the ability to render a nearly correct focus cues and accommodation response.

Towards a Switchable AR/VR Near-eye Display with Accommodation-Vergence and Eyeglass Prescription Support

In this paper, we present our novel design for switchable AR/VR near-eye displays which can help solve the vergence-accommodation-conflict issue. The principal idea is to time-multiplex virtual imagery and real-world imagery and use a tunable lens to adjust focus for the virtual display and the see-through scene separately. With this novel design, prescription eyeglasses for near- and far-sighted users become unnecessary. This is achieved by integrating the wearer's corrective optical prescription into the tunable lens for both virtual display and see-through environment. We built a prototype based on the design, comprised of micro-display, optical systems, a tunable lens, and active shutters. The experimental results confirm that the proposed near-eye display design can switch between AR and VR and can provide correct accommodation for both.

Ferroelectric Liquid Crystals: Physics and Applications

Electrooptic modes with fast response and high contrast ratio are highly desirable in modern photonics and displays. Ferroelectric liquid crystals (FLCs) are especially promising for fulfilling these demands by employing photoalignment technology in FLC cells. Three electrooptic modes including surface stabilized FLC (SSFLC), deformed helix ferroelectric (DHF) mode, and electrically suppressed helix (ESH) mode are reviewed with the corresponding electrooptic effects like bi- and multi-stable switching, continuous modulation of grayscale or phase, and high contrast switching. The general operation principles FLC electrooptic modes are described, and then the characteristics of each modes for potential applications are summarized. With the advantages of controllable anchoring energy, the photoalignment provides FLC samples with uniform alignment and high contrast ratio. The fast FLCs with a high resolution and high contrast can be used in the next generation display including field sequential color FLC microdisplays, as well as switchable 2D/3D televisions.

Device simulation of liquid crystal polarization gratings

Liquid crystal polarization gratings manifest several unique features, such as high diffraction efficiency, polarization selectivity, and fast switching time. However, few works address the chiral-doped liquid crystal alignment issue in such gratings. Here, we develop an improved relaxation method to analyze the liquid crystal director distribution in chiral-doped polarization gratings. Our simulation result agrees well with experimental data on a polarization volume grating. The criteria for forming planar or slanted polarization grating are discussed.

Liquid Crystal Beam Steering Devices: Principles, Recent Advances, and Future Developments

Continuous, wide field-of-view, high-efficiency, and fast-response beam steering devices are desirable in a plethora of applications. Liquid crystals (LCs)—soft, bi-refringent, and self-assembled materials which respond to various external stimuli—are especially promising for fulfilling these demands. In this paper, we review recent advances in LC beam steering devices. We first describe the general operation principles of LC beam steering techniques. Next, we delve into different kinds of beam steering devices, compare their pros and cons, and propose a new LC-cladding waveguide beam steerer using resistive electrodes and present our simulation results. Finally, two future development challenges are addressed: Fast response time for mid-wave infrared (MWIR) beam steering, and device hybridization for large-angle, high-efficiency, and continuous beam steering. To achieve fast response times for MWIR beam steering using a transmission-type optical phased array, we develop a low-loss polymer-network liquid crystal and characterize its electro-optical properties.

Improving near-eye display resolution by polarization multiplexing

We present here an optical approach to boost the apparent pixel density by utilizing the superimposition of two shifted-pixel grids generated by a Pancharatnam-Berry deflector (PBD). The content of the two shifted pixel grids are presented to the observer's eye simultaneously using a polarization-multiplexing method. Considering the compact and lightweight nature of PBD, this approach has potential applications in near-eye display systems. Moreover, the same concept can be extended to projection displays with proper modifications.

All-passive transformable optical mapping near-eye display

We present an all-passive, transformable optical mapping (ATOM) near-eye display based on the “human-centric design” principle. By employing a diffractive optical element, a distorted grating, the ATOM display can project different portions of a two-dimensional display screen to various depths, rendering a real three-dimensional image with correct focus cues. Thanks to its all-passive optical mapping architecture, the ATOM display features a reduced form factor and low power consumption. Moreover, the system can readily switch between a real-three-dimensional and a high-resolution two-dimensional display mode, providing task-tailored viewing experience for a variety of VR/AR applications.

A multidirectional beam steering reflector actuated by hydraulic control

This paper presents a multidirectional beam steering reflector (MBSR) actuated by hydraulic control. It consists of three substrates, an elastic membrane, a magnetic base and a mirror reflector (MR). The MR is fixed on the magnetic base and covered upon the top substrate. The bottom substrate is designed with three channels for pulling in/out the liquid. When liquid volume changes, the shape of the elastic membrane changes to form a liquid piston, accordingly. The liquid piston can make the MR rotate to different directions. When a light beam irradiates the MR, it can achieve the function of beam steering in latitude and longitude, simultaneously. Our experiments show that the proposed MBSR can deflect the light beam through a maximum angle of 0~12.7° in latitude and six-directions in longitude. The MBSR has potential applications in the fields of free-space optical communications, laser detections and solar cells.

Progress in virtual reality and augmented reality based on holographic display

The past, present, and future industry prospects of virtual reality (VR) and augmented reality (AR) are presented. The future of VR/AR technology based on holographic display is predicted by analogy with the VR/AR based on binocular vision display and light field display. The investigations on holographic display that can be used in VR/AR are reviewed. The breakthroughs of holographic display are promising in VR/AR with high resolution. The challenges faced by VR/AR based on holographic display are analyzed.

Fabrication of Pancharatnam-Berry phase optical elements with highly stable polarization holography

Polarization-dependent diffraction based on Pancharatnam-Berry phase optical elements (PBOEs) offers considerable benefits compared to conventional metasurfaces, such as negligible absorption, nearly 100% diffraction efficiency and an inexpensive fabrication process. Polarization holography is a simple way to fabricate PBOEs, which entails the interference of beams with different polarizations to generate a spatial-varying polarization field. Thus, the quality of recorded PBOEs manifests high sensitivity to the length change and phase shift between polarized beams, usually caused by environmental vibration and air flow. Here, new polarization holography based on modified Sagnac interferometry is developed for fabricating liquid crystal-based PB gratings and lenses, where the pitch of grating and optical power of lens could be easily tuned. This approach offers high tolerance to environmental disturbance during the exposure process. Detailed design parameters are analyzed, and the fabricated PBOEs with high optical quality are also demonstrated.

Enhanced see-through near-eye display using time-division multiplexing of a Maxwellian-view and holographic display

In this paper, we suggest the improved integration of a holographic display and a Maxwellian-view display using time-division multiplexing and describe an image rendering process for the proposed system. In general, the holographic displays have a resolution limit when used to represent a virtual 3D scene. In the proposed system, the holographic display processed relatively few layers of the virtual 3D scene, while the remaining objects were processed with a Maxwellian-view display to which was applied a Gaussian smoothing filter. Hence, we obtained the retaining holographic image quality, expanding the field of view, and reducing the computation time of the proposed system. The holographic display of the proposed system had an image size of 28 mm × 28 mm with a field of view of 1.02° and a 10.8 mm eye box. The Maxwellian-view display had an image size of 230 mm × 230 mm with a field of view of 22.6 ° and a 0.9 mm eye box diameter. Each display was integrated in time-division multiplexing of 40 Hz, and the proposed system was successfully verified.

Polarization-independent Pancharatnam-Berry phase lens system

The conventional liquid crystal-based Pancharatnam-Berry (PB) phase lens exhibits distinct polarization selectivity, manifesting opposite optical power to circularly polarized light with opposite handedness. Here, a polarization-independent liquid crystal PB lens system is theoretically predicted and experimentally verified. Such a lens system consists of at least four PB lenses, with specific distances in between them. This enables the PB lens to be applied in polarization-independent optical systems.

Liquid-Crystal-on-Silicon for Augmented Reality Displays

In this paper, we review liquid-crystal-on-silicon (LCoS) technology and focus on its new application in emerging augmented reality (AR) displays. In the first part, the LCoS working principles of three commonly adopted LC modes—vertical alignment and twist nematic for amplitude modulation, and homogeneous alignment for phase modulation—are introduced and their pros and cons evaluated. In the second part, the fringing field effect is analyzed, and a novel pretilt angle patterning method for suppressing the effect is presented. Moreover, we illustrate how to integrate the LCoS panel in an AR display system. Both currently available intensity modulators and under-developing holographic displays are covered, with special emphases on achieving high image quality, such as a fast response time and high-resolution. The rapidly increasing application of LCoS in AR head-mounted displays and head-up displays is foreseeable.

Polarization-multiplexed multiplane display

We demonstrate a polarization-multiplexed multiplane display system for near-eye applications. A polarization-sensitive Pancharatnam-Berry phase lens is implemented to generate two focal depths simultaneously. A spatial polarization modulator is utilized to direct the two images to designated focal planes. Based on this design, a dual-focal-plane display system is constructed without space- or time-multiplexing operations, to suppress the vergence-accommodation conflict successfully.