Depth extraction with offset pixels

CMOS Depth Image Sensor with Offset Pixel Aperture Using a Back-Side Illumination Structure for Improving Disparity

This paper presents a CMOS depth image sensor with offset pixel aperture (OPA) using a back-side illumination structure to improve disparity. The OPA method is an efficient way to obtain depth information with a single image sensor without additional external factors. Two types of apertures (i.e., left-OPA (LOPA) and right-OPA (ROPA)) are applied to pixels. The depth information is obtained from the disparity caused by the phase difference between the LOPA and ROPA images. In a CMOS depth image sensor with OPA, disparity is important information. Improving disparity is an easy way of improving the performance of the CMOS depth image sensor with OPA. Disparity is affected by pixel height. Therefore, this paper compared two CMOS depth image sensors with OPA using front-side illumination (FSI) and back-side illumination (BSI) structures. As FSI and BSI chips are fabricated via different processes, two similar chips were used for measurement by calculating the ratio of the OPA offset to pixel size. Both chips were evaluated for chief ray angle (CRA) and disparity in the same measurement environment. Experimental results were then compared and analyzed for the two CMOS depth image sensors with OPA.

Unsupervised deep learning for depth estimation with offset pixels

Offset Pixel Aperture (OPA) camera has been recently proposed to estimate disparity of a scene with a single shot. Disparity is obtained in the image by offsetting the pixels by a fixed distance. Previously, correspondence matching schemes have been used for disparity estimation with OPA. To improve disparity estimation we use a data-oriented approach. Specifically, we use unsupervised deep learning to estimate the disparity in OPA images. We propose a simple modification to the training strategy which solves the vanishing gradients problem with the very small baseline of the OPA camera. Training degenerates to poor disparity maps if the OPA images are used directly for left-right consistency check. By using images obtained from displaced cameras at training, accurate disparity maps are obtained. The performance of the OPA camera is significantly improved compared to previously proposed single-shot cameras and unsupervised disparity estimation methods. The approach provides 8 frames per second on a single Nvidia 1080 GPU with 1024×512 OPA images. Unlike conventional approaches, which are evaluated in controlled environments, our paper shows the utility of deep learning for disparity estimation with real life sensors and low quality images. By combining OPA with deep learning, we obtain a small depth sensor capable of providing accurate disparity at usable frame rates. Also the ideas in this work can be used in small-baseline stereo systems for short-range depth estimation and multi-baseline stereo to increase the depth range.

Effects of Offset Pixel Aperture Width on the Performances of Monochrome CMOS Image Sensors for Depth Extraction

This paper presents the effects of offset pixel aperture width on the performance of monochrome (MONO) CMOS image sensors (CISs) for a three-dimensional image sensor. Using a technique to integrate the offset pixel aperture (OPA) inside each pixel, the depth information can be acquired using a disparity from OPA patterns. The OPA is classified into two pattern types: Left-offset pixel aperture (LOPA) and right-offset pixel aperture (ROPA). These OPAs are divided into odd and even rows and integrated in a pixel array. To analyze the correlation between the OPA width and the sensor characteristics, experiments were conducted by configuring the test elements group (TEG) regions. The OPA width of the TEG region for the measurement varied in the range of 0.3–0.5 μm. As the aperture width decreased, the disparity of the image increased, while the sensitivity decreased. It is possible to acquire depth information by the disparity obtained from the proposed MONO CIS using the OPA technique without an external light source. Therefore, the proposed MONO CIS with OPA could easily be applied to miniaturized devices. The proposed MONO CIS was designed and manufactured using the 0.11 μm CIS process.

Analysis of Disparity Information for Depth Extraction Using CMOS Image Sensor with Offset Pixel Aperture Technique

A complementary metal oxide semiconductor (CMOS) image sensor (CIS), using offset pixel aperture (OPA) technique, was designed and fabricated using the 0.11-µm CIS process. In conventional cameras, an aperture is located on the camera lens. However, in a CIS camera using OPA technique, apertures are integrated as left-offset pixel apertures (LOPAs) and right-offset pixel apertures (ROPAs). A color pattern is built, comprising LOPA, blue, red, green, and ROPA pixels. The disparity information can be acquired from the LOPA and ROPA channels. Both disparity information and two-dimensional (2D) color information can be simultaneously acquired from the LOPA, blue, red, green, and ROPA channels. A geometric model of the OPA technique is constructed to estimate the disparity of the image, and the measurement results are compared with the estimated results. Depth extraction is thus achieved by a single CIS using the OPA technique, which can be easily adapted to commercial CIS cameras.