Accidental Pinhole and Pinspeck Cameras

Towards a more accurate light transport model for non-line-of-sight imaging

Non-line-of-sight (NLOS) imaging systems involve the measurement of an optical signal at a diffuse surface. A forward model encodes the physics of these measurements mathematically and can be inverted to generate a reconstruction of the hidden scene. Some existing NLOS imaging techniques rely on illuminating the diffuse surface and measuring the photon time of flight (ToF) of multi-bounce light paths. Alternatively, some methods depend on measuring high-frequency variations caused by shadows cast by occluders in the hidden scene. While forward models for ToF-NLOS and Shadow-NLOS have been developed separately, there has been limited work on unifying these two imaging modalities. Dove et al introduced a unified mathematical framework capable of modeling both imaging techniques [Opt. Express27, 18016 (2019)10.1364/OE.27.018016]. The authors utilize this general forward model, known as the two frequency spatial Wigner distribution (TFSWD), to discuss the implications of reconstruction resolution for combining the two modalities but only when the occluder geometry is known a priori. In this work, we develop a graphical representation of the TFSWD forward model and apply it to novel experimental setups with potential applications in NLOS imaging. Furthermore, we use this unified framework to explore the potential of combining these two imaging modalities in situations where the occluder geometry is not known in advance.

Two-edge-resolved three-dimensional non-line-of-sight imaging with an ordinary camera

We introduce an approach for three-dimensional full-colour non-line-of-sight imaging with an ordinary camera that relies on a complementary combination of a new measurement acquisition strategy, scene representation model, and tailored reconstruction method. From an ordinary photograph of a matte line-of-sight surface illuminated by the hidden scene, our approach reconstructs a three-dimensional image of the scene hidden behind an occluding structure by exploiting two orthogonal edges of the structure for transverse resolution along azimuth and elevation angles and an information orthogonal scene representation for accurate range resolution. Prior demonstrations beyond two-dimensional reconstructions used expensive, specialized optical systems to gather information about the hidden scene. Here, we achieve accurate three-dimensional imaging using inexpensive, and ubiquitous hardware, without requiring a calibration image. Thus, our system may find use in indoor situations like reconnaissance and search-and-rescue.

Imaging privacy threats from an ambient light sensor

Embedded sensors in smart devices pose privacy risks, often unintentionally leaking user information. We investigate how combining an ambient light sensor with a device display can capture an image of touch interaction without a camera. By displaying a known video sequence, we use the light sensor to capture reflected light intensity variations partially blocked by the touching hand, formulating an inverse problem similar to single-pixel imaging. Because of the sensors' heavy quantization and low sensitivity, we propose an inversion algorithm involving an ℓp-norm dequantizer and a deep denoiser as natural image priors, to reconstruct images from the screen's perspective. We demonstrate touch interactions and eavesdropping hand gestures on an off-the-shelf Android tablet. Despite limitations in resolution and speed, we aim to raise awareness of potential security/privacy threats induced by the combination of passive and active components in smart devices and promote the development of ways to mitigate them.

Non-line-of-sight snapshots and background mapping with an active corner camera

The ability to form reconstructions beyond line-of-sight view could be transformative in a variety of fields, including search and rescue, autonomous vehicle navigation, and reconnaissance. Most existing active non-line-of-sight (NLOS) imaging methods use data collection steps in which a pulsed laser is directed at several points on a relay surface, one at a time. The prevailing approaches include raster scanning of a rectangular grid on a vertical wall opposite the volume of interest to generate a collection of confocal measurements. These and a recent method that uses a horizontal relay surface are inherently limited by the need for laser scanning. Methods that avoid laser scanning to operate in a snapshot mode are limited to treating the hidden scene of interest as one or two point targets. In this work, based on more complete optical response modeling yet still without multiple illumination positions, we demonstrate accurate reconstructions of foreground objects while also introducing the capability of mapping the stationary scenery behind moving objects. The ability to count, localize, and characterize the sizes of hidden objects, combined with mapping of the stationary hidden scene, could greatly improve indoor situational awareness in a variety of applications.

Computational Imaging on the Electric Grid

Night beats with alternating current (AC) illumination. By passively sensing this beat, we reveal new scene information which includes: the type of bulbs in the scene, the phases of the electric grid up to city scale, and the light transport matrix. This information yields unmixing of reflections and semi-reflections, nocturnal high dynamic range, and scene rendering with bulbs not observed during acquisition. The latter is facilitated by a dataset of bulb response functions for a range of sources, which we collected and provide. To do all this, we built a novel coded-exposure high-dynamic-range imaging technique, specifically designed to operate on the grid's AC lighting.

Deep camera obscura: an image restoration pipeline for pinhole photography

Modern machine learning has enhanced the image quality for consumer and mobile photography through low-light denoising, high dynamic range (HDR) imaging, and improved demosaicing among other applications. While most of these advances have been made for normal lens-based cameras, there has been an emerging body of research for improved photography for lensless cameras using thin optics such as amplitude or phase masks, diffraction gratings, or diffusion layers. These lensless cameras are suited for size and cost-constrained applications such as tiny robotics and microscopy that prohibit the use of a large lens. However, the earliest and simplest camera design, the camera obscura or pinhole camera, has been relatively overlooked for machine learning pipelines with minimal research on enhancing pinhole camera images for everyday photography applications. In this paper, we develop an image restoration pipeline of the pinhole system to enhance the pinhole image quality through joint denoising and deblurring. Our pipeline integrates optics-based filtering and reblur losses for reconstructing high resolution still images (2600 × 1952) as well as temporal consistency for video reconstruction to enable practical exposure times (30 FPS) for high resolution video (1920 × 1080). We demonstrate high 2D image quality on real pinhole images that is on-par or slightly improved compared to other lensless cameras. This work opens up the potential of pinhole cameras to be used for photography in size-limited devices such as smartphones in the future.

Seeing around corners with edge-resolved transient imaging

Non-line-of-sight (NLOS) imaging is a rapidly growing field seeking to form images of objects outside the field of view, with potential applications in autonomous navigation, reconnaissance, and even medical imaging. The critical challenge of NLOS imaging is that diffuse reflections scatter light in all directions, resulting in weak signals and a loss of directional information. To address this problem, we propose a method for seeing around corners that derives angular resolution from vertical edges and longitudinal resolution from the temporal response to a pulsed light source. We introduce an acquisition strategy, scene response model, and reconstruction algorithm that enable the formation of 2.5-dimensional representations—a plan view plus heights—and a 180^∘ field of view for large-scale scenes. Our experiments demonstrate accurate reconstructions of hidden rooms up to 3 meters in each dimension despite a small scan aperture (1.5-centimeter radius) and only 45 measurement locations.

Non-line-of-sight imaging is typically limited by loss of directional information due to diffuse reflections scattering light in all directions. Here, the authors see around corners by using vertical edges and temporal response to pulsed light to obtain angular and longitudinal resolution, respectively.

Imaging around corners in the mid-infrared using speckle correlations

Speckle correlation imaging offers the ability to see objects through diffusive materials and around corners. Imaging self-illuminating thermal objects in non-line-of-sight scenarios is of particular interest. Here, using bispectrum and phase retrieval methods, we demonstrate speckle correlation imaging of mid-infrared objects through diffusers and around corners at resolutions near the diffraction limit. The images agree well with those recorded by conventional cameras with line-of-sight to the same objects.

Thermal non-line-of-sight imaging from specular and diffuse reflections

This paper presents a non-line-of-sight technique to estimate the position and temperature of an occluded object from a camera via reflection on a wall. Because objects with heat emit far infrared light with respect to their temperature, positions and temperatures are estimated from reflections on a wall. A key idea is that light paths from a hidden object to the camera depend on the position of the hidden object. The position of the object is recovered from the angular distribution of specular and diffuse reflection component, and the temperature of the heat source is recovered from the estimated position and the intensity of reflection. The effectiveness of our method is evaluated by conducting real-world experiments, showing that the position and the temperature of the hidden object can be recovered from the reflection destination of the wall by using a conventional thermal camera.

Beyond perspective dual photography with illumination masks

Scene appearance from the point of view of a light source is called a reciprocal or dual view. Since there exists a large diversity in illumination, these virtual views may be nonperspective and multiviewpoint in nature. In this paper, we demonstrate the use of occluding masks to recover these dual views, which we term shadow cameras. We first show how to render a single reciprocal scene view by swapping the camera and light source positions. We then extend this technique for multiple views by both building a virtual shadow camera array and by exploiting area sources. We also capture nonperspective views such as orthographic, cross-slit and a pushbroom variant, while introducing novel applications such as converting between camera projections and removing refractive and catadioptric distortions. Finally, since a shadow camera is artificial, we can manipulate any of its intrinsic parameters, such as camera skew, to create perspective distortions. We demonstrate a variety of indoor and outdoor results and show a rendering application for capturing the light-field of a light-source.