Depth director: a system for adding depth to movies

Perceptual Real-Time 2D-to-3D Conversion Using Cue Fusion

We propose a system to infer binocular disparity from a monocular video stream in real-time. Different from classic reconstruction of physical depth in computer vision, we compute perceptually plausible disparity, that is numerically inaccurate, but results in a very similar overall depth impression with plausible overall layout, sharp edges, fine details and agreement between luminance and disparity. We use several simple monocular cues to estimate disparity maps and confidence maps of low spatial and temporal resolution in real-time. These are complemented by spatially-varying, appearance-dependent and class-specific disparity prior maps, learned from example stereo images. Scene classification selects this prior at runtime. Fusion of prior and cues is done by means of robust MAP inference on a dense spatio-temporal conditional random field with high spatial and temporal resolution. Using normal distributions allows this in constant-time, parallel per-pixel work. We compare our approach to previous 2D-to-3D conversion systems in terms of different metrics, as well as a user study and validate our notion of perceptually plausible disparity.

High-Quality Depth Estimation Using an Exemplar 3D Model for Stereo Conversion

High-quality depth painting for each object in a scene is a challenging task in 2D to 3D stereo conversion. One way to accurately estimate the varying depth within the object in an image is to utilize existing 3D models. Automatic pose estimation approaches based on 2D-3D feature correspondences have been proposed to obtain depth from a given 3D model. However, when the 3D model is not identical to the target object, previous methods often produce erroneous depth in the vicinity of the silhouette of the object. This paper introduces a novel 3D model-based depth estimation method that effectively produces high-quality depth information for rigid objects in a stereo conversion workflow. Given an exemplar 3D model and user correspondences, our method generates detailed depth of an object by optimizing the initial depth obtained by the application of structural fitting and silhouette matching in the image domain. The final depth is accurate up to the given 3D model, while consistent with the image. Our method was applied to various image sequences containing objects with different appearances and varying poses. The experiments show that our method can generate plausible depth information that can be utilized for high-quality 2D to 3D stereo conversion.

Depth Transfer: Depth Extraction from Video Using Non-Parametric Sampling

We describe a technique that automatically generates plausible depth maps from videos using non-parametric depth sampling. We demonstrate our technique in cases where past methods fail (non-translating cameras and dynamic scenes). Our technique is applicable to single images as well as videos. For videos, we use local motion cues to improve the inferred depth maps, while optical flow is used to ensure temporal depth consistency. For training and evaluation, we use a Kinect-based system to collect a large data set containing stereoscopic videos with known depths. We show that our depth estimation technique outperforms the state-of-the-art on benchmark databases. Our technique can be used to automatically convert a monoscopic video into stereo for 3D visualization, and we demonstrate this through a variety of visually pleasing results for indoor and outdoor scenes, including results from the feature film Charade.

WYSIWYG Stereo Paintingwith Usability Enhancements

Despite increasing popularity of stereo capture and display systems, creating stereo artwork remains a challenge. This paper presents a stereo painting system, which enables effective from-scratch creation of high-quality stereo artwork. A key concept of our system is a stereo layer, which is composed of two RGBAd (RGBA + depth) buffers. Stereo layers alleviate the need for fully formed representational 3D geometry required by most existing 3D painting systems, and allow for simple, essential depth specification. RGBAd buffers also provide scalability for complex scenes by minimizing the dependency of stereo painting updates on the scene complexity. For interaction with stereo layers, we present stereo paint and stereo depth brushes, which manipulate the photometric (RGBA) and depth buffers of a stereo layer, respectively. In our system, painting and depth manipulation operations can be performed in arbitrary order with real-time visual feedback, providing a flexible WYSIWYG workflow for stereo painting. Our data structures allow for easy interoperability with existing image and geometry data, enabling a number of applications beyond from-scratch art creation, such as stereo conversion of monoscopic artwork and mixed-media art. Comments from artists and experimental results demonstrate that our system effectively aides in the creation of compelling stereo paintings.

StereoPasting: interactive composition in stereoscopic images

We propose "StereoPasting," an efficient method for depth-consistent stereoscopic composition, in which a source 2D image is interactively blended into a target stereoscopic image. As we paint "disparity" on a 2D image, the disparity map of the selected region is gradually produced by edge-aware diffusion, and then blended with that of the target stereoscopic image. By considering constraints of the expected disparities and perspective scaling, the 2D object is warped to generate an image pair, which is then blended into the target image pair to get the composition result. The warping is formulated as an energy minimization, which could be solved in real time. We also present an interactive composition system, in which users can edit the disparity maps of 2D images by strokes, while viewing the composition results instantly. Experiments show that our method is intuitive and efficient for interactive stereoscopic composition. A lot of applications demonstrate the versatility of our method.

Monocular depth ordering using T-junctions and convexity occlusion cues

This paper proposes a system that relates objects in an image using occlusion cues and arranges them according to depth. The system does not rely on a priori knowledge of the scene structure and focuses on detecting special points, such as T-junctions and highly convex contours, to infer the depth relationships between objects in the scene. The system makes extensive use of the binary partition tree as hierarchical region-based image representation jointly with a new approach for candidate T-junction estimation. Since some regions may not involve T-junctions, occlusion is also detected by examining convex shapes on region boundaries. Combining T-junctions and convexity leads to a system which only relies on low level depth cues and does not rely on semantic information. However, it shows a similar or better performance with the state-of-the-art while not assuming any type of scene. As an extension of the automatic depth ordering system, a semi-automatic approach is also proposed. If the user provides the depth order for a subset of regions in the image, the system is able to easily integrate this user information to the final depth order for the complete image. For some applications, user interaction can naturally be integrated, improving the quality of the automatically generated depth map.