Three-dimensional vision based on a combination of gray-code and phase-shift light projection: analysis and compensation of the systematic errors

BimodalPS: Causes and Corrections for Bimodal Multi-Path in Phase-Shifting Structured Light Scanners

Structured light illumination is an active 3D scanning technique based on projecting and capturing a set of striped patterns and measuring the warping of the patterns as they reflect off a target object's surface. As designed, each pixel in the camera sees exactly one pixel from the projector; however, there are multi-path situations where a camera pixel sees light from multiple projector positions. In the case of bimodal multi-path, the camera pixel receives light from exactly two positions, which occurs along a step edge where the edge slices through a pixel which, therefore, sees both a foreground and background surface. In this paper, we present a general mathematical model to address this bimodal multi-path issue in a phase-shifting or so-called phase-measuring-profilometry scanner to measure the constructive and destructive interference between the two light paths, and by taking advantage of this interference, separate the paths and make two decoupled depth measurements. We validate our algorithm with both simulations and a number of challenging real-world scenarios, significantly outperforming the state-of-the-art methods.

Dynamic phase-differencing profilometry with number-theoretical phase unwrapping and interleaved projection

High-speed 3D measurement is receiving increasing attention. However, simultaneously achieving high computational efficiency, algorithmic robustness, and reconstructing ratio is challenging. Therefore, a dynamic phase-differencing profilometry (DPDP) is proposed. By capturing the minimum three phase-shifting sinusoidal deformed patterns and establishing a brand-new model, the phase difference between the object on the reference plane and the reference plane is directly resolved to effectively improve computational efficiency. Although it is wrapped, by using only two auxiliary complementary gratings with a purposely designed lower frequency, a DPDP-based number-theoretical temporal phase unwrapping (NT-TPU) algorithm is also proposed to unwrap the wrapped phase difference rather than the phase itself with high robustness. Furthermore, compared to existing PSP-based NT-TPU, the proposed NT-TPU can normally work under more relaxed restrictions. In order to accomplish a high reconstructing ratio, a pentabasic interleaved projection (PIP) strategy based on time division multiplexing is proposed. It can improve the reconstructing ratio from one reconstruction per every five patterns to an equivalent of one reconstruction per every 1.67 patterns. Experimental results demonstrate that the proposed method achieves high computational efficiency, high algorithmic robustness, and high reconstructing ratio simultaneously and has prospective application in high-speed 3D measurement.

Phase-aided online self-correction method for high-accuracy three-dimensional measurement

The binocular structured light 3D measurement system is widely used in situ industrial inspection and shape measurement, where the system structure is generally unstable due to mechanical loosening or environmental disturbance. Timely corrections to the changing structural parameters thus is an essential task for online high-accuracy measurement, which is difficult for traditional unidirectional fringe projection methods to self-correct the structural change. To this end, we propose an online self-correction method based on the investigation that orthogonal fringe projection can intrinsically relax the constraint on the epipolar geometry relationship and provide bidirectional phases for accurate corresponding point searching. Since orthogonal fringe projection may sacrifice the measurement efficiency, we further design a searching strategy by locally unwrapping one directional phase to reduce the number of projection patterns. Experimental results demonstrate that the proposed method is effective for online self-correction of unstable system structure to achieve high-accuracy 3D measurement under complex measurement environments.

Robust structured light 3D measurement method based on polarization-encoded projection patterns

Fringe projection profilometry (FPP) is widely used in 3D vision measurement because of its high robustness and measurement accuracy. In the case of HDR objects, due to the problem of surface reflectivity, the obtained image will be overexposed. This will cause the sinusoidality of the fringes projected on the surface of the object in the acquired image to be interfered, resulting in a phase error in the calculated wrapped phase. Therefore, a polarization-encoded sinusoidal structured light is proposed to enhance the sinusoidality of the fringe. The phase information contained in the polarized sinusoidal structured light fringe is only related to the polarization state, not to the light intensity. A polarization coding assisted structured light measurement strategy (PASM) is proposed. This method uses polarization coding assisted polarization phase-shifting fringes for phase unwrapping. The angle of the linear polarizer is set to zero in this method, and it does not require rotating the polarizer. It only needs a single exposure to improve the fringe quality and obtain a more stable unwrapping phase. The experimental results show that the obtained polarization fringes have better sinusoidality, and the phase unwrapping can be more accurate. The reconstructed 3D point cloud also does not appear missing and has better accuracy. It is a reliable method for vision measurement of HDR objects.

Three-Dimensional Reconstruction Based on Multiple Views of Structured Light Projectors and Point Cloud Registration Noise Removal for Fusion

Structured light technology is typical for capturing 3D point cloud data. This paper proposes a 3D reconstruction system to obtain point cloud data of complex objects based on nine-order Gray code and an eight-step structured light projection combined with a phase shift and phase unwrapping method. In this system, two projectors serve as bilateral projectors for structured light, along with a camera and rotating platforms. These components were used to obtain point cloud data from multiple perspectives, which helps avoid the shadow areas caused by a single projection angle and provides complementary point cloud data. The point clusters scanned under each perspective were transformed into the same coordinate system. Furthermore, a distance-based point cloud noise removal algorithm was proposed to optimize platform noise and facilitate point cloud data fusion. The experimental results proved that the system effectively captures 3D point cloud data for complex objects. The dimensional quantitative analysis of an aero engine blade was also performed.

Geometric-feature-based approach to human face reconstruction with high measurement speed

This paper presents a method based on geometry for three-dimensional (3D) face reconstruction without the need for additional images, hardware components, or objects. In our proposed method, we consider part of the nose as the feature region because its shape remains almost constant during the measurement. The geometry of this region was used to provide cues for phase unwrapping. We first spatially unwrap the phase and determine the integer multiple of 2π to be added by comparing the recovered result of the feature region and its actual shape. Then, the face can be reconstructed with the acquired absolute phase. Experimental results demonstrated that our method is capable of reconstructing a dynamic face with high measurement speed, and only three phase-shifted fringes are required per frame.

Digital Fringe Projection-Based Clamping Force Estimation Algorithm for Railway Fasteners

The inspection of railway fasteners to assess their clamping force can be used to evaluate the looseness of the fasteners and improve railway safety. Although there are various methods for inspecting railway fasteners, there is still a need for non-contact, fast inspection without installing additional devices on fasteners. In this study, a system that uses digital fringe projection technology to measure the 3D topography of the fastener was developed. This system inspects the looseness through a series of algorithms, including point cloud denoising, coarse registration based on fast point feature histograms (FPFH) features, fine registration based on the iterative closest point (ICP) algorithm, specific region selection, kernel density estimation, and ridge regression. Unlike the previous inspection technology, which can only measure the geometric parameters of fasteners to characterize the tightness, this system can directly estimate the tightening torque and the bolt clamping force. Experiments on WJ-8 fasteners showed a root mean square error of 9.272 N·m and 1.94 kN for the tightening torque and clamping force, demonstrating that the system is sufficiently precise to replace manual measurement and can substantially improve inspection efficiency while evaluating railway fastener looseness.

Temporal phase unwrapping based on unequal phase-shifting code

In fringe projection profilometry (FPP) based on temporal phase unwrapping (TPU), reducing the number of projecting patterns has become one of the most important works in recent years. To remove the 2π ambiguity independently, this paper proposes a TPU method based on unequal phase-shifting code. Wrapped phase is still calculated from N-step conventional phase-shifting patterns with equal phase-shifting amount to guarantee the measuring accuracy. Particularly, a series of different phase-shifting amounts relative to the first phase-shifting pattern are set as codewords, and encoded to different periods to generate one coded pattern. When decoding, Fringe order with a large number can be determined from the conventional and coded wrapped phases. In addition, we develop a self-correction method to eliminate the deviation between the edge of fringe order and the 2π discontinuity. Thus, the proposed method can achieve TPU but need to only project one additional coded pattern (e. g. 3+1), which can significantly benefit dynamic 3D shape reconstruction. The theoretical and experimental analysis verify that the proposed method performs high robustness on the reflectivity of the isolated object while ensuring the measuring speed.

Large-scale calibration method for MEMS-based projector 3D reconstruction

Projectors based on Micro-Electro-Mechanical System (MEMS) have the advantages of small size and low cost. Moreover, uniaxial MEMS projectors have high projection accuracy, and have been widely used in structured light 3D reconstruction. However, the existing calibration methods for uniaxial MEMS projectors are not effective in large-scale scenes. To solve this problem, this paper proposes a novel efficient large-scale calibration method, which is easily implemented. The proposed method first calibrates a partial light plane for a fixed sampling period, then obtains the rest of the light plane by exploiting a non-fixed rotating shaft linear interpolation method. Experimental results verify that the proposed method attains high accuracy in a large depth field with only 11 sets of calibration data. Specifically, at a distance of 3000mm, the standard deviation of the plane fitting error reaches 0.2584mm on the standard plane, and the measurement accuracy attains 0.9124mm on the standard step object with 200mm interval.

Adaptive horizontal scaling method for speckle-assisted fringe projection profilometry

Phase-shifting method is widely used in fringe projection profilometry to obtain high-precision wrapped phase maps. The wrapped phase map needs to be converted to an absolute phase map to recover 3D information. The speckle pattern based phase unwrapping method requires only one additional auxiliary pattern, showing great potential for fast 3D measurements. In this paper, a speckle assisted four-steps phase-shifting method was proposed for 3D measurements. This method requires five structured light patterns to complete 3D measurements, including four-steps phase-shifting fringe patterns and a speckle pattern which is used to remove phase ambiguity. The main challenge of speckle based phase unwrapping method is to overcome the mismatch problem which often occurs in some very steep surfaces. In order to improve the speckle matching accuracy, an adaptive horizontal scaling method was proposed. A voting strategy based on phase-connected regions was proposed to reduce the computational overhead. The experiments demonstrate its superior performance, and an accuracy of 0.21 mm was achieved.

Three-dimensional shape measurement method based on composite cyclic phase coding

Phase coding is widely used in 3D measurement due to its good anti-interference and robustness. However, the measurement accuracy is affected by the limitation of the number of codewords. To solve this problem, we propose a 3D shape measurement method based on composite cyclic phase coding. The traditional phase coding is quantized cyclic without adding extra patterns, further adopting composite coding, using the composite cyclic phase coding grayscale values to distinguish the same cyclic codewords, and finally integrating them into a new fringe order sequentially for phase unwrapping to achieve effective expansion of codewords. The related experimental results show that the proposed method stably achieves high accuracy 3D reconstruction, which overcomes the misjudgment of codewords caused by traditional phase coding under high-frequency fringes due to system nonlinearity and noise. Meanwhile, compared with the improved phase coding method of temporal domain combined with spatial domain information, such as the method of quantized phase coding and connected region labeling, it can effectively avoid the phenomenon of error propagation, with high robustness and low algorithm complexity.

Semi-Global Matching Assisted Absolute Phase Unwrapping

Measuring speed is a critical factor to reduce motion artifacts for dynamic scene capture. Phase-shifting methods have the advantage of providing high-accuracy and dense 3D point clouds, but the phase unwrapping process affects the measurement speed. This paper presents an absolute phase unwrapping method capable of using only three speckle-embedded phase-shifted patterns for high-speed three-dimensional (3D) shape measurement on a single-camera, single-projector structured light system. The proposed method obtains the wrapped phase of the object from the speckle-embedded three-step phase-shifted patterns. Next, it utilizes the Semi-Global Matching (SGM) algorithm to establish the coarse correspondence between the image of the object with the embedded speckle pattern and the pre-obtained image of a flat surface with the same embedded speckle pattern. Then, a computational framework uses the coarse correspondence information to determine the fringe order pixel by pixel. The experimental results demonstrated that the proposed method can achieve high-speed and high-quality 3D measurements of complex scenes.

Digital image correlation assisted absolute phase unwrapping

This paper presents an absolute phase unwrapping method for high-speed three-dimensional (3D) shape measurement. This method uses three phase-shifted patterns and one binary random pattern on a single-camera, single-projector structured light system. We calculate the wrapped phase from phase-shifted images and determine the coarse correspondence through the digital image correlation (DIC) between the captured binary random pattern of the object and the pre-captured binary random pattern of a flat surface. We then developed a computational framework to determine fringe order number pixel by pixel using the coarse correspondence information. Since only one additional pattern is used, the proposed method can be used for high-speed 3D shape measurement. Experimental results successfully demonstrated that the proposed method can achieve high-speed and high-quality measurement of complex scenes.

Hue-indexing-based absolute phase retrieval method using a discrete hue sequence

To achieve an effective phase unwrapping for hue-based fringe projection profilometry, this paper proposes a hue-indexing-based absolute phase retrieval method using a discrete hue sequence. First, the hue component is extracted as the wrapped phase for 3D reconstruction by projecting a programmed hue fringe pattern. Afterward, a hue-indexing sequence with a random combination of six unique hues from the hue map is designed for hue unwrapping. By assigning a hue index of the fringe geometric center, the defocusing effect in the hue unwrapping is corrected, where the fringe order of the wrapped hue is then uniquely identified. The simulations show that the root mean square (RMS) of the residual error is 2.2185×10^-4 r a d, and the effectiveness of the proposed method is further verified through experiments using a plaster statue and a compressor blade. The measurement comparison of the proposed method and the coordinate measuring machine is provided, where the RMS error of the measured deviation is 4.09×10^-2 m m.

Deep learning in optical metrology: a review

With the advances in scientific foundations and technological implementations, optical metrology has become versatile problem-solving backbones in manufacturing, fundamental research, and engineering applications, such as quality control, nondestructive testing, experimental mechanics, and biomedicine. In recent years, deep learning, a subfield of machine learning, is emerging as a powerful tool to address problems by learning from data, largely driven by the availability of massive datasets, enhanced computational power, fast data storage, and novel training algorithms for the deep neural network. It is currently promoting increased interests and gaining extensive attention for its utilization in the field of optical metrology. Unlike the traditional "physics-based" approach, deep-learning-enabled optical metrology is a kind of "data-driven" approach, which has already provided numerous alternative solutions to many challenging problems in this field with better performances. In this review, we present an overview of the current status and the latest progress of deep-learning technologies in the field of optical metrology. We first briefly introduce both traditional image-processing algorithms in optical metrology and the basic concepts of deep learning, followed by a comprehensive review of its applications in various optical metrology tasks, such as fringe denoising, phase retrieval, phase unwrapping, subset correlation, and error compensation. The open challenges faced by the current deep-learning approach in optical metrology are then discussed. Finally, the directions for future research are outlined.

Advances and Prospects of Vision-Based 3D Shape Measurement Methods

Vision-based three-dimensional (3D) shape measurement techniques have been widely applied over the past decades in numerous applications due to their characteristics of high precision, high efficiency and non-contact. Recently, great advances in computing devices and artificial intelligence have facilitated the development of vision-based measurement technology. This paper mainly focuses on state-of-the-art vision-based methods that can perform 3D shape measurement with high precision and high resolution. Specifically, the basic principles and typical techniques of triangulation-based measurement methods as well as their advantages and limitations are elaborated, and the learning-based techniques used for 3D vision measurement are enumerated. Finally, the advances of, and the prospects for, further improvement of vision-based 3D shape measurement techniques are proposed.

Phase-Shifting Projected Fringe Profilometry Using Binary-Encoded Patterns

A phase unwrapping method for phase-shifting projected fringe profilometry is presented. It did not require additional projections to identify the fringe orders. The pattern used for the phase extraction could be used for phase unwrapping directly. By spatially encoding the fringe patterns that were used to perform the phase-shifting technique with binary contrasts, fringe orders could be discerned. For spatially isolated objects or surfaces with large depth discontinuities, unwrapping could be identified without ambiguity. Even though the surface color or reflectivity varied periodically with position, it distinguished the fringe order very well.

Generalized phase unwrapping method that avoids jump errors for fringe projection profilometry

Jump errors easily occur on the discontinuity of the wrapped phase because of the misalignment between wrapped phase and fringe order in fringe projection profilometry (FPP). In this paper, a phase unwrapping method that avoids jump errors is proposed for FPP. By building two other staggered wrapped phases from the original wrapped phase and dividing each period of fringe order into three parts, the proposed generalized tripartite phase unwrapping (Tri-PU) method can be used to avoid rather than compensatorily correct jump errors. It is suitable for the phase unwrapping method assisted by fringe order with a basic wrapped phase and fringe order, no matter which method is used to recover them. The experimental results demonstrate the effectiveness and generality of the proposed method, which is simple to implement and superior to measure complex objects with sharp edges.

Phase Demodulation Method for Fringe Projection Measurement Based on Improved Variable-Frequency Coded Patterns

The phase-to-height imaging model, as a three-dimensional (3D) measurement technology, has been commonly applied in fringe projection to assist surface profile measurement, where the efficient and accurate calculation of phase plays a critical role in precise imaging. To deal with multiple extra coded patterns and 2π jump error caused to the existing absolute phase demodulation methods, a novel method of phase demodulation is proposed based on dual variable-frequency (VF) coded patterns. In this paper, the frequency of coded fringe is defined as the number of coded fringes within a single sinusoidal fringe period. First, the effective wrapped phase (EWP) as calculated using the four-step phase shifting method was split into the wrapped phase region with complete period and the wrapped phase region without complete period. Second, the fringe orders in wrapped phase region with complete period were decoded according to the frequency of the VF coded fringes and the continuous characteristic of the fringe order. Notably, the sampling frequency of fast Fourier transform (FFT) was determined by the length of the decoding interval and can be adjusted automatically with the variation in height of the object. Third, the fringe orders in wrapped phase region without complete period were decoded depending on the consistency of fringe orders in the connected region of wrapped phase. Last, phase demodulation was performed. The experimental results were obtained to confirm the effectiveness of the proposed method in the phase demodulation of both discontinuous objects and highly abrupt objects.

Spatial-temporal phase unwrapping algorithm for fringe projection profilometry

In this paper, a generalized spatial-temporal phase unwrapping algorithm (STPUA) is proposed for extracting the absolute phase of the isolated objects with intricate surfaces. This proposed algorithm can eliminate thoroughly the order jumps of various temporal phase unwrapping algorithms (TPUAs), while inheriting the high measuring accuracy of quality-guided phase unwrapping algorithms (QGPUAs). Differing from the traditional phase unwrapping algorithms, wrapped phase is first divided into several regional wrapped phases, which can be extracted successively according to its areas and unwrapped individually by QGPUAs. Meanwhile, a series of reliable points from the fringe order map obtained from the code deformed patterns are selected to map the corresponding regional unwrapped phases into an absolute phase. The radii of selecting reliable points can provide the high measuring robustness compared with the classical point-to-point TPUAs for the complex surfaces and the motion blur, while keeping the same number of patterns. Therefore, the proposed STPUA combining SPUAs and TPUAs also can be employed in real-time three-dimensional (3D) reconstruction. Theoretical analysis and experimental results are performed to verify the effectiveness and capability of the proposed algorithm.

Color three-dimensional imaging based on patterned illumination using a negative pinhole array

Reflectance confocal microscopy is widely used for non-destructive optical three-dimensional (3D) imaging. In confocal microscopy, a stack of sequential two-dimensional (2D) images with respect to the axial position is typically needed to reconstruct a 3D image. As a result, in conventional confocal microscopy, acquisition speed is often limited by the rate of mechanical scanning in both the transverse and axial directions. We previously reported a high-speed parallel confocal detection method using a pinhole array for color 3D imaging without any mechanical scanners. Here, we report a high-speed color 3D imaging method based on patterned illumination employing a negative pinhole array, whose optical characteristics are the reverse of the conventional pinhole array for transmitting light. The negative pinhole array solves the inherent limitation of a conventional pinhole array, i.e., low transmittance, meaning brighter color images with abundant color information can be acquired. We also propose a 3D image processing algorithm based on the 2D cross-correlation between the acquired image and filtering masks, to produce an axial response. By using four-different filtering masks, we were able to increase the sampling points in calculation of height and enhance the lateral resolution of the color acquisition by a factor of four. The feasibility of high-speed non-contact color 3D measurement with the improved lateral resolution and brightness provided by the negative pinhole array was demonstrated by imaging various specimens. We anticipate that this high-speed color 3D measurement technology with negative pinhole array will be a useful tool in a variety of fields where rapid and accurate non-contact measurement are required, such as industrial inspection and dental scanning.

3-D shape reconstruction of non-uniform reflectance surface based on pixel intensity, pixel color and camera exposure time adaptive adjustment

High dynamic range 3-D shape measurement is a challenge. In this work, we propose a novel method to solve the 3-D shape reconstruction of high-reflection and colored surfaces. First, we propose a method to establish a fast pixel-level mapping between the projected image and the captured image. Secondly, we propose a color texture extraction method using a black-and-white (B/W) camera and a pixel-level projection color adjustment method. Third, we give an optimal projection fringe modulation/background intensity ratio. Fourth, we propose a method for estimating the reflectivity of the object surface and ambient light interference, and a method for adjusting the projection intensity at the pixel level and a method for estimating the optimal exposure time. Experiments show that, compared with the existing methods, the proposed method not only can obtain high-quality captured images, but also has higher measurement efficiency and wider application range.

Absolute phase retrieval for colored objects based on three phase-shifting amount codes

We propose an absolute phase retrieval method based on three phase-shifting amount codes (3-PSA-codes) to measure the colored object with one additional pattern. 3-PSA-codes adopt the coding concept of 3-digit-codes, in which the code elements of three consecutive periods are treated as a unique code word for one period. However, to measure the colored object more effectively in the proposed method, each code element is embedded into the PSA domain and retrieved from the phase difference. Fringe patterns for the wrapped phase are artfully employed in the code element retrieval. Hence, for the first time, to the best of our knowledge, the code element related to the phase can be determined by one additional pattern. It breaks the constraint that temporal methods require multiple additional patterns to overcome the adverse effect of the surface color of objects on absolute phase retrieval. Experimental results demonstrate that the proposed 3-PSA-codes have strong robustness in the measurement of the colored object.

Fringe projection profilometry by conducting deep learning from its digital twin

High-accuracy and high-speed three-dimensional (3D) fringe projection profilometry (FPP) has been widely applied in many fields. Recently, researchers discovered that deep learning can significantly improve fringe analysis. However, deep learning requires numerous objects to be scanned for training data. In this paper, we propose to build the digital twin of an FPP system and perform virtual scanning using computer graphics, which can significantly save cost and labor. The proposed method extracts 3D geometry directly from a single-shot fringe image, and real-world experiments have demonstrated the success of the virtually trained model. Our virtual scanning method can automatically generate 7,200 fringe images and 800 corresponding 3D scenes within 1.5 hours.

Efficient intensity-based fringe projection profilometry method resistant to global illumination

Intensity-based fringe projection profilometry (IBFPP) is used widely because of its simple structure, high robustness, and noise resilience. Most IBFPP methods assume that any scene point is illuminated by direct illumination only, but global illumination effects introduce strong biases in the reconstruction result for many real-world scenes. To solve this problem, this paper describes an efficient IBFPP method for reconstructing three-dimensional geometry in the presence of global illumination. First, the average intensity of two sinusoidal patterns is used as a pixel-wise threshold to binarize the codeword patterns. The binarized template pattern is then used to convert other binarized fringe patterns into traditional Gray-code patterns. A proprietary compensation algorithm is then applied to eliminate fringe errors caused by environmental noise and lens defocusing. Finally, simple, efficient, and robust phase unwrapping can be achieved despite the effects of subsurface scattering and interreflection. Experimental results obtained in different environments show that the proposed method can obtain three-dimensional information reliably when influenced by global illumination.

Encoding technology of an asymmetric combined structured light for 3D measurement

Sinusoidal phase-shifting symmetrically combined with cyclic code is one of the most important encoding methods in the field of 3D measurement. Due to the modulation of the object surface and the influence of the noise of the image acquisition system, the periods of the cyclic code and the sinusoidal phase-shifting in the intensity image do not coincide completely, and they lead to large absolute phase decoding errors near the cycle boundaries, which are called cycle dislocation errors. In order to eliminate these errors in principle, the concept and method of region encoding for four-step sinusoidal phase-shifting are proposed, and the sinusoidal phase-shifting is combined with cyclic code asymmetrically. Under the premise that the cyclic code and the region code change at different times, the cycle dislocation error is reduced from one cycle of cyclic code to one pixel by the dual constraint of cyclic code and region code. The simulation measurement results of 3 ds max and the physical measurement results show that the asymmetric combination encoding method effectively eliminates the cycle dislocation errors; the maximum measurement error is reduced by an order of magnitude, and the root mean square measurement error is reduced by 70%.

Curved light surface model for calibration of a structured light 3D modeling system based on striped patterns

Structured light is an optical 3D surface measurement technique with the merits of high speed and high robustness. However, the huge size of traditional digital light processing (DLP) projectors limits its convenience in numerous applications. In this paper, a one-axis MEMS mirror is used as the structured light projector in 3D modeling systems, and has the advantages of small volume and low cost. Limited by the inability to project orthogonal patterns and projection distortion, it is difficult for the one-axis MEMS mirror based 3D modeling system to obtain high accuracy through existing calibration methods. This paper proposed a calibration method for structured light 3D modeling systems that can only project stripes in one direction with projection distortion. A curved surface equation called curved light surface model was proposed to replace the ideal plane equation as the mathematic model of the projected structured light stripes. Experiment results verified that this method can significantly reduce the effect of projection distortion and an accuracy of 0.11 mm was achieved when measuring a standard dumbbell-shaped object with 201.10 mm center-to-center distance.

State-of-the-art active optical techniques for three-dimensional surface metrology: a review [Invited]

This paper reviews recent developments of non-contact three-dimensional (3D) surface metrology using an active structured optical probe. We focus primarily on those active non-contact 3D surface measurement techniques that could be applicable to the manufacturing industry. We discuss principles of each technology, and its advantageous characteristics as well as limitations. Towards the end, we discuss our perspectives on the current technological challenges in designing and implementing these methods in practical applications.

Real-time 3D shape measurement with dual-frequency composite grating and motion-induced error reduction

Phase-shifting profilometry has been increasingly sought and applied in dynamic three-dimensional (3D) shape measurement. However, the object motion will lead to extra phase shift error and thus measurement error. In this paper, a real-time 3D shape measurement method based on dual-frequency composite phase-shifting grating and motion-induced error reduction is proposed for a complex scene containing dynamic and static objects. The proposed method detects the motion region of a complex scene through the phase relations of the dual-frequency composite grating and reduces the motion-induced error with the combination of the phase calculated by a phase-shifting algorithm and the phase extracted by Fourier fringe analysis. It can correctly reconstruct the 3D shape of a complex dynamic scene and ensure high measurement accuracy of its static object as well. With the aid of the phase-shifting image ordering approach, the dynamic 3D shape of complex scenes can be reconstructed and the motion-induced error can also be suppressed in real time. Experimental results well proved that the proposed method is effective and practical.

Accurate infrared structured light sensing system for dynamic 3D acquisition

3D real-time acquisition plays a vital role in computer graphics and computer vision. In this paper, we present a dynamic IR structured light sensing system with high resolution and accuracy for real-time 3D scanning. We adopt the Gray code combined with stripe shifting as our 3D acquisition's coding strategy and parallelize the algorithm via the GPU in our IR 3D scanning system. Our built-up system can capture dense and high-precision 3D model sequences with a speed of 29 Hz. Furthermore, we propose a practical calibration method to obtain accurate calibration parameters for our system. Finally, various experiments are performed to verify the feasibility and accuracy of our proposed IR structured light sensing system.

Spatial binary coding method for stripe-wise phase unwrapping

Binary coding methods have been widely used for phase unwrapping. However, traditional temporal binary coding methods require a sequence of binary patterns to encode the fringe order information. This paper presents a spatial binary coding (SBC) method that encodes the fringe order into only one binary pattern. Each stripe of the sinusoidal phase-shifting patterns is corresponding to an N-bit codeword of the binary pattern. A robust stripe-wise decoding scheme is also developed to extract the N-bit codeword, then fringe order can be determined, and stripe-wise phase unwrapping can be performed. Experiment results confirm that the SBC method can correctly recover the absolute phase of measured objects with only one additional binary pattern.

Dynamic 3-D measurement based on fringe-to-fringe transformation using deep learning

Fringe projection profilometry (FPP) has become increasingly important in dynamic 3-D shape measurement. In FPP, it is necessary to retrieve the phase of the measured object before shape profiling. However, traditional phase retrieval techniques often require a large number of fringes, which may generate motion-induced error for dynamic objects. In this paper, a novel phase retrieval technique based on deep learning is proposed, which uses an end-to-end deep convolution neural network to transform a single or two fringes into the phase retrieval required fringes. When the object's surface is located in a restricted depth, the presented network only requires a single fringe as the input, which otherwise requires two fringes in an unrestricted depth. The proposed phase retrieval technique is first theoretically analyzed, and then numerically and experimentally verified on its applicability for dynamic 3-D measurement.

Real-Time Dynamic 3D Shape Reconstruction with SWIR InGaAs Camera

In this paper, a real-time, dynamic three-dimensional (3D) shape reconstruction scheme based on the Fourier-transform profilometry (FTP) method is achieved with a short-wave infrared (SWIR) indium gallium arsenide (InGaAs) camera for monitoring applications in low illumination environments. A SWIR 3D shape reconstruction system is built for generating and acquiring the SWIR two-dimensional (2D) fringe pattern of the target. The depth information of the target is reconstructed by employing an improved FTP method, which has the advantages of high reconstruction accuracy and speed. The maximum error in depth for static 3D shape reconstruction is 1.15 mm for a plastic model with a maximum depth of 36 mm. Meanwhile, a real-time 3D shape reconstruction with a frame rate of 25 Hz can be realized by this system, which has great application prospects in real-time dynamic 3D shape reconstruction, such as low illumination monitoring. In addition, for real-time dynamic 3D shape reconstruction, without considering the edge areas, the maximum error in depth among all frames is 1.42 mm for a hemisphere with a depth of 35 mm, and the maximum error of the average of all frames in depth is 0.52 mm.

Improved unwrapped phase retrieval method of a fringe projection profilometry system based on fewer phase-coding patterns

In this paper, based on two additional phase-coding patterns, an improved phase demodulation method is proposed. First, six equally spaced coding phases in the interval [$ - \pi $-π, $\pi $π] are embedded in different periods of the coded fringes following a certain sequence. Subsequently, since a group of phase orders can be uniquely determined by the four adjacent coding phases, the phase-order map of the object can be generated. To ensure the accuracy of decoding results, the interference coding numbers should be corrected in advance. In the meantime, the connected regions exhibiting the same orders are classified and then labeled for simplifying the decoding process. The simulation results verify the feasibility of the proposed method. By two groups of 3D imaging experiments, the applicability of this method to multiple objects and discontinuous objects is confirmed.

Curvature and height corrections of hyperspectral images using built-in 3D laser profilometry

Optical imaging systems use a light source that illuminates a sample and a photodetector that detects light reflected from or transmitted through the sample. The sample surface curvature, surface-to-camera distance, and illumination-source-to-surface distance significantly affect the measured signal, resulting in image artifacts. To correct the images, a three-dimensional (3D) profilometry system was used to obtain 3D surface information. The 3D information enables image correction using Lambert cosine law and height correction. In this study, the feasibility of the correction method for push-broom hyperspectral imaging of three different objects is presented. Results show a significant reduction of image artifacts, making further image analysis more accurate and robust. The presented 3D profilometry method is applicable to all push-broom imaging systems and the described correction procedure can be applied to all spectral or monochromatic imaging systems.

High-speed three-dimensional shape measurement based on shifting Gray-code light

The measuring technique combining a phase-shifting algorithm and Gray-code light has been widely used in three-dimensional (3D) shape measurement for static scenes because of its high robustness and anti-noise ability. However, in the high-speed measurement, phase unwrapping errors occur easily on the boundaries of adjacent Gray-code words because of the defocus of the projector, the motion of the objects and the non-uniform reflectivity of the surface. To overcome this challenge, a high-speed 3D shape measurement method based on shifting Gray-code light has been proposed in this paper. Firstly, the average intensity of three captured phase-shifting fringe images are used as a pixel-wise threshold to binarize the Gray codes and to eliminate most phase unwrapping errors caused by the non-uniform reflectivity, ambient light variations, and the defocus of projector. Then, the shifting Gray-code (SGC) coding strategy is proposed to avoid the remaining errors of phase unwrapping on the edge of the code words. In this strategy, no additional patterns are projected, and two sets of decoding words with staggered boundaries are built in the temporal sequences for one wrapped phase. Finally, the simple, efficient, and robust phase unwrapping can be achieved in the high-speed dynamic measurement. This proposed method has been applied to reconstruct 3D shape of randomly collapsing objects in a large depth range, and the experimental results demonstrate that it can reliably obtain high-quality shape and texture information at 310 frames per second.

Robust structured-light depth mapping via recursive decomposition of binary codes

Structured-light depth cameras rely on projecting and resolving coded patterns on a three-dimensional scene with high contrast. The front-end optics of such depth cameras impose a fundamental restriction on the depth-sensing range and accuracy: the patterns only remain sharp within the depth of field jointly determined by the camera and projector. We present here a robust method to improve the depth-sensing range and accuracy for a structured-light depth camera without changing the underlying optical design. Moreover, it shows the unique advantage in macrophotography of highly light-scattering objects. We analyze the proposed method theoretically and validate it in experiments.

Pixel-by-pixel absolute phase retrieval assisted by an additional three-dimensional scanner

This paper presents a novel absolute phase unwrapping method assisted by a low-cost three-dimensional (3D) scanner. The proposed absolute phase unwrapping method leverages a low-cost 3D scanner to capture rough 3D data of the scene and transforms the rough 3D data to the world coordinate system to generate an artificial reference phase map ϕ_ref. By referring to ϕ_ref, we can do absolute phase unwrapping directly without projecting any additional patterns, such that the digital fringe projection (DFP) system can achieve higher measurement speed. We develop a multiresolution system consisting of a DFP system and a Kinect V2 to validate our method. Experiments demonstrate that our method works for a large depth range, and the speed of the low-cost 3D scanner is not necessarily the maximum speed of our proposed method. Assisted by the Kinect V2, whose maximum speed is only 30 Hz, our DFP system achieves 53 Hz with a resolution 1600×1000 pixels when we measure dynamic objects that are moving in a large depth range of 400 mm.

Adaptive Structured Light with Scatter Correction for High-Precision Underwater 3D Measurements

High-precision underwater 3D cameras are required to automate many of the traditional subsea inspection, maintenance and repair (IMR) operations. In this paper we introduce a novel multi-frequency phase stepping (structured light) method for high-precision 3D estimation even in turbid water. We introduce an adaptive phase-unwrapping procedure which uses the phase-uncertainty to determine the highest frequency that can be reliably unwrapped. Light scattering adversely affects the phase estimate. We propose to remove the effect of forward scatter with an unsharp filter and a model-based method to remove the backscatter effect. Tests in varying turbidity show that the scatter correction removes the adverse effect of scatter on the phase estimates. The adaptive frequency unwrapping with scatter correction results in images with higher accuracy and precision and less phase unwrap errors than the Gray-Code Phase Stepping (GCPS) approach.

Motion-artifact-free dynamic 3D shape measurement with hybrid Fourier-transform phase-shifting profilometry

Fourier-transform profilometry (FTP) and phase-shifting profilometry (PSP) are two mainstream fringe projection techniques widely used for three-dimensional (3D) shape measurement. The former is well known for its single-shot nature and the latter for its higher measurement resolution and precision. However, when it comes to measuring the dynamic objects, neither approach is able to produce high-resolution, high-accuracy measurement results that are free from any depth ambiguities and motion-related artifacts. Furthermore, for scenes consisting of both static and dynamic objects, a trade-off between measurement precision and efficiency has to be made, suggesting that using a single approach can yield only suboptimal results. To this end, we propose a novel hybrid Fourier-transform phase-shifting profilometry method to integrate the advantages of both approaches. The motion vulnerability of multi-shot PSP can be overcome, or at least significantly alleviated, through the combination of single-shot FTP, while the high accuracy of PSP can also be preserved when the object is motionless. We design a phase-based, pixel-wise motion detection strategy that can accurately outline the moving object regions from their motionless counterparts. The final measurement result is obtained by fusing the determined regions where the PSP or FTP is applied correspondingly. To validate the proposed hybrid approach, we develop a real-time 3D shape measurement system for measuring multiple isolated moving objects. Experimental results demonstrate that our method achieves significantly higher precision and better robustness compared with conventional approaches where PSP or FTP is applied separately.

Three-dimensional measurement based on optimized circular fringe projection technique

Circular fringe projection profilometry (CFPP) is a recently proposed optical three-dimensional (3D) measurement technique. Theoretically, its optimal measurement accuracy should precede that of the conventional fringe projection profilometry. In practice, the measurement accuracy is impacted by many factors, and much research remains to be done in order to make CFPP reach its optimal precision. One of the dominant factors is the zero-phase point. For the usage of the cotangent function, error near the zero-phase point will be significantly amplified. This makes the overall measurement accuracy very low for CFPP with coaxial layout. To address this critical issue, CFPP with off-axis layout (called OCFPP for simplicity) is presented in this paper. The core theory of OCFPP is briefly introduced. The zero-phase point detection problem coming with OCFPP is explained. Then, two methods, one based on a two-dimensional ruler and the other based on plane constraint, are proposed to solve this additional problem. Simulation and experiments validate the effectiveness of the proposed zero-phase point detection methods, and convince the advantage of OCFPP. This paper contributes to distinctly improving the 3D measurement capability of CFPP, and lays an indispensible foundation for its practical application.

High-speed 3D shape measurement using the optimized composite fringe patterns and stereo-assisted structured light system

In this paper, we propose a high-speed 3D shape measurement technique based on the optimized composite fringe patterns and stereo-assisted structured light system. Stereo phase unwrapping, as a new-fashioned method for absolute phase retrieval based on the multi-view geometric constraints, can eliminate the phase ambiguities and obtain a continuous phase map without projecting any additional patterns. However, in order to ensure the stability of phase unwrapping, the period of fringe is generally around 20, which limits the accuracy of 3D measurement. To solve this problem, we develop an optimized method for designing the composite pattern, in which the speckle pattern is embedded into the conventional 4-step phase-shifting fringe patterns without compromising the fringe modulation, and thus the phase measurement accuracy. We also present a simple and effective evaluation criterion for the correlation quality of the designed speckle pattern in order to improve the matching accuracy significantly. When the embedded speckle pattern is demodulated, the periodic ambiguities in the wrapped phase can be eliminated by combining the adaptive window image correlation with geometry constraint. Finally, some mismatched regions are further corrected based on the proposed regional diffusion compensation technique (RDC). These proposed techniques constitute a complete computational framework that allows to effectively recover an accurate, unambiguous, and distortion-free 3D point cloud with only 4 projected patterns. Experimental results verify that our method can achieve high-speed, high-accuracy, robust 3D shape measurement with dense (64-period) fringe patterns at 5000 frames per second.

Absolute Phase Retrieval Using One Coded Pattern and Geometric Constraints of Fringe Projection System

Fringe projection technologies have been widely used for three-dimensional (3D) shape measurement. One of the critical issues is absolute phase recovery, especially for measuring multiple isolated objects. This paper proposes a method for absolute phase retrieval using only one coded pattern. A total of four patterns including one coded pattern and three phase-shift patterns are projected, captured, and processed. The wrapped phase, as well as average intensity and intensity modulation, are calculated from three phase-shift patterns. A code word encrypted into the coded pattern can be calculated using the average intensity and intensity modulation. Based on geometric constraints of fringe projection system, the minimum fringe order map can be created, upon which the fringe order can be calculated from the code word. Compared with the conventional method, the measurement depth range is significantly improved. Finally, the wrapped phase can be unwrapped for absolute phase map. Since only four patterns are required, the proposed method is suitable for real-time measurement. Simulations and experiments have been conducted, and their results have verified the proposed method.

Light-field-based absolute phase unwrapping

Ambiguity caused by a wrapped phase is an intrinsic problem in fringe projection-based 3D shape measurement. Among traditional methods for avoiding phase ambiguity, spatial phase unwrapping is sensitive to sensor noise and depth discontinuity, and temporal phase unwrapping requires additional encoding information that leads to an increase of image sequence acquisition time or a reduction of fringe contrast. Here, to the best of our knowledge, we report a novel method of absolute phase unwrapping based on light field imaging. In a recorded light field under structured illumination, i.e., a structured light field, a wrapped phase-encoded field can be retrieved and resampled in diverse image planes associated with several possible fringe orders in a measurement volume. Then, by leveraging phase consistency constraint in the resampled wrapped phase-encoded field, correct fringe orders can be determined to unwrap the wrapped phase without any additional encoding information. Experimental results demonstrated that the proposed method was suitable for accurate and robust absolute phase unwrapping.

A simple and practical jump error removal method for fringe projection profilometry based on self-alignment technique

The code-based method is one of the frequently adopted fringe projection profilometry techniques because of its robustness and high speed. However, the abnormal jump errors caused by the misalignment between the wrapped phase and the fringe order impact the phase unwrapping quality and are more serious in binary defocusing measurement with significant defocusing. This paper proposes a self-alignment technique (SAT) with high speed, no additional patterns, and no accuracy loss to eliminate such jump errors. After analyzing the relation between the grating changes and the misalignment, we designed an assist-code that can perfectly align with the wrapped phase even in significant defocusing. After that, this assist-code is used to adjust the misalignment and eliminate the jump errors. The comparison between the median filter and the proposed SAT method in simulations and experiments demonstrates that the proposed SAT method has the ability to completely remove jump errors for complex textures and step-height objects, even in significant defocusing, indicating its potential to be applied to other phase unwrapping fields, e.g., color-based methods with color coupling problems.

Reference image based phase unwrapping framework for a structured light system

A novel real-time full-field phase unwrapping framework is proposed for the one-projector and one-camera structured light system. In this framework, only four patterns (including three fringe patterns and a binary speckle pattern) are required to measure the absolute 3D shape of the targets. We use the structured light system to capture four images of a nearly planar target (e.g. wall), of which the speckle image is taken as the reference image, and the corresponding absolute phase map is computed and stored, before measuring. So each pixel in the reference image can be mapped to an absolute phase value. In this way, if we can create the correspondences between the current and the reference speckle images in the process of measurement by using a matching algorithm, we can directly map the absolute values for the pixels of the current image. The mapped absolute phases can be used to determine the period of the relative phases. The experimental results verified the effectiveness and efficiency of the proposed framework. On a consumer-grade GPU (Nvidia GTX1060), our method can run at 187 fps.

GOBO projection for 3D measurements at highest frame rates: a performance analysis

Aperiodic sinusoidal patterns that are cast by a GOBO (GOes Before Optics) projector are a powerful tool for optically measuring the surface topography of moving or deforming objects with very high speed and accuracy. We optimised the first experimental setup that we were able to measure inflating car airbags at frame rates of more than 50 kHz while achieving a 3D point standard deviation of ~500 µm. Here, we theoretically investigate the method of GOBO projection of aperiodic sinusoidal fringes. In a simulation-based performance analysis, we examine the parameters that influence the accuracy of the measurement result and identify an optimal pattern design that yields the highest measurement accuracy. We compare the results with those that were obtained via GOBO projection of phase-shifted sinusoidal fringes. Finally, we experimentally verify the theoretical findings. We show that the proposed technique has several advantages over conventional fringe projection techniques, as the easy-to-build and cost-effective GOBO projector can provide a high radiant flux, allows high frame rates, and can be used over a wide spectral range.

High-speed real-time 3D shape measurement based on adaptive depth constraint

Stereo phase unwrapping (SPU) has been increasingly applied to high-speed real-time fringe projection profilometry (FPP) because it can retrieve the absolute phase or matching points in a stereo FPP system without projecting or acquiring additional fringe patterns. Based on a pre-defined measurement volume, artificial maximum/minimum phase maps can be created solely using geometric constraints of the FPP system, permitting phase unwrapping on a pixel-by-pixel basis. However, when high-frequency fringes are used, the phase ambiguities will increase which makes SPU unreliable. Several auxiliary techniques have been proposed to enhance the robustness of SPU, but their flexibility still needs to be improved. In this paper, we proposed an adaptive depth constraint (ADC) approach for high-speed real-time 3D shape measurement, where the measurement depth volume for geometric constraints is adaptively updated according to the current reconstructed geometry. By utilizing the spatio-temporal correlation of moving objects under measurement, a customized and tighter depth constraint can be defined, which helps enhance the robustness of SPU over a large measurement volume. Besides, two complementary techniques, including simplified left-right consistency check and feedback mechanism based on valid area, are introduced to further increase the robustness and flexibility of the ADC. Experimental results demonstrate the success of our proposed SPU approach in recovering absolute 3D geometries of both simple and complicated objects with only three phase-shifted fringe images.

Reference-plane-based fast pixel-by-pixel absolute phase retrieval for height measurement

Absolute phase retrieval is essential for height measurement in digital fringe projection. However, projections of additional structured patterns that are normally required for phase unwrapping increase the measurement complexity. In this paper, we propose two reference-plane-based pixel-by-pixel absolute phase retrieval techniques with as few projections as possible, suitable for different object depth ranges. The wrapped phase on the object is absolutely unwrapped by referring just to the absolute phase map on the reference plane. Single-frequency absolute phase retrieval with one-reference-plane-based calibration is first proposed for objects within a height limit that equals a calibrated system constant. To extend the measurement depth range, dual-frequency absolute phase retrieval with two parallel reference planes is further proposed. The additional low frequency is used to choose the unwrapping reference from the two reference plane phases for unwrapping the high-frequency phase. Moreover, the proposed techniques are capable of high-frequency absolute phase unwrapping for objects with step-height surface discontinuities. Experiments have been conducted to demonstrate the efficiency of the proposed two techniques.

Local blur analysis and phase error correction method for fringe projection profilometry systems

We introduce a flexible error correction method for fringe projection profilometry (FPP) systems in the presence of local blur phenomenon. Local blur caused by global light transport such as camera defocus, projector defocus, and subsurface scattering will cause significant systematic errors in FPP systems. Previous methods, which adopt high-frequency patterns to separate the direct and global components, fail when the global light phenomenon occurs locally. In this paper, the influence of local blur on phase quality is thoroughly analyzed, and a concise error correction method is proposed to compensate the phase errors. For defocus phenomenon, this method can be directly applied. With the aid of spatially varying point spread functions and local frontal plane assumption, experiments show that the proposed method can effectively alleviate the system errors and improve the final reconstruction accuracy in various scenes. For a subsurface scattering scenario, if the translucent object is dominated by multiple scattering, the proposed method can also be applied to correct systematic errors once the bidirectional scattering-surface reflectance distribution function of the object material is measured.