Automated Defect Detection From Ultrasonic Images Using Deep Learning

Leveraging Segment Anything Model (SAM) for Weld Defect Detection in Industrial Ultrasonic B-Scan Images

Automated ultrasonic testing (AUT) is a critical tool for infrastructure evaluation in industries such as oil and gas, and, while skilled operators manually analyze complex AUT data, artificial intelligence (AI)-based methods show promise for automating interpretation. However, improving the reliability and effectiveness of these methods remains a significant challenge. This study employs the Segment Anything Model (SAM), a vision foundation model, to design an AI-assisted tool for weld defect detection in real-world ultrasonic B-scan images. It utilizes a proprietary dataset of B-scan images generated from AUT data collected during automated girth weld inspections of oil and gas pipelines, detecting a specific defect type: lack of fusion (LOF). The implementation includes integrating knowledge from the B-scan image context into the natural image-based SAM 1 and SAM 2 through a fully automated, promptable process. As part of designing a practical AI-assistant tool, the experiments involve applying both vanilla and low-rank adaptation (LoRA) fine-tuning techniques to the image encoder and mask decoder of different variants of both models, while keeping the prompt encoder unchanged. The results demonstrate that the utilized method achieves improved performance compared to a previous study on the same dataset.

Application of Object Detection Algorithms in Non-Destructive Testing of Pressure Equipment: A Review

Non-destructive testing (NDT) techniques play a crucial role in industrial production, aerospace, healthcare, and the inspection of special equipment, serving as an indispensable part of assessing the safety condition of pressure equipment. Among these, the analysis of NDT data stands as a critical link in evaluating equipment safety. In recent years, object detection techniques have gradually been applied to the analysis of NDT data in pressure equipment inspection, yielding significant results. This paper comprehensively reviews the current applications and development trends of object detection algorithms in NDT technology for pressure-bearing equipment, focusing on algorithm selection, data augmentation, and intelligent defect recognition based on object detection algorithms. Additionally, it explores open research challenges of integrating GAN-based data augmentation and unsupervised learning to further enhance the intelligent application and performance of object detection technology in NDT for pressure-bearing equipment while discussing techniques and methods to improve the interpretability of deep learning models. Finally, by summarizing current research and offering insights for future directions, this paper aims to provide researchers and engineers with a comprehensive perspective to advance the application and development of object detection technology in NDT for pressure-bearing equipment.

Advancing Carbon Fiber Composite Inspection: Deep Learning-Enabled Defect Localization and Sizing via 3-D U-Net Segmentation of Ultrasonic Data

In nondestructive evaluation (NDE), accurately characterizing defects within components relies on accurate sizing and localization to evaluate the severity or criticality of defects. This study presents for the first time a deep learning (DL) methodology using 3-D U-Net to localize and size defects in carbon fiber reinforced polymer (CFRP) composites through volumetric segmentation of ultrasonic testing (UT) data. Using a previously developed approach, synthetic training data, closely representative of experimental data, was used for the automatic generation of ground truth segmentation masks. The model's performance was compared to the conventional amplitude 6 dB drop analysis method used in the industry against ultrasonic defect responses from 40 defects fabricated in CFRP components. The results showed good agreement with the 6 dB drop method for in-plane localization and excellent through-thickness localization, with mean absolute errors (MAEs) of 0.57 and 0.08 mm, respectively. Initial sizing results consistently oversized defects with a 55% higher mean average error than the 6 dB drop method. However, when a correction factor was applied to account for variation between the experimental and synthetic domains, the final sizing accuracy resulted in a 35% reduction in MAE compared to the 6 dB drop technique. By working with volumetric ultrasonic data (as opposed to 2-D images), this approach reduces preprocessing (such as signal gating) and allows for the generation of 3-D defect masks which can be used for the generation of computer-aided design files; greatly reducing the qualification reporting burden of NDE operators.

Utilizing improved YOLOv8 based on SPD-BRSA-AFPN for ultrasonic phased array non-destructive testing

Non-destructive testing (NDT) is a technique for inspecting materials and their defects without causing damage to the tested components. Phased array ultrasonic testing (PAUT) has emerged as a hot topic in industrial NDT applications. Currently, the collection of ultrasound data is mostly automated, while the analysis of the data is still predominantly carried out manually. Manual analysis of scan image defects is inefficient and prone to instability, prompting the need for computer-based solutions. Deep learning-based object detection methods have shown promise in addressing such challenges recently. This approach typically demands a substantial amount of high-resolution, well-annotated training data, which is challenging to obtain in NDT. Consequently, it becomes difficult to detect low-resolution images and defects with varying positional sizes. This work proposes improvements based on the state-of-the-art YOLOv8 algorithm to enhance the accuracy and efficiency of defect detection in phased-array ultrasonic testing. The space-to-depth convolution (SPD-Conv) is imported to replace strided convolution, mitigating information loss during convolution operations and improving detection performance on low-resolution images. Additionally, this paper constructs and incorporates the bi-level routing and spatial attention module (BRSA) into the backbone, generating multiscale feature maps with richer details. In the neck section, the original structure is replaced by the asymptotic feature pyramid network (AFPN) to reduce model parameters and computational complexity. After testing on public datasets, in comparison to YOLOv8 (the baseline), this algorithm achieves high-quality detection of flat bottom holes (FBH) and aluminium blocks on the simulated dataset. More importantly, for the challenging-to-detect defect side-drilled holes (SDH), it achieves F1 scores (weighted average of precision and recall) of 82.50% and intersection over union (IOU) of 65.96%, representing an improvement of 17.56% and 0.43%. On the experimental dataset, the F1 score and IOU for FBH reach 75.68% (an increase of 9.01%) and 83.79%, respectively. Simultaneously, the proposed algorithm demonstrates robust performance in the presence of external noise, while maintaining exceptionally high computational efficiency and inference speed. These experimental results validate the high detection performance of the proposed intelligent defect detection algorithm for ultrasonic images, which contributes to the advancement of the smart industry.

An Adaptive Image Segmentation Network for Surface Defect Detection

Surface defect detection plays an essential role in industry, and it is challenging due to the following problems: 1) the similarity between defect and nondefect texture is very high, which eventually leads to recognition or classification errors and 2) the size of defects is tiny, which are much more difficult to be detected than larger ones. To address such problems, this article proposes an adaptive image segmentation network (AIS-Net) for pixelwise segmentation of surface defects. It consists of three main parts: multishuffle-block dilated convolution (MSDC), dual attention context guidance (DACG), and adaptive category prediction (ACP) modules, where MSDC is designed to merge the multiscale defect features for avoiding the loss of tiny defect feature caused by model depth, DACG is designed to capture more contextual information from the defect feature map for locating defect regions and obtaining clear segmentation boundaries, and ACP is used to make classification and regression for predicting defect categories. Experimental results show that the proposed AIS-Net is superior to the state-of-the-art approaches on four actual surface defect datasets (NEU-DET: 98.38% ± 0.03%, DAGM: 99.25% ± 0.02%, Magnetic-tile: 98.73% ± 0.13%, and MVTec: 99.72% ± 0.02%).

Systematic Evaluation of Ultrasonic In-Line Inspection Techniques for Oil and Gas Pipeline Defects Based on Bibliometric Analysis

The global reliance on oil and gas pipelines for energy transportation is increasing. As the pioneering review in the field of ultrasonic defect detection for oil and gas pipelines based on bibliometric methods, this study employs visual analysis to identify the most influential countries, academic institutions, and journals in this domain. Through cluster analysis, it determines the primary trends, research hotspots, and future directions in this critical field. Starting from the current global industrial ultrasonic in-line inspection (ILI) detection level, this paper provides a flowchart for selecting detection methods and a table for defect comparison, detailing the comparative performance limits of different detection devices. It offers a comprehensive perspective on the latest ultrasonic pipeline detection technology from laboratory experiments to industrial practice.

Online evolution of a phased array for ultrasonic imaging by a novel adaptive data acquisition method

Ultrasonic imaging, using ultrasonic phased arrays, has an enormous impact in science, medicine and society and is a widely used modality in many application fields. The maximum amount of information which can be captured by an array is provided by the data acquisition method capturing the complete data set of signals from all possible combinations of ultrasonic generation and detection elements of a dense array. However, capturing this complete data set requires long data acquisition time, large number of array elements and transmit channels and produces a large volume of data. All these reasons make such data acquisition unfeasible due to the existing phased array technology or non-applicable to cases requiring fast measurement time. This paper introduces the concept of an adaptive data acquisition process, the Selective Matrix Capture (SMC), which can adapt, dynamically, to specific imaging requirements for efficient ultrasonic imaging. SMC is realised experimentally using Laser Induced Phased Arrays (LIPAs), that use lasers to generate and detect ultrasound. The flexibility and reconfigurability of LIPAs enable the evolution of the array configuration, on-the-fly. The SMC methodology consists of two stages: a stage for detecting and localising regions of interest, by means of iteratively synthesising a sparse array, and a second stage for array optimisation to the region of interest. The delay-and-sum is used as the imaging algorithm and the experimental results are compared to images produced using the complete generation-detection data set. It is shown that SMC, without a priori knowledge of the test sample, is able to achieve comparable results, while preforming ∼ 10 times faster data acquisition and achieving ∼ 10 times reduction in data size.

Ultrasound image super-resolution reconstruction based on semi-supervised CycleGAN

In ultrasonic testing, diffraction artifacts generated around defects increase the challenge of quantitatively characterizing defects. In this paper, we propose a label-enhanced semi-supervised CycleGAN network model, referred to as LESS-CycleGAN, which is a conditional cycle generative adversarial network designed for accurately characterizing defect morphology in ultrasonic testing images. The proposed method introduces paired cross-domain image samples during model training to achieve a defect transformation between the ultrasound image domain and the morphology image domain, thereby eliminating artifacts. Furthermore, the method incorporates a novel authenticity loss function to ensure high-precision defect reconstruction capability. To validate the effectiveness and robustness of the model, we use simulated 2D images of defects and corresponding ultrasonic detection images as training and test sets, and an actual ultrasonic phased array image of a test block as the validation set to evaluate the model's application performance. The experimental results demonstrate that the proposed method is convenient and effective, achieving subwavelength-scale defect reconstruction with good robustness.

Fetal Ultrasound Segmentation and Measurements Using Appearance and Shape Prior Based Density Regression with Deep CNN and Robust Ellipse Fitting

Accurately segmenting the structure of the fetal head (FH) and performing biometry measurements, including head circumference (HC) estimation, stands as a vital requirement for addressing abnormal fetal growth during pregnancy under the expertise of experienced radiologists using ultrasound (US) images. However, accurate segmentation and measurement is a challenging task due to image artifact, incomplete ellipse fitting, and fluctuations due to FH dimensions over different trimesters. Also, it is highly time-consuming due to the absence of specialized features, which leads to low segmentation accuracy. To address these challenging tasks, we propose an automatic density regression approach to incorporate appearance and shape priors into the deep learning-based network model (DR-ASPnet) with robust ellipse fitting using fetal US images. Initially, we employed multiple pre-processing steps to remove unwanted distortions, variable fluctuations, and a clear view of significant features from the US images. Then some form of augmentation operation is applied to increase the diversity of the dataset. Next, we proposed the hierarchical density regression deep convolutional neural network (HDR-DCNN) model, which involves three network models to determine the complex location of FH for accurate segmentation during the training and testing processes. Then, we used post-processing operations using contrast enhancement filtering with a morphological operation model to smooth the region and remove unnecessary artifacts from the segmentation results. After post-processing, we applied the smoothed segmented result to the robust ellipse fitting-based least square (REFLS) method for HC estimation. Experimental results of the DR-ASPnet model obtain 98.86% dice similarity coefficient (DSC) as segmentation accuracy, and it also obtains 1.67 mm absolute distance (AD) as measurement accuracy compared to other state-of-the-art methods. Finally, we achieved a 0.99 correlation coefficient (CC) in estimating the measured and predicted HC values on the HC18 dataset.

A multiscale residual U-net architecture for super-resolution ultrasonic phased array imaging from full matrix capture data

Ultrasonic phased array imaging using full-matrix capture (FMC) has raised great interest among various communities, including the nondestructive testing community, as it makes full use of the echo space to provide preferable visualization performance of inhomogeneities. The conventional way of FMC data postprocessing for imaging is through beamforming approaches, such as delay-and-sum, which suffers from limited imaging resolution and contrast-to-noise ratio. To tackle these difficulties, we propose a deep learning (DL)-based image forming approach, termed FMC-Net, to reconstruct high-quality ultrasonic images directly from FMC data. Benefitting from the remarkable capability of DL to approximate nonlinear mapping, the developed FMC-Net automatically models the underlying nonlinear wave-matter interactions; thus, it is trained end-to-end to link the FMC data to the spatial distribution of the acoustic scattering coefficient of the inspected object. Specifically, the FMC-Net is an encoder-decoder architecture composed of multiscale residual modules that make local perception at different scales for the transmitter-receiver pair combinations in the FMC data. We numerically and experimentally compared the DL imaging results to the total focusing method and wavenumber algorithm and demonstrated that the proposed FMC-Net remarkably outperforms conventional methods in terms of exceeding resolution limit and visualizing subwavelength defects. It is expected that the proposed DL approach can benefit a variety of ultrasonic array imaging applications.

Performance Comparison of Object Detection Networks for Shrapnel Identification in Ultrasound Images

Ultrasound imaging is a critical tool for triaging and diagnosing subjects but only if images can be properly interpreted. Unfortunately, in remote or military medicine situations, the expertise to interpret images can be lacking. Machine-learning image interpretation models that are explainable to the end user and deployable in real time with ultrasound equipment have the potential to solve this problem. We have previously shown how a YOLOv3 (You Only Look Once) object detection algorithm can be used for tracking shrapnel, artery, vein, and nerve fiber bundle features in a tissue phantom. However, real-time implementation of an object detection model requires optimizing model inference time. Here, we compare the performance of five different object detection deep-learning models with varying architectures and trainable parameters to determine which model is most suitable for this shrapnel-tracking ultrasound image application. We used a dataset of more than 16,000 ultrasound images from gelatin tissue phantoms containing artery, vein, nerve fiber, and shrapnel features for training and evaluating each model. Every object detection model surpassed 0.85 mean average precision except for the detection transformer model. Overall, the YOLOv7tiny model had the higher mean average precision and quickest inference time, making it the obvious model choice for this ultrasound imaging application. Other object detection models were overfitting the data as was determined by lower testing performance compared with higher training performance. In summary, the YOLOv7tiny object detection model had the best mean average precision and inference time and was selected as optimal for this application. Next steps will implement this object detection algorithm for real-time applications, an important next step in translating AI models for emergency and military medicine.

Deep learning-based anomaly detection from ultrasonic images

Non-destructive testing is a group of methods for evaluating the integrity of components. Among them, ultrasonic inspection stands out due to its ability to visualize both shallow and deep sections of the material in the search for flaws. Testing of the critical components can be a tiring and time-consuming task. Therefore, human experts in analyzing inspection data could use a hand in discarding anomaly-free data and reviewing only suspicious data. Using such a tool, errors would be less common, inspection times would shorten and non-destructive testing would be more efficient. In this work, we evaluate multiple state-of-the-art deep-learning anomaly detection methods on the ultrasonic non-destructive testing dataset. We achieved an average performance of almost 82% of ROC AUC. We discuss in detail the advantages and disadvantages of the presented methods.

A review of ultrasonic sensing and machine learning methods to monitor industrial processes

Supervised machine learning techniques are increasingly being combined with ultrasonic sensor measurements owing to their strong performance. These techniques also offer advantages over calibration procedures of more complex fitting, improved generalisation, reduced development time, ability for continuous retraining, and the correlation of sensor data to important process information. However, their implementation requires expertise to extract and select appropriate features from the sensor measurements as model inputs, select the type of machine learning algorithm to use, and find a suitable set of model hyperparameters. The aim of this article is to facilitate implementation of machine learning techniques in combination with ultrasonic measurements for in-line and on-line monitoring of industrial processes and other similar applications. The article first reviews the use of ultrasonic sensors for monitoring processes, before reviewing the combination of ultrasonic measurements and machine learning. We include literature from other sectors such as structural health monitoring. This review covers feature extraction, feature selection, algorithm choice, hyperparameter selection, data augmentation, domain adaptation, semi-supervised learning and machine learning interpretability. Finally, recommendations for applying machine learning to the reviewed processes are made.

Uncertainty Quantification for Deep Learning in Ultrasonic Crack Characterization

Deep learning for nondestructive evaluation (NDE) has received a lot of attention in recent years for its potential ability to provide human level data analysis. However, little research into quantifying the uncertainty of its predictions has been done. Uncertainty quantification (UQ) is essential for qualifying NDE inspections and building trust in their predictions. Therefore, this article aims to demonstrate how UQ can best be achieved for deep learning in the context of crack sizing for inline pipe inspection. A convolutional neural network architecture is used to size surface breaking defects from plane wave imaging (PWI) images with two modern UQ methods: deep ensembles and Monte Carlo dropout. The network is trained using PWI images of surface breaking defects simulated with a hybrid finite element / ray-based model. Successful UQ is judged by calibration and anomaly detection, which refer to whether in-domain model error is proportional to uncertainty and if out of training domain data is assigned high uncertainty. Calibration is tested using simulated and experimental images of surface breaking cracks, while anomaly detection is tested using experimental side-drilled holes and simulated embedded cracks. Monte Carlo dropout demonstrates poor uncertainty quantification with little separation between in and out-of-distribution data and a weak linear fit ( R=0.84 ) between experimental root-mean-square-error and uncertainty. Deep ensembles improve upon Monte Carlo dropout in both calibration ( R=0.95 ) and anomaly detection. Adding spectral normalization and residual connections to deep ensembles slightly improves calibration ( R=0.98 ) and significantly improves the reliability of assigning high uncertainty to out-of-distribution samples.

Generating ultrasonic images indistinguishable from real images using Generative Adversarial Networks

Ultrasonic imaging is widely used for non-destructive evaluation in various industry applications. Early detection of defects in materials is the key to keeping the integrity of inspected structures. Currently, there have been some attempts to develop models for automated defect detection on ultrasonic data. To push the performance of these models even further more data is needed to train deep convolutional neural networks. A lot of data is also needed for training human experts. However, gathering a sufficient amount of data for training is a challenge due to the rare occurrence of defects in real inspection scenarios. This is why inspection results heavily depend on the inspector's previous experience. To overcome these challenges, we propose the use of Generative Adversarial Networks for generating realistic ultrasonic images. To the best of our knowledge, this work is the first one to show that a Generative Adversarial Network is able to generate images indistinguishable from real ultrasonic images. The most thorough statistical quality analysis to date of generated ultrasonic images has been conducted with the participation of human expert inspectors. The experimental results show that images generated using our Generative Adversarial Network provide the highest quality images compared to other published methods.

Crack Detection in Images of Masonry Using CNNs

While there is a significant body of research on crack detection by computer vision methods in concrete and asphalt, less attention has been given to masonry. We train a convolutional neural network (CNN) on images of brick walls built in a laboratory environment and test its ability to detect cracks in images of brick-and-mortar structures both in the laboratory and on real-world images taken from the internet. We also compare the performance of the CNN to a variety of simpler classifiers operating on handcrafted features. We find that the CNN performed better on the domain adaptation from laboratory to real-world images than these simple models. However, we also find that performance is significantly better in performing the reverse domain adaptation task, where the simple classifiers are trained on real-world images and tested on the laboratory images. This work demonstrates the ability to detect cracks in images of masonry using a variety of machine learning methods and provides guidance for improving the reliability of such models when performing domain adaptation for crack detection in masonry.

An Improved Gesture Segmentation Method for Gesture Recognition Based on CNN and YCbCr

With the continuous improvement of people’s requirements for interactive experience, gesture recognition is widely used as a basic human-computer interaction. However, due to the environment, light source, cover, and other factors, the diversity and complexity of gestures have a great impact on gesture recognition. In order to enhance the features of gesture recognition, firstly, the hand skin color is filtered through YCbCr color space to separate the gesture region to be recognized, and the Gaussian filter is used to process the noise of gesture edge; secondly, the morphological gray open operation is used to process the gesture data, the watershed algorithm based on marker is used to segment the gesture contour, and the eight-connected filling algorithm is used to enhance the gesture features; finally, the convolution neural network is used to recognize the gesture data set with fast convergence speed. The experimental results show that the proposed method can recognize all kinds of gestures quickly and accurately with an average recognition success rate of 96.46% and does not significantly increase the recognition time.