Fetal Ultrasound Image Segmentation for Measuring Biometric Parameters Using Multi-Task Deep Learning

Artificial Intelligence in Obstetrics and Gynecology: Transforming Care and Outcomes

The integration of artificial intelligence (AI) in obstetrics and gynecology (OB/GYN) is revolutionizing the landscape of women's healthcare. This review article explores the transformative impact of AI technologies on the diagnosis, treatment, and management of obstetric and gynecological conditions. We examine key advancements in AI-driven imaging techniques, predictive analytics, and personalized medicine, highlighting their roles in enhancing prenatal care, improving maternal and fetal outcomes, and optimizing gynecological interventions. The article also addresses the challenges and ethical considerations associated with the implementation of AI in clinical practice. This paper highlights the potential of AI to greatly improve the standard of care in OB/GYN, ultimately leading to better health outcomes for women, by offering a thorough overview of present AI uses and future prospects.

Combining deep learning and intelligent biometry to extract ultrasound standard planes and assess early gestational weeks

OBJECTIVES

To develop and validate a fully automated AI system to extract standard planes, assess early gestational weeks, and compare the performance of the developed system to sonographers.

METHODS

In this three-center retrospective study, 214 consecutive pregnant women that underwent transvaginal ultrasounds between January and December 2018 were selected. Their ultrasound videos were automatically split into 38,941 frames using a particular program. First, an optimal deep-learning classifier was selected to extract the standard planes with key anatomical structures from the ultrasound frames. Second, an optimal segmentation model was selected to outline gestational sacs. Third, novel biometry was used to measure, select the largest gestational sac in the same video, and assess gestational weeks automatically. Finally, an independent test set was used to compare the performance of the system with that of sonographers. The outcomes were analyzed using the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and mean similarity between two samples (mDice).

RESULTS

The standard planes were extracted with an AUC of 0.975, a sensitivity of 0.961, and a specificity of 0.979. The gestational sacs' contours were segmented with a mDice of 0.974 (error less than 2 pixels). The comparison showed that the relative error of the tool in assessing gestational weeks was 12.44% and 6.92% lower and faster (min, 0.17 vs. 16.6 and 12.63) than that of the intermediate and senior sonographers, respectively.

CONCLUSIONS

This proposed end-to-end tool allows automatic assessment of gestational weeks in early pregnancy and may reduce manual analysis time and measurement errors.

CLINICAL RELEVANCE STATEMENT

The fully automated tool achieved high accuracy showing its potential to optimize the increasingly scarce resources of sonographers. Explainable predictions can assist in their confidence in assessing gestational weeks and provide a reliable basis for managing early pregnancy cases.

KEY POINTS

• The end-to-end pipeline enabled automatic identification of the standard plane containing the gestational sac in an ultrasound video, as well as segmentation of the sac contour, automatic multi-angle measurements, and the selection of the sac with the largest mean internal diameter to calculate the early gestational week. • This fully automated tool combining deep learning and intelligent biometry may assist the sonographer in assessing the early gestational week, increasing accuracy and reducing the analyzing time, thereby reducing observer dependence.

A multicenter study on two-stage transfer learning model for duct-dependent CHDs screening in fetal echocardiography

Duct-dependent congenital heart diseases (CHDs) are a serious form of CHD with a low detection rate, especially in underdeveloped countries and areas. Although existing studies have developed models for fetal heart structure identification, there is a lack of comprehensive evaluation of the long axis of the aorta. In this study, a total of 6698 images and 48 videos are collected to develop and test a two-stage deep transfer learning model named DDCHD-DenseNet for screening critical duct-dependent CHDs. The model achieves a sensitivity of 0.973, 0.843, 0.769, and 0.759, and a specificity of 0.985, 0.967, 0.956, and 0.759, respectively, on the four multicenter test sets. It is expected to be employed as a potential automatic screening tool for hierarchical care and computer-aided diagnosis. Our two-stage strategy effectively improves the robustness of the model and can be extended to screen for other fetal heart development defects.

A Machine Learning Method for Automated Description and Workflow Analysis of First Trimester Ultrasound Scans

Obstetric ultrasound assessment of fetal anatomy in the first trimester of pregnancy is one of the less explored fields in obstetric sonography because of the paucity of guidelines on anatomical screening and availability of data. This paper, for the first time, examines imaging proficiency and practices of first trimester ultrasound scanning through analysis of full-length ultrasound video scans. Findings from this study provide insights to inform the development of more effective user-machine interfaces, of targeted assistive technologies, as well as improvements in workflow protocols for first trimester scanning. Specifically, this paper presents an automated framework to model operator clinical workflow from full-length routine first-trimester fetal ultrasound scan videos. The 2D+t convolutional neural network-based architecture proposed for video annotation incorporates transfer learning and spatio-temporal (2D+t) modelling to automatically partition an ultrasound video into semantically meaningful temporal segments based on the fetal anatomy detected in the video. The model results in a cross-validation A1 accuracy of 96.10% , F1=0.95 , precision =0.94 and recall =0.95 . Automated semantic partitioning of unlabelled video scans (n=250) achieves a high correlation with expert annotations ( ρ = 0.95, p=0.06 ). Clinical workflow patterns, operator skill and its variability can be derived from the resulting representation using the detected anatomy labels, order, and distribution. It is shown that nuchal translucency (NT) is the toughest standard plane to acquire and most operators struggle to localize high-quality frames. Furthermore, it is found that newly qualified operators spend 25.56% more time on key biometry tasks than experienced operators.

D2ANet: Densely Attentional-Aware Network for First Trimester Ultrasound CRL and NT Segmentation

Manual annotation of medical images is time consuming for clinical experts; therefore, reliable automatic segmentation would be the ideal way to handle large medical datasets. In this paper, we are interested in detection and segmentation of two fundamental measurements in the first trimester ultrasound (US) scan: Nuchal Translucency (NT) and Crown Rump Length (CRL). There can be a significant variation in the shape, location or size of the anatomical structures in the fetal US scans. We propose a new approach, namely Densely Attentional-Aware Network for First Trimester Ultrasound CRL and NT Segmentation (DA2Net), to encode variation in feature size by relying on the powerful attention mechanism and densely connected networks. Our results show that the proposed D2ANet offers high pixel agreement (mean JSC = 84.21) with expert manual annotations.

Applications of artificial intelligence in obstetrics

Artificial intelligence, which has been applied as an innovative technology in multiple fields of healthcare, analyzes large amounts of data to assist in disease prediction, prevention, and diagnosis, as well as in patient monitoring. In obstetrics, artificial intelligence has been actively applied and integrated into our daily medical practice. This review provides an overview of artificial intelligence systems currently used for obstetric diagnostic purposes, such as fetal cardiotocography, ultrasonography, and magnetic resonance imaging, and demonstrates how these methods have been developed and clinically applied.

Ensemble Transfer Learning for Fetal Head Analysis: From Segmentation to Gestational Age and Weight Prediction

Ultrasound is one of the most commonly used imaging methodologies in obstetrics to monitor the growth of a fetus during the gestation period. Specifically, ultrasound images are routinely utilized to gather fetal information, including body measurements, anatomy structure, fetal movements, and pregnancy complications. Recent developments in artificial intelligence and computer vision provide new methods for the automated analysis of medical images in many domains, including ultrasound images. We present a full end-to-end framework for segmenting, measuring, and estimating fetal gestational age and weight based on two-dimensional ultrasound images of the fetal head. Our segmentation framework is based on the following components: (i) eight segmentation architectures (UNet, UNet Plus, Attention UNet, UNet 3+, TransUNet, FPN, LinkNet, and Deeplabv3) were fine-tuned using lightweight network EffientNetB0, and (ii) a weighted voting method for building an optimized ensemble transfer learning model (ETLM). On top of that, ETLM was used to segment the fetal head and to perform analytic and accurate measurements of circumference and seven other values of the fetal head, which we incorporated into a multiple regression model for predicting the week of gestational age and the estimated fetal weight (EFW). We finally validated the regression model by comparing our result with expert physician and longitudinal references. We evaluated the performance of our framework on the public domain dataset HC18: we obtained 98.53% mean intersection over union (mIoU) as the segmentation accuracy, overcoming the state-of-the-art methods; as measurement accuracy, we obtained a 1.87 mm mean absolute difference (MAD). Finally we obtained a 0.03% mean square error (MSE) in predicting the week of gestational age and 0.05% MSE in predicting EFW.

Three-Dimensional Multi-Task Deep Learning Model to Detect Glaucomatous Optic Neuropathy and Myopic Features From Optical Coherence Tomography Scans: A Retrospective Multi-Centre Study

Purpose

We aim to develop a multi-task three-dimensional (3D) deep learning (DL) model to detect glaucomatous optic neuropathy (GON) and myopic features (MF) simultaneously from spectral-domain optical coherence tomography (SDOCT) volumetric scans.

Methods

Each volumetric scan was labelled as GON according to the criteria of retinal nerve fibre layer (RNFL) thinning, with a structural defect that correlated in position with the visual field defect (i.e., reference standard). MF were graded by the SDOCT en face images, defined as presence of peripapillary atrophy (PPA), optic disc tilting, or fundus tessellation. The multi-task DL model was developed by ResNet with output of Yes/No GON and Yes/No MF. SDOCT scans were collected in a tertiary eye hospital (Hong Kong SAR, China) for training (80%), tuning (10%), and internal validation (10%). External testing was performed on five independent datasets from eye centres in Hong Kong, the United States, and Singapore, respectively. For GON detection, we compared the model to the average RNFL thickness measurement generated from the SDOCT device. To investigate whether MF can affect the model’s performance on GON detection, we conducted subgroup analyses in groups stratified by Yes/No MF. The area under the receiver operating characteristic curve (AUROC), sensitivity, specificity, and accuracy were reported.

Results

A total of 8,151 SDOCT volumetric scans from 3,609 eyes were collected. For detecting GON, in the internal validation, the proposed 3D model had significantly higher AUROC (0.949 vs. 0.913, p < 0.001) than average RNFL thickness in discriminating GON from normal. In the external testing, the two approaches had comparable performance. In the subgroup analysis, the multi-task DL model performed significantly better in the group of “no MF” (0.883 vs. 0.965, p-value < 0.001) in one external testing dataset, but no significant difference in internal validation and other external testing datasets. The multi-task DL model’s performance to detect MF was also generalizable in all datasets, with the AUROC values ranging from 0.855 to 0.896.

Conclusion

The proposed multi-task 3D DL model demonstrated high generalizability in all the datasets and the presence of MF did not affect the accuracy of GON detection generally.

Efficient fetal ultrasound image segmentation for automatic head circumference measurement using a lightweight deep convolutional neural network

PURPOSE

Fetal head circumference (HC) is an important biometric parameter that can be used to assess fetal development in obstetric clinical practice. Most of existing methods use deep neural network to accomplish the task of automatic fetal HC measurement from two-dimensional ultrasound images, and some of them achieved relatively high prediction accuracy. However, few of these methods focused on optimizing model efficiency performance. Our purpose is to develop a more efficient approach for this task, which could help doctors measure HC faster and would be more suitable for deployment on devices with scarce computing resources.

METHODS

In this paper, we present a very lightweight deep convolutional neural network to achieve automatic fetal head segmentation from ultrasound images. By using sequential prediction network architecture, the proposed model could perform much faster inference while maintaining a high prediction accuracy. In addition, we used depthwise separable convolution to replace part of the standard convolution in the network and shrunk the input image to further improve model efficiency. After getting fetal head segmentation results, post-processing, including morphological processing and least-squares ellipse fitting, was applied to obtain the fetal HC. All experiments in this work were performed on a public dataset, HC18, with 999 fetal ultrasound images for training and 335 for testing. The dataset is publicly available on https://hc18.grand-challenge.org/ and the code for our method is also publicly available on https://github.com/ApeMocker/CSM-for-fetal-HC-measurement.

RESULTS

Our model has only 0.13 million [M] parameters and can achieve an inference speed of 28[ms] per frame on a CPU and 0.194 [ms] per frame on a GPU, which far exceeds all existing deep learning-based models as far as we know. Experimental results showed that the method achieved a mean absolute difference of 1.97 (±1.89) [mm] and a Dice similarity coefficient of 97.61(±1.72) [%] on HC18 test set, which were comparable to the state-of-the-art.

CONCLUSION

We presented a very lightweight deep learning-based model to realize fast and accurate fetal head segmentation from two-dimensional ultrasound image, which is then used for calculating the fetal HC. The proposed method could help obstetricians measure the fetal head circumference more efficiently with high accuracy, and has the potential to be applied to the situations where computing resources are relatively scarce. This article is protected by copyright. All rights reserved.

RLAS-BIABC: A Reinforcement Learning-Based Answer Selection Using the BERT Model Boosted by an Improved ABC Algorithm

Answer selection (AS) is a critical subtask of the open-domain question answering (QA) problem. The present paper proposes a method called RLAS-BIABC for AS, which is established on attention mechanism-based long short-term memory (LSTM) and the bidirectional encoder representations from transformers (BERT) word embedding, enriched by an improved artificial bee colony (ABC) algorithm for pretraining and a reinforcement learning-based algorithm for training backpropagation (BP) algorithm. BERT can be comprised in downstream work and fine-tuned as a united task-specific architecture, and the pretrained BERT model can grab different linguistic effects. Existing algorithms typically train the AS model with positive-negative pairs for a two-class classifier. A positive pair contains a question and a genuine answer, while a negative one includes a question and a fake answer. The output should be one for positive and zero for negative pairs. Typically, negative pairs are more than positive, leading to an imbalanced classification that drastically reduces system performance. To deal with it, we define classification as a sequential decision-making process in which the agent takes a sample at each step and classifies it. For each classification operation, the agent receives a reward, in which the prize of the majority class is less than the reward of the minority class. Ultimately, the agent finds the optimal value for the policy weights. We initialize the policy weights with the improved ABC algorithm. The initial value technique can prevent problems such as getting stuck in the local optimum. Although ABC serves well in most tasks, there is still a weakness in the ABC algorithm that disregards the fitness of related pairs of individuals in discovering a neighboring food source position. Therefore, this paper also proposes a mutual learning technique that modifies the produced candidate food source with the higher fitness between two individuals selected by a mutual learning factor. We tested our model on three datasets, LegalQA, TrecQA, and WikiQA, and the results show that RLAS-BIABC can be recognized as a state-of-the-art method.

RDHCformer: Fusing ResDCN and Transformers for Fetal Head Circumference Automatic Measurement in 2D Ultrasound Images

Fetal head circumference (HC) is an important biological parameter to monitor the healthy development of the fetus. Since there are some HC measurement errors that affected by the skill and experience of the sonographers, a rapid, accurate and automatic measurement for fetal HC in prenatal ultrasound is of great significance. We proposed a new one-stage network for rotating elliptic object detection based on anchor-free method, which is also an end-to-end network for fetal HC auto-measurement that no need for any post-processing. The network structure used simple transformer structure combined with convolutional neural network (CNN) for a lightweight design, meanwhile, made full use of powerful global feature extraction ability of transformer and local feature extraction ability of CNN to extract continuous and complete skull edge information. The two complement each other for promoting detection precision of fetal HC without significantly increasing the amount of computation. In order to reduce the large variation of intersection over union (IOU) in rotating elliptic object detection caused by slight angle deviation, we used soft stage-wise regression (SSR) strategy for angle regression and added KLD that is approximate to IOU loss into total loss function. The proposed method achieved good results on the HC18 dataset to prove its effectiveness. This study is expected to help less experienced sonographers, provide help for precision medicine, and relieve the shortage of sonographers for prenatal ultrasound in worldwide.

Artificial Intelligence in Prenatal Ultrasound Diagnosis

The application of artificial intelligence (AI) technology to medical imaging has resulted in great breakthroughs. Given the unique position of ultrasound (US) in prenatal screening, the research on AI in prenatal US has practical significance with its application to prenatal US diagnosis improving work efficiency, providing quantitative assessments, standardizing measurements, improving diagnostic accuracy, and automating image quality control. This review provides an overview of recent studies that have applied AI technology to prenatal US diagnosis and explains the challenges encountered in these applications.

Deep Learning-Based Computer-Aided Fetal Echocardiography: Application to Heart Standard View Segmentation for Congenital Heart Defects Detection

Accurate segmentation of fetal heart in echocardiography images is essential for detecting the structural abnormalities such as congenital heart defects (CHDs). Due to the wide variations attributed to different factors, such as maternal obesity, abdominal scars, amniotic fluid volume, and great vessel connections, this process is still a challenging problem. CHDs detection with expertise in general are substandard; the accuracy of measurements remains highly dependent on humans' training, skills, and experience. To make such a process automatic, this study proposes deep learning-based computer-aided fetal heart echocardiography examinations with an instance segmentation approach, which inherently segments the four standard heart views and detects the defect simultaneously. We conducted several experiments with 1149 fetal heart images for predicting 24 objects, including four shapes of fetal heart standard views, 17 objects of heart-chambers in each view, and three cases of congenital heart defect. The result showed that the proposed model performed satisfactory performance for standard views segmentation, with a 79.97% intersection over union and 89.70% Dice coefficient similarity. It also performed well in the CHDs detection, with mean average precision around 98.30% for intra-patient variation and 82.42% for inter-patient variation. We believe that automatic segmentation and detection techniques could make an important contribution toward improving congenital heart disease diagnosis rates.

Mask-R 2 CNN: a distance-field regression version of Mask-RCNN for fetal-head delineation in ultrasound images

BACKGROUND AND OBJECTIVES

Fetal head-circumference (HC) measurement from ultrasound (US) images provides useful hints for assessing fetal growth. Such measurement is performed manually during the actual clinical practice, posing issues relevant to intra- and inter-clinician variability. This work presents a fully automatic, deep-learning-based approach to HC delineation, which we named Mask-R2 CNN. It advances our previous work in the field and performs HC distance-field regression in an end-to-end fashion, without requiring a priori HC localization nor any postprocessing for outlier removal.

METHODS

Mask-R2 CNN follows the Mask-RCNN architecture, with a backbone inspired by feature-pyramid networks, a region-proposal network and the ROI align. The Mask-RCNN segmentation head is here modified to regress the HC distance field.

RESULTS

Mask-R2 CNN was tested on the HC18 Challenge dataset, which consists of 999 training and 335 testing images. With a comprehensive ablation study, we showed that Mask-R2 CNN achieved a mean absolute difference of 1.95 mm (standard deviation = ± 1.92  mm), outperforming other approaches in the literature.

CONCLUSIONS

With this work, we proposed an end-to-end model for HC distance-field regression. With our experimental results, we showed that Mask-R2 CNN may be an effective support for clinicians for assessing fetal growth.

Artificial Intelligence in Obstetric Ultrasound: An Update and Future Applications

Artificial intelligence (AI) can support clinical decisions and provide quality assurance for images. Although ultrasonography is commonly used in the field of obstetrics and gynecology, the use of AI is still in a stage of infancy. Nevertheless, in repetitive ultrasound examinations, such as those involving automatic positioning and identification of fetal structures, prediction of gestational age (GA), and real-time image quality assurance, AI has great potential. To realize its application, it is necessary to promote interdisciplinary communication between AI developers and sonographers. In this review, we outlined the benefits of AI technology in obstetric ultrasound diagnosis by optimizing image acquisition, quantification, segmentation, and location identification, which can be helpful for obstetric ultrasound diagnosis in different periods of pregnancy.

Towards Clinical Application of Artificial Intelligence in Ultrasound Imaging

Artificial intelligence (AI) is being increasingly adopted in medical research and applications. Medical AI devices have continuously been approved by the Food and Drug Administration in the United States and the responsible institutions of other countries. Ultrasound (US) imaging is commonly used in an extensive range of medical fields. However, AI-based US imaging analysis and its clinical implementation have not progressed steadily compared to other medical imaging modalities. The characteristic issues of US imaging owing to its manual operation and acoustic shadows cause difficulties in image quality control. In this review, we would like to introduce the global trends of medical AI research in US imaging from both clinical and basic perspectives. We also discuss US image preprocessing, ingenious algorithms that are suitable for US imaging analysis, AI explainability for obtaining informed consent, the approval process of medical AI devices, and future perspectives towards the clinical application of AI-based US diagnostic support technologies.

Fine-Tuning U-Net for Ultrasound Image Segmentation: Different Layers, Different Outcomes

One way of resolving the problem of scarce and expensive data in deep learning for medical applications is using transfer learning and fine-tuning a network which has been trained on a large data set. The common practice in transfer learning is to keep the shallow layers unchanged and to modify deeper layers according to the new data set. This approach may not work when using a U-Net and when moving from a different domain to ultrasound (US) images due to their drastically different appearance. In this study, we investigated the effect of fine-tuning different sets of layers of a pretrained U-Net for US image segmentation. Two different schemes were analyzed, based on two different definitions of shallow and deep layers. We studied simulated US images, as well as two human US data sets. We also included a chest X-ray data set. The results showed that choosing which layers to fine-tune is a critical task. In particular, they demonstrated that fine-tuning the last layers of the network, which is the common practice for classification networks, is often the worst strategy. It may therefore be more appropriate to fine-tune the shallow layers rather than deep layers in US image segmentation when using a U-Net. Shallow layers learn lower level features which are critical in automatic segmentation of medical images. Even when a large US data set is available, we observed that fine-tuning shallow layers is a faster approach compared to fine-tuning the whole network.

Technology trends and applications of deep learning in ultrasonography: image quality enhancement, diagnostic support, and improving workflow efficiency

In this review of the most recent applications of deep learning to ultrasound imaging, the architectures of deep learning networks are briefly explained for the medical imaging applications of classification, detection, segmentation, and generation. Ultrasonography applications for image processing and diagnosis are then reviewed and summarized, along with some representative imaging studies of the breast, thyroid, heart, kidney, liver, and fetal head. Efforts towards workflow enhancement are also reviewed, with an emphasis on view recognition, scanning guide, image quality assessment, and quantification and measurement. Finally some future prospects are presented regarding image quality enhancement, diagnostic support, and improvements in workflow efficiency, along with remarks on hurdles, benefits, and necessary collaborations.