Quality-improvement program for ultrasound-based fetal anatomy screening using large-scale clinical audit

Development and validation of an artificial intelligence assisted prenatal ultrasonography screening system for trainees

OBJECTIVE

Fetal anomaly screening via ultrasonography, which involves capturing and interpreting standard views, is highly challenging for inexperienced operators. We aimed to develop and validate a prenatal-screening artificial intelligence system (PSAIS) for real-time evaluation of the quality of anatomical images, indicating existing and missing structures.

METHODS

Still ultrasonographic images obtained from fetuses of 18-32 weeks of gestation between 2017 and 2018 were used to develop PSAIS based on YOLOv3 with global (anatomic site) and local (structures) feature extraction that could evaluate the image quality and indicate existing and missing structures in the fetal anatomical images. The performance of the PSAIS in recognizing 19 standard views was evaluated using retrospective real-world fetal scan video validation datasets from four hospitals. We stratified sampled frames (standard, similar-to-standard, and background views at approximately 1:1:1) for experts to blindly verify the results.

RESULTS

The PSAIS was trained using 134 696 images and validated using 836 videos with 12 697 images. For internal and external validations, the multiclass macro-average areas under the receiver operating characteristic curve were 0.943 (95% confidence interval [CI], 0.815-1.000) and 0.958 (0.864-1.000); the micro-average areas were 0.974 (0.970-0.979) and 0.973 (0.965-0.981), respectively. For similar-to-standard views, the PSAIS accurately labeled 90.9% (90.0%-91.4%) with key structures and indicated missing structures.

CONCLUSIONS

An artificial intelligence system developed to assist trainees in fetal anomaly screening demonstrated high agreement with experts in standard view identification.

Memory-based unsupervised video clinical quality assessment with multi-modality data in fetal ultrasound

In obstetric sonography, the quality of acquisition of ultrasound scan video is crucial for accurate (manual or automated) biometric measurement and fetal health assessment. However, the nature of fetal ultrasound involves free-hand probe manipulation and this can make it challenging to capture high-quality videos for fetal biometry, especially for the less-experienced sonographer. Manually checking the quality of acquired videos would be time-consuming, subjective and requires a comprehensive understanding of fetal anatomy. Thus, it would be advantageous to develop an automatic quality assessment method to support video standardization and improve diagnostic accuracy of video-based analysis. In this paper, we propose a general and purely data-driven video-based quality assessment framework which directly learns a distinguishable feature representation from high-quality ultrasound videos alone, without anatomical annotations. Our solution effectively utilizes both spatial and temporal information of ultrasound videos. The spatio-temporal representation is learned by a bi-directional reconstruction between the video space and the feature space, enhanced by a key-query memory module proposed in the feature space. To further improve performance, two additional modalities are introduced in training which are the sonographer gaze and optical flow derived from the video. Two different clinical quality assessment tasks in fetal ultrasound are considered in our experiments, i.e., measurement of the fetal head circumference and cerebellar diameter; in both of these, low-quality videos are detected by the large reconstruction error in the feature space. Extensive experimental evaluation demonstrates the merits of our approach.

Revisiting the utility of prenatal ultrasound in the routine workup of first febrile UTI: A systematic review and meta-analysis of the negative predictive value of prenatal ultrasound for identification of urinary tract abnormalities after first febrile urinary tract infection in children

CONTEXT

The current EAU/ESPU and recently retired AAP pediatric UTI guidelines recommend renal bladder ultrasound after first febrile UTI in children to screen for structural abnormalities, regardless of findings on prenatal ultrasound.

OBJECTIVE

Test the hypothesis that a normal prenatal ultrasound could rule out urinary tract abnormality on post-UTI ultrasound.

DATA SOURCES

Medline, Embase, Cochrane Library.

STUDY SELECTION

Studies including pediatric patients with first febrile UTI who had both prenatal and post-UTI ultrasound.

DATA EXTRACTION

Anatomical abnormalities detected by prenatal and post-UTI ultrasound as reported per individual study criteria were extracted. Meta-analyses of 9 studies (2981 patients) were performed using a random-effects model and composite estimates of the negative predictive value (NPV) of prenatal ultrasound were calculated.

RESULTS

Overall summary NPV of prenatal ultrasound for all pediatric patients was 77%, with heterogeneity score (I2) 97.9%. Summary NPV of prenatal ultrasound for all patients under two years of age was similar at 75%, with I2 98.2% For the 4 studies to which we could apply a more stringent definition of abnormality, summary NPV was 85% and I2 97.5% for prediction of moderate post-UTI ultrasound abnormalities and summary NPV was 93% and I2 90.4% for severe abnormalities.

DISCUSSION

While we calculated an 85% NPV for a normal prenatal ultrasound to rule out significant postnatal abnormality as defined within individual studies, substantial heterogeneity amongst publications limited the precision of our estimates. This highlights the need for more rigorous investigations with attention to timing of ultrasound and the application of clinically meaningful definitions for abnormal prenatal and post-UTI studies. This may allow judicious use of prenatal ultrasound to guide clinical management for children with first febrile UTI and minimize redundant imaging with potential for false positive results. Until then, the current guidelines are justified based on the limited and heterogenous data from the currently available published studies.

Fetal biometry and amniotic fluid volume assessment end-to-end automation using Deep Learning

Fetal biometry and amniotic fluid volume assessments are two essential yet repetitive tasks in fetal ultrasound screening scans, aiding in the detection of potentially life-threatening conditions. However, these assessment methods can occasionally yield unreliable results. Advances in deep learning have opened up new avenues for automated measurements in fetal ultrasound, demonstrating human-level performance in various fetal ultrasound tasks. Nevertheless, the majority of these studies are retrospective in silico studies, with a limited number including African patients in their datasets. In this study we developed and prospectively assessed the performance of deep learning models for end-to-end automation of fetal biometry and amniotic fluid volume measurements. These models were trained using a newly constructed database of 172,293 de-identified Moroccan fetal ultrasound images, supplemented with publicly available datasets. the models were then tested on prospectively acquired video clips from 172 pregnant people forming a consecutive series gathered at four healthcare centers in Morocco. Our results demonstrate that the 95% limits of agreement between the models and practitioners for the studied measurements were narrower than the reported intra- and inter-observer variability among expert human sonographers for all the parameters under study. This means that these models could be deployed in clinical conditions, to alleviate time-consuming, repetitive tasks, and make fetal ultrasound more accessible in limited-resource environments.

Organ-specific learning curves of sonographers performing first-trimester anatomical screening and impact of score-based evaluation on ultrasound image quality

INTRODUCTION

First-trimester anatomical screening (FTAS) by ultrasound has been introduced in many countries as screening for aneuploidies, but also as early screening for fetal structural abnormalities. While a lot of emphasis has been put on the detection rates of FTAS, little is known about the performance of quality control programs and the sonographers' learning curve for FTAS. The aims of the study were to evaluate the performance of a score-based quality control system for the FTAS and to assess the learning curves of sonographers by evaluating the images of the anatomical planes that were part of the FTAS protocol.

METHODS

Between 2012-2015, pregnant women opting for the combined test in the North-Netherlands were also invited to participate in a prospective cohort study extending the ultrasound investigation to include a first-trimester ultrasound performed according to a protocol. All anatomical planes included in the protocol were documented by pictures stored for each examination in logbooks. The logbooks of six sonographers were independently assessed by two fetal medicine experts. For each sonographer, logbooks of examination 25-50-75 and 100 plus four additional randomly selected logbooks were scored for correct visualization of 12 organ-system planes. A plane specific score of at least 70% was considered sufficient. The intra-class correlation coefficient (ICC), was used to measure inter-assessor agreement for the cut-off scores. Organ-specific learning curves were defined by single-cumulative sum (CUSUM) analysis.

RESULTS

Sixty-four logbooks were assessed. Mean duration of the scan was 22 ± 6 minutes and mean gestational age was 12+6 weeks. In total 57% of the logbooks graded as sufficient. Most sufficient scores were obtained for the fetal skull (88%) and brain (70%), while the lowest scores were for the face (29%) and spine (38%). Five sonographers showed a learning curve for the skull and the stomach, four for the brain and limbs, three for the bladder and kidneys, two for the diaphragm and abdominal wall and one for the heart and spine and none for the face and neck.

CONCLUSION

Learning curves for FTAS differ per organ system and per sonographer. Although score-based evaluation can validly assess image quality, more dynamic approaches may better reflect clinical performance.

A National Quality Improvement Program on Ultrasound Department in China: A Controlled Cohort Study of 1297 Public Hospitals

Providing high-quality medical services is of great importance in the imaging department, as there is a growing focus on personal health, and high-quality services can lead to improved patient outcomes. Many quality improvement (QI) programs with good guidance and fine measurement for improvement have been reported to be effective. In order to improve the quality of ultrasound departments in China, we conducted this study of a national quality improvement program. A total of 1297 public hospitals were included in this QI program on ultrasound departments in China from 2017 to 2019. The effect of this QI program was investigated, and potential factors, including hospital level and local economic development, were considered. The outcome indicators, the positive rate and diagnostic accuracy, were improved significantly between the two phases (positive rate, 2017 vs. 2019: 66.21% vs. 73.91%, p < 0.001; diagnostic accuracy, 2017 vs. 2019: 85.37% vs. 89.74%; p < 0.001). Additionally, they were improved in secondary and tertiary hospitals, with the improvement in secondary hospitals being greater. Notably, the enhancement of diagnostic accuracy in low-GDP provinces was almost 20%, which was more significant than the enhancement in high-GDP provinces. However, the important structural indicator, the doctor-to-patient ratio, decreased from 1.05:10,000 to 0.96:10,000 (p = 0.026). This study suggests that the national ultrasound QI program improved the outcome indicators, with secondary-level hospitals improving more than tertiary hospitals and low-GDP provinces improving more than high-GDP regions. Additionally, as there is a growing need for ultrasound examinations, more ultrasound doctors are needed in China.

[Ante- and postnatal growth charts in France - guidelines for clinical practice from the Collège national des gynécologues et obstétriciens français (CNGOF) and from the Société française de néonatologie (SFN)]

OBJECTIVE

To recommend the most appropriate biometric charts for the detection of antenatal growth abnormalities and postnatal growth surveillance.

METHODS

Elaboration of specific questions and selection of experts by the organizing committee to answer these questions; analysis of the literature by experts and drafting conclusions by assigning a recommendation (strong or weak) and a quality of evidence (high, moderate, low, very low) and for each question; all these recommendations have been subject to multidisciplinary external review (obstetrician gynecologists, pediatricians). The objective for the reviewers was to verify the completeness of the literature review, to verify the levels of evidence established and the consistency and applicability of the resulting recommendations. The overall review of the literature, quality of evidence and recommendations were revised to take into consideration comments from external reviewers.

RESULTS

Antenatally, it is recommended to use all WHO fetal growth charts for EFW and common ultrasound biometric measurements (strong recommendation; low quality of evidence). Indeed, in comparison with other prescriptive curves and descriptive curves, the WHO prescriptive charts show better performance for the screening of SGA (Small for Gestational Age) and LGA (Large for Gestational Age) with adequate proportions of fetuses screened at extreme percentiles in the French population. It also has the advantages of having EFW charts by sex and biometric parameters obtained from the same perspective cohort of women screened by qualified sonographers who measured the biometric parameters according to international standards. Postnatally, it is recommended to use the updated Fenton charts for the assessment of birth measurements and for growth monitoring in preterm infants (strong recommendation; moderate quality of evidence) and for the assessment of birth measurements in term newborn (expert opinion).

CONCLUSION

It is recommended to use WHO fetal growth charts for antenatal growth monitoring and Fenton charts for the newborn.

Radiographers' individual perspectives on sonography - A survey of European Federation of Radiographer Societies (EFRS)

INTRODUCTION

Radiographers can elect to work within many different modalities, one being ultrasound. Within Europe there are differing opinions about how much of a role radiographers should take in relation to the ultrasound examination, particularly report writing. This paper provides findings exploring the radiographer's views on working within sonography.

METHODS

In 2019 an electronic survey was disseminated to radiographer members by European Federation of Radiographer Societies (EFRS) national radiographer societies, following a pilot study. A mix of closed questions, free text, and scale responses aimed to investigate radiographers' practice, legal responsibilities, report writing, educational level and experiences of support and mentoring.

RESULTS

Of 561 radiographers participating, most (92%) reported performing ultrasound scans. Challenges with legislation, medical protectionism and lack of high-quality education restricted other radiographers. On average, the respondents have practiced ultrasound for 13.5 years. A total of 60% had postgraduate education and carried out a wide range of examinations. A full interpretative report, including advice on further investigations is performed by 52%, whilst 22% provide a checklist or descriptive report. Over 55% of radiographers took legal responsibility for the examination and the majority had clear protocols, good mentoring and support in the workplace. Peer review of their work was less common.

CONCLUSION

The result shows that in 21 (n = 25) countries radiographers perform ultrasound, however not without challenges. Educational levels range from no formal education or short courses to an MSc in ultrasound. Report writing practice differs across the EFRS countries responding to the survey, as does peer review to enhance skills and clinical practice.

IMPLICATIONS FOR PRACTICE

National Radiographer societies could review findings to support campaigning for a change in legislation and improvements to educational offerings in ultrasound.

Machine learning-based analysis of operator pupillary response to assess cognitive workload in clinical ultrasound imaging

Introduction

Pupillometry, the measurement of eye pupil diameter, is a well-established and objective modality correlated with cognitive workload. In this paper, we analyse the pupillary response of ultrasound imaging operators to assess their cognitive workload, captured while they undertake routine fetal ultrasound examinations. Our experiments and analysis are performed on real-world datasets obtained using remote eye-tracking under natural clinical environmental conditions.

Methods

Our analysis pipeline involves careful temporal sequence (time-series) extraction by retrospectively matching the pupil diameter data with tasks captured in the corresponding ultrasound scan video in a multi-modal data acquisition setup. This is followed by the pupil diameter pre-processing and the calculation of pupillary response sequences. Exploratory statistical analysis of the operator pupillary responses and comparisons of the distributions between ultrasonographic tasks (fetal heart versus fetal brain) and operator expertise (newly-qualified versus experienced operators) are performed. Machine learning is explored to automatically classify the temporal sequences into the corresponding ultrasonographic tasks and operator experience using temporal, spectral, and time-frequency features with classical (shallow) models, and convolutional neural networks as deep learning models.

Results

Preliminary statistical analysis of the extracted pupillary response shows a significant variation for different ultrasonographic tasks and operator expertise, suggesting different extents of cognitive workload in each case, as measured by pupillometry. The best-performing machine learning models achieve receiver operating characteristic (ROC) area under curve (AUC) values of 0.98 and 0.80, for ultrasonographic task classification and operator experience classification, respectively.

Conclusion

We conclude that we can successfully assess cognitive workload from pupil diameter changes measured while ultrasound operators perform routine scans. The machine learning allows the discrimination of the undertaken ultrasonographic tasks and scanning expertise using the pupillary response sequences as an index of the operators’ cognitive workload. A high cognitive workload can reduce operator efficiency and constrain their decision-making, hence, the ability to objectively assess cognitive workload is a first step towards understanding these effects on operator performance in biomedical applications such as medical imaging.

•

Machine learning-based pupillary response analysis is performed to assess operator cognitive workload in clinical ultrasound.

•

A systematic multi-modal data analysis pipeline is proposed using eye-tracking, pupillometry, and sonography data science.

•

Pertinent challenges of natural or real-world clinical datasets are addressed.

•

Pupillary responses around event triggers, different ultrasonographic tasks, and different operator experiences are studied.

•

Machine learning models are learnt to classify undertaken tasks or operator expertise from pupillometric time-series data.

Quality control of fetal biometric evaluation and Doppler ultrasound

In recent years quality control in obstetric ultrasound has become recommended and an essential component of obstetric scanning. This is to minimize the inaccuracy and variability related to fetal measurements, to provide an effective quality assurance system to sonographers to certify their practice and decrease the impact of medical litigations. For a quality control system in obstetric ultrasound to be useful clinically, multiple strategies need to be employed: certified training, practical standardization exercise, image storing, qualitative and quantitative quality control. Qualitative quality control consists of the evaluation of images obtained for fetal biometry and Doppler scans using an objective score against predefined criteria. Quantitative quality control consists of analyzing quantitatively the performance of a sonographer and the impact on measurements values. Quantitative analysis could be performed either using estimates of intraobserver or interobserver reproducibility of plane acquisition and caliper placements.

Towards automated extraction of 2D standard fetal head planes from 3D ultrasound acquisitions: A clinical evaluation and quality assessment comparison

Introduction

Clinical evaluation of deep learning (DL) tools is essential to compliment technical accuracy metrics. This study assessed the image quality of standard fetal head planes automatically-extracted from three-dimensional (3D) ultrasound fetal head volumes using a customised DL-algorithm.

Methods

Two observers retrospectively reviewed standard fetal head planes against pre-defined image quality criteria. Forty-eight images (29 transventricular, 19 transcerebellar) were selected from 91 transabdominal fetal scans (mean gestational age = 26 completed weeks, range = 20⁺⁵–32⁺³ weeks). Each had two-dimensional (2D) manually-acquired (2D-MA), 3D operator-selected (3D-OS) and 3D-DL automatically-acquired (3D-DL) images. The proportion of adequate images from each plane and modality, and the number of inadequate images per plane was compared for each method. Inter and intra-observer agreement of overall image quality was calculated.

Results

Sixty-seven percent of 3D-OS and 3D-DL transventricular planes were adequate quality. Forty-five percent of 3D-OS and 55% of 3D-DL transcerebellar planes were adequate.

Seventy-one percent of 3D-OS and 86% of 3D-DL transventricular planes failed with poor visualisation of intra-cranial structures. Eighty-six percent of 3D-OS and 80% of 3D-DL transcerebellar planes failed due to inadequate visualisation of cerebellar hemispheres. Image quality was significantly different between 2D and 3D, however, no significant difference between 3D-modalities was demonstrated (p < 0.005). Inter-observer agreement of transventricular plane adequacy was moderate for both 3D-modalities, and weak for transcerebellar planes.

Conclusion

The 3D-DL algorithm can automatically extract standard fetal head planes from 3D-head volumes of comparable quality to operator-selected planes. Image quality in 3D is inferior to corresponding 2D planes, likely due to limitations with 3D-technology and acquisition technique.

Implications for practice

Automated image extraction of standard planes from US-volumes could facilitate use of 3DUS in clinical practice, however image quality is dependent on the volume acquisition technique.

Peer review of third trimester abdominal circumference measurements

Background

Third trimester growth scans represent a significant proportion of the workload in obstetric ultrasound departments. The objective of these serial growth scans is to improve the antenatal detection of babies with fetal growth restriction. The aim of this paper is to describe a method of peer review for third trimester abdominal circumference measurements which is realistic within busy obstetric ultrasound departments in the UK.

Method

Twenty-two, third trimester, measured abdominal circumference images were randomly selected. Images were assessed subjectively by 12 sonographers using the image Criteria Achieved Score. For quantitative assessment, termed the Inter-operator Variability Score, three of the abdominal circumference (AC) images were blindly remeasured. Following this, a questionnaire was used to ascertain which image criteria sonographers considered most important and to reach an agreement on correct caliper placement.

Results

The least frequently met image criteria with the lowest Criteria Achieved Score related to an oblique abdominal circumference section. These included fetal kidney present (Criteria Achieved Score 24.6%), multiple oblique ribs (Criteria Achieved Score 39.4%) and oblique spine (Criteria Achieved Score 37.5%). Caliper placement was also identified as inconsistent.

Discussion

This study demonstrates that the perfect AC section is not always possible and sonographers use their professional judgement to determine whether an image is acceptable. Seventy-three percent of the images reviewed were of an acceptable standard. There can be inconsistencies in sonographer opinion regarding what is an acceptable third trimester abdominal circumference image. These differences need to be addressed to maximise the effectiveness of the third trimester ultrasound examination.

Conclusion

Peer review can be used to monitor scan quality and identify areas of inconsistency.

Simulating realistic fetal neurosonography images with appearance and growth change using cycle-consistent adversarial networks and an evaluation

Purpose: We present an original method for simulating realistic fetal neurosonography images specifically generating third-trimester pregnancy ultrasound images from second-trimester images. Our method was developed using unpaired data, as pairwise data were not available. We also report original insights on the general appearance differences between second- and third-trimester fetal head transventricular (TV) plane images. Approach: We design a cycle-consistent adversarial network (Cycle-GAN) to simulate visually realistic third-trimester images from unpaired second- and third-trimester ultrasound images. Simulation realism is evaluated qualitatively by experienced sonographers who blindly graded real and simulated images. A quantitative evaluation is also performed whereby a validated deep-learning-based image recognition algorithm (ScanNav®) acts as the expert reference to allow hundreds of real and simulated images to be automatically analyzed and compared efficiently. Results: Qualitative evaluation shows that the human expert cannot tell the difference between real and simulated third-trimester scan images. 84.2% of the simulated third-trimester images could not be distinguished from the real third-trimester images. As a quantitative baseline, on 3000 images, the visibility drop of the choroid, CSP, and mid-line falx between real second- and real third-trimester scans was computed by ScanNav® and found to be 72.5%, 61.5%, and 67%, respectively. The visibility drop of the same structures between real second-trimester and simulated third-trimester was found to be 77.5%, 57.7%, and 56.2%, respectively. Therefore, the real and simulated third-trimester images were consider to be visually similar to each other. Our evaluation also shows that the third-trimester simulation of a conventional GAN is much easier to distinguish, and the visibility drop of the structures is smaller than our proposed method. Conclusions: The results confirm that it is possible to simulate realistic third-trimester images from second-trimester images using a modified Cycle-GAN, which may be useful for deep learning researchers with a restricted availability of third-trimester scans but with access to ample second trimester images. We also show convincing simulation improvements, both qualitatively and quantitatively, using the Cycle-GAN method compared with a conventional GAN. Finally, the use of a machine learning-based reference (in the case ScanNav®) for large-scale quantitative image analysis evaluation is also a first to our knowledge.