Impact of audit of routine second-trimester cardiac images using a novel image-scoring method

Intra- and interobserver agreement of proposed objective transvaginal ultrasound image-quality scoring system for use in artificial intelligence algorithm development

OBJECTIVES

The development of valuable artificial intelligence (AI) tools to assist with ultrasound diagnosis depends on algorithms developed using high-quality data. This study aimed to test the intra- and interobserver agreement of a proposed image-quality scoring system to quantify the quality of gynecological transvaginal ultrasound (TVS) images, which could be used in clinical practice and AI tool development.

METHODS

A proposed scoring system to quantify TVS image quality was created following a review of the literature. This system involved a score of 1-4 (2 = poor, 3 = suboptimal and 4 = optimal image quality) assigned by a rater for individual ultrasound images. If the image was deemed inaccurate, it was assigned a score of 1, corresponding to 'reject'. Six professionals, including two radiologists, two sonographers and two sonologists, reviewed 150 images (50 images of the uterus and 100 images of the ovaries) obtained from 50 women, assigning each image a score of 1-4. The review of all images was repeated a second time by each rater after a period of at least 1 week. Mean scores were calculated for each rater. Overall interobserver agreement was assessed using intraclass correlation coefficient (ICC), and interobserver agreement between paired professionals and intraobserver agreement for all professionals were assessed using weighted Cohen's kappa and ICC.

RESULTS

Poor levels of interobserver agreement were obtained between the six raters for all 150 images (ICC, 0.480 (95% CI, 0.363-0.586)), as well as for assessment of the uterine images only (ICC, 0.359 (95% CI, 0.204-0.523)). Moderate agreement was achieved for the ovarian images (ICC, 0.531 (95% CI, 0.417-0.636)). Agreement between the paired sonographers and sonologists was poor for all images (ICC, 0.336 (95% CI, -0.078 to 0.619) and 0.425 (95% CI, 0.014-0.665), respectively), as well as when images were grouped into uterine images (ICC, 0.253 (95% CI, -0.097 to 0.577) and 0.299 (95% CI, -0.094 to 0.606), respectively) and ovarian images (ICC, 0.400 (95% CI, -0.043 to 0.669) and 0.469 (95% CI, 0.088-0.689), respectively). Agreement between the paired radiologists was moderate for all images (ICC, 0.600 (95% CI, 0.487-0.693)) and for their assessment of uterine images (ICC, 0.538 (95% CI, 0.311-0.707)) and ovarian images (ICC, 0.621 (95% CI, 0.483-0.728)). Weak-to-moderate intraobserver agreement was seen for each of the raters with weighted Cohen's kappa ranging from 0.533 to 0.718 for all images and from 0.467 to 0.751 for ovarian images. Similarly, for all raters, the ICC indicated moderate-to-good intraobserver agreement for all images overall (ICC ranged from 0.636 to 0.825) and for ovarian images (ICC ranged from 0.596 to 0.862). Slightly better intraobserver agreement was seen for uterine images, with weighted Cohen's kappa ranging from 0.568 to 0.808 indicating weak-to-strong agreement, and ICC ranging from 0.546 to 0.893 indicating moderate-to-good agreement. All measures were statistically significant (P < 0.001).

CONCLUSION

The proposed image quality scoring system was shown to have poor-to-moderate interobserver agreement and mostly weak-to-moderate levels of intraobserver agreement. More refinement of the scoring system may be needed to improve agreement, although it remains unclear whether quantification of image quality can be achieved, given the highly subjective nature of ultrasound interpretation. Although some AI systems can tolerate labeling noise, most will favor clean (high-quality) data. As such, innovative data-labeling strategies are needed. © 2025 The Author(s). Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

The Left Outflow Tract in Fetal Cardiac Screening Examination: Introduction of Quality Criteria Is Not Always Associated With an Improvement of Practice When Supervised by Humans

OBJECTIVES

Since 2016, the French CNEOF included the left ventricular outflow tract (LVOT) in the second and third trimester of pregnancy in addition to the four-chamber view and the parasagittal view of the right outflow tract. The objective of this study was to define quality criteria for fetal LVOT assessment and to perform a human audit of past and current practices, before and after the implementation of those quality criteria at a large scale.

METHODS

Seven quality criteria were investigated and rated from 0 to 1 during three periods of interest. Files were randomly selected from three centers, and average total and specific scores were calculated.

RESULTS

LVOT pictures were present in more than 94.3% of reports. The average quality score was 5.49/7 (95% confidence interval [CI]: 5.36-5.62), 5.91/7 (95% CI: 5.80-6.03), and 5.70/7 (95% CI: 5.58-5.82) for the three centers in the three periods of interest. There was no significant difference following the introduction of the quality criteria, 2017 versus 2020, P = .054.

CONCLUSION

Fetal LVOT images were present in most of ultrasound reports but the introduction of the proposed quality criteria under human supervision seems not associated with a significant change in practice.

Towards automated extraction of 2D standard fetal head planes from 3D ultrasound acquisitions: A clinical evaluation and quality assessment comparison

INTRODUCTION

Clinical evaluation of deep learning (DL) tools is essential to compliment technical accuracy metrics. This study assessed the image quality of standard fetal head planes automatically-extracted from three-dimensional (3D) ultrasound fetal head volumes using a customised DL-algorithm.

METHODS

Two observers retrospectively reviewed standard fetal head planes against pre-defined image quality criteria. Forty-eight images (29 transventricular, 19 transcerebellar) were selected from 91 transabdominal fetal scans (mean gestational age = 26 completed weeks, range = 20+5-32+3 weeks). Each had two-dimensional (2D) manually-acquired (2D-MA), 3D operator-selected (3D-OS) and 3D-DL automatically-acquired (3D-DL) images. The proportion of adequate images from each plane and modality, and the number of inadequate images per plane was compared for each method. Inter and intra-observer agreement of overall image quality was calculated.

RESULTS

Sixty-seven percent of 3D-OS and 3D-DL transventricular planes were adequate quality. Forty-five percent of 3D-OS and 55% of 3D-DL transcerebellar planes were adequate. Seventy-one percent of 3D-OS and 86% of 3D-DL transventricular planes failed with poor visualisation of intra-cranial structures. Eighty-six percent of 3D-OS and 80% of 3D-DL transcerebellar planes failed due to inadequate visualisation of cerebellar hemispheres. Image quality was significantly different between 2D and 3D, however, no significant difference between 3D-modalities was demonstrated (p < 0.005). Inter-observer agreement of transventricular plane adequacy was moderate for both 3D-modalities, and weak for transcerebellar planes.

CONCLUSION

The 3D-DL algorithm can automatically extract standard fetal head planes from 3D-head volumes of comparable quality to operator-selected planes. Image quality in 3D is inferior to corresponding 2D planes, likely due to limitations with 3D-technology and acquisition technique.

IMPLICATIONS FOR PRACTICE

Automated image extraction of standard planes from US-volumes could facilitate use of 3DUS in clinical practice, however image quality is dependent on the volume acquisition technique.

Physician Barriers and Facilitators for Screening for Congenital Heart Disease With Routine Obstetric Ultrasound: A National United States Survey

OBJECTIVES

Prenatal detection of congenital heart disease with obstetric screening remains at less than 50% in most population studies, far from what is thought to be achievable. We sought to identify barriers/facilitators for screening from the perspective of interpreting physicians and to understand how these barriers/facilitators may be associated with interpretation of screening images.

METHODS

Our mixed-methods studies included 4 focus groups in centers across the United States with obstetric, maternal-fetal medicine, and radiology providers who interpreted obstetric ultrasound studies. Themes around barriers/facilitators for fetal heart screening were coded from transcripts. A national Web-based survey was then conducted, which quantitatively measured reported barriers/facilitators and measured physicians' ability to interpret fetal heart-screening images. Multivariable generalized linear random-effect models assessed the association between barriers/facilitators and the accuracy of image interpretation at the image level.

RESULTS

Three main themes were identified in the focus groups: intrinsic barriers (ie, comfort with screening), external barriers (ie, lack of feedback), and organizational barriers (ie, study volumes). Among 190 physician respondents, 104 interpreted ultrasound studies. Perceptions of barriers varied by practice setting, with nontertiary providers having lower self-efficacy and perceived usefulness of cardiac screening. Facilitators associated with the odds of accurate interpretation of screening images were knowledge (odds ratio, 2.54; P = .002) and the volume of scans per week (odds ratio, 1.01 for every additional scan; P = .04).

CONCLUSIONS

Some of the main barriers to cardiac screening identified and prioritized by physicians across the United States were knowledge of screening and minimal volumes of scans. Targeting these barriers will aid in improving prenatal detection of congenital heart disease.

Barriers to Sonographer Screening for Fetal Heart Defects: A U.S. National Survey

OBJECTIVE

We surveyed obstetric sonographers, who are at the forefront of the screening process to determine how barriers to prenatal cardiac screening impacted screening abilities.

METHODS

We performed a cross-sectional national survey of obstetric sonographers in the United States using a sampling frame from American Registry of Diagnostic Medical Sonography mailing lists. The web survey measured the ability to obtain and interpret fetal heart images. Several cognitive, sociodemographic, and system-level factors were measured, including intention to perform cardiac imaging. Regression and mediation analyses determined factors associated with intention to perform and ability to obtain and interpret cardiac images. Subgroup analyses of sonographers in tertiary versus nontertiary centers were also performed.

RESULTS

Survey response rate either due to noncontact or nonresponse was 40%. Of 480 eligible sonographers, ~30% practiced in tertiary settings. Sonographers had lower intention to perform outflow views compared to 4 chambers. Higher self-efficacy and professional expectations predicted higher odds of intention to perform outflow views (OR 2.8, 95% CI 1.9-4.2 and 1.9, 95% CI 1.1-3.0, respectively). Overall accuracy of image interpretation was 65% (±14%). For the overall cohort and nontertiary subgroup, higher intention to perform outflows was associated with increased accuracy in overall image interpretation. For the tertiary subgroup, self-efficacy and feedback were strongly associated with accuracy.

CONCLUSIONS

We identified several modifiable (some heretofore unrecognized) targets to improve prenatal cardiac screening. Priorities identified by sonographers that are associated with screening success should guide future interventions.

An Audit of Second-Trimester Fetal Anomaly Scans Based on a Novel Image-Scoring Method in the Southwest Region of the Netherlands

OBJECTIVES

Since 2007 the second-trimester fetal anomaly scan is offered to all pregnant women as part of the national prenatal screening program in the Netherlands. Dutch population-based screening programs generally have a well-described system to achieve quality assurance. Because of the absence of a uniform system to monitor the actual performance of the fetal anomaly scan in 2012, we developed a standardized image-scoring method. The aim of this study was to evaluate the scanning performance of all sonographers in the southwestern region of the Netherlands using this image-scoring method.

METHODS

Each sonographer was requested to set up a digital portfolio. A portfolio consists of five logbooks from five different pregnant women, each containing 25 fetal anatomical structures and six biometric measures of randomly selected fetal anomaly scans.

RESULTS

During the study period, 425 logbooks of 85 sonographers were assessed as part of the audit process. Seventy-three out of 85 sonographers (86%) met the criteria in the primary audit, and 12 sonographers required individual hands-on training. A successful assessment was achieved for 11 sonographers in the re-audit and one sonographer ceased her contract. Moreover, 2.1% of the required images were not digitally stored and therefore could not be reviewed.

CONCLUSIONS

Quality assessment using the image-scoring method demonstrated that most of the sonographers met the expectations of the audit process, but those who had subpar performance met the expectations after retraining.

Sonographer-Identified Barriers and Facilitators to Prenatal Screening for Congenital Heart Disease

Background: Fetal ultrasound screening for congenital heart disease has high detection rates in efficacy studies but not in practice. We investigated sonographers’ perspectives on barriers to screening and their association with intermediate measures of cardiac imaging. Methods: Barriers to fetal heart screening were identified in focus groups with obstetric sonographers. A web-based survey measured reported barriers and sonographers’ ability to interpret fetal heart screening images. Generalized linear random effect models assessed associations between barriers and image interpretation. Results: Four themes were identified: intrinsic barriers (self-efficacy), external barriers (time), informational needs, and provider and patient expectations. Among 224 sonographers surveyed, an average of 77.6% of fetal heart images were interpreted correctly. Perceptions about self-efficacy ( P = .03) and ease of performing imaging ( P = .01) were associated with accuracy of image interpretation. Conclusions: Several sonographer-identified barriers associated with image interpretation may be novel targets for improving prenatal detection of congenital heart disease.

The effect of preliminary training on quantitative evaluation of sonographer performance in the fetal morphology ultrasound examination

Introduction: The aim of this study is to provide a quantitative scoring system to assess sonographer performance by reviewing images from the fetal morphology examination.

Methods: Ten ultrasound images from patients at 18–22 weeks gestation were assessed and scored for quality according to predefined criteria. One hundred normal cases were randomly selected and 10 images from each case were analysed by four experienced reviewers. The preliminary training incorporated the first 25 cases and involved a training period for reviewers; the remaining 75 cases were allocated to post training. The scores acquired by each reviewer were statistically analysed using Pearson's and intra‐class correlations to determine the reproducibility of the results.

Results: The preliminary training results were calculated separately and compared to the post training study. The preliminary intra‐class correlation coefficient was 0.12. In the post training study the intra‐class correlation coefficient was doubled at 0.24. The greatest correlation was observed between reviewers 1 and 4 with a coefficient of 0.71. Reviewers 3 and 4 demonstrated the lowest correlation coefficient of 0.30.

Discussion: A significant increase in the intra‐class correlation coefficient indicated that training reviewers achieves more reproducible results. Suggested improvements to the study include recording fetal position, maternal BMI and assessing individual reviewer variability. An instruction manual defining each criterion might also yield better results.

Conclusion: The quantitative method used in this study assessed ultrasound images by placing a numerical value on image quality. Analysis of the preliminary training period demonstrates improved reproducibility of the results. Further investigation into the criteria is necessary to refine the quantitative method.

Online audit and feedback improve fetal second-trimester four-chamber view images: a randomised controlled trial

OBJECTIVES

We aimed to study the impact of online audit and feedback on fetal four-chamber view image quality.

METHODS

Ultrasonographers uploaded a set of four-chamber views from 10 consecutive screening examinations and a second set 3 months later. They were randomised to receive (group A) or not (group B) a feedback for their first set. The primary outcome was the difference in image mean score and in percentage of inadequate images between the first set and the second set, comparing the groups with and without feedback.

RESULTS

There were 258 ultrasonographers who completed the trial (group A: 122; group B: 136), and 5160 images were audited. In both groups, the mean score increased. In group A, it rose from 17.0 to 18.5 (p < 0.0001), and in group, B from 17.6 to 18.3 (p < 0.0001). The improvement was greater in group A than in group B (1.5 vs. 0.7, p = 0.0007). The mean percentage of inadequate images decreased in both groups. In group A, it dropped from 32% to 19% (p < 0.0001), and in group B, from 26% to 21% (p = 0.012). The decrease was greater in group A than in group B (13% vs. 5%, p = 0.007).

CONCLUSIONS

Image quality improved following online audit, the improvement being slightly greater with feedback.

A review of fetal volumetry: the need for standardization and definitions in measurement methodology

Volume charts of fetal organs and structures vary considerably among studies. This review identified 42 studies reporting normal volumes, namely for fetal brain (n = 3), cerebellum (n = 4), liver (n = 6), femur (n = 2), lungs (n = 15), kidneys (n = 3) and first-trimester embryo (n = 9). The differences among median volumes were expressed both in percentage form and as standard deviation scores. Wide discrepancies in reported normal volumes make it extremely difficult to diagnose pathological organ growth reliably. Given its magnitude, this variation is likely to be due to inconsistencies in volumetric methodology, rather than population differences. Complicating factors include the absence of clearly defined anatomical landmarks for measurement; inadequate assessment and reporting of method repeatability; the inherent difficulty in validating fetal measurements in vivo against a reference standard; and a multitude of mutually incompatible three-dimensional (3D) imaging formats and software measuring tools. An attempt to standardize these factors would improve intra- and inter-researcher agreement concerning reported volumetric measures, would allow generalization of reference data across different populations and different ultrasound systems, and would allow quality assurance in 3D fetal biometry. Failure to ensure a quality control process may hamper the wide use of 3D ultrasound.

Online evaluation of fetal second-trimester four-chamber view images: a comparison of six evaluation methods

OBJECTIVE

To compare six online evaluation methods for auditing routine second-trimester four-chamber view still images.

METHODS

We evaluated three different scoring grids (subjective, five-item score and seven-item score), which were applied with or without access to online help, resulting in a total of six evaluation methods. For the subjective scoring grid, images were rated as excellent, good, fair, poor or very poor. For the five-item score, 1 point was allocated for visualization (vs non-visualization or non-evaluable) of each of: heart crux, atria, ventricles, apex and aorta, yielding a score of 0-5. For the seven-item score, 1 point was allocated for clear (vs unclear) visualization of each of: moderator band at the apex, interventricular septum, atrioventricular valves, non-linear insertion of atrioventricular valves (normal offset), septum primum, aorta and pulmonary vein. Each evaluation method was used via the Internet by three randomly selected reviewers, who evaluated the same set of 80 images. Reviewers were experienced in fetal ultrasound, but were not involved in the design of the study. Interrater agreement was the main outcome.

RESULTS

The five-item scoring grid with online help achieved the best interrater agreement (interrater intraclass correlation coefficient = 0.7).

CONCLUSIONS

Evaluation of the second-trimester sonographic four-chamber view is apparently best achieved with a simple five-item scoring grid.

Perinatal diagnosis of heart defects

IMPORTANCE OF THE FIELD

Heart defects are the most frequent congenital malformations, affecting 8 in every 1000 neonates. As 90% of infants born with a congenital heart defects (CHD) have no known risk factor, screening for heart defects in every routine obstetric ultrasound is essential for antenatal diagnosis.

AREAS COVERED IN THIS REVIEW

Two-dimensional echocardiography allied with three- and four-dimensional technologies such as spatio-temporal image correlation offer means for thoroughly evaluating fetal cardiac anatomy.

WHAT THE READER WILL GAIN

This article reviews the most relevant aspects of antenatal screening and diagnosis of CHD using sonography and echocardiography. It includes first trimester diagnosis and three- and four-dimensional techniques.

TAKE HOME MESSAGE

These antenatal diagnostic tools play an important role in optimization of obstetric care and neonatal management, thus potentially reducing perinatal mortality.