Interobserver agreement in describing adnexal masses using the International Ovarian Tumor Analysis simple rules in a real-time setting and using three-dimensional ultrasound volumes and digital clips

Subjective Assessment of Adnexal Masses Using Various Ultrasonographic Diagnostic Models: An Analysis of Interobserver Variability

OBJECTIVE

To investigate the interobserver agreement of the International Ovarian Tumour Analysis (IOTA) ultrasound-based simple rules risk (SRRisk) score, the logistic regression model 2 (LR2), the Assessment of Different NEoplasias in the adneXa (ADNEX) model and the Ovarian-Adnexal Reporting and Data System (O-RADS) in an Australian, population-based context.

METHODS

A retrospective multi-centre study was performed between January 2020 and January 2021. The study included 198 women with adnexal masses examined with transvaginal grey scale and power Doppler ultrasound. Participants were recruited from the multidisciplinary oncology meetings (MDT) of two tertiary cancer centres. Two independent radiologists described the adnexal masses according to the SRR, LR2 scores, ADNEX model, and O-RADS. Values > 30 units different were considered differential and > 50 units were considered highly differential.

RESULTS

From 198 patients, 128 were diagnosed with benign ovarian masses, 53 with malignant and 17 patients with borderline tumours. There was strong agreement (Cohen's kappa 0.8) for intra-tumour blood flow, number of cysts locules, and presence of blood flow within solid projections. Interobserver agreement was moderate (Cohen's kappa 0.60-0.79) for the presence of free pelvic fluid/ascites, solid components, unilocular cysts and acoustic shadows. Of the 198 cases, 10 (5%) cases were highly differential and (38/198) 19% were differential for SRRisk, (20/198) 10% highly differential and (36/198) 18% differential for LR2, and (10/198) 5% and (24/198) 12% for ADNEXA model, respectively. Comparison of O-RADS scores between the two observers showed a moderate agreement with a kappa of 0.65. In 7/198 (4%) cases, the difference between observers was for 2 or more categories when using the O-RADS score.

CONCLUSIONS

Our results suggested that interobserver variation was present in evaluating adnexal masses using well established ultrasonographic diagnostic models. Implementation of sonographic ovarian cancer risk prediction models will need to consider this issue and ensure examiners have adequate training in the technique, and standard operating procedures are in place to reduce interobserver variability.

Ultrasonographic diagnosis of adnexal masses: interobserver agreement in the interpretation of videos, using IOTA terminology

OBJECTIVES

Aim of this study is to estimate interobserver agreement in classifying adnexal tumors using IOTA terms, simple rules and subjective assessment. In addition, we related observers' accuracy with their experience in gynecological ultrasonography and the year of IOTA certification.

METHODS

Eleven observers with three different levels of experience evaluated videoclips of 70 adnexal masses, defining tumor type according to IOTA terms and definitions, classifying the mass using IOTA Simple rules and Subjective assessment as well as providing Color Score evaluation. Sensitivity, specificity and area under the ROC curve were calculated and the year of IOTA certification was related with operators' accuracy through Pearson correlation coefficient. Interobserver agreement was estimated calculating percentage of agreement, Fleiss kappa and Cohen's kappa.

RESULTS

We found a positive correlation between the year of IOTA certification and operators' accuracy (Pearson coefficient 0.694), especially among the observers with the least experience, the residents (p = 0.003). For tumor type classification, identification of papillary projections and classification of tumors using subjective assessment, agreement among all observers was moderate (Fleiss kappa 0.455, 0.552, and 0.476, respectively) and increased with the years of experience. Agreement in the application of Simple Rules was moderate in all examiners with IOTA certification, with Fleiss kappa in the range of (0.403, 0.498). For Color Score assignment interobserver agreement among all observers was fair (Cohen's kappa 0.380).

CONCLUSIONS

Even among expert examiners, the results of adnexal lesion assessment can be inconsistent. Experience impacts on accuracy and agreement in subjective assessment, while the application of Simple Rules can mitigate the role of experience in interobserver agreement. The knowledge of IOTA models among residents seams to improve their diagnostic accuracy, showing the benefits of IOTA terminology for in training sonographers.

Diagnostic performance of IOTA SR and O-RADS combined with CA125, HE4, and risk of malignancy algorithm to distinguish benign and malignant adnexal masses

PURPOSE

To compare the diagnostic performance of International Ovarian Tumour Analysis Simple Rules (IOTA SR) and Ovarian-Adnexal Reporting and Data System (O-RADS), and to analyse whether combining IOTA SR and O-RADS with the biomarkers cancer antigen 125 (CA125), human epididymis protein 4 (HE4), and risk of malignancy algorithm (ROMA) further improves diagnostic performance in women with different menopause status.

METHODS

This study retrospectively included patients with ovarian adnexal masses confirmed by surgical pathology between September 2021 and February 2022. The area under the curve (AUC), sensitivity, and specificity were calculated to evaluate the diagnostic efficacy of IOTA SR, O-RADS, and their combination with CA125, HE4, and ROMA.

RESULTS

This study included 1,179 ovarian adnexal masses. In all women, the AUC of IOTA SR was comparable to O-RADS (0.879 vs. 0.889, P = 0.361), and O-RADS had a significantly higher sensitivity than IOTA SR (95.77 % vs. 87.32 %, P < 0.001). In premenopausal women, O-RADS had a significantly higher AUC than other diagnostic strategies (all P < 0.05), and the sensitivity, specificity, and accuracy were 93.33 %, 84.74 %, and 85.59 %, respectively. In postmenopausal women, IOTA SR + ROMA had a significantly higher AUC than other diagnostic strategies (all P < 0.05), and the sensitivity, specificity, and accuracy were 85.37 %, 93.88 %, and 90.00 %, respectively.

CONCLUSIONS

Our study supports the high diagnostic value of IOTA SR or O-RADS alone in all women, and O-RADS was more sensitive than IOTA SR. In premenopausal women, O-RADS had the highest diagnostic value. In postmenopausal women, IOTA SR outperformed O-RADS, and IOTA SR + ROMA had the highest diagnostic value.

Ultrasound examination, MRI, or ROMA for discriminating between inconclusive adnexal masses as determined by IOTA Simple Rules: a prospective study

OBJECTIVE

To determine the best second-step approach for discriminating benign from malignant adnexal masses classified as inconclusive by International Ovarian Tumour Analysis Simple Rules (IOTA-SR).

METHODS

Single-center prospective study comprising a consecutive series of patients diagnosed as having an adnexal mass classified as inconclusive according to IOTA-SR. All women underwent Risk of Ovarian Malignancy Algorithm (ROMA) analysis, MRI interpreted by a radiologist, and ultrasound examination by a gynecological sonologist. Cases were clinically managed according to the result of the ultrasound expert examination by either serial follow-up for at least 1 year or surgery. Reference standard was histology (patient was submitted to surgery if any of the tests was suspicious) or follow-up (masses with no signs of malignancy after 12 months were considered benign). Diagnostic performance of all three approaches was calculated and compared. Direct cost analysis of the test used was also performed.

RESULTS

Eighty-two adnexal masses in 80 women (median age 47.6 years, range 16 to 73 years) were included. Seventeen patients (17 masses) were managed expectantly (none had diagnosis of ovarian cancer after at least 12 months of follow-up) and 63 patients (65 masses) underwent surgery and tumor removal (40 benign and 25 malignant tumors). Sensitivity and specificity for ultrasound, MRI, and ROMA were 96% and 93%, 100% and 81%, and 24% and 93%, respectively. The specificity of ultrasound was better than that for MRI (p=0.021), and the sensitivity of ultrasound was better than that for ROMA (p<0.001), sensitivity was better for MRI than for ROMA (p<0.001) and the specificity of ROMA was better than that for MRI (p<0.001). Ultrasound evaluation was the most effective and least costly method as compared with MRI and ROMA.

CONCLUSION

In this study, ultrasound examination was the best second-step approach in inconclusive adnexal masses as determined by IOTA-SR, but the findings require confirmation in multicenter prospective trials.

Learning Curve for Ultrasound Assessment of Myometrial Infiltration in Endometrial Cancer Visualizing Videoclips: Potential Implications for Training

BACKGROUND

Diagnostic accuracy for estimating myometrial infiltration by ultrasound in endometrial cancer requires experience. The objective of this study is to determine the learning curve (LC) for assessing myometrial infiltration in cases of endometrial cancer using transvaginal ultrasound (TVS).

METHODS

Five trainees (one staff radiologist and four fourth-year OB/GYN residents) participated in this study. All trainees had experience in performing TVS, but none of them had specific training on the assessment of myometrial infiltration. Trainees were given one specific lecture about the topic, and then they observed videoclips from 10 cases explained by the trainer. After this, all trainees visualized 45 videoclips of uterine ultrasound scans of endometrial cancer cases. The assessment of myometrial infiltration was based on the subjective impression. Definitive histology was used as a reference standard. Trainees stated whether myometrial infiltration was ≥50% or <50%. LC-CUSUM and standard CUSUM graphics were plotted to determine how many cases were needed to reach competence, allowing a mistake rate of 15%.

RESULTS

All trainees completed the study. LC-CUSUM graphics showed that three trainees reached competence at the 33rd, 35th and 36th case, respectively. All three of them kept the process under control after reaching competence. One trainee reached competence but did not maintain it in the cumulative analysis. One trainee did not reach competence.

CONCLUSION

Our study suggests that 30-40 cases would be needed to be trained for assessing myometrial infiltration by TVS by visual interpretation of videoclips by most trainees.

Performance of IOTA Simple Rules, Simple Rules risk assessment, ADNEX model and O-RADS in differentiating between benign and malignant adnexal lesions in North American women

OBJECTIVES

To apply the International Ovarian Tumor Analysis (IOTA) Simple Rules (SR), the IOTA Simple Rules risk assessment (SRR), the IOTA Assessment of Different NEoplasias in the adneXa (ADNEX) model and the Ovarian-Adnexal Reporting and Data System (O-RADS) in the same cohort of North American patients and to compare their performance in preoperative discrimination between benign and malignant adnexal lesions.

METHODS

This was a single-center diagnostic accuracy study, performed between March 2018 and February 2021, which included 150 women with an adnexal lesion. Using the ADNEX model, lesions were classified prospectively, whereas the SR, SRR assessment and O-RADS were applied retrospectively. Surgery with histological analysis was performed within 6 months of the ultrasound exam. Sensitivity and specificity were determined for each testing modality and the performance of the different modalities was compared.

RESULTS

Of the 150 women, 110 (73.3%) had a benign ovarian tumor and 40 (26.7%) had a malignant tumor. The mean risk of malignancy generated by the ADNEX model without CA 125 was significantly higher in malignant vs benign lesions (63.3% vs 11.8%) and the area under the receiver-operating-characteristics curve (AUC) of the ADNEX model for differentiating between benign and malignant adnexal masses at the time of ultrasound examination was 0.937. The mean risk of malignancy generated by SRR assessment was also significantly higher in malignant vs benign lesions (74.1% vs 15.9%) and the AUC was 0.941. To compare the ADNEX model, SRR assessment and O-RADS, the malignancy risk threshold was set at ≥ 10%. This cut-off differentiates O-RADS low-risk categories (Category ≤ 3) from intermediate-to-high-risk categories (Categories 4 and 5). At this cut-off, the sensitivity of the ADNEX model was 97.5% (95% CI, 85.3%-99.9%) and the specificity was 63.6% (95% CI, 53.9%-72.4%), and, for the SRR model, the sensitivity was 100% (95% CI, 89.1%-100%) and the specificity was 51.8% (95% CI, 42.1%-61.4%). In the 113 cases to which the SR could be applied, the sensitivity was 100% (95% CI, 81.5%-100%) and the specificity was 95.6% (95% CI, 88.5%-98.6%). If the remaining 37 cases, which were inconclusive under SR, were designated 'malignant', the sensitivity remained at 100% but the specificity was reduced to 79.1% (95% CI, 70.1%-86.0%). The 150 cases fell into the following O-RADS categories: 17 (11.3%) lesions in Category 2, 34 (22.7%) in Category 3, 66 (44.0%) in Category 4 and 33 (22.0%) in Category 5. There were no histologically proven malignant lesions in Category 2 or 3. There were 14 malignant lesions in Category 4 and 26 in Category 5. The sensitivity of O-RADS using a malignancy risk threshold of ≥ 10% was 100% (95% CI, 89.1%-100.0%) and the specificity was 46.4% (95% CI, 36.9%-56.1%).

CONCLUSIONS

When IOTA terms and techniques are used, the performance of IOTA models in a North American patient population is in line with published IOTA results in other populations. The IOTA SR, SRR assessment and ADNEX model and O-RADS have similar sensitivity in the preoperative discrimination of malignant from benign pelvic tumors; however, the IOTA models have higher specificity and the algorithm does not require the use of magnetic resonance imaging. © 2022 International Society of Ultrasound in Obstetrics and Gynecology.

The learning curve and difficult points of the O-RADS ultrasound risk stratification system in 54 trainees

Purpose

This study aimed to evaluate the learning curve and explore the difficult points of the Ovarian-Adnexal Reporting and Data System (O-RADS) ultrasound risk stratification system.

Methods

One hundred adnexal masses (AMs) were randomly selected for five tests as training data. Two experienced trainers had an inter-rater agreement of 0.95 for the O-RADS scores. Fifty-four trainees (26 level I practitioners [group 1], 17 level II practitioners [group 2], and 11 experienced level II practitioners [group 3]) attended the training. Every trainee received assessment and feedback after 20 scored cases. The outcomes of the five tests were compared among the three groups using repeated-measurements analysis of variance.

Results

Of the 100 AMs, 52 were pathologically benign and 48 were malignant; the O-RADS scores were 2, 3, 4, and 5 in 22, 11, 48, and 19 AMs, respectively. The between-subjects effects test showed no significant differences between groups 1, 2, and 3 for the five tests (P=0.501). For each group, the differences among the five tests were significant (P<0.001, P=0.006, and P=0.044 for groups 1, 2, and 3, respectively). Test 2 was the worst. In 23 cases, more than 40% of trainees gave incorrect answers, which mainly related to classic benign lesions, the color flow score, and solid-appearing masses.

Conclusion

After training, junior doctors at different levels can reach a coincident O-RADS ultrasound risk stratification. The difficulties primarily related to subjective judgments of classic benign lesions, the color flow score, and solid-appearing masses. More experience is needed to improve the applicability of the system.

Comparison of O-RADS, GI-RADS, and IOTA simple rules regarding malignancy rate, validity, and reliability for diagnosis of adnexal masses

OBJECTIVE

The American College of Radiology (ACR) recently published the ovarian-adnexal reporting and data system (O-RADS) to provide guidelines to physicians who interpret ultrasound (US) examinations of adnexal masses (AM). This study aimed to compare the O-RADS with two other well-established US classification systems for diagnosis of AM.

METHODS

This retrospective multicenter study between May 2016 and December 2019 assessed consecutive women with AM detected by the US. Five experienced consultant radiologists independently categorized each AM according to O-RADS, gynecologic imaging reporting and data system (GI-RADS), and international ovarian tumor analysis (IOTA) simple rules. Pathology and adequate follow-up were used as reference standards for calculating the validity of three US classification systems for diagnosis of AM. Kappa statistics were used to assess the inter-reviewer agreement (IRA).

RESULTS

A total of 609 women (mean age, 48 ± 13.7 years; range, 18-72 years) with 647 AM were included. Of the 647 AM, 178 were malignant and 469 were benign. Malignancy rates were comparable to recommended rates by previous literature in O-RADS and IOTA, but higher in GI-RADS. O-RADS had significantly higher sensitivity for malignancy than GI-RAD and IOTA (p = 0.003 and 0.0007, respectively), but non-significant slightly lower specificity (p > 0.05). O-RADS, GI-RADS, and IOTA showed similar overall IRA (κ = 0.77, 0.69, and 0.63, respectively) with a tendency toward higher IRA with O-RADS than with GI-RADS and IOTA.

CONCLUSIONS

O-RADS compares favorably with GI-RADS and IOTA. O-RADS had higher sensitivity than GI-RADS and IOTA simple rules with relatively similar specificity and reliability.

KEY POINTS

• The malignancy rates were comparable to recommended rates by previous literature in O-RADS and IOTA, but higher in GI-RADS. • The O-RADS had significantly higher sensitivity for malignancy than GI-RADS and IOTA (96.8% vs 92.7% and 92.1%; p = 0.003 and 0.0007, respectively), but non-significant slightly lower specificity (92.8% vs 93.6% and 93.2%, respectively; p > 0.05). • The O-RADS, GI-RADS, and IOTA showed similar overall inter-reviewer agreement (IRA) (κ = 0.77, 0.69, and 0.63, respectively), with a tendency toward higher IRA with O-RADS than with GI-RADS and IOTA.

Reproducibility of placental maturity grade classification using a dynamic ultrasonography

OBJECTIVE

To evaluate the agreement between ultrasonographers in classifying placental maturity grade using examination videos.

METHODS

A prospective, cross-sectional study was conducted using 25 videos of ultrasonographic examinations of single, healthy pregnancies ≥ 34 + 0 weeks. The Grannum (grades 0, I, II and III) and Julio (grades 0, 1, 2A, 2B and 3) classifications were used for placental maturity grade. The videos were evaluated on two occasions by 10 experienced ultrasonographers. Intra- and inter-observer agreements were evaluated using the Kappa (k) coefficient of agreement.

RESULTS

According to the Grannum classification, the inter-observer agreement coefficients were good for grades III and 0 (k = 0.69 and 0.738, respectively) and moderate for grades I and II (k = 0.401 and 0.593, respectively) placentas. According to the Julio classification, the inter-observer agreement coefficients were good for grades 0 and 3 (k = 0.738 and 0.651, respectively), moderate for grades 1 and 2A (0.401 and 0.413, respectively) and poor for grade 2B (k = 0.161). Intra-observer coefficients for the Grannum and Julio classifications ranged from 0.567 to 0.890 and from 0.446 to 0.790, respectively.

CONCLUSION

Placental maturity grading using the Grannum and Julio classifications demonstrated moderate/good intra- and inter-observer agreements on examination videos evaluation.

Interobserver agreement in describing the ultrasound appearance of adnexal masses and in calculating the risk of malignancy using logistic regression models

PURPOSE

To estimate interobserver agreement with regard to describing adnexal masses using the International Ovarian Tumor Analysis (IOTA) terminology and the risk of malignancy calculated using IOTA logistic regression models LR1 and LR2, and to elucidate what explained the largest interobserver differences in calculated risk of malignancy.

EXPERIMENTAL DESIGN

One hundred and seventeen women with adnexal masses were examined with transvaginal gray scale and power Doppler ultrasound by two independent experienced sonologists who described the masses using IOTA terminology. The risk of malignancy was calculated using LR1 and LR2. A predetermined risk of malignancy cutoff of 10% indicated malignancy.

RESULTS

There were 94 benign, four borderline, and 19 invasively malignant tumors. There was substantial variability between the two sonologists in measurement results and some variability in assessment of categorical variables (agreement 40%-98%, Kappa 0.30-0.91). Interobserver agreement when classifying tumors as benign or malignant was 84% (98/117), Kappa 0.68 for LR1, and for LR2 85% (99/117), Kappa 0.68. When using LR1 and LR2, the interobserver difference in calculated risk was ≥ 25 percentage units in 9% (11/117) and 12% (14/117) of tumors, respectively. Differences in assessment of wall irregularity, acoustic shadowing, color score, and color flow in papillary projections explained most of these largest differences.

CONCLUSIONS

Interobserver agreement in classifying tumors as benign or malignant using the risk of malignancy cutoff of 10% for LR1 and LR2 was good. However, because risk estimates may differ substantially between sonologists, one should be cautious with using the risk value for counseling patients about their individual risk.

Use of IOTA simple rules for diagnosis of ovarian cancer: meta-analysis

OBJECTIVES

To present data on prospective evaluation of the International Ovarian Tumor Analysis (IOTA) 'simple-rules' tool for the diagnosis of ovarian cancer and to perform a meta-analysis of studies that utilized the same diagnostic method.

METHODS

In the present study a level-II ultrasound operator systematically assessed the tumors of women with an ultrasound diagnosis of adnexal tumor(s) according to the IOTA simple-rules protocol to determine the risk of the tumor being malignant. The results of simple rules were compared with the 'pattern recognition' method and with histological findings. This validation study was included in the subsequent meta-analysis, for which we searched MEDLINE, EMBASE and Cochrane from the publication of the first study in 2008. The terms used were 'simple rules', 'simple rules ovarian', 'ovar tumor' and 'ultrasound'. Quality assessment was performed using the modified Quality Assessment of the Diagnostic Accuracy of Studies (QUADAS-2) checklist. Random effects meta-analysis was used to calculate pooled estimates of sensitivity and specificity for the simple-rules tool, and meta-regression was used to investigate heterogeneity across the studies.

RESULTS

Three hundred and three women were included in the validation study with 168 (55.4%) benign, 19 (6.3%) borderline and 116 (38.3%) malignant tumors on histological examination. The rules were applicable in 237 (78.2%) of the tumors and for these tumors, sensitivity was 96.2% (95% CI, 90.5-99.0%) and specificity was 88.6% (95% CI, 82.0-93.5%). Six of the 88 discovered studies were included in the meta-analysis along with the current validation study, which resulted in inclusion of a total of 3568 patients. When the meta-analysis was performed the pooled sensitivity (when the rules were applicable) was 93% (95% CI, 90-96%) (I(2)  = 32.1%) and the pooled specificity was 95% (95% CI, 93-97%) (I(2)  = 78.1%). Heterogeneity was observed across the studies. Sensitivity was higher and specificity lower in the study populations in which the prevalence of malignant tumors was greatest.

CONCLUSION

The simple rules protocol could be used in 76-89% of tumors and is an accurate test for the diagnosis of ovarian cancer. Assessment by an ultrasound expert is required when the protocol cannot be applied.