Intra- and interobserver agreement with regard to describing adnexal masses using International Ovarian Tumor Analysis terminology: reproducibility study involving seven observers

Ultrasonographic diagnosis of adnexal masses: interobserver agreement in the interpretation of videos, using IOTA terminology

OBJECTIVES

Aim of this study is to estimate interobserver agreement in classifying adnexal tumors using IOTA terms, simple rules and subjective assessment. In addition, we related observers' accuracy with their experience in gynecological ultrasonography and the year of IOTA certification.

METHODS

Eleven observers with three different levels of experience evaluated videoclips of 70 adnexal masses, defining tumor type according to IOTA terms and definitions, classifying the mass using IOTA Simple rules and Subjective assessment as well as providing Color Score evaluation. Sensitivity, specificity and area under the ROC curve were calculated and the year of IOTA certification was related with operators' accuracy through Pearson correlation coefficient. Interobserver agreement was estimated calculating percentage of agreement, Fleiss kappa and Cohen's kappa.

RESULTS

We found a positive correlation between the year of IOTA certification and operators' accuracy (Pearson coefficient 0.694), especially among the observers with the least experience, the residents (p = 0.003). For tumor type classification, identification of papillary projections and classification of tumors using subjective assessment, agreement among all observers was moderate (Fleiss kappa 0.455, 0.552, and 0.476, respectively) and increased with the years of experience. Agreement in the application of Simple Rules was moderate in all examiners with IOTA certification, with Fleiss kappa in the range of (0.403, 0.498). For Color Score assignment interobserver agreement among all observers was fair (Cohen's kappa 0.380).

CONCLUSIONS

Even among expert examiners, the results of adnexal lesion assessment can be inconsistent. Experience impacts on accuracy and agreement in subjective assessment, while the application of Simple Rules can mitigate the role of experience in interobserver agreement. The knowledge of IOTA models among residents seams to improve their diagnostic accuracy, showing the benefits of IOTA terminology for in training sonographers.

Current uptake and barriers to wider use of the International Ovarian Tumor Analysis (IOTA) models in Dutch gynaecological practice

OBJECTIVE

Correct referral of women with an ovarian tumor to an oncology department remains challenging. The International Ovarian Tumor Analysis (IOTA) consortium has developed models with higher diagnostic accuracy than the alternative Risk of Malignancy Index (RMI). This study explores the uptake of the IOTA models in Dutch hospitals and factors that impede or promote implementation. Optimal implementation is crucial to improve pre-operative classification of ovarian tumors, which may lead to better patient referral to the appropriate level of care.

STUDY DESIGN

In February 2021, an electronic questionnaire consisting of 37 questions was sent to all 72 hospitals in the Netherlands. One pre-selected gynaecologist per hospital was asked to respond on behalf of the department.

RESULTS

The study had a response rate of 93% (67/72 hospitals). All respondents (100%) were familiar with the IOTA models with 94% using them in practice. The logistic regression 2 (LR2)-model, Simple ultrasound-based rules (SR) and Assessment of Different NEoplasias in the adneXa (ADNEX) model were used in respectively 40%, 67% and 73% of these hospitals. Respondents rated the models overall with an 8.2 (SD 1.8), 8.3 (SD 1.6) and 8.9 (SD 1.3) respectively for LR2, SR and ADNEX on a scale from 1 to 10. Moreover, 89% indicated to have confidence in the results of the IOTA models. The most important factors to improve further implementation are more training (43%), research on sensitivity, specificity and cost-effectiveness in the Dutch health care system (27%), easier usability (24%) and more consultation time (19%).

CONCLUSION

The IOTA ultrasound models are adopted in the majority of Dutch hospitals with the ADNEX model being used the most. While Dutch gynecologists have a strong familiarity and confidence in the models, the uptake varies in reality. Areas that warrant improvement in the Dutch context are more uniformity, education and more research. These findings can be helpful for other countries considering adopting the IOTA models.

Comparison of O-RADS, GI-RADS, and IOTA simple rules regarding malignancy rate, validity, and reliability for diagnosis of adnexal masses

OBJECTIVE

The American College of Radiology (ACR) recently published the ovarian-adnexal reporting and data system (O-RADS) to provide guidelines to physicians who interpret ultrasound (US) examinations of adnexal masses (AM). This study aimed to compare the O-RADS with two other well-established US classification systems for diagnosis of AM.

METHODS

This retrospective multicenter study between May 2016 and December 2019 assessed consecutive women with AM detected by the US. Five experienced consultant radiologists independently categorized each AM according to O-RADS, gynecologic imaging reporting and data system (GI-RADS), and international ovarian tumor analysis (IOTA) simple rules. Pathology and adequate follow-up were used as reference standards for calculating the validity of three US classification systems for diagnosis of AM. Kappa statistics were used to assess the inter-reviewer agreement (IRA).

RESULTS

A total of 609 women (mean age, 48 ± 13.7 years; range, 18-72 years) with 647 AM were included. Of the 647 AM, 178 were malignant and 469 were benign. Malignancy rates were comparable to recommended rates by previous literature in O-RADS and IOTA, but higher in GI-RADS. O-RADS had significantly higher sensitivity for malignancy than GI-RAD and IOTA (p = 0.003 and 0.0007, respectively), but non-significant slightly lower specificity (p > 0.05). O-RADS, GI-RADS, and IOTA showed similar overall IRA (κ = 0.77, 0.69, and 0.63, respectively) with a tendency toward higher IRA with O-RADS than with GI-RADS and IOTA.

CONCLUSIONS

O-RADS compares favorably with GI-RADS and IOTA. O-RADS had higher sensitivity than GI-RADS and IOTA simple rules with relatively similar specificity and reliability.

KEY POINTS

• The malignancy rates were comparable to recommended rates by previous literature in O-RADS and IOTA, but higher in GI-RADS. • The O-RADS had significantly higher sensitivity for malignancy than GI-RADS and IOTA (96.8% vs 92.7% and 92.1%; p = 0.003 and 0.0007, respectively), but non-significant slightly lower specificity (92.8% vs 93.6% and 93.2%, respectively; p > 0.05). • The O-RADS, GI-RADS, and IOTA showed similar overall inter-reviewer agreement (IRA) (κ = 0.77, 0.69, and 0.63, respectively), with a tendency toward higher IRA with O-RADS than with GI-RADS and IOTA.

Recommendation Patterns among Gynaecologists and Radiologists for Adnexal Masses on Ultrasound

OBJECTIVE

Ultrasound is the primary modality used to evaluate adnexal lesions. Follow-up recommendations for ovarian cysts remain controversial between gynaecologists and radiologists. The objective of this study was to compare practice patterns for adnexal masses described on ultrasound on the basis of the interpreter's field of specialty.

METHODS

This study was conducted within the McGill University Hospital Network at two hospitals that differ in the department of interpretation of pelvic ultrasounds. In one hospital, all studies are reported by gynaecologists, and in the other, by radiologists. The study investigators collected data from pelvic ultrasounds of newly diagnosed ovarian lesions performed from May to June 2014. Multivariate logistic regression analyses were used to compare recommendation patterns between the two groups.

RESULTS

A total of 201 of 1110 pelvic ultrasound studies performed met our inclusion criteria. Gynaecologists interpreted 69 (34%) pelvic ultrasounds, and radiologists reported on 132 (66%). Reported adnexal mass types were not significantly different between the two groups. As compared with gynaecologists, radiologists were more likely to recommend MRI or CT scans (OR 11.76; 95% CI 1.17-117.78), as well as follow-up ultrasound studies (OR 4.67; 95% CI 1.66-13.1), and they were less likely to recommend no further imaging (OR 0.18; 95% CI 0.07-0.45). Groups did not differ in recommendation patterns for referral to a specialist.

CONCLUSION

There are significant differences in recommendation patterns between gynaecologists and radiologists in evaluating new adnexal masses on ultrasound. This difference can have important effects on resource use and patients' concerns.

Ultrasound-based logistic regression model LR2 versus magnetic resonance imaging for discriminating between benign and malignant adnexal masses: a prospective study

BACKGROUND

The diagnostic performances of the International Ovarian Tumor Analysis (IOTA) ultrasound-based logistic regression model (LR2) and magnetic resonance imaging (MRI) in discriminating between benign and malignant adnexal masses have not been directly compared in a single study.

METHODS

Using the IOTA LR2 model and subjective interpretation of MRI findings by experienced radiologists, 265 consecutive patients with adnexal masses were preoperatively evaluated in two hospitals between February 2014 and December 2015. Definitive histological diagnosis of excised tissues was used as a gold standard.

RESULTS

From the 265 study subjects, 54 (20.4%) tumors were histologically diagnosed as malignant (including 11 borderline and 3 metastatic tumors). Preoperative diagnoses of malignant tumors showed 91.7% total agreement between IOTA LR2 and MRI, with a kappa value of 0.77 [95% confidence interval (CI), 0.68-0.86]. Sensitivity of IOTA LR2 (0.94, 95% CI, 0.85-0.98) for predicting malignant tumors was similar to that of MRI (0.96, 95% CI, 0.87-0.99; P = 0.99), whereas specificity of IOTA LR2 (0.98, 95% CI, 0.95-0.99) was significantly higher than that of MRI (0.91, 95% CI, 0.87-0.95; P = 0.002). Combined IOTA LR2 and MRI results gave the greatest sensitivity (1.00, 95% CI, 0.93-1.00) and had similar specificity (0.91, 95% CI, 0.86-0.94) to MRI.

CONCLUSIONS

The IOTA LR2 model had a similar sensitivity to MRI for discriminating between benign and malignant tumors and a higher specificity compared with MRI. Our findings suggest that the IOTA LR2 model, either alone or in conjunction with MRI, should be included in preoperative evaluation of adnexal masses.

Practice Bulletin No. 174: Evaluation and Management of Adnexal Masses

Adnexal masses (ie, masses of the ovary, fallopian tube, or surrounding tissues) commonly are encountered by obstetrician-gynecologists and often present diagnostic and management dilemmas. Most adnexal masses are detected incidentally on physical examination or at the time of pelvic imaging. Less commonly, a mass may present with symptoms of acute or intermittent pain. Management decisions often are influenced by the age and family history of the patient. Although most adnexal masses are benign, the main goal of the diagnostic evaluation is to exclude malignancy. The purpose of this document is to provide guidelines for the evaluation and management of adnexal masses in adolescents, pregnant women, and nonpregnant women and to outline criteria for the identification of adnexal masses that are likely to be malignant and may warrant referral to or consultation with a gynecologic oncologist.

Ultrasound Monitoring of Extant Adnexal Masses in the Era of Type 1 and Type 2 Ovarian Cancers: Lessons Learned From Ovarian Cancer Screening Trials

Women that are positive for an ovarian abnormality in a clinical setting can have either a malignancy or a benign tumor with probability favoring the benign alternative. Accelerating the abnormality to surgery will result in a high number of unnecessary procedures that will place cost burdens on the individual and the health delivery system. Surveillance using serial ultrasonography is a reasonable alternative that can be used to discover if changes in the ovarian abnormality will occur that favor either a malignant or benign interpretation. Several ovarian cancer screening trials have had extensive experiences with changes in subclinical ovarian abnormalities in normal women that can define growth, stability or resolution and give some idea of the time frame over which changes occur. The present report examines these experiences and relates them to the current understanding of ovarian cancer ontology, presenting arguments related to the benefits of surveillance.

Reliability of diagnosis from Mohs slides: interpersonal and intrapersonal agreement on basal cell carcinoma presence and histological subtype

BACKGROUND

The success of Mohs micrographic surgery (MMS) depends partly on the correct diagnosis of slides.

OBJECTIVES

To determine reliability of diagnosis from Mohs slides.

METHODS

This was a prospective study evaluating the reliability of diagnosis from Mohs slides of basal cell carcinoma (BCC) presence, BCC location on the slide and BCC subtype among six raters who independently assessed 50 Mohs slides twice with a 2-month interval. Slides were randomly selected whereby difficult-to-diagnose slides were oversampled. For each slide, a reference diagnosis was established by an expert panel. Cohen's kappa (κ) was calculated to determine levels of agreement interpersonally (rater vs. reference diagnosis) and intrapersonally (rater at T1 vs. T2). Multivariable logistic regression was used to determine independent risk factors for slides with interpersonal discordant diagnosis. The variables studied were BCC presence, whether a slide was scored as easy or difficult to diagnose, review duration of the 50 slides, profession and years of experience in diagnosis from Mohs slides.

RESULTS

Interpersonal and intrapersonal agreement were substantial on BCC presence (κ = 0·66 and 0·68) and moderate on BCC subtype (κ = 0·45 and 0·55). Slides that were scored as difficult to diagnose were an independent risk factor for interpersonal discordant diagnosis on BCC presence (odds ratio 3·54, 95% confidence interval 1·81-6·84).

CONCLUSIONS

Reliability of diagnosis from Mohs slides was substantial on BCC presence and moderate on BCC subtype. For slides that are scored difficult to diagnose, a second opinion is recommended to prevent misinterpretation and thereby recurrence of skin cancer.

Interobserver agreement and accuracy of non-invasive diagnosis of endometriosis by transvaginal sonography

OBJECTIVES

To evaluate interobserver agreement and accuracy of transvaginal sonography (TVS) in diagnosing deep infiltrating endometriosis (DIE) and endometriomas.

METHODS

A total of 67 consecutive patients referred to a pelvic pain clinic and scheduled for laparoscopy were enrolled in the study between January 2013 and January 2014. Patients were independently examined prospectively by two experienced sonographers (Observers A and B) who were blinded to the other's results. For the two observers, Gwet's first-order agreement coefficient (Gwet's AC1) was used to calculate interobserver agreement and diagnostic accuracy, as well as sensitivity, specificity, positive (PPV) and negative (NPV) predictive values using TVS, as compared to laparoscopy, for diagnosing DIE and endometriomas.

RESULTS

Among the 67 patients enrolled, 65 were analyzed. For the diagnosis of DIE and endometriomas by TVS, the level of agreement (Gwet's AC1) between Observers A and B and sensitivity/specificity values for the respective Observers were, by site: vagina (Gwet's AC1, 0.933; 62%/94% and 82%/94%), bladder (Gwet's AC1, 1.00; 67%/97% and 67%/97%), uterosacral ligaments (Gwet's AC1, 0.84; 73%/83% and 53%/90%), adnexa (Gwet's AC1, 0.95; 71%/93% and 71%/93%), rectovaginal septum (Gwet's AC1, 0.95; 40%/90% and 33%/87%) and rectosigmoid (Gwet's AC1, 0.98; 93%/96% and 94%/98%) which reflected high interobserver agreement. With the exception of sensitivity of diagnosis of DIE affecting the RVS, similar results were observed when TVS was compared with laparoscopy.

CONCLUSIONS

TVS is a highly accurate and reproducible method for non-invasive diagnosis of DIE by well-trained professionals.

Borderline ovarian tumors and diagnostic dilemma of intraoperative diagnosis: could preoperative He4 assay and ROMA score assessment increase the frozen section accuracy? A multicenter case-control study

The aim of our study was to assess the value of a preoperative He4-serum-assay and ROMA-score assessment in improving the accuracy of frozen section histology in the diagnosis of borderline ovarian tumors (BOT). 113 women presenting with a unilateral ovarian mass diagnosed as serous/mucinous BOT at frozen-section-histology (FS) and/or confirmed on final pathology were recruited. Pathologists were informed of the results of preoperative clinical/instrumental assessment of all patients. For Group_A patients, additional information regarding He4, CA125, and ROMA score was available (in Group_B only CA125 was known). The comparison between Group A and Group B in terms of FS accuracy, demonstrated a consensual diagnosis in 62.8% versus 58.6% (P: n.s.), underdiagnosis in 25.6% versus 41.4% (P < 0.05), and overdiagnosis in 11.6% versus 0% (P < 0.01). Low FS diagnostic accuracy was associated with menopausal status (OR: 2.13), laparoscopic approach (OR: 2.18), mucinous histotype (OR: 2.23), low grading (OR: 1.30), and FIGO stage I (OR: 2.53). Ultrasound detection of papillae (OR: 0.29), septa (OR: 0.39), atypical vascularization (OR: 0.34), serum He4 assay (OR: 0.39), and ROMA score assessment (OR: 0.44) decreased the probability of underdiagnosis. A combined preoperative assessment through serum markers and ultrasonographic features may potentially reduce the risk of underdiagnosis of BOTs on FS while likely increasing the concomitant incidence of false-positive events.