Interobserver reproducibility and agreement with original diagnosis in the categories “atypical” and “suspicious for malignancy” for bile and pancreatic duct brushings

World Health Organization Reporting System for Soft Tissue Cytopathology: Risk of malignancy and reproducibility of categories among observers

INTRODUCTION

In 2024, the World Health Organization (WHO) is scheduled to publish the WHO Reporting System for Soft Tissue Cytopathology (WHORSSTC). This system establishes categories with well-defined definitions, criteria, and estimated risks of malignancy (ROMs) for soft tissue tumors. The estimates of ROM are based on a relatively small number of published studies. Interobserver reproducibility is not addressed in the reporting system even though reproducibility of a reporting system is highly important.

METHODS

A manual search of one authors personal consultation files and teaching set (L.J.L.) was conducted for all cytologic specimens of soft tissue tumors accessioned between January 1, 1985 and December 31, 2022. Only cases with documented surgical pathology follow-up were included in the study. Slides from each case were evaluated independently by three cytopathologists with each case assigned to one of the WHORSSTC categories. A ROM for each of the WHORSSTC categories was calculated. Interobserver agreement was evaluated by the kappa and weighted kappa statistics.

RESULTS

Risk for malignancy by category were: Category 1: 0%, Category 2: 28%, Category 3: 57%, Category 4: 47%, Category 5: 63%, and Category 6: 88%. Kappa statistics for agreement between raters varied from 0.2183 to 0.3465 and weighted kappa varied from 0.3778 to 0.5217.

CONCLUSIONS

The WHORSSTC showed a progression of malignancy risk from the category "benign" (28%) to the category "malignant" (88%). Interobserver agreement was only fair.

The World Health Organization Reporting System for Pancreaticobiliary Cytopathology: Overview and Summary

The recently published WHO Reporting System for Pancreaticobiliary Cytopathology (World Health Organization [WHO] System) is an international approach to the standardized reporting of pancreaticobiliary cytopathology, updating the Papanicolaou Society of Cytopathology System for Reporting Pancreaticobiliary Cytology (PSC System). Significant changes were made to the categorization of benign neoplasms, intraductal neoplasms, mucinous cystic neoplasms, and malignant neoplasms considered low grade. Benign neoplasms, such as serous cystadenoma, categorized as Neoplastic: benign in the PSC system, are categorized as Benign/negative for malignancy in the WHO system. Pancreatic neuroendocrine tumor, solid-pseudopapillary neoplasm, and gastrointestinal stromal tumor, categorized as Neoplastic: other in the PSC system, are categorized as Malignant in the WHO System in accord with their classification in the 5th edition WHO Classification of Digestive System Tumours (2019). The two new categories of Pancreaticobiliary Neoplasm Low-risk/grade and Pancreaticobiliary Neoplasm High-risk/grade are mostly limited to intraductal neoplasms and mucinous cystic neoplasms. Low-risk/grade lesions are mucinous cysts, with or without low-grade epithelial atypia. High-risk/grade lesions contain neoplastic epithelium with high-grade epithelial atypia. Correlation with clinical, imaging, and ancillary studies remains a key tenet. The sections for each entity are written to highlight key cytopathological features and cytopathological differential diagnoses with the pathologist working in low resource setting in mind. Each section also includes the most pertinent ancillary studies useful for the differential diagnosis. Sample reports are provided for each category. Finally, the book provides a separate section with risk of malignancy and management recommendations for each category to facilitate decision-making for clinicians.

Interobserver variability in cytopathology: How much do we agree?

Interobserver variability remains a major challenge for cytopathologists despite the development of standardized reporting and classification systems. Indeed, whereas moderate-to-good interobserver agreement is generally achievable when the differential diagnosis between benign and malignant entities is straightforward, high levels of variability make the diagnostic interpretation of atypical and suspicious samples not consistent. This review explores the landscape of interobserver agreement in cytopathology across different anatomical sites.

The Milan System for Reporting Salivary Gland Cytopathology

OBJECTIVES

Our study assesses whether the Milan System for Reporting Salivary Gland Cytopathology (MSRSGC) offers any benefit over the original cytology classification, and measures interobserver agreement.

METHODS

Four cytopathologists retrospectively blindly classified preoperative cytology by MSRSGC from 101 resected salivary tumors. Consensus MSRSGC diagnoses were correlated with surgical pathology diagnoses and compared with the original cytology classification. Diagnostic parameters were calculated for both systems. Interobserver variability was assessed.

RESULTS

The original cytology classification vs MSRSGC had sensitivity, specificity, positive predictive value, and negative predictive value of 75.0% vs 78.3%, 97.1% vs 98.0%, 91.2% vs 94.7%, and 90.1% vs 90.0%, respectively. The original cytology classification risk of neoplasm (RON) was 91.7% for "negative for malignancy" and 100.0% for other categories. The MSRSGC RON was 71.4% in category II (nonneoplastic) and 100.0% in all other categories. The original cytology classification risk of malignancy (ROM) ranged from 0.0% for "atypical" to 100.0% for "positive for malignancy." The MSRSGC ROM ranged from 0.0% in categories I (nondiagnostic) and III (nonneoplastic) to 100.0% in category VI (malignant). Weighted agreement using the MSRSGC was 92% (Gwet AC1, 0.84); unweighted agreement was 69% (Gwet AC1, 0.64). MSRSGC category IVA (benign neoplasm) was most likely to show interobserver agreement, with complete agreement in 67% of cases.

CONCLUSIONS

The MSRSGC performs similarly to the original cytology classification and shows relatively high interobserver agreement.

The international system for serous fluid cytopathology: Interobserver agreement

BACKGROUND

A number of categorization systems had been developed for the reporting of cytology specimens with the aim of providing uniform definitions, criteria, and diagnostic terminology. The intention of these systems is to improve reproducibility of diagnostic categorization with standardized estimates of malignancy risk. Required for the success of these systems is a high level of interobserver reproducibility for category assignment. Recently, the international system for serous fluid cytopathology (TIS) was proposed using the categories nondiagnostic, negative for malignancy, atypia of undetermined significance (AUS), suspicious for malignancy, and malignant. Little data exists documenting the interobserver agreement for these categories.

DESIGN

A search of the cytology records at the University of Missouri was performed for all pleural fluid specimens obtained between January 2014 and December 2019. A total of 200 specimens were reviewed independently by three board-certified cytopathologists. Specimens were characterized as nondiagnostic, negative, AUS, suspicious for malignancy, and malignant. Interobserver agreement was analyzed using Cohen's kappa.

RESULTS

Overall observer agreement was 68% and chance-corrected weighted agreement (weighted kappa) was 0.63. Agreement was good for categories negative and malignant, but poor for categories atypia of uncertain significance, and suspicious for malignancy.

CONCLUSIONS

The TIS has performance characteristics similar to other cytologic classification schemes. Interobserver agreement is best for the negative (76%) and malignant (81%) categories. Interobserver agreement is poor for the category's AUS, and suspicious for malignancy. This is similar to interobserver agreement associated with other published categorization systems.

Atypical cells in fine needle aspiration biopsies of pancreas: Causes, work-up, and recommendations for management

Endoscopic ultrasound-guided fine needle aspiration (EUS-FNA) is a sensitive and specific method for diagnosing cancer in solid pancreatic masses. However, some cases receive indeterminate atypical diagnoses, which creates management dilemmas. In the 2014 Papanicolaou Society of Cytopathology (PSC) standardized guidelines for pancreatobiliary cytology, specimens in the "Atypical" category show a spectrum of architectural and/or cellular changes beyond normal or reactive, but, quantitatively or qualitatively, insufficient for classification as neoplastic (benign/other), suspicious or positive for malignancy. Implementation of the PSC system decreased atypical diagnoses, particularly for cystic lesions, and redistributed many cases into benign and neoplastic categories. Because no set cytologic criteria exist for the "Atypical" category there is wide variability in its use, and its frequency ranges from 0%-16% (mean 6%). It consists of a heterogeneous mix of cases that occur because of preanalytic, lesion-specific (low cellularity, necrosis, cystic, reactive and premalignant changes), to pathologist-dependent factors (experience, expertise, training and institutional case volume). Outcomes of atypical diagnoses in solid pancreatic masses range from benign to premalignant and malignant and include reactive atypia in pancreatitis, well differentiated adenocarcinoma, and non-ductal malignancies. The associated risk of malignancy (ROM) ranges from 28%-100%, with an overall intermediate ROM in large-scale studies. Cytopathologists and institutions should monitor and keep their personal and/or laboratory's atypical rates low by judiciously using rapid onsite evaluation, ancillary studies, consensus or expert review, as well as correlation with clinical and radiologic findings. Early repeat EUS-FNA is indicated for unresolved cases.

Application of enhancer of zeste homolog 2 immunocytochemistry to bile cytology

BACKGROUND

Bile cytology has low diagnostic sensitivity and requires ancillary techniques. This study assessed the utility of enhancer of zeste homolog 2 (EZH2) immunocytochemistry (ICC) in bile cytology.

METHODS

A total of 141 bile cytology specimens from 141 patients were evaluated retrospectively. Papanicolaou-stained slides were immunostained with an antibody to EZH2. After calculation of the EZH2 labeling index (LI), the cutoff value was determined via receiver operating characteristic curve analysis. Cytological performance with and without EZH2 ICC was evaluated with reference to the final diagnosis.

RESULTS

The area under the curve for the EZH2 LI was 0.955, and the cutoff value for identifying benign bile samples versus malignant ones was 24.0%. The sensitivity and specificity values for malignancy were 53.4% and 100% for routine cytology only, 89.0% and 95.7% for EZH2 ICC only, and 89.8% and 95.7% for a combination of routine cytology and EZH2 ICC. The sensitivities of EZH2 ICC only and a combination of routine cytology and EZH2 ICC were significantly improved in comparison with routine cytology only (P < .001). EZH2 ICC alone had a sensitivity of 68.0% and a specificity of 85.7% in bile samples with atypical cytology, a sensitivity of 87.0% in samples that were suspicious for malignancy, and a sensitivity of 85.7% and a specificity of 100% in samples that were negative for malignancy.

CONCLUSIONS

EZH2 ICC improved the diagnostic sensitivity for pancreatobiliary adenocarcinoma in bile cytology. This method is particularly meaningful in samples of indeterminate cytology and may be useful as an initial assessment to ensure that no cancer cells are missed.

Effect of single operator cholangioscopy on accuracy of bile duct cytology

INTRODUCTION

This is the first study to evaluate the accuracy of bile duct brushings since the introduction of single operator cholangioscopy SpyGlass DS system in 2015. The primary aim of our study was to compare the accuracy of cytology brushings against biopsies obtained at endoscopic retrograde cholangiopancreatography (ERCP) and cholangioscopy.

METHOD

A retrospective search for bile duct brushing specimens was performed and the charts reviewed. The gold standard for definitive diagnosis of malignancy was surgical tissue or compelling clinical evidence of malignancy. Definite negative diagnosis relied on lack of clinical/imaging features of malignancy on follow-up.

RESULTS

There was no significant difference in diagnostic accuracy of cytology specimens obtained at different procedures. Overall sensitivity for all methods was 59%, specificity 90% and accuracy of 82%. Notably, all four false-positive cases except for one were from patients with primary sclerosing cholangitis or primary biliary sclerosis. There was no significant difference in sensitivity or specificity between biopsies procured by the two methods. The combined biopsy sensitivity for all modalities was 44% with a specificity of 100%. There was 70% concordance between cytology and biopsy cases. The overall specificity and sensitivity of the combined cytology and biopsy result was 85% and 65.3%, respectively. Ranking of factors that predict clinical diagnosis shows cytology results to surpass other parameters including pancreatic mass, age and stenosis length. Similarly, the presence of two cell populations and three-dimensional clusters was not nearly as predictive of malignancy as single malignant cells.

CONCLUSION

Biliary brush cytology specimens performed better than biopsies irrespective of whether they are procured during ERCP or cholangioscopy. The combination of cytology and biopsy had the best accuracy than either one alone.

Interobserver Agreement for the International Academy of Cytology Yokohama System for Reporting Breast Fine-Needle Aspiration Biopsy Cytopathology

BACKGROUND

A number of guidelines have been developed to improve standardization of the terminology and criteria for cytologic specimens obtained from the thyroid, pancreas, lung, and salivary glands. A major goal of these guidelines is to improve reproducibility and understanding of the reporting of diagnostic results among cytopathologists and between cytopathologists and clinicians. The International Academy of Cytology Yokohama System for Reporting Breast Fine-Needle Aspiration Biopsy Cytopathology (IAC YSRB) is the most recent of these guidelines. The value of this system is, in part, dependent upon interobserver reproducibility.

DESIGN

Ninety consecutive fine-needle aspiration biopsies (FNAB) of the breast, performed over a 6-year period, were independently evaluated by 4 board-certified pathologists blinded to the original diagnoses. The 5 diagnostic categories used were those of the IAC YSRB according to published criteria for these categories. Observed agreement and chance corrected agreement (Fliess κ) were calculated. Differences in κ values were evaluated using the T statistic of Gwent. Statistical calculations were performed using STATA v16.0 (STATA Corp., College Station, TX, USA).

RESULTS

Overall agreement between observers was good. Observed unweighted agreement was 69% and weighted agreement was 91%. The majority of diagnoses were concordant (68.6%).

CONCLUSIONS

Interobserver agreement of 4 cytopathologists was good using the 5 categories of the IAC YRSB (69%). Agreement was greater among pathologists with more years of experience. The IAC YSRB system appears to provide greater agreement among viewers than guidelines for cytologic specimens obtained from some other body sites (salivary gland and lung). Most discrepancies were only by a single category, with only 22/113 (19%) differing by more than one category.

The Papanicolaou Society of Cytopathology guidelines for respiratory cytology: Reproducibility of categories among observers

Introduction:

The Papanicolaou Society of Cytopathology (PSC) has developed a set of guidelines for reporting respiratory cytology. While the malignancy risk for each category is known, the interobserver reproducibility of these diagnostic categories has not been well described.

Methods:

Fifty-five cytologic specimens obtained by fine needle aspiration from the pulmonary nodules were independently reviewed by four board-certified cytopathologists and assigned to the diagnostic categories described by the PSC guidelines for respiratory specimens. Statistical analysis for diagnostic accuracy was performed for absolute agreement and chance-corrected agreement (kappa). Differences in frequency of distribution of diagnoses between raters were assessed using the Kruskal–Wallis test.

Results:

No significant differences in distribution of scores by raters were observed. On average, the absolute agreement was 49.5% and the chance-corrected agreement (kappa) was 20%. 34.5% of interrater comparisons were in full agreement, and total lack of agreement between the four categories was found in 3% of cases. Combining the “suspicious for malignancy” category with the “malignant” category did not significantly alter interrater agreement statistics.

Conclusions:

The PSC categories showed only fair reproducibility among four cytopathologists. Agreement between raters was at best fair and did not improve significantly when the categories “suspicious for malignancy” and “malignant” were combined. The most common source of disagreement appeared to be between the categories “suspicious” and “malignant.”

Factors Impacting the Performance Characteristics of Bile Duct Brushings: A Clinico-Cytopathologic Analysis of 253 Patients

Context.—

Literature on factors impacting bile duct brushings (BDBs) performance characteristics remain limited.

Objective.—

To capture the current state of daily practice with BDB sign-out.

Design.—

Two hundred fifty-three of 444 BDBs signed out by more than 7 cytopathologists, with histopathologic and/or clinical follow-up of at least 18 months, were examined.

Results.—

One hundred thirty-five of 253 BDBs (53%) had histologically confirmed malignancies, 22 (9%) had cancer-related deaths, and 96 (38%) were benign. Cytologic diagnoses in the 444 BDBs were nondiagnostic (11 [2.5%]), negative (284 [64%]), atypical (62 [13.9%]), suspicious (34 [7.7%]), and malignant (53 [11.9%]). Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy of malignancy detection were 35%, 100%, 100%, 58%, and 66%, respectively. When atypical, suspicious, and malignant (ASM) categories were combined, sensitivity increased (58%), specificity and PPV dropped (97%), and accuracy increased (73%). Carcinoma type (bile-duct versus pancreatic-ductal) had no effect on accuracy (P = .60) or diagnostic class (P = .84), nor did time of performance (first 7.5 versus latter 7.5 years, P = .13). Interestingly, ThinPrep + cell block (n = 41) had higher sensitivity (61%) and lower specificity (80%) than ThinPrep only (versus 51% and 100%, respectively). Sensitivity and specificity were higher (47% and 100%) in nonstented than stented specimens (59% and 97%). Relative risk of malignancy for “suspicious” (2.30) and “atypical” (2.28) categories was lower but not very different from that of “malignant” category (2.41).

Conclusions.—

Bile duct brushings had fairly low sensitivity but high specificity and PPV with no false positives. Sensitivity almost doubled and specificity dipped minimally when ASM categories were combined, highlighting the need for better classification criteria for atypical/suspicious cases. Higher specificity, PPV, NPV, and accuracy but lower sensitivity in stented BDBs suggest that they be called malignant only when evidence is overwhelmingly convincing.

Bile duct brushing cytology: Cytohistologic/fine-needle aspiration correlation and diagnostic pitfalls

INTRODUCTION

Bile duct brushings are a mainstay in the work-up of patients who present with clinical findings of pancreaticobiliary disease. This study presents the University of Wisconsin Hospital and Clinics Cytopathology Laboratory's experience over a 10.5-year period with histologic and fine-needle aspiration (FNA) correlation.

MATERIALS AND METHODS

From January 1, 2005, through June 30, 2015, 676 bile duct brushings were processed, of which 408 (60%) were negative, 84 (12%) were atypical, 65 (10%) were suspicious, and 119 (18%) were malignant. Histologic and FNA follow-up was available in 285 (42%) of the cases and forms the basis of this study.

RESULTS

There were 59 (21%) positive, 38 (13%) suspicious, 54 (19%) atypical, and 134 (47%) negative bile duct brushings with follow-up. Of the 59 positive cases, 57 (97%) were malignant and 2 (3%) were negative on histologic/FNA follow-up. Thirty-three (87%) of the suspicious brushing were malignant on follow-up and 5 (13%) were negative; 43 (80%) of the atypical brushings were malignant and 11 (20%) were negative; and 94 (70%) of the negative brushings were malignant and 40 (30%) were negative on histologic/FNA follow-up. The sensitivity was 38%; specificity, 95%; positive predictive value, 97%; negative predictive value, 30%.

CONCLUSIONS

Bile duct brushing cytology is highly specific in the detection of malignant pancreatobiliary lesions with few false positives, but has low sensitivity with many false negatives.