A practical application of analysing weighted kappa for panels of experts and EQA schemes in pathology

Standardization of Gleason grading among 337 European pathologists

AIMS

  The 2005 International Society of Urological Pathology (ISUP) modification of Gleason grading recommended that the highest grade should always be included in the Gleason score (GS) in prostate biopsies. We analysed the impact of this recommendation on reporting of GS 6 versus 7.

METHODS AND RESULTS

  Fifteen expert uropathologists reached two-thirds consensus on 15 prostate biopsies with GS 6-7 cancer. Eighty-five microphotographs were graded by 337 of 617 members of the European Network of Uropathology (ENUP), representing 19 countries. There was agreement between expert and majority member GS in 12 of 15 cases, while members upgraded in three cases. Among members and the expert consensus, a GS >6 was assigned by 64.5% and 60%, respectively. Mean member GS was higher than consensus GS in nine of 15 cases. A Gleason pattern (GP) 5 was reported by 0.3-5.6% in 10 cases. Agreement between consensus and member GS was 58.2-89.3% (mean 71.4%) in GS 6 cases and 46.3-63.8% (mean 56.4%) in GS 7 cases (P = 0.009).

CONCLUSIONS

  While undergrading of prostate cancer used to be prevalent, some now tend to overgrade. Minimum diagnostic criteria for GP 4 and 5 in biopsies need to be better defined. Image libraries reviewed by experts may be useful for standardization.

Measuring interobserver variation in a pathology EQA scheme using weighted κ for multiple readers

Background

A Urological Pathology External Quality Assurance (EQA) Scheme in the UK has reported observer variation in the diagnosis and grading of adenocarcinoma in prostatic biopsies using basic κ statistics, which rate all disagreements equally.

Aim

The aim of this study is to use customised weighting schemes to report κ statistics that reflect the closeness of interobserver agreement in the prostate EQA scheme.

Methods

A total of 83, 114 and 116 pathologists took part, respectively, in three web-based circulations and were classified as either expert or other readers. For analyses of diagnosis, there were 10, 8 and 8 cases in the three circulations, respectively. For analyses of Gleason Sum Score, only invasive cases were included, leaving 5, 5 and 6 cases, respectively. Analyses were conducted using customised weighting schemes with ‘pairwise-weighted’ κ for multiple readers.

Results

Analysis of diagnosis for all circulations and all readers gave a composite κ value of 0.86 and pairwise-weighted κ (κp–w) value of 0.91, both regarded as ‘almost perfect’ agreement. This was due to the high proportion of responses that showed partial agreement. Analysis of Gleason Sum Score gave κ=0.38 and κp–w=0.58 over all circulations and all readers, indicating that discrepancies occur at the boundary between adjacent grades and may not be as clinically significant as suggested by composite κ.

Conclusion

Weighted κ show higher levels of agreement than previously reported as they have the advantage of applying weighting, which reflects the relative importance of different types of discordance in diagnosis or grading. Agreement on grading remained low.