Interobserver variability in cytopathology: How much do we agree?

Interobserver variability remains a major challenge for cytopathologists despite the development of standardized reporting and classification systems. Indeed, whereas moderate-to-good interobserver agreement is generally achievable when the differential diagnosis between benign and malignant entities is straightforward, high levels of variability make the diagnostic interpretation of atypical and suspicious samples not consistent. This review explores the landscape of interobserver agreement in cytopathology across different anatomical sites.

Reliability of Magnetic Resonance Imaging Criteria for the Preoperative Assessment of Rotator Cuff Tears: A Systematic Review

BACKGROUND

Magnetic resonance (MR) imaging is the most common modality for assessment of the rotator cuff before and after surgery. Several classifications have been described aiming to define main tear characteristics. However, there is still confusion when it comes to the reliability of those classifications.

PURPOSE

(1) To identify all MR classifications available in the literature for preoperative assessment of rotator cuff tears, (2) to summarize available data on the reliability of identified classifications, and (3) to assess the methodological quality of reliability studies.

STUDY DESIGN

Systematic review; Level of evidence, 4.

METHODS

This systematic review was conducted following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. All studies reporting MR assessment in patients with a superior or posterosuperior rotator cuff tear were included. After identification of the available MR criteria, reliability studies were analyzed. Descriptive statistics were used to summarize findings. Methodological quality was assessed using the Quality Appraisal of Reliability Studies checklist.

RESULTS

A total of 75 studies were included in this review. Eight categories of outcomes could be identified. Of the total, 62 studies reported interobserver reliability whereas 32 reported intraobserver reliability of some of the identified criteria. Each category reflected a variety of reliability, ranging from poor to excellent agreement. MR proved to be a reliable imaging modality to detect the structural integrity of the posterosuperior cuff, especially in cases of full-thickness tear; it was also reliable in terms of tear width and length and muscle atrophy based on a tangent sign or Thomazeau classification. All other classifications did not prove acceptable reliability. Methodological quality was high for 23 articles and moderate for 14.

CONCLUSION

Preoperative MR is a reliable imaging modality to identify full-thickness tears, measure tear size and morphology, and identify muscle atrophy with tangent sign or Thomazeau classification. All other outcomes and classifications did not show acceptable reliability; therefore, caution is needed when using them for preoperative evaluation of a rotator cuff tear.

PD-L1 Assessment in Needle Core Biopsies of Non-Small Cell Lung Cancer: Interpathologist Agreement and Potential Associated Histopathological Features

OBJECTIVE

Immune checkpoint inhibitors are used in the treatment of non-small cell lung cancer (NSCLC). Programmed cell death-ligand 1 (PD-L1) immunohistochemistry (IHC) assessed by pathologists is subject to interobserver variability. In advanced/metastatic disease and inoperable patients, PD-L1 assessment relies on biopsy specimens, commonly needle core biopsies (NCB). We aimed to determine the interobserver agreement for PD-L1 tumor proportion score (TPS) in NSCLC NCBs and identify histopathological features that may be related to interobserver variability.

MATERIAL AND METHODS

Sixty NSCLC NCBs with PD-L1 IHC were evaluated independently by four pathologists from different institutions. PD-L1 TPS was evaluated in three categories: no/low expression ( < 1%), intermediate expression (1%49%), and high expression (≥50%). Histological tumor type, necrosis, tumor-infiltrating lymphocytes, tumor length/percentage in the biopsy, and crush/squeeze artifact was evaluated.

RESULTS

The statistical analysis of the three PD-L1 TPS categories demonstrated moderate agreement (Fleiss Kappa 0.477) in the no/low category, fair agreement (Fleiss Kappa 0.390) in the intermediate category, and almost perfect agreement (Fleiss Kappa 0.952) in the high category. A significant correlation (p=0.003) was found between the crush/squeeze artifact in NCB and rate of discordant TPS categories. There was no significant correlation between pathologists' agreement in the TPS categories and histological tumor type, tumor length, tumor ratio, necrosis, and tumor-infiltrating lymphocytes.

CONCLUSION

Our results demonstrated moderate agreement among pathologists for the PD-L1 TPS 1% cut-off in NSCLC NCB, which is lower than that reported in resection materials. The presence of crush/squeeze artifact in NCBs is significantly related to the rate of discordant TPS categories, suggesting that PD-L1 assessment of pulmonary NCBs requires an awareness of this artifact.

Ultrasound Strain Elastography Reliability in the Assessment of the Plantar Fascia and Its Relationship with the Plantar Thickness in Healthy Adults: An Intra and Interobserver Reliability Study in Novice Evaluators

PURPOSE

This study was aimed at verifying both the intraobserver and interobserver reliability of measuring plantar fascia stiffness for a given image in healthy active adults.

METHODOLOGY

This study is reported following the Guidelines of Reporting Reliability and Agreement Studies. A total of 20 plantar fascia from healthy volunteers were analyzed. The thickness of the plantar fascia was measured vertically from the anterior edge of the inferior calcaneal border to the inferior border of the plantar fascia and the ultrasound elastography measurement was taken at the calcaneal insertion of the plantar fascia with the region of interest one centimeter away from the insertion.

RESULTS

The ultrasound strain elastography measurements: the right intraobserver 1 showed an ICC value of 0.9 and the left intraobserver 1 showed an ICC value of 0.78, while the right intraobserver 2 showed an ICC value of 0.91 and the left intraobserver 2 showed an ICC value of 0.83. Interobserver measurements showed excellent reliability with a right ICC value of 0.8 and a left ICC value of 0.9 for the plantar fascia thickness measurements.

DISCUSSION

The results of this study showed a strong correlation between left and right plantar fascia thickness. The intraobserver reliability was excellent for both plantar fascia ultrasound strain elastography and thickness measurements, with interobserver measurements showing excellent reliability.

Interobserver Agreement Rates on C-X-C Motif Chemokine Receptor 4-Directed Molecular Imaging and Therapy

BACKGROUND

We aimed to evaluate the interobserver agreement rates in patients scanned with C-X-C motif chemokine receptor 4 (CXCR4)-directed PET/CT, including the rate of patients eligible for CXCR4-targeted radioligand therapy (RLT) based on scan results.

METHODS

Four independent observers reviewed 50 CXCR4-targeted [ 68 Ga]pentixafor PET/CT of patients with various solid cancers. On a visual level, the following items were assessed by each reader: overall scan impression, number of organ and lymph node (LN) metastases and number of affected organs and LN regions. For a quantitative investigation, readers had to choose a maximum of 3 target lesions, defined as largest in size and/or most intense uptake per organ compartment. Reference tissues were also quantified, including unaffected hepatic parenchyma and blood pool. Last, all observers had to decide whether patients were eligible for CXCR4-targeted RLT. Concordance rates were tested using intraclass correlation coefficients (ICCs). For interpretation, we applied the definition of Cicchetti (with 0.4-0.59 indicating fair; 0.6-0.74, good; 0.75-1, excellent agreement).

RESULTS

On a visual level, fair agreement was achieved for an overall scan impression (ICC, 0.58; 95% confidence interval, 0.45-0.71). Organ and LN involvement (ICC, ≥0.4) demonstrated fair, whereas CXCR4 density and number of LN and organ metastases showed good agreement rates (ICC, ≥0.65). Number of affected organs and affected LN areas, however, showed excellent concordance (ICC, ≥0.76). Quantification in LN and organ lesions also provided excellent agreement rates (ICC, ≥0.92), whereas quantified uptake in reference organs provided fair concordance (ICC, ≥0.54). Again, excellent agreement rates were observed when deciding on patients eligible for CXCR4-RLT (ICC, 0.91; 95% confidence interval, 0.85-0.95).

CONCLUSIONS

In patients scanned with CXCR4-targeted PET/CT, we observed fair to excellent agreement rates for both molecular imaging and therapy parameters, thereby favoring a more widespread adoption of [ 68 Ga]pentixafor in the clinic.

Agreement and reliability of hepatic transient elastography in patients with chronic hepatitis C: A cross‐sectional test–retest study

Background and Aims

Transient elastography (TE) has largely replaced liver biopsy to evaluate fibrosis stage and cirrhosis in chronic hepatitis C. Previous studies have reported excellent reliability of TE but agreement metrics have not been reported. This study aimed to assess interrater agreement and reliability of repeated TE measurements.

Methods

Two operators performed TE independently, directly after each other. The primary outcome was disagreement, defined as a difference in TE results between operators of ≥33%, as well as the smallest detectable change, SDC₉₅ (i.e., the difference between measurements needed to state with 95% certainty that there is a difference in underlying stiffness). Secondary outcomes included reliability, measured as intraclass correlation (ICC), and patient and examination characteristics associated with the agreement.

Results

In total, 65 patients were included, with a mean liver stiffness of 9.7 kPa. Of these, 21 (32%) had a disagreement in TE results of ≥33% between the two operators. The SDC₉₅ on the log scale was 1.97, indicating that an almost twofold increase or decrease in liver stiffness would be required to confidently represent a change in the underlying fibrosis. Reliability, estimated using the ICC, was acceptable at 0.86. In a post hoc analysis, fasting less than 5 h before TE was associated with a higher degree of disagreement (48% vs. 19%, p = 0.03).

Conclusions

In our clinical setting, interrater agreement in directly repeated TE measurements was surprisingly low. It is essential to further investigate the reliability and agreement of TE to determine its validity and usefulness.

The impact of motion induced artifacts in the evaluation of HR-pQCT scans of the scaphoid bone: an assessment of inter- and intraobserver variability and quantitative parameters

Background

In-vivo high-resolution peripheral quantitative computed tomography (HR-pQCT) has high potential in scaphoid bone pathologies' scientific and clinical fields. The manufacturer's visual grading scale (VGS) classifies motion artifacts and divides scans into five quality grades ranging from grade 1 (good quality) to grade 5 (poor quality). This prospective study aimed to investigate the feasibility of the VGS and the influence of image quality on bone density and microarchitecture parameters for the scaphoid bone.

Methods

Within one year, twenty-two patients with scaphoid fractures received up to six scans of their fractured and contralateral wrist (each consisting of three stacks) using second-generation HR-pQCT (total 256 scans). Three experienced observers graded each stack following the visual grading system, and inter- and intraobserver variability were assessed. The contralateral uninjured scaphoids were then compared pairwise within each patient to high-quality grade 1 scans to determine the influence of image quality on density and microarchitecture parameters.

Results

Inter- and intraobserver variability among the three observers significantly revealed fair to moderate agreement, P<0.001 and P<0.05, respectively. Bone volume (BV) fraction tended to increase with poorer image quality but did not exceed four percent. Trabecular bone mineral density (Tb.BMD) decreased with poorer image quality but did not exceed five percent. Trabecular number and trabecular thickness significantly increased by 15.5% and 6.8% at grade five (P<0.001), respectively, and trabecular separation significantly decreased by 13.7% at grade five (P<0.001).

Conclusions

This study revealed a considerable influence of motion on bone morphometry parameters of the scaphoid. Therefore, high image quality must be a central point in studies focusing on the histomorphometry of small objects. The high inter- and intraobserver variability limit the VGS. Future research may focus on other grading systems or automated techniques leading to more consistent and reproducible results. Currently, the use of microarchitectural analysis should be limited to cases without motion artefacts or, at most low graded motion artefacts.

Cd34 Immunostaining Adds Specificity To Microvascular invasion Analysis in Hepatocellular Carcinoma

Introduction

Hepatocellular carcinoma is the most common primary neoplasia of the liver. Microvascular invasion predicts outcome and defines tumor staging. However, its diagnosis is still a challenge. The present study aims to evaluate inter and intraobserver agreement in identifying the presence of microvascular invasion using conventional and immunohistochemistry histology.

Methods

Three pathologists performed the analysis of 76 hepatocellular carcinoma explants to characterize the presence of microvascular invasion using the hematoxylin/eosin stain and immunohistochemistry for CD34. The evaluations were made individually, in two distinct moments. Results were analyzed by the Kappa's coefficient and ROC curves.

Results

Our study demonstrated similar agreement for microvascular invasion between hematoxylin/eosin and CD34 methods. However, the intraobserver agreement values for both methods were higher than the interobserver ones. The accuracy of CD34 in relation to hematoxylin/eosin by ROC curves in intraobserver analysis tends to a high specificity, ranging from 82.1 to almost 100%, with sensitivity of 46.9% to 81.1%. In interobserver analysis, CD34 also has a high specificity (84.3% to 85.5%) while its sensitivity is a little shorter (81.2% to 84.3%).

Conclusion

Intraobserver higher agreement allows us to suppose that pathologists employed own criteria to evaluate vascular invasion, reinforcing the need of standardization. ROC Curves analysis showed that the CD34 method is more specific than sensitive. Therefore, immunohistochemistry for CD34 should not be used routinely, but it could be useful to help confirming invasion previously seen by conventional histology.

Interobserver agreement for the Chest Wall Injury Society taxonomy of rib fractures using computed tomography images

BACKGROUND

In 2020, a universal nomenclature for rib fractures was proposed by the international Chest Wall Injury Society taxonomy collaboration. The purpose of this study is to validate this taxonomy. We hypothesized that there would be at least moderate agreement, regardless of the observers' background.

METHODS

An international group of independent observers evaluated axial, coronal, and sagittal computed tomography images on an online platform from 11 rib fractures for location (anterior, lateral, or posterior), type (simple, wedge, or complex), and displacement (undisplaced, offset, or displaced) of rib fractures. The multirater κ and Gwet's first agreement coefficient (AC1) were calculated to estimate agreement among the observers.

RESULTS

A total of 90 observers participated, with 76 complete responses (84%). Strong agreement was found for the classification of fracture location ( κ = 0.83 [95% confidence interval (CI) 0.69-0.97]; AC1, 0.84 [95% CI, 0.81-0.88]), moderate for fracture type ( κ = 0.46 [95% CI, 0.32-0.59]; AC1, 0.50 [95% CI, 0.45-0.55]), and fair for rib fracture displacement ( κ = 0.38 [95% CI, 0.21-0.54], AC1, 0.38 [95% CI, 0.34-0.42]).

CONCLUSION

Agreement on rib fracture location was strong and moderate for fracture type. Agreement on displacement was lower than expected. Evaluating strategies such as comprehensive education, additional imaging techniques, or further specification of the definitions will be needed to increase agreement on the classification of rib fracture type and displacement as defined by the Chest Wall Injury Society taxonomy.

LEVEL OF EVIDENCE

Diagnostic Test or Criteria; Level IV.

Acromioclavicular joint separation: retrospective study of nonoperative and surgical treatment in 38 patients with grade III or higher injuries and a minimum follow-up of 1 year

INTRODUCTION

Acromioclavicular (AC) joint separation is a common shoulder injury. When the injury is graded as type III or higher in the Rockwood classification, surgical treatment can be proposed. However, an increasing number of practitioners are shifting back to conservative treatment as it is associated with fewer complications and seemingly close functional outcomes. The aim of our study was to evaluate the functional recovery of operated and non-operated patients with grade III or higher AC joint injuries. Secondarily, the reliability and relevance of the Rockwood classification was evaluated within and between raters.

MATERIALS AND METHODS

We did a retrospective two-center study of 38 patients treated between 2014 and 2020. The clinical evaluation involved various functional outcome scores (Constant, QuickDASH, ASES, UCLA, SSV, STT) and a pain assessment (VAS). Return to sports and to work was also documented. The radiological evaluation consisted of Zanca AP and lateral axillary views immediately after the injury and at each radiographic follow-up visit until the final visit. An intra- and inter-rater analysis was also done for the Rockwood classification.

RESULTS

There was no significant difference in the functional scores (Constant score surgery group = 91, nonoperative group = 83; p = 0.09) or the pain on VAS at the final assessment. Return to work and to sports was significantly faster in patients treated nonoperatively. No complications were found in the non-operated patients, while nine of the operated patients suffered a complication. The inter-rater reliability of the Rockwood classification was found to be poor (kappa = 0.08) to fair (kappa = 0.35), while the intra-rater reliability was moderate (kappa = 0.6) to good (kappa = 0.63).

DISCUSSION/CONCLUSION

No matter which treatment is used, the functional outcomes and patient satisfaction level a minimum of 1 year after the injury appear to be identical. Thus, surgery should be only for patients whose AC joint is painful 7 days after the injury (VAS > 7) and whose function has not improved. For young and athletic patients, or for patients who simply want to regain nearly normal function, it is important to remember that the time to return to work and sports is longer with surgical management, and to take into consideration the potential postoperative complications. While none of the patients who received the nonoperative treatment required a secondary stabilizing surgery, this is a possible recourse.

LEVEL OF EVIDENCE

III.

Intra- and Interobserver Variability of Shear Wave Elastography in Rectal Cancer

Simple Summary

For a diagnostic tool to be important, there must be a high level of agreement between different observers. If not, the reliability of the measurement can not be considered valid. Previous studies have evaluated the reproducibility of tissue stiffness measurements of rectal tumors, but only on previously obtained images. Therefore, we aimed to investigate the issue in a clinical setting and found a high level of agreement between observers.

Abstract

Background: Endorectal ultrasound (ERUS) is an important tool when evaluating complex rectal adenomas and rectal cancer, and the accuracy is improved by adding elastography measurements. A high interobserver agreement is imperative in clinical practice. Therefore, the aim of this study was to evaluate interobserver agreement assessed on real-time images. Additionally, we investigated the intra- and interobserver agreement between experienced and inexperienced observers. Materials and methods: We prospectively included patients referred to an ERUS at the Department of Radiology with a complex rectal polyp or suspected rectal malignancy. Two operators independently scanned each patient in turn. Furthermore, four observers assessed previously obtained images using three different methods for placing the region of interest (ROI). Three months later, the four observers reassessed the images to assess intraobserver variability. Results: A total of 19 patients were included for live assessment. Agreement of tumor classification was substantial for T stage (kappa: 0.86) and fair for N stage (kappa: 0.73), with an absolute agreement for T and N stages of 84% and 89%, respectively. Agreement of SWE was good for Emean (ICC 0.94, 95% CI 0.86–0.98) and fair for Emax (ICC 0.85, 95% CI 0.66–0.94). Intra- and interobserver agreement between inexperienced and experienced observers showed good to excellent agreement with all ROI methods. Conclusion: Interobserver agreement is high in SWE when performed in a clinical setting. We found the best agreement using the mean value of several ROIs. Intra- and interobserver agreement was high regardless of operator experience.

CT measurement of prostate volume using OsiriX® viewer is reliable, repeatable, and not dependent on observer, CT protocol, or contrast enhancement in dogs

Computed tomography (CT) is an established method for evaluating dogs with suspected prostatic disease; however, publications assessing the effects of varying factors on prostate volume measurements are lacking. The objectives of this two-part, observer agreement, methods comparison study were to assess observer agreement and the effects of varying CT technical parameters for volume measurements of canine prostate glands on CT images using OsiriX^® DICOM viewer software. In the first retrospective study, two observers measured prostate volumes of 13 client-owned dogs thrice on noncontrast and contrast CT images. In the second prospective study, two observers measured the prostate volume of 10 cadavers using five different CT protocols and eight cadavers using three slice thicknesses. Observer agreement analyses were performed, and prostatic CT volume measurements were compared with water displacement volume measurements. Intra- and interobserver variability and the effect of contrast enhancement were found to be minimal when a one-way analysis of variance model and intraclass correlation coefficients were used. No significant differences emerged between different protocols and slice thicknesses using a linear mixed effects model. When the prostate CT volume was compared using a Bland-Altman plot with the reference volume acquired by the water displacement method, agreement without consistent bias between the methods was shown, and over 90% of measurements were located within the 95% limits of agreement. The findings supported using OsiriX^® software for CT prostatic volume measurements in dogs.

Validation of the AO Spine Sacral Classification System: Reliability Among Surgeons Worldwide

OBJECTIVES

To (1) demonstrate that the AO Spine Sacral Classification System can be reliably applied by general orthopaedic surgeons and subspecialists universally around the world and (2) delineate those injury subtypes that are most difficult to classify reliably to refine the classification before evaluating clinical outcomes.

DESIGN

Agreement study.

SETTING

All-level trauma centers, worldwide.

PARTICIPANTS

One hundred seventy-two members of the AO Trauma and AO Spine community.

INTERVENTION

The AO Sacral Classification System was applied by each surgeon to 26 cases in 2 independent assessments performed 3 weeks apart.

MAIN OUTCOME MEASUREMENTS

Interobserver reliability and intraobserver reproducibility.

RESULTS

A total of 8097 case assessments were performed. The kappa coefficient for interobserver agreement for all cases was 0.72/0.75 (assessment 1/assessment 2), representing substantial reliability. When comparing classification grading (A/B/C) regardless of subtype, the kappa coefficient was 0.84/0.85, corresponding to excellent reliability. The kappa coefficients for interobserver reliability were 0.95/0.93 for type A fractures, 0.78/0.79 for type B fractures, and 0.80/0.83 for type C fractures. The overall kappa statistic for intraobserver reliability was 0.82 (range 0.18-1.00), representing excellent reproducibility. When only evaluating morphology type (A/B/C), the average kappa value was 0.87 (range 0.18-1.00), representing excellent reproducibility.

CONCLUSION

The AO Spine Sacral Classification System is universally reliable among general orthopaedic surgeons and subspecialists worldwide, with substantial interobserver and excellent intraobserver reliability.

Equilibrium radionuclide angiography: Intra- and inter-observer repeatability and reproducibility in the assessment of cardiac systolic and diastolic function

BACKGROUND

This study aimed to assess intra- and inter-observer agreement in assessing the systolic and diastolic function with equilibrium radionuclide angiography (ERNA).

MATERIALS AND METHODS

Thirty-two adults underwent baseline and repeat ERNA. An experienced and a trainee operator analyzed the data by assigning regions of interest manually, fully automatically, and semi-automatically. The Bland-Altman statistic (mean ± 1.96 standard deviations of the differences) was used to assess the repeatability (two different assessments of a single acquisition) and reproducibility (assessments of two different acquisitions).

RESULTS

Using the semi-automated technique the intraobserver repeatability and reproducibility of left ventricular ejection fraction for the experienced physician were - 0.1 ± 3.7 and 0.0 ± 3.8 and for the trainee 2.2 ± 10.6 and 1.9 ± 8.4, respectively. The inter-observer repeatability and reproducibility were - 1.8 ± 6.4 and 0.4 ± 9.0, respectively. Among the parameters of diastolic function, the intraobserver repeatability and reproducibility of the peak filling rate for the experienced physician were - 0.0 ± 1.1 and - 0.1 ± 1.1 and for the trainee 0.2 ± 3.5 and 0.4 ± 3.7, respectively. The inter-observer repeatability and reproducibility were 0.3 ± 1.5 and 0.5 ± 4.0, respectively. Similar was the pattern for the other diastolic indices. In all cases the limits of agreement varied according to the quantification approach.

CONCLUSION

A good repeatability but a moderate reproducibility was found in the assessment of the LVEF. Less good were the findings in the assessment of diastolic function.

Good Interrater Reliability for Standardized MRI Assessment of Tendon Discontinuity and Tendon Retraction in Acute Proximal Full-Thickness Hamstring Tendon Injury

BACKGROUND

Proximal full-thickness free hamstring tendon injury (ie, tendon avulsion or rupture) is a severe injury. Treatment decision making relies on clinical factors and magnetic resonance imaging (MRI) variables; it specifically relies on which tendons are injured as well as the extent of tendon retraction. According to a worldwide evaluation of current practice, discontinuity of both proximal tendons and retraction of >2 cm are used as surgical indications. However, both the diagnosis and the use of MRI variables in decision making may be fraught with uncertainty in clinical practice. A reliable standardized MRI assessment is required.

PURPOSE

To propose an MRI assessment for acute proximal full-thickness free hamstring tendon injury and to evaluate its interater reliability.

STUDY DESIGN

Cohort study (diagnosis); Level of evidence, 2.

METHODS

We included 40 MRI scans of patients with acute (≤4 weeks of injury) proximal full-thickness free hamstring tendon injury. Three musculoskeletal radiologists assessed proximal full-thickness free hamstring tendon discontinuity using the novel "dropped ice cream sign" and tendon retraction (in mm). Quantification of tendon retraction (in mm) was performed using 2 different methods: (1) a direct (ie, shortest distance between the center of the hamstring origin and the tendon stump) method and (2) a combined craniocaudal/mediolateral measurement method. Absolute and relative interrater reliability were calculated.

RESULTS

We found an almost perfect interrater agreement (kappa = 0.87) for assessment of full-thickness tendon discontinuity using the dropped ice cream sign. Interrater agreement for the direct and craniocaudal retraction measurements was good for both the conjoint (intraclass correlation coefficient [ICC], 0.88 and 0.83) and the semimembranosus tendons (ICC, 0.81 and 0.79). The mediolateral retraction measurement yielded only moderate to poor reliability for the conjoint (ICC, 0.53) and semimembranosus tendons (ICC, 0.41).

CONCLUSION

The standardized MRI assessment to identify proximal hamstring tendon discontinuity and quantify tendon retraction is reliable. We recommend using the novel dropped ice cream sign and the direct retraction measurement in clinical practice and research.

Test-Retest Reliability of the Assessment of Fatty Liver Disease Using Low-Dose Computed Tomography in Cardiac Patients

Non-alcoholic fatty liver disease (NAFLD) is a common disorder that is associated with the risk of cardiovascular diseases. Therefore, its prevalence is high in patients with coronary artery disease. In myocardial perfusion imaging (MPI), low-dose computed tomography (CT) scans are used for attenuation correction in separate stress and rest studies. Here, the test–retest reliability of CT-based quantification of NAFLD was evaluated using these two CT scans. The study population consisted of 261 patients (156 men and 105 women, age 66 ± 10 years). Quantification of liver fat content was based on the radiodensity of the liver in Hounsfield units as well as in relation to corresponding values of the spleen. NAFLD was observed in 47 subjects (18%). CT quantification has good test–retest reliability in assessing NAFLD, with concordance correlation coefficient (CCC) ranging from 0.512 to 0.923, intraclass correlation coefficient (ICC) ranging from 0.513 to 0.923, and coefficient of variation ranging from 3.1 to 7.0%. Regarding the liver to spleen ratio, CCC for non-NAFLD patients and NAFLD patients was 0.552 and 0.911, respectively. For non-NAFLD patients ICC was 0.553 and NAFLD patients it was 0.913. The coefficient of variation for non-NAFLD and NAFLD patients was 4.9% and 3.1%, respectively. Our results suggest that low-dose CT is a feasible and well repeatable method but amount of liver fat contributes to repeatability. In NAFLD patients CCC and ICC were high reflecting excellent reliability, whereas in non-NAFLD patients test-retest reliability was moderate. Assessment of liver fat content can be used as additional information in studies where a CT scan has been done for other medical reasons, such as for low-dose attenuation correction CT along with MPI.

Reliability and Reproducibility of Landmark Identification in Unilateral Cleft Lip and Palate Patients: Digital Lateral Vis-A-Vis CBCT-Derived 3D Cephalograms

BACKGROUND

The aim of the retrospective observational study was to compare the precision of landmark identification and its reproducibility using cone beam computed tomography-derived 3D cephalograms and digital lateral cephalograms in unilateral cleft lip and palate patients.

METHODS

Cephalograms of thirty-one (31) North Indian children (18 boys and 13 girls) with a unilateral cleft lip and palate, who were recommended for orthodontic treatment, were selected. After a thorough analysis of peer-reviewed articles, 20 difficult-to-trace landmarks were selected, and their reliability and reproducibility were studied. These were subjected to landmark identification to evaluate interobserver variability; the coordinates for each point were traced separately by three different orthodontists (OB_A, OB_B, OB_C). Statistical analysis was performed using descriptive and inferential statistics with paired t-tests to compare the differences measured by the two methods. Real-scale data are presented in mean ± SD. A p-value less than 0.05 was considered as significant at a 95% confidence level.

RESULTS

When comparing, the plotting of points posterior nasal spine (PNS) (p < 0.05), anterior nasal spine (ANS) (p < 0.01), upper 1 root tip (p < 0.05), lower 1 root tip (p < 0.05), malare (p < 0.05), pyriforme (p < 0.05), porion (p < 0.01), and basion (p < 0.05) was statistically significant.

CONCLUSION

In patients with a cleft lip and palate, the interobserver identification of cephalometric landmarks was significantly more precise and reproducible with cone beam computed tomography -derived cephalograms vis-a-vis digital lateral cephalograms.

Validation of PD-L1 clone 22C3 immunohistochemical stain on two Ventana DISCOVERY autostainer models: detailed protocols, test performance characteristics, and interobserver reliability analyses

Immunohistochemical (IHC) stain for PD-L1 as a biomarker for immunotherapy is recommended in non-small cell lung cancer (NSCLC). Under the FDA, the selection of patients for pembrolizumab requires companion diagnostic testing using the Dako Agilent PD-L1 IHC 22C3 pharmDx kit performed on the Dako Autostainer Link 48 platform. However, because it is not widely available, there is need for cross-platform validation. Existing studies provide incomplete protocol detail. In our study, 73 lung tumors were stained using the FDA-approved test ('gold standard'). The same blocks were stained using two different models of the Ventana DISCOVERY platform (ULTRA, n = 73 and XT, n = 70) using different parameters, and interpreted by three pathologists. The ULTRA group met College of American Pathologists (CAP) validation criteria (concordance 91.8%) while the XT group did not (concordance 67.1%). Using tumor proportion score (TPS) ≥1% and TPS ≥50% as cut-offs, the ULTRA protocol had higher sensitivity (97.8% and 91.7%) than XT (73.3% and 60.9%) and similar specificity (ULTRA 88.9% and 100%, XT 88% and 100%). Discordance between ULTRA and XT was 27%, and in all these cases ULTRA was concordant with gold standard. Interobserver reliability was substantial for ULTRA and almost perfect for XT, providing evidence that staining rather than observer variability accounts for XT's inferior performance. Cross-validation of the clinically used 22C3 anti PD-L1 antibody test with substantial interobserver agreement is possible on the commonly used the Ventana DISCOVERY ULTRA automated instrument, while the validation failed on the XT. Cautious attention to detail must be paid when choosing cross-validation parameters.

Interobserver reliability of the classification of capitellar osteochondritis dissecans using magnetic resonance imaging

AIM

(1) To determine the interobserver reliability of magnetic resonance classifications and lesion instability criteria for capitellar osteochondritis dissecans lesions and (2) to assess differences in reliability between subgroups.

METHODS

Magnetic resonance images of 20 patients with capitellar osteochondritis dissecans were reviewed by 33 observers, 18 orthopaedic surgeons and 15 musculoskeletal radiologists. Observers were asked to classify the osteochondritis dissecans according to classifications developed by Hepple, Dipaola/Nelson, Itsubo, as well as to apply the lesion instability criteria of DeSmet/Kijowski and Satake. Interobserver agreement was calculated using the multirater kappa (k) coefficient.

RESULTS

Interobserver agreement ranged from slight to fair: Hepple (k = 0.23); Dipaola/Nelson (k = 0.19); Itsubo (k = 0.18); DeSmet/Kijowksi (k = 0.16); Satake (k = 0.12). When classifications/instability criteria were dichotomized into either a stable or unstable osteochondritis dissecans, there was more agreement for Hepple (k = 0.52; p = .002), Dipaola/Nelson (k = 0.38; p = .015), DeSmet/Kijowski (k = 0.42; p = .001) and Satake (k = 0.41; p < .001). Overall, agreement was not associated with the number of years in practice or the number of osteochondritis dissecans cases encountered per year (p > .05).

CONCLUSION

One should be cautious when assigning grades using magnetic resonance classifications for capitellar osteochondritis dissecans. When making treatment decisions, one should rather use relatively simple distinctions (e.g. stable versus unstable osteochondritis dissecans; lateral wall intact versus not intact), as these are more reliable.

Intra- and Interobserver Reliability of Size Measurement of Morton Neuromas on Sonography

OBJECTIVES

Sonography is often used for analysis of Morton neuromas, but the measurement reliability is unknown. The aim of this study was to determine the intra- and interobserver reliability and precision of sonographic measurement of Morton neuromas.

METHODS

Three observers with different levels of sonography experience (observer A, expert; observer B, intermediate; observer C, beginner) performed repeated measurements of the size of Morton's neuroma in 20 patients. Intraclass correlation coefficient and limit of agreement were used to evaluate reliability and measurement precision.

RESULTS

The intraclass correlation coefficient for intraobserver reliability was 0.80, 0.71, and 0.43 (limit of agreement, 20%, 18%, and 47%) in observers A, B, and C, respectively. The intraclass correlation coefficient for interobserver reliability was 0.72, 0.38, and 0.38 (limit of agreement, 25%, 36%, and 35%) between observers A/B, A/C, and B/C, respectively.

CONCLUSIONS

The intra- and interobserver reliability values for sonographic measurement of Morton neuroma size were dependent on observer experience. Experienced observers had higher reliability, while observers who were less familiar with sonography showed moderate to low reliability.

Repeatability, reproducibility, and agreement of three computational methods to approximate the functional flexion-extension axis of the tibiofemoral joint using 3D bone models of the femur

Background: Closely approximating the functional flexion-extension (FE) axis of the tibiofemoral joint in 3D models of the femur is important when computing joint motions which are physiologic. The objectives were to 1) develop methods to approximate the functional FE axis based on fitting circles, a tapered cylinder, and spheres to the posterior condyles, 2) determine the repeatability and reproducibility of each method, and 3) determine limits of agreement between pairs of axes. Methods: For each method, the respective axis was determined in forty 3D bone models of the distal femur. Varus-valgus angles and internal-external axial angles were computed relative to standard planes. Results: Repeatability and reproducibility were comparable for the tapered cylinder-based and sphere-based methods and better than that for the circle-based method. Limits of agreement were tightest when comparing the sphere-based and tapered cylinder-based axes. However, limits of agreement for the internal-external axial angle were wide at +3.6° to -3.9° whereas limits of agreement were tighter at +1.4° to -0.7° for the varus-valgus angle. Conclusion: The tapered cylinder-based and sphere-based methods offer advantages of better repeatability and reproducibility over the circle-based method. However, the sphere-based and tapered cylinder-based axes are not interchangeable owing to wide limits of agreement for the internal-external axial angle. The tapered cylinder-based axis is preferred intuitively over the sphere-based axis because the spheres require fitting in both the sagittal and coronal planes whereas the tapered cylinder requires fitting in the sagittal plane only which is the plane of motion in flexion-extension.

Agreement among equine veterinarians and between equine veterinarians and inertial sensor system during clinical examination of hindlimb lameness in horses

BACKGROUND

Hindlimb lameness evaluation is known to be challenging. Experience is essential for the ability of equine veterinarians to detect lameness. Nevertheless, even an experienced veterinarian is still subject to bias. Objective lameness detecting methods have been established to aid veterinarians.

OBJECTIVES

1) To estimate the effect of experience on the interobserver agreement and the agreement between a body-mounted inertial sensor system (BMISS) and veterinarians on detecting hindlimb lameness, and 2) to estimate the agreement between the BMISS and highly experienced veterinarians on change in lameness after diagnostic analgesia.

STUDY DESIGN

Cross-sectional study.

METHODS

Twenty-six horses with hindlimb lameness were evaluated in clinical conditions by clinicians and simultaneously measured by the BMISS. Videos of their lameness examination were recorded and shown to 13 veterinarians from three groups of varying experience for evaluation. The interobserver agreement and the agreement between veterinarians and the BMISS were calculated.

RESULTS

Interobserver agreement from all three groups was recorded as 'fair'. The strength of agreement between veterinarians and BMISS was 'fair' for the highly experienced group, 'slight to fair' for the moderately experienced group and 'slight' in the inexperienced group. The BMISS and the highly experienced veterinarians declared a 'strong' agreement in assigning an improvement in lameness after diagnostic analgesia.

MAIN LIMITATIONS

Lameness evaluation through video viewing might be more challenging for some evaluators than live situations.

CONCLUSIONS

Given the task of evaluating videos of horses trotting in a straight line, the more experienced veterinarians did not show more reliability than those with less experience. Due to 1) the moderate agreement between the BMISS and clinicians (highly experienced and moderately experienced) in the live clinical evaluation in determining hindlimb lameness, and 2) the strong association between the BMISS and highly experienced veterinarians in determining improvement of lameness after anaesthesia, therefore the use of the BMISS as a supporting tool for veterinarians is encouraged.

Diagnosis of Papillary Breast Lesions on Core Needle Biopsy: Upgrade Rates and Interobserver Variability

The histologic distinction between papillary breast lesions remains challenging, especially with core biopsy (CB) specimens. A retrospective review of the clinical, imaging, and histologic findings was performed for patients with papillary breast lesions on CB from 2013 to 2017. The interpretation accuracy was expressed as upgrade rate relative to the excision diagnosis. Diagnostic reproducibility with and without immunohistochemistry was analyzed as interobserver variability among 3 board-certified pathologists. Among 57 papillary lesions with biopsies and excisions available for review, the upgrade rates were 0% for benign papilloma, 30% for papilloma with atypical ductal hyperplasia, and 25% for papilloma with ductal carcinoma in situ, resulting in an overall upgrade rate of 11.1%. There were no statistical differences between patients in an upgrade group and others, when comparing the patient age, clinical presentation, BI-RADS (Breast Imaging Reporting and Database System) category, location, and histologic grade. The overall interobserver variability of the 60 consecutive core biopsies of papillary breast lesions by morphology alone was in the "substantial" agreement range (κ = 0.79, 86% agreement), with an excellent κ score of 0.88 for papilloma (92% agreement). "Substantial" and "fair" κ values were seen for papilloma with atypical ductal hyperplasia/ductal carcinoma in situ (0.74, 84% agreement) and invasive carcinoma (0.40, 60% agreement). Use of immunohistochemical stains improved the κ values into "excellent" range (0.92, 94% agreement). Our study favors a conservative approach in the management of benign papillomas, at least in cases of good radiologic-pathologic concordance. Papillary breast lesions with atypia/malignancy show lower diagnostic reproducibility on CB, and utility of immunohistochemistry is recommended in challenging cases.

A simulation study assessing the accuracy and reliability of orchidometer estimation of testicular volume

CONTEXT

Measuring testicular volume (TV) by orchidometer is the standard method of male pubertal staging. A paucity of evidence exists as to its inter- and intra-observer reliability and the impact of clinicians' gender, training and experience on accuracy.

OBJECTIVE

Prosthetic testicular models were engineered to investigate accuracy and reliability of TV estimation.

DESIGN

Simulation study.

SETTING

Conducted over three-day 2015 British Society for Paediatric Endocrinology and Diabetes (BSPED) meeting.

PARTICIPANTS

Two hundred fifteen meeting delegates (161F, 54M): 50% consultants, 30% trainees, 9% clinical nurse specialists, 11% other professionals.

INTERVENTION

Three child-sized mannequins displayed latex scrotum containing prosthetic testicles of 3, 4, 5, 10 and 20 mL. Demographic data, paediatric endocrinology experience, TV examination training, examination technique and TV estimations were collected. Delegates were asked to repeat their measurements later during the meeting. Scrotum order was changed daily.

MAIN OUTCOME MEASURES

Accuracy by variance from the simulated TV. Inter- and intra-observer variability.

RESULTS

One thousand two hundred eighty four individual estimations were obtained. Eighty-five participants repeated measurements. Delegates measured TV accurately on 33.4% (±2.6) of occasions: overestimations 37% (±2.3), underestimations 28% (±1.8) (Fleiss' Kappa score 0.04). The accuracy of assessing a 4 mL testis was 36%-39%. Observers underestimated the volume when paired with a 3 mL testis and overestimated when paired with a 5 mL testis demonstrating a tendency impose biological symmetry. Intra-observer reliability was lacking; individuals giving different estimations for the same size testicle on 61% (±4.2) of occasions, 20% (±3.5) of estimations were more than 1 size outside the previous measurement. On only 39% (±4.2) of occasions did individuals agree with their previous estimation (irrespective of whether or not it was initially accurate). Training did not impact on results but experience did improve accuracy.

CONCLUSIONS

Overall TV estimation accuracy was poor. Considerable variation exists between and within subjects. Seniority slightly improved measurement estimation.

Intraobserver and interobserver reliability of recategorized Neer classification in differentiating 2-part surgical neck fractures from multi-fragmented proximal humeral fractures in 116 patients

BACKGROUND

Optimal fracture classification should be simple and reproducible and should guide treatment. For proximal humeral fractures, the Neer classification is commonly used. However, intraobserver and interobserver reliability of the Neer classification has been shown to be poor. In clinical practice, it is essential to differentiate 2-part surgical neck fractures from multi-fragmented fractures. Thus, the aim of this study was to evaluate whether surgeons can differentiate 2-part surgical neck fractures from multi-fragmented fractures using plain radiographs and/or computed tomography (CT).

METHODS

Three experienced upper limb specialists and trauma surgeons (B.O.S., A.P.L., and V.L.) independently reviewed and classified blinded plain radiographs and CT scans of 116 patients as showing 2-part surgical neck fractures or multi-fragmented fractures. Each imaging modality was reviewed and classified separately by each surgeon, after which each surgeon reviewed both modalities at the same time. This process was repeated by all surgeons after 24 weeks. Intraobserver and interobserver analyses were conducted using Cohen and Fleiss κ values, respectively.

RESULTS

The κ coefficient for interobserver reliability showed substantial correlation (0.61-0.73) and was as follows: 0.73 for radiographs alone, 0.61 for CT scans alone, and 0.72 for radiographs and CT scans viewed together. After 24 weeks, the process was repeated and intraobserver reliability was calculated.The κ coefficient for intraobserver reliability showed substantial correlation (0.62-0.75) and was as follows: 0.62 for radiographs alone, 0.64 for CT scans alone, and 0.75 for radiographs and CT scans viewed together.

CONCLUSION

Clinicians were able to differentiate 2-part surgical neck fractures from multi-fragmented fractures based on plain radiographs reliably.

Interobserver Agreement for the Standardized Reporting System PSMA-RADS 1.0 on 18F-DCFPyL PET/CT Imaging

Recently, the standardized reporting and data system for prostate-specific membrane antigen (PSMA)-targeted PET imaging studies, termed PSMA-RADS version 1.0, was introduced. We aimed to determine the interobserver agreement for applying PSMA-RADS to imaging interpretation of ¹⁸F-DCFPyL (2-(3-{1-carboxy-5-[(6-¹⁸F-fluoro-pyridine-3-carbonyl)-amino]-pentyl}-ureido)-pentanedioic acid) PET examinations in a prospective setting mimicking the typical clinical workflow at a prostate cancer referral center. Methods: Four readers (2 experienced readers (ERs, >3 y of PSMA-targeted PET interpretation experience) and 2 inexperienced readers (IRs, <1 y of experience)), who had all read the initial publication on PSMA-RADS 1.0, assessed 50 ¹⁸F-DCFPyL PET/CT studies independently. Per scan, a maximum of 5 target lesions was selected by the observers, and a PSMA-RADS score for every target lesion was recorded. No specific preexisting conditions were placed on the selection of the target lesions, although PSMA-RADS 1.0 suggests that readers focus on the most avid or largest lesions. An overall scan impression based on PSMA-RADS was indicated, and interobserver agreement rates on a target lesion-based, on an organ-based, and on an overall PSMA-RADS score-based level were computed. Results: The number of target lesions identified by each observer was as follows: ER 1, 123; ER 2, 134; IR 1, 123; and IR 2, 120. Among those selected target lesions, 125 were chosen by at least 2 individual observers (all 4 readers selected the same target lesion in 58 of 125 [46.4%] instances, 3 readers in 40 of 125 [32%], and 2 observers in 27 of 125 [21.6%]). The interobserver agreement for PSMA-RADS scoring among identical target lesions was good (intraclass correlation coefficient [ICC] for 4, 3, and 2 identical target lesions, ≥0.60, respectively). For lymph nodes, an excellent interobserver agreement was derived (ICC, 0.79). The interobserver agreement for an overall scan impression based on PSMA-RADS was also excellent (ICC, 0.84), with a significant difference for ER (ICC, 0.97) vs. IR (ICC, 0.74) (P = 0.005). Conclusion: PSMA-RADS demonstrated a high concordance rate in this study, even among readers with different levels of experience. This finding suggests that PSMA-RADS can be effectively used for communication with clinicians and can be implemented in the collection of data for large prospective trials.

Interobserver reliability of radiographic assessment after radial head arthroplasty

BACKGROUND

Radiographs are part of routine clinical care after radial head arthroplasty (RHA). Therefore, the aim of this diagnostic study was to assess the interobserver reliability of radiographic assessment following RHA.

METHODS

Anteroposterior (AP) and lateral radiographs of 24 consecutive patients who underwent press-fit bipolar RHA were evaluated with respect to 14 parameters by 14 orthopaedic surgeons specializing in the elbow: shaft loosening (AP, lateral), subcollar bone resorption, nonbridging heterotopic ossification, capitellar erosion, capitellar osteopenia, implant size, ulnohumeral joint gapping, ulnohumeral joint degeneration, proximal radio-ulnar joint congruency, stem size, stem positioning (AP, lateral) and component dissociation or polyethylene wear of the head with increased angulation. Observer agreement was evaluated using the multirater kappa (κ) measure.

RESULTS

Nine of 14 parameters had poor interobserver agreement [κ = 0.0 to 0.20, confidence interval (CI) = 0.0 to 0.31). Four parameters had fair agreement: subcollar bone resorption (κ = 0.27, CI = 0.12 to 0.40), capitellar erosion (κ = 0.30, CI = 0.20 to 0.40), ulnohumeral joint degeneration (κ = 0.35, CI = 0.22 to 0.51) and stem positioning in AP view (κ = 0.24, CI = 0.14 to 0.36). One parameter had moderate agreement: nonbridging heterotopic ossification (κ = 0.47, CI = 0.31 to 0.64).

CONCLUSIONS

The overall interobserver reliability of radiographic assessment following press-fit bipolar RHA was poor among experienced elbow surgeons. Therefore, radiographic evaluation after RHA should be interpreted with caution when making treatment decisions.

Interobserver Variation of the Bolus-and-Burst Method for Pancreatic Perfusion with Dynamic - Contrast-Enhanced Ultrasound

PURPOSE

Dynamic contrast-enhanced ultrasound (DCE-US) can be used for calculating organ perfusion. By combining bolus injection with burst replenishment, the actual mean transit time (MTT) can be estimated. Blood volume (BV) can be obtained by scaling the data to a vessel on the imaging plane. The study aim was to test interobserver agreement for repeated recordings using the same ultrasound scanner and agreement between results on two different scanner systems.

MATERIALS AND METHODS

Ten patients under evaluation for exocrine pancreatic failure were included. Each patient was scanned two times on a GE Logiq E9 scanner, by two different observers, and once on a Philips IU22 scanner, after a bolus of 1.5 ml Sonovue. A 60-second recording of contrast enhancement was performed before the burst and the scan continued for another 30 s for reperfusion. We performed data analysis using MATLAB-based DCE-US software. An artery in the same depth as the region of interest (ROI) was used for scaling. The measurements were compared using the intraclass correlation coefficient (ICC) and Bland Altman plots.

RESULTS

The interobserver agreement on the Logiq E9 for MTT (ICC=0.83, confidence interval (CI) 0.46-0.96) was excellent. There was poor agreement for MTT between the Logiq E9 and the IU22 (ICC=-0.084, CI -0.68-0.58). The interobserver agreement for blood volume measurements was excellent on the Logiq E9 (ICC=0.9286, CI 0.7250-0.98) and between scanners (ICC=0.86, CI=0.50-0.97).

CONCLUSION

Interobserver agreement was excellent using the same scanner for both parameters and between scanners for BV, but the comparison between two scanners did not yield acceptable agreement for MTT. This was probably due to incomplete bursting of bubbles in some of the recordings on the IU22.

Prospective Comparison of Diagnostic Accuracy Between Point-of-Care and Conventional Ultrasound in a General Diagnostic Department: Implications for Resource-Limited Settings

OBJECTIVES

To compare the diagnostic accuracy of hand-held point-of-care (POC) versus conventional sonography in a general diagnostic setting with the intention to inform medical providers or clinicians on the rational use of POC ultrasound in resource limited settings.

METHODS

Over 3 months in 2010, 47 patients were prospectively enrolled at a single academic center to obtain 54 clinical conventional ultrasound examinations and 54 study-only POC ultrasound examinations. Indications were 48% abdominal, 26% retroperitoneal, and 24% obstetrical. Nine blinded readers (sonographers, residents, and attending radiologists) sequentially assigned diagnoses to POC and then conventional studies, yielding 476 interpreted study pairs. Diagnostic accuracy was obtained by comparing POC and conventional diagnoses to a reference diagnosis established by the unblinded, senior author. Analysis was stratified by study type, body mass index (BMI), diagnostic confidence, and image quality.

RESULTS

The mean diagnostic accuracy of conventional sonography was 84% compared with 74% for POC (P < .001). This difference was constant regardless of reader, exam type, or BMI. The sensitivity and specificity to detect abnormalities with conventional was 85 and 83%, compared with 75 and 68% for POC. The POC sonography demonstrated greater variability in image quality and diagnostic confidence, and this accounted for lower diagnostic accuracy. When image quality and diagnostic confidence were similar between POC and conventional examinations, there was no difference in accuracy.

CONCLUSIONS

Point-of-care was nearly as accurate as conventional sonography for basic, focused examinations. Observed differences in accuracy were attributed to greater variation in POC image quality.

Accuracy of Grading Gleason Score 7 Prostatic Adenocarcinoma on Needle Biopsy: Influence of Percent Pattern 4 and Other Histological Factors

BACKGROUND

Recognition of Gleason pattern 4 in prostatic needle biopsies is crucial for both prognosis and therapy. Recently, it has been recommended to record percent pattern 4 when Gleason score 7 cancer is the highest grade in a case.

METHODS

Four hundred and five prostate needle core biopsies received for a second opinion at our institution from February-June, 2015 were prospectively diagnosed with prostatic adenocarcinoma Gleason score 7 as the highest score on review by a consultant urological pathologist. Percentage of core involvement by cancer, percentage of Gleason pattern 4 per core, distribution of Gleason pattern 4 (clustered, scattered), morphology of pattern 4 (cribriform, non-cribriform), and whether the cancer was continuous or discontinuous were recorded.

RESULTS

Better agreement was noted between the consultant and referring pathologists when pattern 4 was clustered as opposed to dispersed in biopsies (P = 0.009). The percentage of core involvement by cancer, morphology of pattern 4, and continuity of cancer did not affect the agreement between the consultant and referring pathologists. There was a trend (P = 0.06) for better agreement based on the percent of pattern 4.

CONCLUSIONS

When pattern 4 is scattered amongst pattern 3 as opposed to being discrete foci, there is less interobserver reproducibility in grading Gleason score 7 cancer, and in this setting pathologists should consider obtaining second opinions either internally within their group or externally. Prostate 77: 681-685, 2017. © 2017 Wiley Periodicals, Inc.

68Ga-PSMA-11 PET/CT Interobserver Agreement for Prostate Cancer Assessments: An International Multicenter Prospective Study

The interobserver agreement for ⁶⁸Ga-PSMA-11 PET/CT study interpretations in patients with prostate cancer is unknown. Methods:⁶⁸Ga-PSMA-11 PET/CT was performed in 50 patients with prostate cancer for biochemical recurrence (n = 25), primary diagnosis (n = 10), biochemical persistence after primary therapy (n = 5), or staging of known metastatic disease (n = 10). Images were reviewed by 16 observers who used a standardized approach for interpretation of local (T), nodal (N), bone (Mb), or visceral (Mc) involvement. Observers were classified as having a low (<30 prior ⁶⁸Ga-PSMA-11 PET/CT studies; n = 5), intermediate (30-300 studies; n = 5), or high level of experience (>300 studies; n = 6). Histopathology (n = 25, 50%), post-external-beam radiation therapy prostate-specific antigen response (n = 15, 30%), or follow-up PET/CT (n = 10, 20%) served as a standard of reference. Observer groups were compared by overall agreement (% patients matching the standard of reference) and Fleiss' κ with mean and corresponding 95% confidence interval (CI). Results: Agreement among all observers was substantial for T (κ = 0.62; 95% CI, 0.59-0.64) and N (κ = 0.74; 95% CI, 0.71-0.76) staging and almost perfect for Mb (κ = 0.88; 95% CI, 0.86-0.91) staging. Level of experience positively correlated with agreement for T (κ = 0.73/0.66/0.50 for high/intermediate/low experience, respectively), N (κ = 0.80/0.76/0.64, respectively), and Mc staging (κ = 0.61/0.46/0.36, respectively). Interobserver agreement for Mb was almost perfect irrespective of prior experience (κ = 0.87/0.91/0.88, respectively). Observers with low experience, when compared with intermediate and high experience, demonstrated significantly lower median overall agreement (54% vs. 66% and 76%, P = 0.041) and specificity for T staging (73% vs. 88% and 93%, P = 0.032). Conclusion: The interpretation of ⁶⁸Ga-PSMA-11 PET/CT for prostate cancer staging is highly consistent among observers with high levels of experience, especially for nodal and bone assessments. Initial training on at least 30 patient cases is recommended to ensure acceptable performance.

Reproducibility of atypia of undetermined significance/follicular lesion of undetermined significance category using the bethesda system for reporting thyroid cytology when reviewing slides from different institutions: A study of interobserver variability among cytopathologists

BACKGROUND

The Bethesda System for Reporting Thyroid Cytopathology (TBSRTC) offers a six-tiered diagnostic scheme for thyroid Fine Needle Aspiration (FNA): Benign, Atypia of Undetermined Significance/Follicular Lesion of Undetermined Significance (AUS/FLUS), suspicious for follicular neoplasm, suspicious for malignancy, malignant, and unsatisfactory with an aim to standardize diagnostic criteria. Reported rate of AUS/FLUS category in the literature has varied from 3% to 20.5%.

METHODS

The aim of this study was to assess interobserver variability among cytopathologists to assess reproducibility of the AUS/FLUS category. Seven cytopathologists brought FNA cases (a mixture of atypical and non-atypical FNA diagnosis) diagnosed using TBSRTC from their respective institutions which were reviewed and diagnosed by the participants. The analysis assessed interobserver variability among 7 cytopathologists and determined characteristics on the slides which were associated with concordance to the institutional diagnosis.

RESULTS

Seventy eight of 125 (62.4%) benign cases were classified as benign by the reviewers and 26 (21%) were called AUS/FLUS on review. A third of the AUS/FLUS cases were called benign on review and 28.2% were classified as suspicious for neoplasia/malignancy. Roughly a third each of the suspicious for follicular neoplasm/suspicious for malignancy cases were classified as AUS/FLUS.

DISCUSSION

When pathologists from different institutions shared their slides, concordance was high for specimens with adequate cellularity and those that were clearly benign but thresholds varied for the other indeterminate categories. Most definite categorization of the AUS/FLUS category was seen on review. Diagn. Cytopathol. 2017;45:399-405. © 2017 Wiley Periodicals, Inc.

Templating in uncemented total hip arthroplasty-on intra- and interobserver reliability and professional experience

BACKGROUND

This study examines the intra-, and interobserver reliability of digital templating in uncemented total hip arthroplasty (THA), and assesses whether these values are dependent on professional experience.

METHODS

Three independent observers retrospectively examined digital X-rays of 34 consecutive hips scheduled for uncemented THA. These were templated using templating software. Evaluations were carried out on two occasions at least 6 weeks apart. Findings were compared to each surgeon's own findings, and then to the other surgeons' findings. Data underwent statistical analysis to assess and describe reliability.

RESULTS

The intraobserver reliability of the method was found to be good. The intra-class correlation coefficient (ICC) for individual surgeons ranged from 0.81 to 0.87 for acetabular components and 0.74 to 0.91 for femoral components. However, it was somewhat lower for neck length with kappa statistics (κ) from 0.41 to 0.51 with agreement in about 70% of the cases. Interobserver reliability was similar, with an ICC of 0.87 for the acetabular component and 0.79 for the femoral component, but somewhat lower for neck length with κ of 0.27 and agreement in 41% of the cases. We found no association between increasing experience and increasing precision, as the least experienced observer showed the highest intraobserver reliability.

CONCLUSIONS

The reliability of digital templating of uncemented THA is good for acetabular and femoral components, but inferior for neck length. Precision does not rely on professional experience. Digital templating provides surgeons with a valuable tool for preoperative planning, but cannot supersede the intraoperative assessment and final decision.

Validation of diagnostic accuracy with whole-slide imaging compared with glass slide review in dermatopathology

BACKGROUND

Teledermatopathology has evolved from static images to whole slide imaging (WSI), which allows for remote viewing and manipulation of tissue sections. Previous studies of WSI in teledermatopathology predated College of American Pathologists (CAP) telepathology validation guidelines.

OBJECTIVE

We conducted a comprehensive retrospective WSI validation study of routine dermatopathology cases, adhering to CAP guidelines.

METHOD

In all, 181 consecutive cases arranged into 3 categories (inflammatory, melanocytic, nonmelanocytic proliferations) were reviewed by 3 board-certified dermatopathologists via traditional microscopy (TM) and WSI. Intraobserver (TM vs WSI), TM intraobserver and interobserver (TM vs TM), and WSI interobserver (WSI vs WSI) concordance was interpreted using a 3-tier system.

RESULTS

TM versus WSI intraobserver concordance (86.9%; 95% confidence interval [CI] 83.7-89.6) did not differ from TM versus TM intraobserver concordance (90.3%; 95% CI 86.7-93.1) or interobserver concordance (WSI: 89.9%; 95% CI 87.0-92.2, and TM: 89.5%; 95% CI 86.5-91.9). Melanocytic proliferations had the lowest TM versus WSI intraobserver concordance (75.6%; 95% CI 68.5-81.5), whereas inflammatory lesions had the highest TM versus WSI intraobserver concordance (96.1%; 95% CI 91.8-98.3). Nonmelanocytic proliferations had an intraobserver concordance of 89.1% (95% CI 83.4-93.0).

LIMITATIONS

Efficiency and other logistical WSI parameters were not evaluated.

CONCLUSION

Intraobserver and interobserver diagnostic concordance between WSI and TM was equivalent. Therefore, WSI appears to be a reliable diagnostic modality for dermatopathology.

68Ga-DOTATATE PET/CT Interobserver Agreement for Neuroendocrine Tumor Assessment: Results of a Prospective Study on 50 Patients

We evaluated observer agreement for ⁶⁸Ga-DOTATATE PET/CT interpretations in patients with neuroendocrine tumor (NET).

METHODS

⁶⁸Ga-DOTATATE PET/CT was performed on 50 patients with known or suspected NET of the small bowel (n = 19), pancreas (n = 14), lung (n = 4), or other location (n = 13). The images were reviewed by 7 observers, who used a standardized interpretation approach. The observers were classified as having a low level of experience (<500 scans or <5 y experience with ⁶⁸Ga-DOTATATE PET/CT; n = 4) or a high level of experience (≥500 scans or ≥5 y experience with ⁶⁸Ga-DOTATATE PET/CT; n = 3). Interpretation by the primary nuclear medicine physician, who had access to all clinical and imaging data, served as the reference standard. Interobserver agreement was determined by the Cohen κ statistic and intraclass correlation coefficient (ICC) with corresponding 95% confidence interval (95%CI).

RESULTS

Interobserver agreement was substantial, and the median number of false findings was low for the overall scan result: that is, positive versus negative scan result (κ = 0.80; 95%CI, 0.74-0.86; false findings, 3), organ involvement (κ = 0.70; 95%CI, 0.64-0.76; false findings, 5), and lymph node involvement (κ = 0.71; 95%CI, 0.65-0.78; false findings, 6). Interobserver agreement was substantial to almost perfect, and the average absolute difference (Δ) from the reference observer was low for number of organ and lymph node metastases (organ: ICC, 0.84; 95%CI, 0.77-0.89; Δ = 0.45; lymph node: ICC, 0.77; 95%CI, 0.69-0.84; Δ = 0.45), tumor SUV_max (ICC, 0.99; 95%CI, 0.97-0.99; Δ = 0.44), and reference SUV (spleen: ICC, 0.81; Δ = 1.10; liver: ICC, 0.79; Δ = 0.62). Interpretations of appropriateness for peptide-receptor radionuclide therapy varied more significantly among observers (κ = 0.64; 95%CI, 0.57-0.70), and a higher frequency of false-positive recommendations for peptide-receptor radionuclide therapy occurred in observers with low experience than in those with high experience (range, 7-12 vs. 4-8).

CONCLUSION

The interpretation of ⁶⁸Ga-DOTATATE PET/CT images for NET staging is consistent among observers with low and high levels of experience. However, image-based recommendations for or against peptide-receptor radionuclide therapy require experience and training.

Interobserver agreement in the cytologic grading of atypia in neoplastic pancreatic mucinous cysts with the 2-tiered approach

BACKGROUND

The accurate cytologic grading of epithelial atypia in fine-needle aspirates of pancreatic mucinous cysts has important implications for clinical management. The Papanicolaou Society of Cytopathology has recommended a 2-tiered system of low-grade (LG) and high-grade (HG) for grading this atypia. Using this approach, this study examined the interobserver agreement within a group of cytopathologists at the Cleveland Clinic.

METHODS

Twenty cases of fine-needle aspiration of pancreatic neoplastic mucinous cysts with documented histologic follow-up and representative lesional cells were selected. Blinded to the histologic outcome, 4 cytopathologists were independently asked to assign the highest grade of atypia with the 2-tiered system of LG and HG atypia for these cases. The interobserver agreement was calculated with the κ statistic.

RESULTS

The overall raw agreement in the grading of atypia was 60%. The overall chance-adjusted agreement was fair (κ = 0.28). On the basis of the histologic outcomes, the cases were stratified into group A (HG dysplasia or worse) and group B (LG or intermediate-grade [IG] dysplasia on follow-up). Group A (n = 12) showed good chance-adjusted agreement (κ = 0.65). For group B, the chance-adjusted agreement among the observers was poor (κ = 0.03).

CONCLUSIONS

This study shows that the cytologic recognition of HG dysplasia or worse as HG atypia in pancreatic mucinous cysts has a good degree of interobserver reproducibility among cytopathologists. In contrast, a problematic area with a lack of agreement appears to be the cytologic recognition of LG and IG dysplasia as LG atypia. Additional studies with the development of reproducible criteria and educational tools may help with this challenging distinction. Cancer Cytopathol 2016;124:909-916. © 2016 American Cancer Society.

Corticosteroid pre-treated primary CNS lymphoma: a detailed analysis of stereotactic biopsy findings and consideration of interobserver variability

Prior corticosteroid therapy presents a major challenge in the diagnosis of CNS lymphomas, particularly in stereotactic biopsies. In this study we analysed the cytological, histopathological and immunohistochemical features in stereotactic biopsies of 25 primary CNS lymphoma cases pre-treated with corticosteroids. We documented the extent and the frequency of each finding. We also investigated the significance of subjectivity in evaluation of these biopsies in 3 seperate sessions including the final diagnostic decision. In 48% of our cases the diagnosis was straightforward. These cases were characterized by prominent blasts either in diffuse paranchymal infiltrates or in perivascular regions. The remaining 52% demonstrated some degree of variability among pathologists. Lymphoid atypia other than the typical blastic morphology appeared as a subjective finding and this was more pronounced in cytology preparations. In our study, corticosteroid pre-treatment in primary CNS lymphoma was associated with a large spectrum of histopathological, immunohistochemical and cytological findings. Combined use of an extended immunohistochemical panel would increase the possibility of conclusive diagnosis. Nevertheless some of these findings and therefore the diagnosis are open to subjectivity.

Reliability of the classification and treatment of dislocations of the acromioclavicular joint

BACKGROUND

We evaluated interobserver and intraobserver reliability of the classification and treatment of acromioclavicular (AC) joint dislocations and assessed the impact of adding 3-dimensional computed tomography (3D CT) on the reliability of classification and treatment choice.

METHODS

Ten surgeons independently reviewed plain radiographs and 3D CT in 28 cases with AC joint dislocation. Images from each case were randomly presented to the observers, with plain radiographs alone being presented first, followed by plain radiographs plus 3D CT 2 weeks later. Four weeks later, they repeated the same survey to evaluate intraobserver reliability. Reliability was assessed on the basis of Fleiss κ values.

RESULTS

On the basis of plain radiographs alone, interobserver and intraobserver reliability of the Rockwood classification were fair (κ = .214) and moderate (κ = .474), respectively. Interobserver and intraobserver reliability of treatment were both fair (κ = .213 and .399, respectively). On the basis of a combination of plain radiographs and 3D CT, interobserver and intraobserver reliability of the Rockwood classification were slight (κ = .177) and moderate (κ = .565), respectively. Interobserver and intraobserver reliability of treatment were fair (κ = .253) and moderate (κ = .554), respectively. There were no significant differences in reliability between the two groups in terms of any κ values.

CONCLUSION

This study suggests an overall lack of reliability of the Rockwood classification of AC joint dislocations and of decisions regarding their treatment. There is especially poor agreement between experienced shoulder surgeons. The addition of 3D CT did not improve reliability of classification and treatment of AC joint dislocations.

Decision-making in the treatment of diaphyseal clavicle fractures: is there agreement among surgeons? Results of a survey on surgeons' treatment preferences

BACKGROUND AND HYPOTHESIS

Nonoperative treatment is standard for most diaphyseal clavicle fractures, but recent studies have demonstrated improved outcomes with operative treatment of displaced fractures. The objectives of this diagnostic study were to assess agreement of orthopaedic surgeons regarding their treatment preferences for diaphyseal clavicle fractures and to compare them with recent recommendations. Interobserver and intraobserver agreement in treatment decisions were hypothesized to be only slight.

METHODS

Anonymized case vignettes of 50 acute diaphyseal clavicle fractures including medical history, physical examination findings, and radiographs were independently reviewed by 32 orthopaedic surgeons from the United States. Four treatment options were offered and decisions were compared with current treatment recommendations. Interobserver agreement was calculated using Fleiss' kappa coefficient. Average intraobserver agreement for surgeons who completed a retest review (minimum interval of 8 weeks) was calculated.

RESULTS

Thirty-two surgeons completed the first round of reviewing and 27 completed the retest (mean interval, 22 weeks). Interobserver agreement was overall fair (kappa = 0.36) and moderate (kappa = 0.56) when operative options were compared with nonoperative options. Median intraobserver agreement was 74% for the 4 treatment options offered and 84% in deciding on operative vs. nonoperative means. Concordance with recent recommendations for operative vs. nonoperative treatment was seen in 91% of decisions (median).

DISCUSSION AND CONCLUSIONS

Recent recommendations appear to have been adopted by a selected subgroup of U.S. orthopaedic surgeons, showing a surprisingly high median concordance of 91% in this study. However, only fair to moderate interobserver and intraobserver agreement was present, leaving potential for improvement.

An assessment of intra- and interobserver agreement of reporting orthopaedic findings on presale radiographs of Thoroughbred yearlings

REASONS FOR PERFORMING STUDY

Radiography is commonly used in clinical practice but agreement of reporting of radiographically detected orthopaedic findings in horses has rarely been studied.

OBJECTIVES

To assess agreement within and between observers for reporting of orthopaedic findings on presale radiographs of Thoroughbred yearlings.

STUDY DESIGN

Retrospective analysis of archived radiographs.

METHODS

Four veterinary radiology specialists each twice examined 167 sets of radiographs for orthopaedic findings in the fore feet, fore and hind fetlocks, carpi, tarsi and stifles. There were 27 findings analysed for agreement. Kappa statistic (κ), percentage of positive agreement (Ppos) and percentage of negative agreement are reported.

RESULTS

An excellent percentage of negative agreement was observed for all findings, with the exception of regular vascular channels of the proximal sesamoid bones. Ppos and κ results were variable. The presence of extra carpal bones, osseous cyst-like lesions of the ulnar carpal bone, sagittal ridge defects of the third metacarpus, fracture of the fore and hind proximal sesamoid bones, regular vascular channels in the hind proximal sesamoids, osteochondrosis lesions of the distal intermediate ridge and/or medial malleoli of the tibia, and osseous cyst-like lesions in the medial femoral condyle was consistently observed with an intra- and interagreement κ≥0.5 and Ppos≥50%. Lucency within the proximal sesamoids consistently had an observed intra- and interagreement κ<0.4 and Ppos<40%.

CONCLUSIONS

Observation of orthopaedic findings on yearling repository radiographs showed generally excellent agreement on the absence of findings, but variable agreement on the presence of findings. Agreement was good for larger and easy to categorise radiographic findings. More accurate definitions and training need to be developed to improve agreement within and between observers for orthopaedic findings with poor or fair to good agreement.

Measuring hindfoot alignment radiographically: the long axial view is more reliable than the hindfoot alignment view

BACKGROUND

Hindfoot malalignment is a recognized cause of foot and ankle disability. For preoperative planning and clinical follow-up, reliable radiographic assessment of hindfoot alignment is important. The long axial radiographic view and the hindfoot alignment view are commonly used for this purpose. However, their comparative reliabilities are unknown. As hindfoot varus or valgus malalignment is most pronounced during mid-stance of gait, a unilateral weight-bearing stance, in comparison with a bilateral stance, could increase measurement reliability. The purpose of this study was to compare the intra- and interobserver reliability of hindfoot alignment measurements of both radiographic views in bilateral and unilateral stance.

MATERIALS AND METHODS

A hindfoot alignment view and a long axial view were acquired from 18 healthy volunteers in bilateral and unilateral weight-bearing stances. Hindfoot alignment was defined as the angular deviation between the tibial anatomical axis and the calcaneus longitudinal axis from the radiographs. Repeat measurements of hindfoot alignment were performed by nine orthopaedic examiners.

RESULTS

Measurements from the hindfoot alignment view gave intra- and interclass correlation coefficients (CCs) of 0.72 and 0.58, respectively, for bilateral stance and 0.91 and 0.49, respectively, for unilateral stance. The long axial view showed, respectively, intra- and interclass CCs of 0.93 and 0.79 for bilateral stance and 0.91 and 0.58 for unilateral stance.

CONCLUSION

The long axial view is more reliable than the hindfoot alignment view or the angular measurement of hindfoot alignment. Although intra-observer reliability is good/excellent for both methods, only the long axial view leads to good interobserver reliability. A unilateral weight-bearing stance does not lead to greater reliability of measurement.

CT-based classification of long spinal allograft fusion

Anterior column reconstruction of the thoracolumbar spine by structural allograft has an increased potential for biological fusion when compared to synthetic reconstructive options. Estimation of cortical union and trabecular in-growth is, however, traditionally based on plain radiography, a technique lacking in sensitivity. A new assessment method of bony union using high-speed spiral CT imaging is proposed which reflects the gradually increasing biological stability of the construct. Grade I (complete fusion) implies cortical union of the allograft and central trabecular continuity. Grade II (partial fusion) implies cortical union of the structural allograft with partial trabecular incorporation. Grade III (unipolar pseudarthrosis) denotes superior or inferior cortical non-union of the central allograft with partial trabecular discontinuity centrally and Grade IV (bipolar pseudarthrosis) suggests both superior and inferior cortical non-union with a complete lack of central trabecular continuity. Twenty-five patients underwent anterior spinal reconstruction for a single level burst fracture between T4 and L5. At a minimum of two years follow up the subjects underwent high-speed spiral CT scanning through the reconstructed region of the thoracolumbar spine. The classification showed satisfactory interobserver (kappa score = 0.91) and intraobserver (kappa score = 0.95) reliability. The use of high-speed CT imaging in the assessment of structural allograft union may allow a more accurate assessment of union. The classification system presented allows a reproducible categorization of allograft incorporation with implications for treatment.