Variability in observer performance studies experimental observations

Reader performance in radiographic diagnosis of signs of mitral regurgitation in cavalier King Charles spaniels

OBJECTIVES

To measure accuracy and variability of diagnosis by radiography of heart enlargement (HE) and heart failure (HF) in mitral regurgitation (MR).

METHODS

Sixteen readers representing four levels of experience evaluated 50 sets of radiographs with varying severity of MR for presence or absence of HE, left atrial enlargement (LAE) and HF. The performance of the readers was compared with a reference standard, using area under the curve (AUC) of receiver operating characteristic (ROC) curves. The interreader agreement value kappa (K) was calculated. A subset of difficult cases of HF was analysed before and after removing an outlying reader from each group.

RESULTS

AUC for HE was 0.89, for LAE it was 0.93 and for HF it was 0.92. Experience increased certainty of diagnosis but not accuracy. K ranges were HE, 0.53 to 0.67; LAE, 0.61 to 0.69 and HF, 0.49 to 0.58. When only difficult cases of HF were read, accuracy decreased and experienced readers performed better than inexperienced. When outlying readers were excluded, the differences between experienced and inexperienced readers increased.

CLINICAL SIGNIFICANCE

LAE, not HE, should be used to evaluate the heart size and indirectly the severity of MR on radiographs. For HF, agreement among individual readers was only moderate. Studies of reader accuracy should consider the effects of interreader variability.

The "laboratory" effect: comparing radiologists' performance and variability during prospective clinical and laboratory mammography interpretations

PURPOSE

To compare radiologists' performance during interpretation of screening mammograms in the clinic with their performance when reading the same mammograms in a retrospective laboratory study.

MATERIALS AND METHODS

This study was conducted under an institutional review board-approved, HIPAA-compliant protocol; the need for informed consent was waived. Nine experienced radiologists rated an enriched set of mammograms that they had personally read in the clinic (the "reader-specific" set) mixed with an enriched "common" set of mammograms that none of the participants had previously read in the clinic by using a screening Breast Imaging Reporting and Data System (BI-RADS) rating scale. The original clinical recommendations to recall the women for a diagnostic work-up, for both reader-specific and common sets, were compared with their recommendations during the retrospective experiment. The results are presented in terms of reader-specific and group-averaged sensitivity and specificity levels and the dispersion (spread) of reader-specific performance estimates.

RESULTS

On average, the radiologists' performance was significantly better in the clinic than in the laboratory (P = .035). Interreader dispersion of the computed performance levels was significantly lower during the clinical interpretations (P < .01).

CONCLUSION

Retrospective laboratory experiments may not represent either expected performance levels or interreader variability during clinical interpretations of the same set of mammograms in the clinical environment well.

Reliability studies of diagnostic tests are not using enough observers for robust estimation of interobserver agreement: a simulation study

OBJECTIVE

Any attempt to generalize the performance of a subjective diagnostic method should take into account the sample variation in both cases and readers. Most current measures of the performance of a test, especially the indices of reliability, only tackle the variation of cases, and hence are not suitable for generalizing results across the population of readers. We attempted to study the effect of readers' variation on two measures of multireader reliability: pair-wise agreement and Fleiss' kappa.

STUDY DESIGN AND SETTING

We used a normal hierarchical model with a latent trait (signal) variable to simulate a binary decision-making task by different number of readers on an infinite sample of cases.

RESULTS

It could be shown that both measures, especially Fleiss' kappa, have a large sample variance when estimated by a small number of readers, casting doubt on their accuracy given the number of readers typically used in current reliability studies.

CONCLUSION

The majority of the current agreement studies is likely limited by the number of readers and is unlikely to produce a reliable estimate of reader agreement.

Reader variance in ROC studies--generalizability to reader population at high and low performance levels

RATIONALE AND OBJECTIVES

To investigate the variability between discriminative performances of readers as a function of average performance levels during receiver operating characteristic (ROC) studies.

MATERIALS AND METHODS

Four subsets of cases from previously ascertained ROC rating data by 12 observers when detecting interstitial disease and pneumothorax on posteroanterior chest films were selected for each abnormality and reanalyzed to assess changes in "reader" variance component. The subsets were selected based on a prestudy subjective assessment of the subtleness of depicted abnormality (positive cases) and the difficulty in determining its absence (negative cases). Reader variance component was estimated using a bootstrap approach for each subset and the results were used to assess a general relationship between variability and average performance level.

RESULTS

The reader variance component decreased substantially (from 0.007704 to 0.000426), as expected, when the areas under the ROC curves (AUC) for detecting pneumothoraces increased from 84% to 97%. On the other hand, reader variance component increased substantially (from 0.000890 to 0.005181) when AUC for detecting interstitial disease increased from 59% to 87%. The large magnitude of and changes in the reader variance component resulted in a consistent nonmonotone relationship as a function of AUC when other related variance components were included in addition to the reader component.

CONCLUSION

Among several factors affecting generalizability of ROC results to the population of readers, the reader variance component depended nonmonotonically on the average diagnostic performance and is lowest at both very high and very low levels of performance.

The effect of image display size on observer performance an assessment of variance components

RATIONALE AND OBJECTIVE

Our goal was to investigate the effect of the displayed image size on variance components during the performance of an observer performance study to detect masses on abdominal computed tomography (CT) examinations.

MATERIALS AND METHODS

A previously performed receiver operating characteristic (ROC) study with eight observers to detect abdominal masses on 166 CT examinations was reanalyzed to assess variance components when comparing two similar modes with displayed image sizes varying by a factor of 2. Case, mode, and reader-related variance components were estimated for the group of eight observers and subsets of readers after excluding each of the participants.

RESULTS

There was no significant difference in the average area under the ROC curves between the two modes using the two image sizes (P > .05). Reader and reader-by-case variability were substantially larger for the mode displaying enlarged images for the group and all subsets formed by excluding a single reader. Reader variability was affected by one observer who actually performed better with the enlarged images.

CONCLUSION

Sequential viewing of enlarged CT images for the detection of abdominal masses did not improve performance and increased reader variability.