Another look at interrater agreement

Interrater reliability estimators tested against true interrater reliabilities

BACKGROUND

Interrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate. Almost all agree that percent agreement (a_o), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters' random rating. The experts, however, disagree on which chance estimators are legitimate or better. The experts also disagree on which of the three factors, rating category, distribution skew, or task difficulty, an index should rely on to estimate chance agreement, or which factors the known indices in fact rely on. The most popular chance-adjusted indices, according to a functionalist view of mathematical statistics, assume that all raters conduct intentional and maximum random rating while typical raters conduct involuntary and reluctant random rating. The mismatches between the assumed and the actual rater behaviors cause the indices to rely on mistaken factors to estimate chance agreement, leading to the numerous paradoxes, abnormalities, and other misbehaviors of the indices identified by prior studies.

METHODS

We conducted a 4 × 8 × 3 between-subject controlled experiment with 4 subjects per cell. Each subject was a rating session with 100 pairs of rating by two raters, totaling 384 rating sessions as the experimental subjects. The experiment tested seven best-known indices of interrater reliability against the observed reliabilities and chance agreements. Impacts of the three factors, i.e., rating category, distribution skew, and task difficulty, on the indices were tested.

RESULTS

The most criticized index, percent agreement (a_o), showed as the most accurate predictor of reliability, reporting directional r² = .84. It was also the third best approximator, overestimating observed reliability by 13 percentage points on average. The three most acclaimed and most popular indices, Scott's π, Cohen's κ and Krippendorff's α, underperformed all other indices, reporting directional r² = .312 and underestimated reliability by 31.4 ~ 31.8 points. The newest index, Gwet's AC₁, emerged as the second-best predictor and the most accurate approximator. Bennett et al's S ranked behind AC₁, and Perreault and Leigh's I_r ranked the fourth both for prediction and approximation. The reliance on category and skew and failure to rely on difficulty explain why the six chance-adjusted indices often underperformed a_o, which they were created to outperform. The evidence corroborated the notion that the chance-adjusted indices assume intentional and maximum random rating while the raters instead exhibited involuntary and reluctant random rating.

CONCLUSION

The authors call for more empirical studies and especially more controlled experiments to falsify or qualify this study. If the main findings are replicated and the underlying theories supported, new thinking and new indices may be needed. Index designers may need to refrain from assuming intentional and maximum random rating, and instead assume involuntary and reluctant random rating. Accordingly, the new indices may need to rely on task difficulty, rather than distribution skew or rating category, to estimate chance agreement.

Reliability in evaluator-based tests: using simulation-constructed models to determine contextually relevant agreement thresholds

BACKGROUND

Indices of inter-evaluator reliability are used in many fields such as computational linguistics, psychology, and medical science; however, the interpretation of resulting values and determination of appropriate thresholds lack context and are often guided only by arbitrary "rules of thumb" or simply not addressed at all. Our goal for this work was to develop a method for determining the relationship between inter-evaluator agreement and error to facilitate meaningful interpretation of values, thresholds, and reliability.

METHODS

Three expert human evaluators completed a video analysis task, and averaged their results together to create a reference dataset of 300 time measurements. We simulated unique combinations of systematic error and random error onto the reference dataset to generate 4900 new hypothetical evaluators (each with 300 time measurements). The systematic errors and random errors made by the hypothetical evaluator population were approximated as the mean and variance of a normally-distributed error signal. Calculating the error (using percent error) and inter-evaluator agreement (using Krippendorff's alpha) between each hypothetical evaluator and the reference dataset allowed us to establish a mathematical model and value envelope of the worst possible percent error for any given amount of agreement.

RESULTS

We used the relationship between inter-evaluator agreement and error to make an informed judgment of an acceptable threshold for Krippendorff's alpha within the context of our specific test. To demonstrate the utility of our modeling approach, we calculated the percent error and Krippendorff's alpha between the reference dataset and a new cohort of trained human evaluators and used our contextually-derived Krippendorff's alpha threshold as a gauge of evaluator quality. Although all evaluators had relatively high agreement (> 0.9) compared to the rule of thumb (0.8), our agreement threshold permitted evaluators with low error, while rejecting one evaluator with relatively high error.

CONCLUSIONS

We found that our approach established threshold values of reliability, within the context of our evaluation criteria, that were far less permissive than the typically accepted "rule of thumb" cutoff for Krippendorff's alpha. This procedure provides a less arbitrary method for determining a reliability threshold and can be tailored to work within the context of any reliability index.

On the Performance of the Marginal Homogeneity Test to Detect Rater Drift

When constructed response items are administered repeatedly, "trend scoring" can be used to test for rater drift. In trend scoring, raters rescore responses from the previous administration. Two simulation studies evaluated the utility of Stuart's Q measure of marginal homogeneity as a way of evaluating rater drift when monitoring trend scoring. In the first study, data were generated based on trend scoring tables obtained from an operational assessment. The second study tightly controlled table margins to disentangle certain features present in the empirical data. In addition to Q, the paired t test was included as a comparison, because of its widespread use in monitoring trend scoring. Sample size, number of score categories, interrater agreement, and symmetry/asymmetry of the margins were manipulated. For identical margins, both statistics had good Type I error control. For a unidirectional shift in margins, both statistics had good power. As expected, when shifts in the margins were balanced across categories, the t test had little power. Q demonstrated good power for all conditions and identified almost all items identified by the t test. Q shows substantial promise for monitoring of trend scoring.

The Exact Variance of Weighted Kappa with Multiple Raters

Weighted kappa described by Cohen in 1968 is widely used in psychological research to measure agreement between two independent raters. Everitt then provided the exact variance for weighted kappa for two raters. In this paper, Everitt's exact variance is extended to three or more raters.

Estimating the Accuracy of Decisions Based on Cutting Scores

Classifications based on test results are used routinely in educational research and practice. Although test validity usually is expressed as a correlation coefficient, this does not indicate the expected accuracy of pass-fail or eligible-not eligible decisions based on test scores. This paper describes expectancy tables for converting a validity or test-retest reliability coefficient, r, into measures of classification accuracy for dichotomous categories. Results for a sample of correlation coefficients and cut-off scores are reported using several indicators of accuracy: sensitivity, efficiency, specificity, hit rate, and kappa. It appears that a validity coefficient above .90 is required to achieve a kappa above .70 and to keep false positive and false negative error rates below 25%. This suggests that many tests and measures considered to have adequate validity (between .60 and .90) often will have limited utility in making diagnostic, placement, or treatment decisions.

Field Reliability of Comprehensive System Scoring in an Adolescent Inpatient Sample

The extent to which the Comprehensive System for the Rorschach is reliably scored has been a topic of some controversy. Although several studies have concluded it can be scored reliably in research settings, little is known about its reliability in field settings. This study evaluated the reliability of both response-level codes and protocol-level scores among 84 adolescent psychiatric inpatients in a clinical setting. Rorschachs were originally administered and scored for clinical purposes. Among response codes, 87% demonstrated acceptable reliability(> .60), and most coefficients exceeded .80. Results were similar for protocol-level scores, with only one score demonstrating less than adequate reliability. The findings are consistent with previous evidence, indicating reliable scoring is possible even in field settings.

Argument Integration and Attitude Change: Suppression Effects in the Integration of One-Sided Arguments that Vary in Persuasiveness

Petty and Cacioppo's elaboration likelihood model of persuasion and Chaiken, Liberman, and Eagly `s heuristic-systematic model suggest that for highly involved message recipients, adding weaker arguments to strong arguments could suppress or dilute the overall persuasiveness of a message. The few previous studies addressing this prediction, however, have provided conflicting evidence. In the present study, involvement level, number of weak arguments, and number of strong arguments were varied factorially in a message advocating the institution of senior comprehensive exams. Results provided clear support for the predicted weak-argument suppression of attitude change. Analyses of thought-listing data supported the notion that suppression results from the integration of favorable and unfavorable cognitive responses to the communication. Further research questions regarding the processes by which mixed-quality messages exert theirpersuasive impact are discussed.

Performance of ACMG-AMP Variant-Interpretation Guidelines among Nine Laboratories in the Clinical Sequencing Exploratory Research Consortium

Evaluating the pathogenicity of a variant is challenging given the plethora of types of genetic evidence that laboratories consider. Deciding how to weigh each type of evidence is difficult, and standards have been needed. In 2015, the American College of Medical Genetics and Genomics (ACMG) and the Association for Molecular Pathology (AMP) published guidelines for the assessment of variants in genes associated with Mendelian diseases. Nine molecular diagnostic laboratories involved in the Clinical Sequencing Exploratory Research (CSER) consortium piloted these guidelines on 99 variants spanning all categories (pathogenic, likely pathogenic, uncertain significance, likely benign, and benign). Nine variants were distributed to all laboratories, and the remaining 90 were evaluated by three laboratories. The laboratories classified each variant by using both the laboratory's own method and the ACMG-AMP criteria. The agreement between the two methods used within laboratories was high (K-alpha = 0.91) with 79% concordance. However, there was only 34% concordance for either classification system across laboratories. After consensus discussions and detailed review of the ACMG-AMP criteria, concordance increased to 71%. Causes of initial discordance in ACMG-AMP classifications were identified, and recommendations on clarification and increased specification of the ACMG-AMP criteria were made. In summary, although an initial pilot of the ACMG-AMP guidelines did not lead to increased concordance in variant interpretation, comparing variant interpretations to identify differences and having a common framework to facilitate resolution of those differences were beneficial for improving agreement, allowing iterative movement toward increased reporting consistency for variants in genes associated with monogenic disease.

Inter-Coder Agreement in One-to-Many Classification: Fuzzy Kappa

Content analysis involves classification of textual, visual, or audio data. The inter-coder agreement is estimated by making two or more coders to classify the same data units, with subsequent comparison of their results. The existing methods of agreement estimation, e.g., Cohen’s kappa, require that coders place each unit of content into one and only one category (one-to-one coding) from the pre-established set of categories. However, in certain data domains (e.g., maps, photographs, databases of texts and images), this requirement seems overly restrictive. The restriction could be lifted, provided that there is a measure to calculate the inter-coder agreement in the one-to-many protocol. Building on the existing approaches to one-to-many coding in geography and biomedicine, such measure, fuzzy kappa, which is an extension of Cohen’s kappa, is proposed. It is argued that the measure is especially compatible with data from certain domains, when holistic reasoning of human coders is utilized in order to describe the data and access the meaning of communication.

Use of Google Street View to Assess Environmental Contributions to Pedestrian Injury

OBJECTIVES

To demonstrate an information technology-based approach to assess characteristics of streets and intersections associated with injuries that is less costly and time-consuming than location-based studies of pedestrian injury.

METHODS

We used imagery captured by Google Street View from 2007 to 2011 to assess 9 characteristics of 532 intersections within New York City. We controlled for estimated pedestrian count and estimated the relation between intersections' characteristics and frequency of injurious collisions.

RESULTS

The count of pedestrian injuries at intersections was associated with the presence of marked crosswalks (80% increase; 95% confidence interval [CI] = 2%, 218%), pedestrian signals (156% increase; 95% CI = 69%, 259%), nearby billboards (42% increase; 95% CI = 7%, 90%), and bus stops (120% increase; 95% CI = 51%, 220%). Injury incidence per pedestrian was lower at intersections with higher estimated pedestrian volumes.

CONCLUSIONS

Consistent with in-person study observations, the information-technology approach found traffic islands, visual advertising, bus stops, and crosswalk infrastructures to be associated with elevated counts of pedestrian injury in New York City. Virtual site visits for pedestrian injury control studies are a viable and informative methodology.

Multimethod latent class analysis

Correct and, hence, valid classifications of individuals are of high importance in the social sciences as these classifications are the basis for diagnoses and/or the assignment to a treatment. The via regia to inspect the validity of psychological ratings is the multitrait-multimethod (MTMM) approach. First, a latent variable model for the analysis of rater agreement (latent rater agreement model) will be presented that allows for the analysis of convergent validity between different measurement approaches (e.g., raters). Models of rater agreement are transferred to the level of latent variables. Second, the latent rater agreement model will be extended to a more informative MTMM latent class model. This model allows for estimating (i) the convergence of ratings, (ii) method biases in terms of differential latent distributions of raters and differential associations of categorizations within raters (specific rater bias), and (iii) the distinguishability of categories indicating if categories are satisfyingly distinct from each other. Finally, an empirical application is presented to exemplify the interpretation of the MTMM latent class model.

A sequential examination of parent-child interactions at anesthetic induction

Parental presence is often employed to alleviate distress in children within the context of surgery under general anesthesia. The critical component of this intervention may not be the presence of the parent per se, but more importantly the behaviors in which the parent and child engage when the parent is present. The purpose of the current study was to examine the sequential and reciprocal relationships between parental behaviors and child distress during induction of general anesthesia. Participants were 32 children (3-6 years) receiving dental surgery as a day surgery procedure, and their parents. A modified Child Adult Medical Procedures Interaction Scale-Revised was used to code parent and child behaviors. Initial child distress led to increased parental provision of reassurance and decreased provision of physical comfort. Our findings may inform the development of preoperative preparation programs whereby parents can be appropriately educated about what behaviors will be helpful/unhelpful for their child during induction of general anesthesia.

The Effect of the Raters' Marginal Distributions on Their Matched Agreement: A Rescaling Framework for Interpreting Kappa

Cohen's κ measures the improvement in classification above chance level and it is the most popular measure of interjudge agreement. Yet, there is considerable confusion about its interpretation. Specifically, researchers often ignore the fact that the observed level of matched agreement is bounded from above and below and the bounds are a function of the particular marginal distributions of the table. We propose that these bounds should be used to rescale the components of κ (observed and expected agreement). Rescaling κ in this manner results in κ', a measure that was originally proposed by Cohen (1960) and was largely ignored in both research and practice. This measure provides a common scale for agreement measures of tables with different marginal distributions. It reaches the maximal value of 1 when the judges show the highest level of agreement possible, given their marginal disagreements. We conclude that κ' should be used to measure the level of matched agreement contingent on a particular set of marginal distributions. The article provides a framework and a set of guidelines that facilitate comparisons between various types of agreement tables. We illustrate our points with simulations and real data from two studies-one involving judges' ratings of baseball players and one involving ratings of essays in high-stakes tests.

Online evaluation of fetal second-trimester four-chamber view images: a comparison of six evaluation methods

OBJECTIVE

To compare six online evaluation methods for auditing routine second-trimester four-chamber view still images.

METHODS

We evaluated three different scoring grids (subjective, five-item score and seven-item score), which were applied with or without access to online help, resulting in a total of six evaluation methods. For the subjective scoring grid, images were rated as excellent, good, fair, poor or very poor. For the five-item score, 1 point was allocated for visualization (vs non-visualization or non-evaluable) of each of: heart crux, atria, ventricles, apex and aorta, yielding a score of 0-5. For the seven-item score, 1 point was allocated for clear (vs unclear) visualization of each of: moderator band at the apex, interventricular septum, atrioventricular valves, non-linear insertion of atrioventricular valves (normal offset), septum primum, aorta and pulmonary vein. Each evaluation method was used via the Internet by three randomly selected reviewers, who evaluated the same set of 80 images. Reviewers were experienced in fetal ultrasound, but were not involved in the design of the study. Interrater agreement was the main outcome.

RESULTS

The five-item scoring grid with online help achieved the best interrater agreement (interrater intraclass correlation coefficient = 0.7).

CONCLUSIONS

Evaluation of the second-trimester sonographic four-chamber view is apparently best achieved with a simple five-item scoring grid.

Validation of a novel method to identify healthcare-associated infections

Despite its potential for use in large-scale analyses, previous attempts to utilise administrative data to identify healthcare-associated infections (HAI) have been shown to be unsuccessful. In this study, we validate the accuracy of a novel method of HAI identification based on antibiotic utilisation patterns derived from administrative data. We contemporaneously and independently identified HAIs using both chart review analysis and our method from four Japanese hospitals (N=584). The accuracy of our method was quantified using sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) relative to chart review analysis. We also analysed the inter-rater agreement between both identification methods using Cohen's kappa coefficient. Our method showed a sensitivity of 0.93 (95% CI: 0.87-0.96), specificity of 0.91 (0.89-0.94), PPV of 0.75 (0.68-0.81) and NPV of 0.98 (0.96-0.99). A kappa coefficient of 0.78 indicated a relatively high level of agreement between the two methods. Our results show that our method has sufficient validity for identification of HAIs in large groups of patients, though the relatively lower PPV may imply limited utilisation in the pinpointing of individual infections. Our method may have applications in large-scale HAI identification, risk-adjusted multicentre studies involving cost of illness, or even as the starting point of future cost-effectiveness analyses of HAI control measures.

Assessment of diagnostic tools for identifying cattle shedding and super-shedding Escherichia coli O157:H7 in a longitudinal study of naturally infected feedlot steers in Ohio

The objectives of this study were to compare the performance of different diagnostic protocols (rectoanal mucosal swabs and immunomagnetic separation [RAMS-IMS], fecal samples and IMS [fecal-IMS], and direct plating) to determine the prevalence of Escherichia coli O157:H7 and to evaluate the pattern of E. coli O157:H7 shedding and super-shedding (defined as having a direct plating count equal to or >10(4) colony forming units of E. coli O157:H7 per gram of feces) in a longitudinal study of naturally infected feedlot steers. RAMS and fecal grab samples were obtained at 14-day intervals from 168 Angus-cross beef steers over a period of 22 weeks. Fecal samples were assessed by direct plating and IMS, whereas RAMS were tested only by enrichment followed by IMS to recover E. coli O157:H7. The period prevalence for shedding was high (62%) among feedlot steers and super-shedding was higher (23%) than anticipated. Although direct plating was the least sensitive method to detect E. coli O157:H7-positive samples, over 20% of high bacterial load samples were not detected by RAMS-IMS and/or fecal-IMS. The sensitivity of RAMS-IMS, fecal-IMS, and direct plating protocols was estimated using simple and multilevel mixed-effects logistic regression models, in which the dependent variable was the dichotomous results of each test and gold standard (i.e., parallel interpretation of the three protocols)-positive individuals were included as an independent variable along with other factors such as dietary supplements, time of sampling, and being exposed to a super-shedding pen-mate. The associations between these factors and the sensitivity of the diagnostic protocols were not statistically significant. In conclusion, differences in the reported impact of diet and probiotics on the shedding of E. coli O157:H7 in previous studies using RAMS-IMS or fecal-IMS were unlikely due to their impact on test performance.

Peirce'siand Cohen'sκfor2×2Measures of Rater Reliability

This study examined a historical mixture model approach to the evaluation of ratings made in “gold standard” and two-rater2×2contingency tables. Peirce'siand the derivediaverage were discussed in relation to a widely used index of reliability in the behavioral sciences, Cohen'sκ. Sample size, population base rate of occurrence, the true “science of the method”, and guessing rates were manipulated across simulations. In “gold standard” situations, Peirce'sitended to recover the true reliability of ratings as well as better thanκ. In two-rater situations,iavetended to recover the true reliability as well as better thanκin most situations. The empirical utility and potential theoretical benefits of mixture model methods in estimating reliability are discussed, as are the associations between theistatistics and other modern mixture model approaches.

Resampling Probability Values for Weighted Kappa with Multiple Raters

A new procedure to compute weighted kappa with multiple raters is described. A resampling procedure to compute approximate probability values for weighted kappa with multiple raters is presented. Applications of weighted kappa are illustrated with an example analysis of classifications by three independent raters.

Evaluation of a new binary system of grading oral epithelial dysplasia for prediction of malignant transformation

The aim of this paper is to assess the reproducibility of a novel binary grading system (high/low risk) of oral epithelial dysplasia and to compare it with the WHO classification 2005. The accuracy of the new system for predicting malignant transformation was also assessed. Ninety-six consecutive oral epithelial dysplasia biopsies with known clinical outcomes were retrieved from the Oral Pathology archives. A pilot study was conducted on 28 cases to determine the process of classification. Four observers then reviewed the same set of H&E stained slides of 68 oral dysplastic lesions using the two grading systems blinded to the clinical outcomes. The overall inter-observer unweighted and weighted kappa agreements for the WHO grading system were Ks = 0.22 (95% CI: 0.11-0.35), Kw = 0.63 (95% CI: 0.42-0.78), respectively, versus K = 0.50 (95% CI: 0.35-0.67) for the new binary system. Interestingly, all pathologists showed satisfactory agreement on the distinction of mild dysplasia from severe dysplasia and from carcinoma in situ using the new WHO classification. However, assessment of moderate dysplasia remains problematic. The sensitivity and specificity of the new binary grading system for predicting malignant transformation in oral epithelial dysplasia were 85% and 80%, respectively and the accuracy was 82%. The new binary grading system complemented the WHO Classification 2005 and may have merit in helping clinicians to make critical clinical decisions particularly for the cases of moderate dysplasia. Histological grading of dysplasia using established criteria is a reproducible prognosticator in oral epithelial dysplasia. Furthermore, the present study showed that more consensus scoring on either the degree of dysplasia, assessment of risk or the presence of each morphological characteristic by a panel should be encouraged.

Inter-observer reliability and intra-observer reproducibility of the Weber classification of ankle fractures

Our aim was to assess the reproducibility and the reliability of the Weber classification system for fractures of the ankle based on anteroposterior and lateral radiographs. Five observers with varying clinical experience reviewed 50 sets of blinded radiographs. The same observers reviewed the same radiographs again after an interval of four weeks. Inter- and intra-observer agreement was assessed based on the proportion of agreement and the values of the kappa coefficient. For inter-observer agreement, the mean kappa value was 0.61 (0.59 to 0.63) and the proportion of agreement was 78% (76% to 79%) and for intra-observer agreement the mean kappa value was 0.74 (0.39 to 0.86) with an 85% (60% to 93%) observed agreement. These results show that the Weber classification of fractures of the ankle based on two radiological views has substantial inter-observer reliability and intra-observer reproducibility.

Comparison of validity of assessment methods using indices of adjusted agreement

For comparing the validity of rating methods, the adjusted kappa (S coefficient) and Yule's Y index are better than Cohen's kappa which is affected by marginal probabilities. We consider a validity study in which a subject is assessed as exposed or not-exposed by two competing rating methods and the gold standard. We are interested in one of the methods, which is closer in agreement with the gold standard. We present statistical methods taking correlations into account for comparing the validity of the rating methods using S coefficient and Y index. We show how the S coefficient and Yule's Y index are related to sensitivity and specificity. In comparing the two rating methods, the preference is clear when the inference is the same for both S and Y. If the inference using S differs from that using Y, then it is not obvious how to decide a preference. This may occur when one rating method is better than the other in sensitivity but not in specificity. Numerical examples for comparing asbestos-exposure assessment methods are illustrated.