The Conundrum of Kappa and Why Some Musculoskeletal Tests Appear Unreliable Despite High Agreement: A Comparison of Cohen Kappa and Gwet AC to Assess Observer Agreement When Using Nominal and Ordinal Data

A comparison between perceived rurality and established geographic rural status among Indiana residents

The study assessed the association and concordance of the traditional geography-based Rural-Urban Commuting Area (RUCA) codes to individuals' self-reported rural status per a survey scale. The study included residents from rural and urban Indiana, seen at least once in a statewide health system in the past 12 months. Surveyed self-reported rural status of individuals obtained was measured using 6 items with a 7-point Likert scale. Cronbach's alpha was used to measure the internal consistency between the 6 survey response items, along with exploratory factor analysis to evaluate their construct validity. Perceived rurality was compared with RUCA categorization, which was mapped to residential zip codes. Association and concordance between the 2 measures were calculated using Spearman's rank correlation coefficient and Gwet's Agreement Coefficient (Gwet's AC), respectively. Primary self-reported data were obtained through a cross-sectional, statewide, mail-based survey, administered from January 2018 through February 2018, among a random sample of 7979 individuals aged 18 to 75, stratified by rural status and race. All 970 patients who completed the survey answered questions regarding their perceived rurality. Cronbach's alpha value of 0.907 was obtained indicating high internal consistency among the 6 self-perceived rurality items. Association of RUCA categorization and self-reported geographic status was moderate, ranging from 0.28 to 0.41. Gwet's AC ranged from -0.11 to 0.26, indicating poor to fair agreement between the 2 measures based on the benchmark scale of reliability. Geography-based and self-report methods are complementary in assessing rurality. Individuals living in areas of relatively high population density may still self-identify as rural, or individuals with long commutes may self-identify as urban.

Comparison of health information exchange data with self-report in measuring cancer screening

BACKGROUND

Efficient measurement of the receipt of cancer screening has been attempted with electronic health records (EHRs), but EHRs are commonly implemented within a single health care setting. However, health information exchange (HIE) includes EHR data from multiple health care systems and settings, thereby providing a more population-based measurement approach. In this study, we set out to understand the value of statewide HIE data in comparison to survey self-report (SR) to measure population-based cancer screening.

METHODS

A statewide survey was conducted among residents in Indiana who had been seen at an ambulatory or inpatient clinical setting in the past year. Measured cancer screening tests included colonoscopy and fecal immunochemical test (FIT) for colorectal cancer, human papilloma virus (HPV) and Pap tests for cervical cancer, and mammogram for breast cancer. For each screening test, the self-reported response for receipt of the screening (yes/no) and 'time since last screening' were compared with the corresponding information from patient HIE to evaluate the concordance between the two measures.

RESULTS

Gwet's AC for HIE and self-report of screening receipt ranged from 0.24-0.73, indicating a fair to substantial concordance. For the time since receipt of last screening test, the Gwet's AC ranged from 0.21-0.90, indicating fair to almost perfect concordance. In comparison with SR data, HIE data provided relatively more additional information about laboratory-based tests: FIT (19% HIE alone vs. 4% SR alone) and HPV tests (27% HIE alone vs. 12% SR alone) and less additional information about procedures: colonoscopy (8% HIE alone vs. 23% SR alone), Pap test (13% HIE alone vs. 19% SR alone), or mammography (9% HIE alone vs. 10% SR alone).

CONCLUSION

Studies that use a single data source should consider the type of cancer screening test to choose the optimal data collection method. HIE and self-report both provided unique information in measuring cancer screening, and the most robust measurement approach involves collecting screening information from both HIE and patient self-report.

Initial validation of an intelligent video surveillance system for automatic detection of dairy cattle lameness

Introduction

Lameness is a major welfare challenge facing the dairy industry worldwide. Monitoring herd lameness prevalence, and early detection and therapeutic intervention are important aspects of lameness control in dairy herds. The objective of this study was to evaluate the performance of a commercially available video surveillance system for automatic detection of dairy cattle lameness (CattleEye Ltd).

Methods

This was achieved by first measuring mobility score agreement between CattleEye and two veterinarians (Assessor 1 and Assessor 2), and second, by investigating the ability of the CattleEye system to detect cows with potentially painful foot lesions. We analysed 6,040 mobility scores collected from three dairy farms. Inter-rate agreement was estimated by calculating percentage agreement (PA), Cohen's kappa (κ) and Gwet's agreement coefficient (AC). Data regarding the presence of foot lesions were also available for a subset of this dataset. The ability of the system to predict the presence of potentially painful foot lesions was tested against that of Assessor 1 by calculating measures of accuracy, using lesion records during the foot trimming sessions as reference.

Results

In general, inter-rater agreement between CattleEye and either human assessor was strong and similar to that between the human assessors, with PA and AC being consistently above 80% and 0.80, respectively. Kappa agreement between CattleEye and the human scorers was in line with previous studies (investigating agreement between human assessors) and within the fair to moderate agreement range. The system was more sensitive than Assessor 1 in identifying cows with potentially painful lesions, with 0.52 sensitivity and 0.81 specificity compared to the Assessor's 0.29 and 0.89 respectively.

Discussion

This pilot study showed that the CattleEye system achieved scores comparable to that of two experienced veterinarians and was more sensitive than a trained veterinarian in detecting painful foot lesions.

The interrater reliability of Modic changes among a potential basivertebral nerve ablation population: Why AC1 may be preferred to kappa

Interrater reliability of Modic changes is subject to variables which affect consistency in reporting. Given the importance of Modic change identification for basivertebral nerve ablation (BVNA) candidacy, interrater reliability for this specific cohort has not yet been reported. Twenty lumbar magnetic resonance images of potential basivertebral nerve candidates were independently reviewed by two neuroradiologists and two interventional spine physiatrists for the presence and characterization of Modic changes. The kappa value of their agreement on the presence of Modic changes was 0.52 (95% confidence interval [CI] 0.37-0.67), whereas agreement on the type of Modic change was 0.51 (95% CI 0.37-0.65). Using an alternative methodology for measuring interrater reliability (Gwet's AC1) yielded the identification of the presence of Modic changes at AC1 0.51 (95% CI 0.36-0.66), whereas agreement on the type of Modic change was AC1 0.75 (95% CI 0.66-0.83). While less common, AC1 may be preferred in the appropriate cohort to kappa as it mitigates some of the pitfalls to which kappa values may be victim. Ultimately, our results are in-line with previous reports of interrater reliability results for Modic changes in other cohorts and should serve to caution those who perform BVNA regarding interrater agreement of the imaging crux of the procedure.

Intra and inter observer agreement in the mobility assessment of the upper thoracic costovertebral joints

BACKGROUND

Determining the mobility of the costovertebral joints might be important in patients with neck-upper thoracic pain. Little research has been performed on observer agreement when assessing the mobility of these joints.

PURPOSE

The purpose of this study was to determine intra- and inter-observer agreements when assessing costovertebral joint mobility of the upper three ribs in those with and without neck pain and to compare the difference between the Kappa and AC statistic.

METHODS

Forty-four participants, with and without current neck/upper-thoracic pain, were assessed by two raters. Raters applied a posterior to anterior pressure to the anatomical neck of the first three ribs bilaterally. Mobility was graded: normal, increased, or decreased. An AC1, for nominal data, AC2 for ordinal data, and the Kappa statistics were used to analyze the results. The AC statistics determines chance agreement different than Kappa.

RESULTS

AC1 showed "moderate to very good" (0.74-1.0) intra-rater reliability, while inter-rater reliability showed "fair to good" agreement (0.51-0.79). Using the AC2 Intra-rater reliability was "very good" and "almost perfect" (AC2: 0.93-1.0), while using the AC2 was "good to very good" and "good to almost perfect" (0.76-0.94). Kappa values for intra-rater reliability ranged from "fair to moderate" (0.38-0.54), while inter-rater reliability ranged from "poor to fair" (-0.10-0.26).

CONCLUSION

Posterior/anterior pressure is a reliable method to assess the mobility of the upper costovertebral joints. Assessing costovertebral mobility is important when establishing a movement diagnosis in patients with neck/upper thoracic pain.

Effect of application time of 38% silver diamine fluoride solution on arresting early childhood caries in preschool children: a randomised double-blinded controlled trial protocol

Background

To study the caries lesion activity response to topical 38% silver diamine fluoride (SDF) therapy with increasing treatment application time.

Methods/design

The design is a stratified-randomised, double-blind, active-controlled, parallel-group clinical trial with nine treatment arms. The trial will involve recruiting at least 414 3- to 5-year-old kindergarten children with caries, who will receive approximately 0.004 mL of 38% SDF (the typical amount applied per the manufacturer’s instructions) to treat each caries lesion. The children will be stratified by caries status, randomised by blocks, and allocated to nine groups of SDF application times: 3, 5, 10, 15, 30, 45, 60, 120, and 180 s. The outcome measure is caries lesion activity (active/arrest) at the tooth-surface level at 6 months post-initial treatment. A calibrated dentist will conduct the blinded clinical examinations at baseline and at the 6-month follow-up. In addition, the parents will be surveyed to examine the effects of the moderating variables, such as oral hygiene, on caries lesion activity. The hypothesis is that a monotonically increasing trend can be found between the SDF application time and the proportion of caries lesions that are arrested. The Cochran-Armitage test for trends in proportions, corrected for clustering within children, will be used to determine the relationship between the exposure to SDF (the SDF application time) and the response (proportion of lesions arrested) in children, taking into consideration the effect of the moderating variables as well as the nesting of multiple caries lesions within an individual child. An EC 80 analysis (an 80% maximal concentration) will be used to determine the exposure (the SDF application time) for 80% caries lesion arrest. Bootstrap methods will be used for clustered data and will be resampled by clustering to determine the 95% confidence interval.

Discussion

This study will help with determining the optimal application time for SDF treatment. It will provide an evidence-based protocol for the use of SDF to arrest tooth decay in the primary teeth of young children. The results will inform an evidence-based SDF protocol to arrest caries, which affects 573 million children with tooth decay worldwide.

Trial registration

ClinicalTrials.gov NCT04655430. Registered on 7th December 2020.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13063-022-06130-1.