Overall indices for assessing agreement among multiple raters

Using multiple agreement methods for continuous repeated measures data: a tutorial for practitioners

Background

Studies of agreement examine the distance between readings made by different devices or observers measuring the same quantity. If the values generated by each device are close together most of the time then we conclude that the devices agree. Several different agreement methods have been described in the literature, in the linear mixed modelling framework, for use when there are time-matched repeated measurements within subjects.

Methods

We provide a tutorial to help guide practitioners when choosing among different methods of assessing agreement based on a linear mixed model assumption. We illustrate the use of five methods in a head-to-head comparison using real data from a study involving Chronic Obstructive Pulmonary Disease (COPD) patients and matched repeated respiratory rate observations. The methods used were the concordance correlation coefficient, limits of agreement, total deviation index, coverage probability, and coefficient of individual agreement.

Results

The five methods generated similar conclusions about the agreement between devices in the COPD example; however, some methods emphasized different aspects of the between-device comparison, and the interpretation was clearer for some methods compared to others.

Conclusions

Five different methods used to assess agreement have been compared in the same setting to facilitate understanding and encourage the use of multiple agreement methods in practice. Although there are similarities between the methods, each method has its own strengths and weaknesses which are important for researchers to be aware of. We suggest that researchers consider using the coverage probability method alongside a graphical display of the raw data in method comparison studies. In the case of disagreement between devices, it is important to look beyond the overall summary agreement indices and consider the underlying causes. Summarising the data graphically and examining model parameters can both help with this.

Brazilian-Portuguese Linguistic Validation of the Velopharyngeal Insufficiency Effects on Life Outcome Instrument

BACKGROUND

The purpose of this study was to conduct a linguistic validation of the velopharyngeal insufficiency (VPI) Effects on Life Outcome (VELO) instrument for use in Brazilian-Portuguese patients with VPI.

METHODS

The original English version of the VELO instrument was translated into Brazilian-Portuguese, back-translated, and adapted among the Brazilian patients (n = 21) with VPI and their parents, based on the standardized guidelines for the cross-culture adaption process. Discrepancies in the forward and backward translation steps were computed. Comprehension rates were captured for each debriefing interview. The content validity index (CVI) per item (I-CVI) and of the scale (S-CVI universal agreement [S-CVI/UA] and averaging [S-CVI/Ave]) were calculated.

RESULTS

Reconciliation of the 2 forward translations and the comparison between the back translation and the original VELO version resulted in some item wordings with discrepancies which were reviewed by the research team (translators, expert committee, and original developers of instrument). Three rounds of cognitive interviews also led to some revisions of wording. Comprehension rates of patients and their parents were 60% to 100%, 80% to 100%, and 100% in the first, second, and third rounds of cognitive interviews, respectively. The I-CVI, S-CVI/AU, and SCI/Ave for the Brazilian-Portuguese VELO version were 0.83 (or higher), 0.83 (or higher), and 0.97, respectively.

CONCLUSIONS

The linguistic validation process of the VELO instrument created a cross-culturally equivalent Brazilian-Portuguese version for use in Brazilian-Portuguese speaking patients with VPI.