The Composite Quality Score (CQS) as an Appraisal Tool for Prospective, Controlled Clinical Therapy Trials: Rationale and Current Evidence

BACKGROUND

Current evidence appraisal concepts, such as the Assessment, Development and Evaluation (GRADE) approach and Cochrane's Risk of Bias (RoB) tool, rely on assumptions related to the classic problem of inductive reasoning and may suffer from insufficient inter-rater reliability.

DISCUSSION

The Composite Quality Score (CQS) has emerged as a possible trial appraisal tool that does not rely on inductive assumptions and has been shown to be of potentially very high inter-rater reliability.

CONCLUSION

Although the current CQS concept is still under development, its current evidence is encouraging and justifies further study. This article presents the rationale and currently available research concerning the CQS and shows where further research is required.

The Composite Quality Score (CQS) as a trial appraisal tool: inter-rater reliability and rating time

OBJECTIVE

To establish the CQS inter-rater reliability and rating time and to compare both against that of the Jadad scale and Cochrane's Risk of Bias Tool (ROBT).

MATERIAL AND METHODS

Four independent raters rated 45 trial reports. The inter-rater reliability was established by use of the Brennan-Prediger coefficient (BPC). The coefficients were compared using the two-sample z-test. Secondary analysis included comparison of the inter-rater reliability of the randomization component of all tools, as well as of the allocation concealment component of the CQS to that of the ROBT. The mean rating time with standard deviation (SD) for each tool was determined using one-way repeated measures analysis of variance. Post hoc comparisons were made using the Tukey-Kramer adjustment for three pair-wise multiple comparisons.

RESULTS

The inter-rater reliability was significantly higher for the CQS (BPC, 95% CI: 0.95, 0.87-1.00) compared to Jadad (0.70, 0.58-0.82) (adjusted p = 0.0005) and most components of ROBT. The mean (SD) time to complete the CQS (4.0 (1.0) min) did not differ significantly from that of the Jadad scale 4.8 (1.1) min (adjusted p = 0.11), but was significantly shorter compared to that of the ROBT 15.3 (5.9) min (adjusted p < 0.0001).

CONCLUSIONS

The results suggest the CQS to be a very reliable and fast trial appraisal tool. Clinical relevance The higher the inter-rater reliability, the higher the probability that trial results reflect therapeutic truth. The CQS will need to take further bias sources into consideration, in order to increase its utility.