Evaluating the accuracy and economic value of a new test in the absence of a perfect reference test

Is the Comparator in Your Diagnostic Cost-Effectiveness Model "Standard of Care"? Recommendations from Literature Reviews and Expert Interviews on How to Identify and Operationalize It

OBJECTIVES

This research aimed to develop best-practice recommendations for identifying the "standard of care" (SoC) and integrate it when it is the comparator in diagnostic economic models (SoC comparator).

METHODS

A multi-methods approach comprising 2 pragmatic literature reviews and 9 expert interviews was used. Experts rated their agreement with draft recommendations based on the authors' analysis of the reviews. These were refined iteratively to produce final recommendations.

RESULTS

Fourteen best-practice recommendations are provided. Care pathway mapping (using quantitative, qualitative, or mixed-methods approaches) should be used for identifying the SoC comparator. Guidelines analysis can be integrated with expert opinion to identify pathway variability and discrepancies from clinical practice. For integrating the SoC comparator into the model, recommendations around structure, input sourcing, data aggregation and reporting, input uncertainty, and model variability are presented. For example, modelers should consider that the reference standard is not synonymous with the SoC, and the SoC may not be the only comparator. The comparator limitations should be discussed with clinical experts, but elicitation of its diagnostic accuracy is not recommended. Probabilistic sensitivity analysis is recommended when evaluating the overall input uncertainty, and deterministic sensitivity analysis is useful when there is high model uncertainty or SoC variability. Consensus could not be reached for some topics (eg, the role of real-world data, model averaging, and alternative model structures), but the reported discussions provide points for consideration.

CONCLUSIONS

To our knowledge, this is the first guidance to support modelers when identifying and operationalizing the SoC comparator in diagnostic cost-effectiveness models.

Validation of RT-qPCR test for SARS-CoV-2 in saliva specimens

Background

Saliva samples may be an easier, faster, safer, and cost-saving alternative to NPS samples, and can be self-collected by the patient. Whether SARS-CoV-2 RT-qPCR in saliva is more accurate than in nasopharyngeal swaps (NPS) is uncertain. We evaluated the accuracy of the RT-qPCR in both types of samples, assuming both approaches were imperfect.

Methods

We assessed the limit of detection (LoD) of RT-qPCR in each type of sample. We collected paired NPS and saliva samples and tested them using the Berlin Protocol to detect SARS-CoV-2 envelope protein (E). We used a Bayesian latent class analysis (BLCA) to estimate the sensitivity and specificity of each test, while accounting for their conditional dependence.

Results

The LoD were 10 copies/mL in saliva and 100 copies/mL in NPS. Paired samples of saliva and NPS were collected in 412 participants. Out of 68 infected cases, 14 were positive only in saliva. RT-qPCR sensitivity ranged from 82.7% (95% CrI: 54.8, 94.8) in NPS to 84.5% (50.9, 96.5) in saliva. Corresponding specificities were 99.1 (95% CrI: 95.3, 99.8) and 98.4 (95% CrI: 92.8, 99.7).

Conclusions

SARS-CoV-2 RT-qPCR test in saliva specimens has a similar or better accuracy than RT-qPCR test in NPS. Saliva specimens may be ideal for surveillance in general population, particularly in children, and in healthcare or other personnel in need of serial testing.

Developing Economic Models for Assessing the Cost-Effectiveness of Multiple Diagnostic Tests: Methods and Applications

Background

Clinical pathways with multiple diagnostic tests are complex to model, but problematic and simplistic approaches are often used in economic evaluations.

Methods

We analyzed statistical methods of handling multiple diagnostic tests and provided guidance on applying these methods in economic modeling. We first introduced a statistical model to quantify the correlations between 2 tests and how those correlations can be incorporated within an economic model. We also presented the general form of conditional dependence among multiple tests. We then introduced net reclassification improvement (NRI), a measure that evaluates the added value of a new risk factor (e.g., biomarker) for risk prediction. We further provided 2 examples to illustrate the application of these methods.

Results

Our first example illustrated how to model an add-on test to an existing test, in the absence of a perfect reference standard. After accounting for the imperfect nature of both tests and the conditional dependence between tests, the potential health benefits from the additional test were reduced. This led to differential cost-effectiveness results when comparing models using the perfect test and conditional independence assumptions. The second example illustrated how to evaluate the added value of a new risk factor using the NRI measure. Using the new risk classification provides greater precision in risk prediction, and in the example, the strategy using the new risk classification with treatment for selected individuals led to more favorable cost-effectiveness results.

Conclusions

These innovative methods for handling multiple diagnostic tests have improved the methodology within the field and should be adopted to provide more accurate estimates within cost-effectiveness analyses.

Highlights

Economic evaluations of multiple diagnostic tests often apply problematic simplistic approaches, such as ignoring conditional dependence between 2 tests or assuming a perfect final test in the diagnostic pathway. We provided guidance on how to apply improved methods for economic modeling. We introduced methods to model conditional dependence between 2 imperfect tests. We used an example to illustrate how assumptions about perfect diagnostic test accuracy and conditional independence between tests affect cost-effectiveness. Compared with the results of the area under the receiver-operating-characteristic curve, net reclassification improvement has distinct advantages in measuring the added value of a new risk factor for model-based economic evaluation. Economic evaluations that appropriately account for the complexities of diagnostic test pathways can help decision makers ensure efficient use of resources.

Economic Analyses of Respiratory Tract Infection Diagnostics: A Systematic Review

BACKGROUND

Diagnostic testing for respiratory tract infections is a tool to manage the current COVID-19 pandemic, as well as the rising incidence of antimicrobial resistance. At the same time, new European regulations for market entry of in vitro diagnostics, in the form of the in vitro diagnostic regulation, may lead to more clinical evidence supporting health-economic analyses.

OBJECTIVE

The objective of this systematic review was to review the methods used in economic evaluations of applied diagnostic techniques, for all patients seeking care for infectious diseases of the respiratory tract (such as pneumonia, pulmonary tuberculosis, influenza, sinusitis, pharyngitis, sore throats and general respiratory tract infections).

METHODS

Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, articles from three large databases of scientific literature were included (Scopus, Web of Science and PubMed) for the period January 2000 to May 2020.

RESULTS

A total of 70 economic analyses are included, most of which use decision tree modelling for diagnostic testing for respiratory tract infections in the community-care setting. Many studies do not incorporate a generally comparable clinical outcome in their cost-effectiveness analysis: fewer than half the studies (33/70) used generalisable outcomes such as quality-adjusted life-years. Other papers consider outcomes related to the accuracy of the test or outcomes related to the prescribed treatment. The time horizons of the studies generally are limited.

CONCLUSIONS

The methods to economically assess diagnostic tests for respiratory tract infections vary and would benefit from clear recommendations from policy makers on the assessed time horizon and outcomes used.

A New Approach for Classifying Fetal Growth Restriction

BACKGROUND

Fetal growth restriction is commonly defined using small for gestational age (SGA) birth (birthweight < 10th percentile) as a proxy, but this approach is problematic because most SGA infants are small but healthy. In this proof-of-concept study, we sought to develop a new approach for identifying fetal growth restriction at birth that combines information on multiple, imperfect measures of fetal growth restriction in a probabilistic manner.

METHODS

We combined information on birthweight, placental weight, placental malperfusion lesions, maternal disease, and fetal acidemia using latent profile analysis to classify fetal growth in births at the Royal Victoria Hospital in Montreal, Canada, 2001-2009. We examined the clinical characteristics and health outcomes of infants classified as growth-restricted and nongrowth-restricted by our model, and among the subgroup of growth-restricted infants who had a birthweight ≥10th percentile (i.e., would have been missed by the conventional SGA proxy).

RESULTS

Among 26,077 births, 345 (1.3%) were classified as growth-restricted by our latent profile model. Growth-restricted infants were more likely than nongrowth-restricted infants to have an Apgar score <7 (10% vs. 2%), have hypoglycemia at birth (17% vs. 3%), require neonatal intensive care unit admission (59% vs. 6%), die in the perinatal period (3.8% vs. 0.2%), and require an emergency cesarean delivery (42% vs. 15%). Risks remained elevated in growth-restricted infants who were not SGA, suggesting our model identified at-risk infants not detected using the SGA proxy.

CONCLUSIONS

Latent profile analysis is a promising strategy for classifying growth restriction at birth in fetal growth restriction research.

Composite diagnostic criteria are problematic for linking potentially distinct populations: the case of frailty

Composite diagnostic criteria are common in frailty research. We worry distinct populations may be linked to each other due to complicated criteria. We aim to investigate whether distinct populations might be considered similar based on frailty diagnostic criteria. The Functional Domains Model for frailty diagnosis included four domains: physical, nutritive, cognitive and sensory functioning. Health and Retirement Study participants with two or more deficiencies in the domains were diagnosed frail. The survival distributions were analyzed using discrete-time survival analysis. The distributions of the demographic characteristics and survival across the groups diagnosed with frailty were significantly different (p < 0.05). A deficiency in cognitive functioning was associated with the worst survival pattern compared with a deficiency in the other domains (adjusted p < 0.05). The associations of the domains with mortality were cumulative without interactions. Cognitive functioning had the largest effect size for mortality prediction (Odds ratios, OR = 2.37), larger than that of frailty status (OR = 1.92). The frailty diagnostic criteria may take distinct populations as equal and potentially assign irrelevant interventions to individuals without corresponding conditions. We think it necessary to review the adequacy of composite diagnostic criteria in frailty diagnosis.