Randomized test-treatment studies with an outlook on adaptive designs

Towards Recommendations for Cost-Effectiveness Analysis of Predictive, Prognostic, and Serial Biomarker Tests in Oncology

BACKGROUND

Cost-effectiveness analysis (CEA) of biomarkers is challenging due to the indirect impact on health outcomes and the lack of sufficient fit-for-purpose data. Hands-on guidance is lacking.

OBJECTIVE

We aimed firstly to explore how CEAs in the context of three different types of biomarker applications have addressed these challenges, and secondly to develop recommendations for future CEAs.

METHODS

A scoping review was performed for three biomarker applications: predictive, prognostic, and serial testing, in advanced non-small cell lung cancer, early-stage colorectal cancer, and all-stage colorectal cancer, respectively. Information was extracted on the model assumptions and uncertainty, and the reported outcomes. An in-depth analysis of the literature was performed describing the impact of model assumptions in the included studies.

RESULTS

A total of 43 CEAs were included (31 predictive, 6 prognostic, and 6 serial testing). Of these, 40 utilized different sources for test and treatment parameters, and three studies utilized a single source. Test performance was included in 78% of these studies utilizing different sources, but this parameter was differently expressed across biomarker applications. Sensitivity analyses for test performance was only performed in half of these studies. For the linkage of test results to treatments outcomes, a minority of the studies explored the impact of suboptimal adherence to test results, and/or explored potential differences in treatment effects for different biomarker subgroups. Intermediate outcomes were reported by 67% of studies.

CONCLUSIONS

We identified various approaches for dealing with challenges in CEAs of biomarker tests for three different biomarker applications. Recommendations on assumptions, handling uncertainty, and reported outcomes were drafted to enhance modeling practices for future biomarker cost-effectiveness evaluations.

Sample size recalculation based on the overall success rate in a randomized test-treatment trial with restricting randomization to discordant pairs

BACKGROUND

Randomized test-treatment studies are performed to evaluate the clinical effectiveness of diagnostic tests by assessing patient-relevant outcomes. The assumptions for a sample size calculation for such studies are often uncertain.

METHODS

An adaptive design with a blinded sample size recalculation based on the overall success rate in a randomized test-treatment trial with restricting randomization to discordant pairs is proposed and evaluated by a simulation study. The results of the adaptive design are compared to those of the fixed design.

RESULTS

The empirical type I error rate is sufficiently controlled in the adaptive design as well as in the fixed design and the estimates are unbiased. The adaptive design achieves the desired theoretical power, whereas the fixed design tends to be over- or under-powered.

CONCLUSIONS

It may be advisable to consider blinded recalculation of sample size in a randomized test-treatment study with restriction of randomization to discordant pairs in order to improve the conduct of the study. However, there are a number of study-related limitations that affect the implementation of the method which need to be considered.

A unified framework for diagnostic test development and evaluation during outbreaks of emerging infections

Evaluating diagnostic test accuracy during epidemics is difficult due to an urgent need for test availability, changing disease prevalence and pathogen characteristics, and constantly evolving testing aims and applications. Based on lessons learned during the SARS-CoV-2 pandemic, we introduce a framework for rapid diagnostic test development, evaluation, and validation during outbreaks of emerging infections. The framework is based on the feedback loop between test accuracy evaluation, modelling studies for public health decision-making, and impact of public health interventions. We suggest that building on this feedback loop can help future diagnostic test evaluation platforms better address the requirements of both patient care and public health.

A diagnostic phase III/IV seamless design to investigate the diagnostic accuracy and clinical effectiveness using the example of HEDOS and HEDOS II

The development process of medical devices can be streamlined by combining different study phases. Here, for a diagnostic medical device, we present the combination of confirmation of diagnostic accuracy (phase III) and evaluation of clinical effectiveness regarding patient-relevant endpoints (phase IV) using a seamless design. This approach is used in the Thyroid HEmorrhage DetectOr Study (HEDOS & HEDOS II) investigating a post-operative hemorrhage detector named ISAR-M THYRO® in patients after thyroid surgery. Data from the phase III trial are reused as external controls in the control group of the phase IV trial. An unblinded interim analysis is planned between the two study stages which includes a recalculation of the sample size for the phase IV part after completion of the first stage of the seamless design. The study concept presented here is the first seamless design proposed in the field of diagnostic studies. Hence, the aim of this work is to emphasize the statistical methodology as well as feasibility of the proposed design in relation to the planning and implementation of the seamless design. Seamless designs can accelerate the overall trial duration and increase its efficiency in terms of sample size and recruitment. However, careful planning addressing numerous methodological and procedural challenges is necessary for successful implementation as well as agreement with regulatory bodies.

Evaluation of point-of-care multiplex polymerase chain reaction in guiding antibiotic treatment of patients acutely admitted with suspected community-acquired pneumonia in Denmark: A multicentre randomised controlled trial

BACKGROUND

Rapid and accurate detection of pathogens is needed in community-acquired pneumonia (CAP) to enable appropriate antibiotics and to slow the development of antibiotic resistance. We aimed to compare the effect of point-of-care (POC) polymerase chain reaction (PCR) detection of respiratory pathogens added to standard care with standard care only (SCO) on antibiotic prescriptions after acute hospital admission.

METHODS AND FINDINGS

We performed a superiority, parallel-group, open-label, multicentre, randomised controlled trial (RCT) in 3 Danish medical emergency departments (EDs) from March 2021 to February 2022. Adults acutely admitted with suspected CAP during the daytime on weekdays were included and randomly assigned (1:1) to POC-PCR (The Biofire FilmArray Pneumonia Panel plus added to standard care) or SCO (routine culture and, if requested by the attending physician, target-specific PCR) analysis of respiratory samples. We randomly assigned 294 patients with successfully collected samples (tracheal secretion 78.4% or expectorated sputum 21.6%) to POC-PCR (n = 148, 50.4%) or SCO (146, 49.6%). Patients and investigators owning the data were blinded to the allocation and test results. Outcome adjudicators and clinical staff at the ED were not blinded to allocation and test results but were together with the statistician, blinded to data management and analysis. Laboratory staff performing standard care analyses was blinded to allocation. The study coordinator was not blinded. Intention-to-treat and per protocol analysis were performed using logistic regression with Huber-White clustered standard errors for the prescription of antibiotic treatment. Loss to follow-up comprises 3 patients in the POC-PCR (2%) and none in the SCO group. Intention-to-treat analysis showed no difference in the primary outcome of prescriptions of no or narrow-spectrum antibiotics at 4 h after admission for the POC-PCR (n = 91, 62.8%) odds ratio (OR) 1.13; (95% confidence interval (CI) [0.96, 1.34] p = 0.134) and SCO (n = 87, 59.6%). Secondary outcomes showed that prescriptions were significantly more targeted at 4-h OR 5.68; (95% CI [2.49, 12.94] p < 0.001) and 48-h OR 4.20; (95% CI [1.87, 9.40] p < 0.001) and more adequate at 48-h OR 2.11; (95% CI [1.23, 3.61] p = 0.006) and on day 5 in the POC-PCR group OR 1.40; (95% CI [1.18, 1.66] p < 0.001). There was no difference between the groups in relation to intensive care unit (ICU) admissions OR 0.54; (95% CI [0.10, 2.91] p = 0.475), readmission within 30 days OR 0.90; (95% CI [0.43, 1.86] p = 0.787), length of stay (LOS) IRR 0.82; (95% CI [0.63, 1.07] p = 0.164), 30 days mortality OR 1.24; (95% CI [0.32, 4.82] p = 0.749), and in-hospital mortality OR 0.98; (95% CI [0.19, 5.06] p = 0.986).

CONCLUSIONS

In a setting with an already restrictive use of antibiotics, adding POC-PCR to the diagnostic setup did not increase the number of patients treated with narrow-spectrum or without antibiotics. POC-PCR may result in a more targeted and adequate use of antibiotics. A significant study limitation was the concurrent Coronavirus Disease 2019 (COVID-19) pandemic resulting in an unusually low transmission of respiratory virus.

TRIAL REGISTRATION

ClinicalTrials.gov (NCT04651712).

2-[18F]FDG-PET/CT in Cancer of Unknown Primary Tumor-A Retrospective Register-Based Cohort Study

We investigated the impact of 2-[18F]FDG-PET/CT on detection rate (DR) of the primary tumor and survival in patients with suspected cancer of unknown primary tumor (CUP), comparing it to the conventional diagnostic imaging method, CT. Patients who received a tentative CUP diagnosis at Odense University Hospital from 2014-2017 were included. Patients receiving a 2-[18F]FDG-PET/CT were assigned to the 2-[18F]FDG-PET/CT group and patients receiving a CT only to the CT group. DR was calculated as the proportion of true positive findings of 2-[18F]FDG-PET/CT and CT scans, separately, using biopsy of the primary tumor, autopsy, or clinical decision as reference standard. Survival analyses included Kaplan-Meier estimates and Cox proportional hazards regression adjusted for age, sex, treatment, and propensity score. We included 193 patients. Of these, 159 were in the 2-[18F]FDG-PET/CT group and 34 were in the CT group. DR was 36.5% in the 2-[18F]FDG-PET/CT group and 17.6% in the CT group, respectively (p = 0.012). Median survival was 7.4 (95% CI 0.4-98.7) months in the 2-[18F]FDG-PET/CT group and 3.8 (95% CI 0.2-98.1) in the CT group. Survival analysis showed a crude hazard ratio of 0.63 (p = 0.024) and an adjusted hazard ratio of 0.68 (p = 0.087) for the 2-[18F]FDG-PET/CT group compared with CT. This study found a significantly higher DR of the primary tumor in suspected CUP patients using 2-[18F]FDG-PET/CT compared with patients receiving only CT, with possible immense clinical importance. No significant difference in survival was found, although a possible tendency towards longer survival in the 2-[18F]FDG-PET/CT group was observed.

Modelling the Cost-Effectiveness of Diagnostic Tests

Diagnostic tests are used to determine whether a disease or condition is present or absent in a patient, who will typically be suspected of having the disease or condition due to symptoms or clinical signs. Economic evaluations of diagnostic tests (e.g. cost-effectiveness analyses) can be used to determine whether a test produces sufficient benefit to justify its cost. Evidence on the benefits conferred by a test is often restricted to its accuracy, which means mathematical models are required to estimate the impact of a test on outcomes that matter to patients and health payers. It is important to realise the case for introducing a new test may not be restricted to its accuracy, but extend to factors such as time to diagnosis and acceptability for patients. These and other considerations may mean the common modelling approach, the decision tree, is inappropriate for underpinning an economic evaluation. There are no consensus guidelines on how economic evaluations of diagnostic tests should be conducted-this article attempts to explore the common challenges encountered in economic evaluations, suggests solutions to those challenges, and identifies some areas where further methodological work may be necessary.

Sample size recalculation based on the prevalence in a randomized test-treatment study

BACKGROUND

Randomized test-treatment studies aim to evaluate the clinical utility of diagnostic tests by providing evidence on their impact on patient health. However, the sample size calculation is affected by several factors involved in the test-treatment pathway, including the prevalence of the disease. Sample size planning is exposed to strong uncertainties in terms of the necessary assumptions, which have to be compensated for accordingly by adjusting prospectively determined study parameters during the course of the study.

METHOD

An adaptive design with a blinded sample size recalculation in a randomized test-treatment study based on the prevalence is proposed and evaluated by a simulation study. The results of the adaptive design are compared to those of the fixed design.

RESULTS

The adaptive design achieves the desired theoretical power, under the assumption that all other nuisance parameters have been specified correctly, while wrong assumptions regarding the prevalence may lead to an over- or underpowered study in the fixed design. The empirical type I error rate is sufficiently controlled in the adaptive design as well as in the fixed design.

CONCLUSION

The consideration of a blinded recalculation of the sample size already during the planning of the study may be advisable in order to increase the possibility of success as well as an enhanced process of the study. However, the application of the method is subject to a number of limitations associated with the study design in terms of feasibility, sample sizes needed to be achieved, and fulfillment of necessary prerequisites.

Bland–Altman Limits of Agreement from a Bayesian and Frequentist Perspective

Bland–Altman agreement analysis has gained widespread application across disciplines, last but not least in health sciences, since its inception in the 1980s. Bayesian analysis has been on the rise due to increased computational power over time, and Alari, Kim, and Wand have put Bland–Altman Limits of Agreement in a Bayesian framework (Meas. Phys. Educ. Exerc. Sci. 2021, 25, 137–148). We contrasted the prediction of a single future observation and the estimation of the Limits of Agreement from the frequentist and a Bayesian perspective by analyzing interrater data of two sequentially conducted, preclinical studies. The estimation of the Limits of Agreement θ1 and θ2 has wider applicability than the prediction of single future differences. While a frequentist confidence interval represents a range of nonrejectable values for null hypothesis significance testing of H0: θ1 ≤ −δ or θ2 ≥ δ against H1: θ1 > −δ and θ2 < δ, with a predefined benchmark value δ, Bayesian analysis allows for direct interpretation of both the posterior probability of the alternative hypothesis and the likelihood of parameter values. We discuss group-sequential testing and nonparametric alternatives briefly. Frequentist simplicity does not beat Bayesian interpretability due to improved computational resources, but the elicitation and implementation of prior information demand caution. Accounting for clustered data (e.g., repeated measurements per subject) is well-established in frequentist, but not yet in Bayesian Bland–Altman analysis.