Asymptotic Properties of the Sequential Empirical ROC, PPV and NPV Curves Under Case-Control Sampling

A new risk score for the assessment of outcomes for Chinese patients with peripartum cardiomyopathy

BACKGROUND

Peripartum cardiomyopathy (PPCM) is a potentially life-threatening complication of pregnancy, but identifying patients at higher risk of this condition remains difficult.

OBJECTIVES

We conducted a study to identify new risk factors associated with PPCM and predictors of poor outcomes.

METHODS

This retrospective analysis included a total of 44 women with PPCM. As a control group, 79 women who gave birth around the same time as the PPCM patients and who did not have organic disease were included. A multivariate regression analysis was conducted to identify risk factors associated with PPCM and with delayed recovery.

RESULTS

All PPCM patients were discharged within 28 days. In comparison to the control group, PPCM patients had higher rates of preeclampsia (20.4% vs. 1.27%, P<0.001), autoimmune disease (27.3% vs. 11.4%, P = 0.018), and cesarean delivery with preterm labor (31.8% vs. 17.7%, P = 0.037). The neonates of PPCM patients had lower birth weight (2.70±0.66 kg vs. 3.21±0.57 kg, P<0.001). PPCM patients had higher levels of C-reactive protein, d-dimer, brain natriuretic peptide (BNP), and serum phosphorus, but lower levels of albumin and serum calcium (all P<0.001). In all patients with PPCM, the left ventricular ejection fraction (LVEF) returned to normal (≥50%) within 28 days after admission. Subjects with early recovery (n = 34) had lower BNP than those with delayed recovery (n = 10) (649.7 ± 526.0 pg/mL vs. 1444.1 ± 1040.8 pg/mL, P = 0.002). Multivariate regression led to a three-point score system to predict PPCM (1 point each for the presence of pericardial effusion, left ventricular dilatation, and d-dimer level ≥0.5 μg/mL). At a cutoff of ≥2, this scoring system predicted delayed recovery with 95.5% sensitivity and 96.1% specificity. The negative predictive value was 97.4% and the positive predictive value was 93.3%. Binary logistic regression indicated that PPCM patients with pulmonary hypertension, lower hemoglobin, or worse LVEF tended to require longer hospital stay (minimum 14 days).

CONCLUSIONS

A risk score that consists of pericardial effusion, left ventricular dilatation, and d-dimer level ≥ 0.5 μg/mL could help streamline the diagnosis of PPCM prior to confirmatory investigations. Moreover, a risk score that consists of pulmonary hypertension, lower hemoglobin and worse LVEF could help to predict poor outcomes in PPCM patients.

Group sequential comparison of positive predictive value curves for correlated biomarker data

Clinical studies of predictive diagnostic tests consider the evaluation of a single test and comparison of two tests regarding their predictive accuracy of disease status. The positive predictive value (PPV) curve is used for assessing the probability of predicting the disease given a positive test result. The sequential property of one PPV curve had been studied. However, in later stages of diagnostic test development, it is more interesting to compare predictive accuracy of two tests. In this article, we propose a group sequential test for the comparison of PPV curves for paired designs when both diagnostic tests are applied to the same subject. We first derive asymptotic properties of the sequential differences of two correlated empirical PPV curves under the common case-control sampling. We then apply these results to develop a group sequential test procedure. The asymptotic results are also critical for deriving both the optimal sample size ratio and minimal required sample sizes for the proposed procedure. Our simulation studies show that the proposed sequential testing maintains the nominal type I error rate in finite samples. The proposed design is illustrated in a hypothetical lung cancer predictive trial and in a cancer diagnostic trial.

Adaptive trial designs in diagnostic accuracy research

The aim of diagnostic accuracy studies is to evaluate how accurately a diagnostic test can distinguish diseased from nondiseased individuals. Depending on the research question, different study designs and accuracy measures are appropriate. As the prior knowledge in the planning phase is often very limited, modifications of design aspects such as the sample size during the ongoing trial could increase the efficiency of diagnostic trials. In intervention studies, group sequential and adaptive designs are well established. Such designs are characterized by preplanned interim analyses, giving the opportunity to stop early for efficacy or futility or to modify elements of the study design. In contrast, in diagnostic accuracy studies, such flexible designs are less common, even if they are as important as for intervention studies. However, diagnostic accuracy studies have specific features, which may require adaptations of the statistical methods or may lead to specific advantages or limitations of sequential and adaptive designs. In this article, we summarize the current status of methodological research and applications of flexible designs in diagnostic accuracy research. Furthermore, we indicate and advocate future development of adaptive design methodology and their use in diagnostic accuracy trials from an interdisciplinary viewpoint. The term "interdisciplinary viewpoint" describes the collaboration of experts of the academic and nonacademic research.

Strategies for validating biomarkers using data from a reference set

Candidate biomarkers discovered in the laboratory need to be rigorously validated before advancing to clinical application. However, it is often expensive and time-consuming to collect the high quality specimens needed for validation; moreover, such specimens are often limited in volume. The Early Detection Research Network has developed valuable specimen reference sets that can be used by multiple labs for biomarker validation. To optimize the chance of successful validation, it is critical to efficiently utilize the limited specimens in these reference sets on promising candidate biomarkers. Towards this end, we propose a novel two-stage validation strategy that partitions the samples in the reference set into two groups for sequential validation. The proposed strategy adopts the group sequential testing method to control for the type I error rate and rotates group membership to maximize the usage of available samples. We develop analytical formulas for performance parameters of this strategy in terms of the expected numbers of biomarkers that can be evaluated and the truly useful biomarkers that can be successfully validated, which can provide valuable guidance for future study design. The performance of our proposed strategy for validating biomarkers with respect to the points on the receiver operating characteristic curve are evaluated via extensive simulation studies and compared with the default strategy of validating each biomarker using all samples in the reference set. Different types of early stopping rules and boundary shapes in the group sequential testing method are considered. Compared with the default strategy, our proposed strategy makes more efficient use of the limited resources in the reference set by allowing more candidate biomarkers to be evaluated, giving a better chance of having truly useful biomarkers successfully validated.

Unbiased estimation of biomarker panel performance when combining training and testing data in a group sequential design

Motivated by an ongoing study to develop a screening test able to identify patients with undiagnosed Sjögren's Syndrome in a symptomatic population, we propose methodology to combine multiple biomarkers and evaluate their performance in a two-stage group sequential design that proceeds as follows: biomarker data is collected from first stage samples; the biomarker panel is built and evaluated; if the panel meets pre-specified performance criteria the study continues to the second stage and the remaining samples are assayed. The design allows us to conserve valuable specimens in the case of inadequate biomarker panel performance. We propose a nonparametric conditional resampling algorithm that uses all the study data to provide unbiased estimates of the biomarker combination rule and the sensitivity of the panel corresponding to specificity of 1-t on the receiver operating characteristic curve (ROC). The Copas and Corbett (2002) correction, for bias resulting from using the same data to derive the combination rule and estimate the ROC, was also evaluated and an improved version was incorporated. An extensive simulation study was conducted to evaluate finite sample performance and propose guidelines for designing studies of this type. The methods were implemented in the National Cancer Institutes Early Detection Network Urinary PCA3 Evaluation Trial.

Group sequential testing of the predictive accuracy of a continuous biomarker with unknown prevalence

Group sequential testing procedures have been proposed as an approach to conserving resources in biomarker validation studies. Previously, we derived the asymptotic properties of the sequential empirical positive predictive value (PPV) and negative predictive value (NPV) curves, which summarize the predictive accuracy of a continuous marker, under case-control sampling. A limitation of this approach is that the prevalence cannot be estimated from a case-control study and must be assumed known. In this paper, we consider group sequential testing of the predictive accuracy of a continuous biomarker with unknown prevalence. First, we develop asymptotic theory for the sequential empirical PPV and NPV curves when the prevalence must be estimated, rather than assumed known in a case-control study. We then discuss how our results can be combined with standard group sequential methods to develop group sequential testing procedures and bias-adjusted estimators for the PPV and NPV curve. The small sample properties of the proposed group sequential testing procedures and estimators are evaluated by simulation, and we illustrate our approach in the context of a study to validate a novel biomarker for prostate cancer.

Prospective Evaluation of Rapid Antigen Tests for Diagnosis of Respiratory Viral Pathogens

Acute respiratory infection is a frequently transmitted illness of concern to doctors and patients. Considering its airborne transmission, early diagnosis of such disease is particularly important. This study explored respiratory viral infections with influenza virus, parainfluenza virus, respiratory syncytial virus, human metapneumovirus, human bocavirus, coronavirus, and other early diagnostic substances as confirmed by literature resources. This study also used the corresponding monoclonal antibodies that were produced with the use of hybridoma technology, which were fixed on the chip after purification, for further serum detection. Using this method, a new technique to simultaneously detect 6 kinds of febrile respiratory viruses in a protein chip was developed. The accuracy rate of this method can be >99.65%. This product is inexpensive and capable of high-precision and high-throughput screening, which are prominent advantages.

•

Six diagnostic methods on respiratory viral infection are explored in this study.

•

Monoclonal antibodies produced with hybridoma technology are used.

•

A new technique to simultaneously detect 6 kinds of febrile respiratory viruses in a protein chip was developed.

•

The accuracy rate for this method is >99.65%, and is inexpensive and capable of high-precision and high-throughout screening.

Early termination of a two-stage study to develop and validate a panel of biomarkers

Two-stage designs to develop and validate a panel of biomarkers present a natural setting for the inclusion of stopping rules for futility in the event of poor preliminary estimates of performance. We consider the design of a two-stage study to develop and validate a panel of biomarkers where a predictive model is developed using a subset of the samples in stage 1 and the model is validated using the remainder of the samples in stage 2. First, we illustrate how we can implement a stopping rule for futility in a standard, two-stage study for developing and validating a predictive model where samples are separated into a training sample and a validation sample. Simulation results indicate that our design has type I error rate and power similar to the fixed-sample design but with a substantially reduced sample size under the null hypothesis. We then illustrate how we can include additional interim analyses in stage 2 by applying existing group sequential methodology, which results in even greater savings in the number of samples required under both the null and the alternative hypotheses. Our simulation results also illustrate that the operating characteristics of our design are robust to changes in the underlying marker distribution.