A parametric comparison of diagnostic accuracy with three ordinal diagnostic groups

A new accuracy metric under three classes when subclasses are involved and its confidence interval estimation

"Compound multi-class classification" refers to the setting where three or more main classes are involved and at least one of the main classes have multiple subclasses. A common practice in evaluating biomarker performance under "compound multi-class classification" is "subclasses pooling." In this article, we first explore the downsides of accuracy metrics based on pooled data. Then we propose a new accuracy measure proper for "compound multi-class classification" with three ordinal main classes, namely "volume under compoundR O C $$ ROC $$ surface (V U S C $$ VU{S}_C $$ )." The proposedV U S C $$ VU{S}_C $$ evaluates the accuracy of a biomarker appropriately by identifying main classes without requiring specification of an ordering for marker values of subclasses within each main class. For confidence interval estimation ofV U S C $$ VU{S}_C $$ , both parametric and nonparametric methods are studied, and simulation studies are carried out to assess coverage probabilities. A subset of Alzheimer's Disease Neuroimaging Initiative study dataset is analyzed.

A family of estimators to diagnostic accuracy when candidate tests are subject to detection limits-Application to diagnosing early stage Alzheimer disease

In disease diagnosis, individuals are usually assumed to be one of the two basic types, healthy or diseased, as typically based on an established gold standard. Candidate markers for diagnosing a disease often are much cheaper and less invasive than the gold standard but must be evaluated against the gold standard for their sensitivity and specificity to accurately diagnose the disease. When candidate diagnostic markers are fully measured, receiver operating characteristic curves have been the standard approaches for assessing diagnostic accuracy. However, full measurements of diagnostic markers may not be available above or below certain limits due to various practical and technical limitations. For example, in the diagnosis of Alzheimer disease using cerebrospinal fluid biomarkers, the Roche Elecsys® immunoassays have a measuring range for multiple cerebrospinal fluid molecular concentrations. Many cognitive tests used in diagnosing dementia due to Alzheimer disease are also subject to detection limits, often referred to as the floor and ceiling effects in the neuropsychological literature. We propose a new statistical methodology for estimating the diagnostic accuracy when a diagnostic marker is subject to detection limits by dividing the entire study sample into two sub-samples by a threshold of the diagnostic marker. We then propose a family of estimators to the area under the receiver operating characteristic curve by combining a conditional nonparametric estimator and another conditional semi-parametric estimator derived from Cox's proportional hazards model. We derive the variance to the proposed estimators, and further, assess the performance of the proposed estimators as a function of possible thresholds through an extensive simulation study, and recommend the optimum thresholds. Finally, we apply the proposed methodology to assess the ability of several cerebrospinal fluid biomarkers and cognitive tests in diagnosing early stage Alzheimer disease dementia.

Combining multiple biomarkers to linearly maximize the diagnostic accuracy under ordered multi-class setting

Either in clinical study or biomedical research, it is a common practice to combine multiple biomarkers to improve the overall diagnostic performance. Despite the fact there exist a large number of statistical methods for biomarker combination under binary classification, research on this topic under multi-class setting is sparse. The overall diagnostic accuracy, i.e. the sum of correct classification rates, directly measures the classification accuracy of the combined biomarkers. Hence the overall accuracy can serve as an important objective function for biomarker combination, especially when the combined biomarkers are used for the purpose of making medical diagnosis. In this paper, we address the problem of combining multiple biomarkers to directly maximize the overall diagnostic accuracy by presenting several grid search methods and derivation-based methods. A comprehensive simulation study was conducted to compare the performances of these methods. An ovarian cancer data set is analyzed in the end.

Measuring diagnostic accuracy for biomarkers under tree-ordering

In the field of diagnostic studies for tree or umbrella ordering, under which the marker measurement for one class is lower or higher than those for the rest unordered classes, there exist a few diagnostic measures such as the naive AUC ( NAUC), the umbrella volume ( UV), and the recently proposed TAUC, i.e. area under a ROC curve for tree or umbrella ordering (TROC). However, an important characteristic about tree or umbrella ordering has been neglected. This paper mainly focuses on promoting the use of the integrated false negative rate under tree ordering ( ITFNR) as an additional diagnostic measure besides TAUC, and proposing the idea of using ( TAUC, ITFNR) instead of TAUC to evaluate the diagnostic accuracy of a biomarker under tree or umbrella ordering. Parametric and non-parametric approaches for constructing joint confidence region of ( TAUC, ITFNR) are proposed. Simulation studies under a variety of settings are carried out to assess and compare the performance of these methods. In the end, a published microarray data set is analyzed.

A new diagnostic accuracy measure and cut-point selection criterion

Most diagnostic accuracy measures and criteria for selecting optimal cut-points are only applicable to diseases with binary or three stages. Currently, there exist two diagnostic measures for diseases with general k stages: the hypervolume under the manifold and the generalized Youden index. While hypervolume under the manifold cannot be used for cut-points selection, generalized Youden index is only defined upon correct classification rates. This paper proposes a new measure named maximum absolute determinant for diseases with k stages ([Formula: see text]). This comprehensive new measure utilizes all the available classification information and serves as a cut-points selection criterion as well. Both the geometric and probabilistic interpretations for the new measure are examined. Power and simulation studies are carried out to investigate its performance as a measure of diagnostic accuracy as well as cut-points selection criterion. A real data set from Alzheimer's Disease Neuroimaging Initiative is analyzed using the proposed maximum absolute determinant.

Estimating confidence intervals for the difference in diagnostic accuracy with three ordinal diagnostic categories without a gold standard

With three ordinal diagnostic categories, the most commonly used measures for the overall diagnostic accuracy are the volume under the ROC surface (VUS) and partial volume under the ROC surface (PVUS), which are the extensions of the area under the ROC curve (AUC) and partial area under the ROC curve (PAUC), respectively. A gold standard (GS) test on the true disease status is required to estimate the VUS and PVUS. However, oftentimes it may be difficult, inappropriate, or impossible to have a GS because of misclassification error, risk to the subjects or ethical concerns. Therefore, in many medical research studies, the true disease status may remain unobservable. Under the normality assumption, a maximum likelihood (ML) based approach using the expectation-maximization (EM) algorithm for parameter estimation is proposed. Three methods using the concepts of generalized pivot and parametric/nonparametric bootstrap for confidence interval estimation of the difference in paired VUSs and PVUSs without a GS are compared. The coverage probabilities of the investigated approaches are numerically studied. The proposed approaches are then applied to a real data set of 118 subjects from a cohort study in early stage Alzheimer's disease (AD) from the Washington University Knight Alzheimer's Disease Research Center to compare the overall diagnostic accuracy of early stage AD between two different pairs of neuropsychological tests.

Diagnostic thresholds with three ordinal groups

In practice, there exist many disease processes with three ordinal disease classes; for example, in the detection of Alzheimer's disease (AD) a patient can be classified as healthy (disease-free stage), mild cognitive impairment (early disease stage), or AD (full disease stage). The treatment interventions and effectiveness of such disease processes will depend on the disease stage. Therefore, it is important to develop diagnostic tests with the ability to discriminate between the three disease stages. Measuring the overall ability of diagnostic tests to discriminate between the three classes has been discussed extensively in the literature. However, there has been little proposed on how to select clinically meaningful thresholds for such diagnostic tests, except for a method based on the generalized Youden index by Nakas et al. (2010). In this article, we propose two new criteria for selecting diagnostic thresholds in the three-class setting. The numerical study demonstrated that the proposed methods may provide thresholds with less variability and more balance among the correct classification rates for the three stages. The proposed methods are applied to two real examples: the clinical diagnosis of AD from the Washington University Alzheimer's Disease Research Center and the detection of liver cancer (LC) using protein segments.

Confidence interval estimation of the difference between two sensitivities to the early disease stage

Although most of the statistical methods for diagnostic studies focus on disease processes with binary disease status, many diseases can be naturally classified into three ordinal diagnostic categories, that is normal, early stage, and fully diseased. For such diseases, the volume under the ROC surface (VUS) is the most commonly used index of diagnostic accuracy. Because the early disease stage is most likely the optimal time window for therapeutic intervention, the sensitivity to the early diseased stage has been suggested as another diagnostic measure. For the purpose of comparing the diagnostic abilities on early disease detection between two markers, it is of interest to estimate the confidence interval of the difference between sensitivities to the early diseased stage. In this paper, we present both parametric and non-parametric methods for this purpose. An extensive simulation study is carried out for a variety of settings for the purpose of evaluating and comparing the performance of the proposed methods. A real example of Alzheimer's disease (AD) is analyzed using the proposed approaches.

DiagTest3Grp: An R Package for Analyzing Diagnostic Tests with Three Ordinal Groups

Medical researchers endeavor to identify potentially useful biomarkers to develop marker-based screening assays for disease diagnosis and prevention. Useful summary measures which properly evaluate the discriminative ability of diagnostic markers are critical for this purpose. Literature and existing software, for example, R packages nicely cover summary measures for diagnostic markers used for the binary case (e.g., healthy vs. diseased). An intermediate population at an early disease stage usually exists between the healthy and the fully diseased population in many disease processes. Supporting utilities for three-group diagnostic tests are highly desired and important for identifying patients at the early disease stage for timely treatments. However, application packages which provide summary measures for three ordinal groups are currently lacking. This paper focuses on two summary measures of diagnostic accuracy-volume under the receiver operating characteristic surface and the extended Youden index, with three diagnostic groups. We provide the R package DiagTest3Grp to estimate, under both parametric and nonparametric assumptions, the two summary measures and the associated variances, as well as the optimal cut-points for disease diagnosis. An omnibus test for multiple markers and a Wald test for two markers, on independent or paired samples, are incorporated to compare diagnostic accuracy across biomarkers. Sample size calculation under the normality assumption can be performed in the R package to design future diagnostic studies. A real world application evaluating the diagnostic accuracy of neuropsychological markers for Alzheimer's disease is used to guide readers through step-by-step implementation of DiagTest3Grp to demonstrate its utility.

Linear combinations of biomarkers to improve diagnostic accuracy with three ordinal diagnostic categories

Many researchers have addressed the problem of finding the optimal linear combination of biomarkers to maximize the area under receiver operating characteristic (ROC) curves for scenarios with binary disease status. In practice, many disease processes such as Alzheimer can be naturally classified into three diagnostic categories such as normal, mild cognitive impairment and Alzheimer's disease (AD), and for such diseases the volume under the ROC surface (VUS) is the most commonly used index of diagnostic accuracy. In this article, we propose a few parametric and nonparametric approaches to address the problem of finding the optimal linear combination to maximize the VUS. We carried out simulation studies to investigate the performance of the proposed methods. We apply all of the investigated approaches to a real data set from a cohort study in early stage AD.

On ROC analysis with nonbinary reference standard

Statistical methods for the evaluation of the accuracy of diagnostic tests usually assume a binary true disease status. However, this assumption may not be realistic in practical settings in which "disease" is defined by dichotomizing continuous or ordinal categorical measures using a pre-specified threshold value. In this paper, we focus on the analysis of studies in which both the diagnostic test and the reference standard are reported as continuous measures. We propose a semiparametric model for estimating the sensitivity, specificity, and the ROC curve as functions of reference standard thresholds. Under suitable order restrictions on the mean of the test result variable, fitting is done via two alternative approaches: isotonic regression and monotone smoothing splines. The model provides the basis to assess the effect of varying reference standard threshold on the performance of a diagnostic test. An example to evaluate the ability of the maximal SUV-lean (standardized uptake value normalized to lean body mass) in predicting axillary node involvement in women diagnosed with breast cancer is presented.

Parametric and non-parametric confidence intervals of the probability of identifying early disease stage given sensitivity to full disease and specificity with three ordinal diagnostic groups

In practice, there exist many disease processes with three ordinal disease classes, that is, the non-diseased stage, the early disease stage, and the fully diseased stage. Because early disease stage is likely the best time window for treatment interventions, it is important to have diagnostic tests that have good diagnostic ability to discriminate the early disease stage from the other two stages. In this paper, we present both parametric and non-parametric approaches for confidence interval estimation of probability of detecting early disease stage given the true classification rates for non-diseased group and diseased group, namely, the specificity and the sensitivity to full disease. We analyze a data set on the clinical diagnosis of early-stage Alzheimer's disease from the neuropsychological database at the Washington University Alzheimer's Disease Research Center using the proposed approaches.

Exact confidence interval estimation for the difference in diagnostic accuracy with three ordinal diagnostic groups

In the cases with three ordinal diagnostic groups, the important measures of diagnostic accuracy are the volume under surface (VUS) and the partial volume under surface (PVUS) which are the extended forms of the area under curve (AUC) and the partial area under curve (PAUC). This article addresses confidence interval estimation of the difference in paired VUS s and the difference in paired PVUS s. To focus especially on studies with small to moderate sample sizes, we propose an approach based on the concepts of generalized inference. A Monte Carlo study demonstrates that the proposed approach generally can provide confidence intervals with reasonable coverage probabilities even at small sample sizes. The proposed approach is compared to a parametric bootstrap approach and a large sample approach through simulation. Finally, the proposed approach is illustrated via an application to a data set of blood test results of anemia patients.