Loss Function Based Ranking in Two-Stage, Hierarchical Models

A Bayesian hierarchical model for signal extraction from protein microarrays

Protein microarrays are a promising technology that measure protein levels in serum or plasma samples. Due to their high technical variability and high variation in protein levels across serum samples in any population, directly answering biological questions of interest using protein microarray measurements is challenging. Analyzing preprocessed data and within-sample ranks of protein levels can mitigate the impact of between-sample variation. As for any analysis, ranks are sensitive to preprocessing, but loss function based ranks that accommodate major structural relations and components of uncertainty are very effective. Bayesian modeling with full posterior distributions for quantities of interest produce the most effective ranks. Such Bayesian models have been developed for other assays, for example, DNA microarrays, but modeling assumptions for these assays are not appropriate for protein microarrays. Consequently, we develop and evaluate a Bayesian model to extract the full posterior distribution of normalized protein levels and associated ranks for protein microarrays, and show that it fits well to data from two studies that use protein microarrays produced by different manufacturing processes. We validate the model via simulation and demonstrate the downstream impact of using estimates from this model to obtain optimal ranks.

MEASURING PERFORMANCE FOR END-OF-LIFE CARE

Although not without controversy, readmission is entrenched as a hospital quality metric with statistical analyses generally based on fitting a logistic-Normal generalized linear mixed model. Such analyses, however, ignore death as a competing risk, although doing so for clinical conditions with high mortality can have profound effects; a hospital's seemingly good performance for readmission may be an artifact of it having poor performance for mortality. in this paper we propose novel multivariate hospital-level performance measures for readmission and mortality that derive from framing the analysis as one of cluster-correlated semi-competing risks data. We also consider a number of profiling-related goals, including the identification of extreme performers and a bivariate classification of whether the hospital has higher-/lower-than-expected readmission and mortality rates via a Bayesian decision-theoretic approach that characterizes hospitals on the basis of minimizing the posterior expected loss for an appropriate loss function. in some settings, particularly if the number of hospitals is large, the computational burden may be prohibitive. To resolve this, we propose a series of analysis strategies that will be useful in practice. Throughout, the methods are illustrated with data from CMS on N = 17,685 patients diagnosed with pancreatic cancer between 2000-2012 at one of J = 264 hospitals in California.

An empirical Bayesian ranking method, with applications to high throughput biology

MOTIVATION

In bioinformatics, genome-wide experiments look for important biological differences between two groups at a large number of locations in the genome. Often, the final analysis focuses on a P-value-based ranking of locations which might then be investigated further in follow-up experiments. However, this strategy may result in small effect sizes, with low P-values, being ranked more favorably than larger more scientifically important effects. Bayesian ranking techniques may offer a solution to this problem provided a good prior distribution for the collective distribution of effect sizes is available.

RESULTS

We develop an Empirical Bayes ranking algorithm, using the marginal distribution of the data over all locations to estimate an appropriate prior. In simulations and analysis using real datasets, we demonstrate favorable performance compared to ordering P-values and a number of other competing ranking methods. The algorithm is computationally efficient and can be used to rank the entirety of genomic locations or to rank a subset of locations, pre-selected via traditional FWER/FDR methods in a 2-stage analysis.

AVAILABILITY AND IMPLEMENTATION

An R-package, EBrank, implementing the ranking algorithm is available on CRAN.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Optimal Bayesian point estimates and credible intervals for ranking with application to county health indices

It is fairly common to rank different geographic units, e.g. counties in the USA, based on health indices. In a typical application, point estimates of the health indices are obtained for each county, and the indices are then simply ranked as if they were known constants. Several authors have considered optimal rank estimators under squared error loss on the rank scale as a default method for general purpose ranking, e.g. situations where ranking units across the full spectrum of performance (low, medium, high) is important. While computationally convenient, squared error loss on the rank scale may not represent the true inferential goals of rank consumers. We construct alternative loss functions based on three components: (1) the inferential goal (rank position or pairwise comparisons), (2) the scale (original, log-transformed or rank) and (3) the (positional or pairwise) loss function (0/1, squared error or absolute error). We can obtain optimal ranks for loss functions based on rank positions and nearly optimal ranks for loss functions based on pairwise comparisons paired with highest posterior density (HPD) credible intervals. We compare inferences produced by the various ranking methods, both optimal and heuristic, using low birth weight data for counties in the Midwestern United States, from 2006 to 2012.

Misclassification Risk of Tier-Based Physician Quality Performance Systems

OBJECTIVE

There is increasing interest in identifying high-quality physicians, such as whether physicians perform above or below a threshold level. To evaluate whether current methods accurately distinguish above- versus below-threshold physicians, we estimate misclassification rates for two-category identification systems.

DATA SOURCES

Claims data for Medicare fee-for-service beneficiaries residing in Florida or New York in 2010.

STUDY DESIGN

Estimate colorectal cancer, glaucoma, and diabetes quality scores for 23,085 physicians. Use a beta-binomial model to estimate physician score reliabilities. Compute the proportion of physicians whose performance tier would be misclassified under three scoring systems.

PRINCIPAL FINDINGS

In the three scoring systems, misclassification ranges were 8.6-25.7 percent, 6.4-22.8 percent, and 4.5-21.7%. True positive rate ranges were 72.9-97.0 percent, 83.4-100.0 percent, and 34.7-88.2 percent. True negative rate ranges were 68.5-91.6 percent, 10.5-92.4 percent, and 81.1-99.9 percent. Positive predictive value ranges were 70.5-91.6 percent, 77.0-97.3 percent, and 55.2-99.1 percent.

CONCLUSIONS

Current methods for profiling physicians on quality may produce misleading results, as the number of eligible events is typically small. Misclassification is a policy-relevant measure of the potential impact of tiering on providers, payers, and patients. Quantifying misclassification rates should inform the construction of high-performance networks and quality improvement initiatives.

Making the cut: improved ranking and selection for large-scale inference

Identifying leading measurement units from a large collection is a common inference task in various domains of large-scale inference. Testing approaches, which measure evidence against a null hypothesis rather than effect magnitude, tend to overpopulate lists of leading units with those associated with low measurement error. By contrast, local maximum likelihood (ML) approaches tend to favor units with high measurement error. Available Bayesian and empirical Bayesian approaches rely on specialized loss functions that result in similar deficiencies. We describe and evaluate a generic empirical Bayesian ranking procedure that populates the list of top units in a way that maximizes the expected overlap between the true and reported top lists for all list sizes. The procedure relates unit-specific posterior upper tail probabilities with their empirical distribution to yield a ranking variable. It discounts high-variance units less than popular non-ML methods and thus achieves improved operating characteristics in the models considered.

Classification accuracy of claims-based methods for identifying providers failing to meet performance targets

Quality assessment is critical for healthcare reform, but data sources are lacking for measurement of many important healthcare outcomes. With over 49 million people covered by Medicare as of 2010, Medicare claims data offer a potentially valuable source that could be used in targeted health care quality improvement efforts. However, little is known about the operating characteristics of provider profiling methods using claims-based outcome measures that may estimate provider performance with error. Motivated by the example of screening mammography performance, we compared approaches to identifying providers failing to meet guideline targets using Medicare claims data. We used data from the Breast Cancer Surveillance Consortium and linked Medicare claims to compare claims-based and clinical estimates of cancer detection rate. We then demonstrated the performance of claim-based estimates across a broad range of operating characteristics using simulation studies. We found that identification of poor performing providers was extremely sensitive to algorithm specificity, with no approach identifying more than 65% of poor performing providers when claims-based measures had specificity of 0.995 or less. We conclude that claims have the potential to contribute important information on healthcare outcomes to quality improvement efforts. However, to achieve this potential, development of highly accurate claims-based outcome measures should remain a priority.

On the accuracy of classifying hospitals on their performance measures

The evaluation, comparison, and public report of health care provider performance is essential to improving the quality of health care. Hospitals, as one type of provider, are often classified into quality tiers (e.g., top or suboptimal) based on their performance data for various purposes. However, potential misclassification might lead to detrimental effects for both consumers and payers. Although such risk has been highlighted by applied health services researchers, a systematic investigation of statistical approaches has been lacking. We assess and compare the expected accuracy of several commonly used classification methods: unadjusted hospital-level averages, shrinkage estimators under a random-effects model accommodating between-hospital variation, and two others based on posterior probabilities. Assuming that performance data follow a classic one-way random-effects model with unequal sample size per hospital, we derive accuracy formulae for these classification approaches and gain insight into how the misclassification might be affected by various factors such as reliability of the data, hospital-level sample size distribution, and cutoff values between quality tiers. The case of binary performance data is also explored using Monte Carlo simulation strategies. We apply the methods to real data and discuss the practical implications.

Hierarchical Rank Aggregation with Applications to Nanotoxicology

The development of high throughput screening (HTS) assays in the field of nanotoxicology provide new opportunities for the hazard assessment and ranking of engineered nanomaterials (ENMs). It is often necessary to rank lists of materials based on multiple risk assessment parameters, often aggregated across several measures of toxicity and possibly spanning an array of experimental platforms. Bayesian models coupled with the optimization of loss functions have been shown to provide an effective framework for conducting inference on ranks. In this article we present various loss-function-based ranking approaches for comparing ENM within experiments and toxicity parameters. Additionally, we propose a framework for the aggregation of ranks across different sources of evidence while allowing for differential weighting of this evidence based on its reliability and importance in risk ranking. We apply these methods to high throughput toxicity data on two human cell-lines, exposed to eight different nanomaterials, and measured in relation to four cytotoxicity outcomes. This article has supplementary material online.

Empirical Bayes ranking and selection methods via semiparametric hierarchical mixture models in microarray studies

The main purpose of microarray studies is screening of differentially expressed genes as candidates for further investigation. Because of limited resources in this stage, prioritizing genes are relevant statistical tasks in microarray studies. For effective gene selections, parametric empirical Bayes methods for ranking and selection of genes with largest effect sizes have been proposed (Noma et al., 2010; Biostatistics 11: 281-289). The hierarchical mixture model incorporates the differential and non-differential components and allows information borrowing across differential genes with separation from nuisance, non-differential genes. In this article, we develop empirical Bayes ranking methods via a semiparametric hierarchical mixture model. A nonparametric prior distribution, rather than parametric prior distributions, for effect sizes is specified and estimated using the "smoothing by roughening" approach of Laird and Louis (1991; Computational statistics and data analysis 12: 27-37). We present applications to childhood and infant leukemia clinical studies with microarrays for exploring genes related to prognosis or disease progression.

Brain network analysis: separating cost from topology using cost-integration

A statistically principled way of conducting brain network analysis is still lacking. Comparison of different populations of brain networks is hard because topology is inherently dependent on wiring cost, where cost is defined as the number of edges in an unweighted graph. In this paper, we evaluate the benefits and limitations associated with using cost-integrated topological metrics. Our focus is on comparing populations of weighted undirected graphs that differ in mean association weight, using global efficiency. Our key result shows that integrating over cost is equivalent to controlling for any monotonic transformation of the weight set of a weighted graph. That is, when integrating over cost, we eliminate the differences in topology that may be due to a monotonic transformation of the weight set. Our result holds for any unweighted topological measure, and for any choice of distribution over cost levels. Cost-integration is therefore helpful in disentangling differences in cost from differences in topology. By contrast, we show that the use of the weighted version of a topological metric is generally not a valid approach to this problem. Indeed, we prove that, under weak conditions, the use of the weighted version of global efficiency is equivalent to simply comparing weighted costs. Thus, we recommend the reporting of (i) differences in weighted costs and (ii) differences in cost-integrated topological measures with respect to different distributions over the cost domain. We demonstrate the application of these techniques in a re-analysis of an fMRI working memory task. We also provide a Monte Carlo method for approximating cost-integrated topological measures. Finally, we discuss the limitations of integrating topology over cost, which may pose problems when some weights are zero, when multiplicities exist in the ranks of the weights, and when one expects subtle cost-dependent topological differences, which could be masked by cost-integration.

Defining and ranking effects of individual agents based on survival times of cancer patients treated with combination chemotherapies

An important problem in oncology is comparing chemotherapy (chemo) agents in terms of their effects on survival or progression-free survival time. When the goal is to evaluate individual agents, a difficulty commonly encountered with observational data is that many patients receive a chemo combination including two or more agents. Because agents given in combination may interact, quantifying the contribution of each individual agent to the combination's overall effect is problematic. Still, if on average combinations including a particular agent confer longer survival, then that agent may be considered superior to agents whose combinations confer shorter survival. Motivated by this idea, we propose a definition of individual agent effects based on observational survival data from patients treated with many different chemo combinations. We define an individual agent effect as the average of the effects of the chemo combinations that include the agent. Similarly, we define the effect of each pair of agents as the average of the effects of the combinations including the pair. Under a Bayesian regression model for survival time in which the chemo combination effects follow a hierarchical structure, these definitions are used as a basis for estimating the posterior effects and ranks of the individual agents, and of all pairs of agents. The methods are illustrated by a data set arising from 224 pediatric brain tumor patients treated with over 27 different chemo combinations involving seven chemo agents.

Efficient evaluation of ranking procedures when the number of units is large, with application to SNP identification

Simulation-based assessment is a popular and frequently necessary approach for evaluating statistical procedures. Sometimes overlooked is the ability to take advantage of underlying mathematical relations and we focus on this aspect. We show how to take advantage of large-sample theory when conducting a simulation using the analysis of genomic data as a motivating example. The approach uses convergence results to provide an approximation to smaller-sample results, results that are available only by simulation. We consider evaluating and comparing various ranking-based methods for identifying the most highly associated SNPs in a genome-wide association study, derive integral equation representations of the pre-posterior distribution of percentiles produced by three ranking methods, and provide examples comparing performance. These results are of interest in their own right and set the framework for a more extensive set of comparisons.

Bayesian ranking and selection methods using hierarchical mixture models in microarray studies

The main purpose of microarray studies is screening to identify differentially expressed genes as candidates for further investigation. Because of limited resources in this stage, prioritizing or ranking genes is a relevant statistical task in microarray studies. In this article, we develop 3 empirical Bayes methods for gene ranking on the basis of differential expression, using hierarchical mixture models. These methods are based on (i) minimizing mean squared errors of estimation for parameters, (ii) minimizing mean squared errors of estimation for ranks of parameters, and (iii) maximizing sensitivity in selecting prespecified numbers of differential genes, with the largest effect. Our methods incorporate the mixture structures of differential and nondifferential components in empirical Bayes models to allow information borrowing across differential genes, with separation from nuisance, nondifferential genes. The accuracy of our ranking methods is compared with that of conventional methods through simulation studies. An application to a clinical study for breast cancer is provided.

Ranking USRDS provider specific SMRs from 1998-2001

Provider profiling (ranking/percentiling) is prevalent in health services research. Bayesian models coupled with optimizing a loss function provide an effective framework for computing non-standard inferences such as ranks. Inferences depend on the posterior distribution and should be guided by inferential goals. However, even optimal methods might not lead to definitive results and ranks should be accompanied by valid uncertainty assessments. We outline the Bayesian approach and use estimated Standardized Mortality Ratios (SMRs) in 1998-2001 from the United States Renal Data System (USRDS) as a platform to identify issues and demonstrate approaches. Our analyses extend Liu et al. (2004) by computing estimates developed by Lin et al. (2006) that minimize errors in classifying providers above or below a percentile cut-point, by combining evidence over multiple years via a first-order, autoregressive model on log(SMR), and by use of a nonparametric prior. Results show that ranks/percentiles based on maximum likelihood estimates of the SMRs and those based on testing whether an SMR = 1 substantially under-perform the optimal estimates. Combining evidence over the four years using the autoregressive model reduces uncertainty, improving performance over percentiles based on only one year. Furthermore, percentiles based on posterior probabilities of exceeding a properly chosen SMR threshold are essentially identical to those produced by minimizing classification loss. Uncertainty measures effectively calibrate performance, showing that considerable uncertainty remains even when using optimal methods. Findings highlight the importance of using loss function guided percentiles and the necessity of accompanying estimates with uncertainty assessments.