Perspectives on incorporating expert feedback into model updates

Machine learning (ML) practitioners are increasingly tasked with developing models that are aligned with non-technical experts' values and goals. However, there has been insufficient consideration of how practitioners should translate domain expertise into ML updates. In this review, we consider how to capture interactions between practitioners and experts systematically. We devise a taxonomy to match expert feedback types with practitioner updates. A practitioner may receive feedback from an expert at the observation or domain level and then convert this feedback into updates to the dataset, loss function, or parameter space. We review existing work from ML and human-computer interaction to describe this feedback-update taxonomy and highlight the insufficient consideration given to incorporating feedback from non-technical experts. We end with a set of open questions that naturally arise from our proposed taxonomy and subsequent survey.

Inferring population structure in biobank-scale genomic data

Inferring the structure of human populations from genetic variation data is a key task in population and medical genomic studies. Although a number of methods for population structure inference have been proposed, current methods are impractical to run on biobank-scale genomic datasets containing millions of individuals and genetic variants. We introduce SCOPE, a method for population structure inference that is orders of magnitude faster than existing methods while achieving comparable accuracy. SCOPE infers population structure in about a day on a dataset containing one million individuals and variants as well as on the UK Biobank dataset containing 488,363 individuals and 569,346 variants. Furthermore, SCOPE can leverage allele frequencies from previous studies to improve the interpretability of population structure estimates.

Explaining Groups of Points in Low-Dimensional Representations

A common workflow in data exploration is to learn a low-dimensional representation of the data, identify groups of points in that representation, and examine the differences between the groups to determine what they represent. We treat this workflow as an interpretable machine learning problem by leveraging the model that learned the low-dimensional representation to help identify the key differences between the groups. To solve this problem, we introduce a new type of explanation, a Global Counterfactual Explanation (GCE), and our algorithm, Transitive Global Translations (TGT), for computing GCEs. TGT identifies the differences between each pair of groups using compressed sensing but constrains those pairwise differences to be consistent among all of the groups. Empirically, we demonstrate that TGT is able to identify explanations that accurately explain the model while being relatively sparse, and that these explanations match real patterns in the data.

SMaSH: a benchmarking toolkit for human genome variant calling

MOTIVATION

Computational methods are essential to extract actionable information from raw sequencing data, and to thus fulfill the promise of next-generation sequencing technology. Unfortunately, computational tools developed to call variants from human sequencing data disagree on many of their predictions, and current methods to evaluate accuracy and computational performance are ad hoc and incomplete. Agreement on benchmarking variant calling methods would stimulate development of genomic processing tools and facilitate communication among researchers.

RESULTS

We propose SMaSH, a benchmarking methodology for evaluating germline variant calling algorithms. We generate synthetic datasets, organize and interpret a wide range of existing benchmarking data for real genomes and propose a set of accuracy and computational performance metrics for evaluating variant calling methods on these benchmarking data. Moreover, we illustrate the utility of SMaSH to evaluate the performance of some leading single-nucleotide polymorphism, indel and structural variant calling algorithms.

AVAILABILITY AND IMPLEMENTATION

We provide free and open access online to the SMaSH tool kit, along with detailed documentation, at smash.cs.berkeley.edu

Joint Link Prediction and Attribute Inference Using a Social-Attribute Network

The effects of social influence and homophily suggest that both network structure and node-attribute information should inform the tasks of link prediction and node-attribute inference. Recently, Yin et al. [2010a, 2010b] proposed an attribute-augmented social network model, which we call Social-Attribute Network (SAN), to integrate network structure and node attributes to perform both link prediction and attribute inference. They focused on generalizing the random walk with a restart algorithm to the SAN framework and showed improved performance. In this article, we extend the SAN framework with several leading supervised and unsupervised link-prediction algorithms and demonstrate performance improvement for each algorithm on both link prediction and attribute inference. Moreover, we make the novel observation that attribute inference can help inform link prediction, that is, link-prediction accuracy is further improved by first inferring missing attributes. We comprehensively evaluate these algorithms and compare them with other existing algorithms using a novel, large-scale Google+ dataset, which we make publicly available (http://www.cs.berkeley.edu/~stevgong/gplus.html).

Metabolic and biochemical responses of probiotic bacteria to oxygen

The interaction between oxygen and probiotic bacteria was studied by growing Lactobacillus acidophilus and Bifidobacterium spp. in 0, 5, 10, 15, and 21% oxygen in a hypoxic glove box. The metabolic responses of each probiotic strain in the different oxygen environments were monitored by measuring the levels of lactic acid and determining the lactate-to-acetate ratio. Biochemical changes induced by oxygen were examined by monitoring the specific activities of NADH oxidase, NADH peroxidase, and superoxide dismutase. In addition, the ability to decompose hydrogen peroxide and the sensitivity of each strain to hydrogen peroxide was also determined. With an increase in oxygen percentage, levels of lactic acid in L. acidophilus strains decreased, whereas the lactate-to-acetate ratio reduced in all the bifidobacteria tested. At 21% oxygen, the specific activities of NADH oxidase and NADH peroxidase, and the hydrogen peroxide decomposing ability of five probiotic strains was significantly higher than at 0% oxygen. The sensitivity of the probiotic strains to hydrogen peroxide however, remained unaffected in all the different oxygen percentages. Superoxide dismutase levels did not reveal any conclusive trend. In both L. acidophilus and Bifidobacterium spp., NADH oxidase and NADH peroxidase functioned optimally at pH 5. Growth in the various oxygen environments did not change this optimum pH.

Application of RBGR--a simple way for screening of oxygen tolerance in probiotic bacteria

Oxygen toxicity is a major problem in the survival of probiotic bacteria in dairy foods. High levels of oxygen in the product are detrimental to the viability of these predominantly anaerobic bacteria. Screening probiotic bacteria for oxygen tolerance before their incorporation could ensure high cell counts in food products during storage. Reported techniques have focused only on qualitative estimations of oxygen tolerance in probiotic bacteria. To characterize the oxygen tolerance of a large number of organisms, a quantitative measurement is essential. For the first time, the oxygen tolerance of several probiotic strains was measured quantitatively using an index known as Relative Bacterial Growth Ratio (RBGR). The tolerance to oxygen varied between organisms, and this technique can therefore be applied for screening probiotic bacteria for oxygen tolerance.