Differential protein expression and peak selection in mass spectrometry data by binary discriminant analysis

Identification of allergens from Aspergillus fumigatus-Potential association with lung damage in asthma

BACKGROUND

Component-resolved diagnosis allows detection of IgE sensitization having the advantage of reproducibility and standardization compared to crude extracts. The main disadvantage of the traditional allergen identification methods, 1- or 2-dimensional western blotting and screening of expression cDNA libraries with patients' IgEs, is that the native structure of the protein is not necessarily maintained.

METHODS

We used a novel immunoprecipitation technique in combination with mass spectrometry to identify new allergens of Aspergillus fumigatus. Magnetic Dynabeads coupled with anti-human IgE antibodies were used to purify human serum IgE and subsequently allergens from A. fumigatus protein extract.

RESULTS

Of the 184 proteins detected by subsequent mass peptide fingerprinting, a subset of 13 were recombinantly expressed and purified. In a panel of 52 A. fumigatus-sensitized people with asthma, 23 non-fungal-sensitized asthmatics and 18 healthy individuals, only the former showed an IgE reaction by immunoblotting and/or ELISA. We discovered 11 proteins not yet described as A. fumigatus allergens, with fructose-bisphosphate aldolase class II (FBA2) (33%), NAD-dependent malate dehydrogenase (31%) and Cu/Zn superoxide dismutase (27%) being the most prevalent. With respect to these three allergens, native versus denatured protein assays indicated a better recognition of the native proteins. Seven of 11 allergens fulfilled the WHO/IUIS criteria and were accepted as new A. fumigatus allergens.

CONCLUSION

In conclusion, we introduce a straightforward method of allergen identification from complex allergenic sources such as A. fumigatus by immunoprecipitation combined with mass spectrometry, which has the advantage over traditional methods of identifying allergens by maintaining the structure of the proteins.

A biological reading of a palimpsest

In the Middle Ages, texts were recorded and preserved on parchment, an animal-derived material. When this resource was scarce, older manuscripts were sometimes recycled to write new manuscripts. In the process, the ancient text was erased, creating what is known as a palimpsest. Here, we explore the potential of peptide mass fingerprinting (PMF), widely applied to identify species, to help reconnect the dispersed leaves of a manuscript and reveal differences in parchment manufacturing. In combination with visual methods, we analyzed a whole palimpsest, the codex AM 795 4to from the Arnamagnæan Collection (Copenhagen, Denmark). We find that both sheep and goat skins were used in this manuscript, and that parchment differed in quality. Notably, the PMF analysis distinguished five groups of folios which match the visual groupings. We conclude that this detailed interrogation of a single mass spectrum can be a promising tool to understand how palimpsest manuscripts were constructed.

Discrimination of the chemotherapy resistance status of human leukemia and glioblastoma cell lines by MALDI-TOF-MS profiling

Chemotherapy mistreatment is partially due to a lack of rapid and reliable tools to discriminate between sensitive and resistant phenotypes. In many cases, the resistance mechanism is not fully understood, contributing to the diagnostic tools' absence. This work aims to determine the capacity of MALDI-TOF-MS profiling to discriminate between chemotherapy-resistant and sensitive phenotypes in leukemia and glioblastoma cells. A multivariate analysis of two therapy-resistant leukemia cell lines (Ki562 and Kv562) and two TMZ-resistant glioblastoma cell lines (U251-R and LN229-R) and their sensitive counterparts was performed. In this work, we first show MALDI-TOF-MS patterns analysis ability to differentiate these cancer cell lines by their chemotherapy-resistant status. We present a rapid and inexpensive tool that would guide and complement the therapeutic decision.

Discrimination of Escherichia coli, Shigella flexneri, and Shigella sonnei using lipid profiling by MALDI-TOF mass spectrometry paired with machine learning

Matrix‐assisted laser desorption/ionization‐time of flight mass spectrometry (MALDI‐TOF MS) has become a staple in clinical microbiology laboratories. Protein‐profiling of bacteria using this technique has accelerated the identification of pathogens in diagnostic workflows. Recently, lipid profiling has emerged as a way to complement bacterial identification where protein‐based methods fail to provide accurate results. This study aimed to address the challenge of rapid discrimination between Escherichia coli and Shigella spp. using MALDI‐TOF MS in the negative ion mode for lipid profiling coupled with machine learning. Both E. coli and Shigella species are closely related; they share high sequence homology, reported for 16S rRNA gene sequence similarities between E. coli and Shigella spp. exceeding 99%, and a similar protein expression pattern but are epidemiologically distinct. A bacterial collection of 45 E. coli, 48 Shigella flexneri, and 62 Shigella sonnei clinical isolates were submitted to lipid profiling in negative ion mode using the MALDI Biotyper Sirius® system after treatment with mild‐acid hydrolysis (acetic acid 1% v/v for 15 min at 98°C). Spectra were then analyzed using our in‐house machine learning algorithm and top‐ranked features used for the discrimination of the bacterial species. Here, as a proof‐of‐concept, we showed that lipid profiling might have the potential to differentiate E. coli from Shigella species using the analysis of the top five ranked features obtained by MALDI‐TOF MS in the negative ion mode of the MALDI Biotyper Sirius® system. Based on this new approach, MALDI‐TOF MS analysis of lipids might help pave the way toward these goals.

Synergy between machine learning and natural products cheminformatics: Application to the lead discovery of anthraquinone derivatives

Cheminformatics utilizing machine learning (ML) techniques have opened up a new horizon in drug discovery. This is owing to vast chemical space expansion with rocketing numbers of expected hits and lead compounds that match druggable macromolecular targets, in particular from natural compounds. Due to the natural products' (NP) structural complexity, uniqueness, and diversity, they could occupy a bigger space in pharmaceuticals, allowing the industry to pursue more selective leads in the nanomolar range of binding affinity. ML is an essential part of each step of the drug design pipeline, such as target prediction, compound library preparation, and lead optimization. Notably, molecular mechanic and dynamic simulations, induced docking, and free energy perturbations are essential in predicting best binding poses, binding free energy values, and molecular mechanics force fields. Those applications have leveraged from artificial intelligence (AI), which decreases the computational costs required for such costly simulations. This review aimed to describe chemical space and compound libraries related to NPs. High-throughput screening utilized for fractionating NPs and high-throughput virtual screening and their strategies, and significance, are reviewed. Particular emphasis was given to AI approaches, ML tools, algorithms, and techniques, especially in drug discovery of macrocyclic compounds and approaches in computer-aided and ML-based drug discovery. Anthraquinone derivatives were discussed as a source of new lead compounds that can be developed using ML tools for diverse medicinal uses such as cancer, infectious diseases, and metabolic disorders. Furthermore, the power of principal component analysis in understanding relevant protein conformations, and molecular modeling of protein-ligand interaction were also presented. Apart from being a concise reference for cheminformatics, this review is a useful text to understand the application of ML-based algorithms to molecular dynamics simulation and in silico absorption, distribution, metabolism, excretion, and toxicity prediction.

Using MALDI-TOF spectra in epidemiological surveillance for the detection of bacterial subgroups with a possible epidemic potential

Background

For the purpose of epidemiological surveillance, the Hospital University Institute Méditerranée infection has implemented since 2013 a system named MIDaS, based on the systematic collection of routine activity materials, including MALDI-TOF spectra, and results. The objective of this paper is to present the pipeline we use for processing MALDI-TOF spectra during epidemiological surveillance in order to disclose proteinic cues that may suggest the existence of epidemic processes in complement of incidence surveillance. It is illustrated by the analysis of an alarm observed for Streptococcus pneumoniae.

Methods

The MALDI-TOF spectra analysis process looks for the existence of clusters of spectra characterized by a double time and proteinic close proximity. This process relies on several specific methods aiming at contrasting and clustering the spectra, presenting graphically the results for an easy epidemiological interpretation, and for determining the discriminating spectra peaks with their possible identification using reference databases.

Results

The use of this pipeline in the case of an alarm issued for Streptococcus pneumoniae has made it possible to reveal a cluster of spectra with close proteinic and temporal distances, characterized by the presence of three discriminant peaks (5228.8, 5917.8, and 8974.3 m/z) and the absence of peak 4996.9 m/z. A further investigation on UniProt KB showed that peak 5228.8 is possibly an OxaA protein and that the absent peak may be a transposase.

Conclusion

This example shows this pipeline may support a quasi-real time identification and characterization of clusters that provide essential information on a potentially epidemic situation. It brings valuable information for epidemiological sensemaking and for deciding on the continuation of the epidemiological investigation, in particular the involving of additional costly resources to confirm or invalidate the alarm.

Clinical trials registration

NCT03626987.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12879-021-06803-3.

Sputum proteomic analysis for distinguishing between pulmonary tuberculosis and non-tuberculosis using matrix-assisted laser desorption ionization time-of-flight mass spectrometry (MALDI-TOF MS): preliminary results

OBJECTIVES

The aim was to evaluate the feasibility and diagnostic contribution of protein profiling using matrix-assisted laser desorption ionization time-of-flight (MALDI-TOF) mass spectrometry (MS) applied to sputum to diagnose pulmonary tuberculosis.

METHODS

Sputum samples collected from patients suspected of having pulmonary tuberculosis were analysed using MALDI-TOF MS. Using the differentially expressed protein peaks, we compared three groups of patients, including those with confirmed pulmonary tuberculosis (PTB), those without tuberculosis but with a lower respiratory tract infection (non-TB LRTI) and those without tuberculosis and without an LRTI (non-TB controls).

RESULTS

A total of 102 patients included 35 PTB, 36 non-TB LRTI and 31 non-TB controls. The model differentiated between the PTB patients and the non-TB controls using the 25 most differentially expressed protein peaks, with a sensitivity of 97%, 95% CI 85-100%, and a specificity of 77%, 95% CI 59-90%. The model distinguished the PTB patients from the non-TB LRTI patients using the ten most differentially expressed protein peaks, with a sensitivity of 80%, 95% CI 63-92%, and a specificity of 89%, 95% CI 74-97%. We observed that the negative predictive value of MALDI-TOF MS sputum analysis was higher (96%, 95% CI 80-100%) than that of direct sputum microscopic examination and sputum culture (78%, 95% CI 62-89%) for non-TB controls. When MALDI-TOF MS sputum analysis and direct microscopic examination were combined, the negative predictive value reached 94%, 95% CI 80-99%, for non-TB LRTI patients.

DISCUSSION

These results suggest that MALDI-TOF MS sputum analysis coupled with microscopic examination could be used as a screening tool for diagnosing pulmonary TB.

Establishing Objective Measures of Clinical Competence in Undergraduate Medical Education Through Immersive Virtual Reality

OBJECTIVE

The Association of American Medical Colleges defines recognition of the need for urgent or emergent escalation of care as a key Entrustable Professional Activity (EPA) for entering residency (EPA#10). This study pilots the use of an immersive virtual reality (VR) platform for defining objective observable behaviors as standards for evaluation of medical student recognition of impending respiratory failure.

METHODS

A cross-sectional observational study was conducted from July 2018 to December 2019, evaluating student performance during a VR scenario of an infant in impending respiratory failure using the OculusRift VR platform. Video recordings were rated by 2 pair of physician reviewers blinded to student identity. One pair provided a consensus global assessment of performance (not competent, borderline, or competent) while the other used a checklist of observable behaviors to rate performance. Binary discriminant analysis was used to identify the observable behaviors that predicted the global assessment rating.

RESULTS

Twenty-six fourth year medical students participated. Student performance of 8 observable behaviors was found to be most predictive of a rating of competent, with a 91% probability. Correctly stating that the patient required an escalation of care had the largest contribution toward predicting a rating of competent, followed by commenting on the patient's increased heart rate, low oxygen saturation, increased respiratory rate, and stating that the patient was in respiratory distress.

CONCLUSIONS

This study demonstrates that VR can be used to establish objective and observable performance standards for assessment of EPA attainment - a key step in moving towards competency based medical education.

Gingival Crevicular Fluid Peptidome Profiling in Healthy and in Periodontal Diseases

Given its intrinsic nature, gingival crevicular fluid (GCF) is an attractive source for the discovery of novel biomarkers of periodontal diseases. GCF contains antimicrobial peptides and small proteins which could play a role in specific immune-inflammatory responses to guarantee healthy gingival status and to prevent periodontal diseases. Presently, several proteomics studies have been performed leading to increased coverage of the GCF proteome, however fewer efforts have been done to explore its natural peptides. To fill such gap, this review provides an overview of the mass spectrometric platforms and experimental designs aimed at GCF peptidome profiling, including our own data and experiences gathered from over several years of matrix-assisted laser desorption ionization/time of flight mass spectrometry (MALDI-TOF MS) based approach in this field. These tools might be useful for capturing snapshots containing diagnostic clinical information on an individual and population scale, which may be used as a specific code not only for the diagnosis of the nature or the stage of the inflammatory process in periodontal disease, but more importantly, for its prognosis, which is still an unmet medical need. As a matter of fact, current peptidomics investigations suffer from a lack of standardized procedures, posing a serious problem for data interpretation. Descriptions of the efforts to address such concerns will be highlighted.

Rapid diagnosis of periodontitis, a feasibility study using MALDI-TOF mass spectrometry

AIM

The aim of the present study was to assess the feasibility and diagnostic contribution of protein profiling using MALDI-TOF mass spectrometry applied to saliva, gingival crevicular fluid (GCF) and dental plaque from periodontitis and healthy subjects. We hypothesized that rapid routine and blinded MALDI-TOF analysis could accurately classify these three types of samples according to periodontal state.

MATERIALS AND METHODS

Unstimulated saliva, GCF and dental plaque, collected from periodontitis subjects and healthy controls, were analyzed by MALDI-TOF MS. Based on the differentially expressed peaks between the two groups, diagnostic decision trees were built for each sample.

RESULTS

Among 141 patients (67 periodontitis and 74 healthy controls), the decision trees diagnosed periodontitis with a sensitivity = 70.3% (± 0.211) and a specificity = 77.8% (± 0.165) for saliva, a sensitivity = 79.6% (± 0.188) and a specificity = 75.7% (± 0.195) for GCF, and a sensitivity = 72.1% (± 0.202) and a specificity = 72.2% (± 0.195) for dental plaque. The sensitivity and specificity of the tests were improved to 100% (CI 95% = [0.91;1]) and 100% (CI 95% = [0.92;1]), respectively, when two samples were tested.

CONCLUSION

We developed, for the first time, diagnostic tests based on protein profiles of saliva, GCF and dental plaque between periodontitis patients and healthy subjects. When at least 2 of these samples were tested, the best results were obtained.

MALDI-TOF mass spectrometry on intact bacteria combined with a refined analysis framework allows accurate classification of MSSA and MRSA

Fast and reliable detection coupled with accurate data-processing and analysis of antibiotic-resistant bacteria is essential in clinical settings. In this study, we use MALDI-TOF on intact cells combined with a refined analysis framework to demonstrate discrimination between methicillin-susceptible (MSSA) and methicillin-resistant (MRSA) Staphylococcus aureus. By combining supervised and unsupervised machine learning methods, we firstly show that the mass spectroscopy data contains strong signal for the clustering of MSSA and MRSA. Then we concentrate on applying supervised learning to extract and verify the important features. A new workflow is proposed that allows for extracting a fixed set of reference peaks so that any new data can be aligned to it and hence consistent feature matrices can be obtained. Also note that by doing so we are able to examine the robustness of the important features that have been found. We also show that appropriate size of the benchmark data, appropriate alignment of the testing data and use of an optimal set of features via feature selection results in prediction accuracy over 90%. In summary, as proof-of-principle, our integrated experimental and bioinformatics study suggests a novel intact cell MALDI-TOF to be of great promise for fast and reliable detection of MRSA strains.

Spatio-temporal flowering patterns in Mediterranean Poaceae. A community study in SW Spain

This study focused on phenological timing and spatial patterns in 30 Poaceae species flowering in spring in different types of plant cover (scrub, riverbank and pasture). Grass community composition was studied, and the influence of species and plant cover on the start date and duration of flowering was assessed from March to June in both 2014 and 2015. Twenty-nine sampling sites were selected for phenological monitoring using the BBCH scale. Data were subjected to GLMM analyses. Binary discriminant analysis revealed differences in grass community composition as a function of plant cover type; scrub cover comprised a considerably larger number of species than those in riverbank and pasture. Moreover, more species diversity was observed in 2014 than in 2015 with a drier and stressed pre-flowering period. Differences on phenology were also recorded between plant cover types and study years. Species in pasture and riverbank flowered before (113.4 days; 116.1 days) than species in scrub (120.9 days), being these species with shorter flowering length because they are more exposed to the characteristic of the Mediterranean region during the summer. In general, flowering onset occurred later in 2014 (118.2 days) than in 2015 (115.8 days), probably attributable to precipitation occurring during March. On the other hand, spatial autocorrelation within some cover types has been observed, showing spatial patterns exist at a smaller scale. The findings of this study contribute to a better understanding of grass phenology in different environments.

Using MALDI-TOF MS typing method to decipher outbreak: the case of Staphylococcus saprophyticus causing urinary tract infections (UTIs) in Marseille, France

Staphylococcus saprophyticus is one of the leading causes of urinary tract infections (UTI). In December 2014, our surveillance system identified an abnormal increase in S. saprophyticus causing UTIs in four university hospitals in Marseille, indicating a suspected community S. saprophyticus UTI outbreak. This was detected by our surveillance system BALYSES (Bacterial real-time Laboratory-based Surveillance System). S. saprophyticus/ Escherichia coli UTI ratio increased three-fold from 0.0084 in 2002 to 0.025 in December 2015 in Marseille with an abnormal peak in December 2014, and with an annual estimated ratio trend of 5.10^-6 (p-value < 10^-3). Matrix-Assisted Laser Desorption Ionisation-Time of Flight Mass Spectrometry (MALDI-TOF MS) spectral analysis of strains was used to analyse strains cluster expansion, comparing strains from Marseille to those from Nice during the same period. MALDI-TOF MS spectral analysis revealed a geographical restricted clonal expansion of the strains clusters in Marseille as compared to Nice. Our finding suggests (i) a geographically restricted expansion of a specific S. saprophyticus strain clusters circulating in Marseille, and (ii) MALDI-TOF MS can be used as a cost-effective tool to investigate an outbreak.

Sparse Proteomics Analysis - a compressed sensing-based approach for feature selection and classification of high-dimensional proteomics mass spectrometry data

BACKGROUND

High-throughput proteomics techniques, such as mass spectrometry (MS)-based approaches, produce very high-dimensional data-sets. In a clinical setting one is often interested in how mass spectra differ between patients of different classes, for example spectra from healthy patients vs. spectra from patients having a particular disease. Machine learning algorithms are needed to (a) identify these discriminating features and (b) classify unknown spectra based on this feature set. Since the acquired data is usually noisy, the algorithms should be robust against noise and outliers, while the identified feature set should be as small as possible.

RESULTS

We present a new algorithm, Sparse Proteomics Analysis (SPA), based on the theory of compressed sensing that allows us to identify a minimal discriminating set of features from mass spectrometry data-sets. We show (1) how our method performs on artificial and real-world data-sets, (2) that its performance is competitive with standard (and widely used) algorithms for analyzing proteomics data, and (3) that it is robust against random and systematic noise. We further demonstrate the applicability of our algorithm to two previously published clinical data-sets.

Comparison of different statistical approaches for urinary peptide biomarker detection in the context of coronary artery disease

Background

When combined with a clinical outcome variable, the size, complexity and nature of mass-spectrometry proteomics data impose great statistical challenges in the discovery of potential disease-associated biomarkers. The purpose of this study was thus to evaluate the effectiveness of different statistical methods applied for urinary proteomic biomarker discovery and different methods of classifier modelling in respect of the diagnosis of coronary artery disease in 197 study subjects and the prognostication of acute coronary syndromes in 368 study subjects.

Results

Computing the discovery sub-cohorts comprising \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ {\scriptscriptstyle \raisebox{1ex}{$2$}\!\left/ \!\raisebox{-1ex}{$3$}\right.} $$\end{document}23 of the study subjects based on the Wilcoxon rank sum test, t-score, cat-score, binary discriminant analysis and random forests provided largely different numbers (ranging from 2 to 398) of potential peptide biomarkers. Moreover, these biomarker patterns showed very little overlap limited to fragments of type I and III collagens as the common denominator. However, these differences in biomarker patterns did mostly not translate into significant differently performing diagnostic or prognostic classifiers modelled by support vector machine, diagonal discriminant analysis, linear discriminant analysis, binary discriminant analysis and random forest. This was even true when different biomarker patterns were combined into master-patterns.

Conclusion

In conclusion, our study revealed a very considerable dependence of peptide biomarker discovery on statistical computing of urinary peptide profiles while the observed diagnostic and/or prognostic reliability of classifiers was widely independent of the modelling approach. This may however be due to the limited statistical power in classifier testing. Nonetheless, our study showed that urinary proteome analysis has the potential to provide valuable biomarkers for coronary artery disease mirroring especially alterations in the extracellular matrix. It further showed that for a comprehensive discovery of biomarkers and thus of pathological information, the results of different statistical methods may best be combined into a master pattern that then can be used for classifier modelling.

Electronic supplementary material

The online version of this article (doi:10.1186/s12859-016-1390-1) contains supplementary material, which is available to authorized users.

Spectra-first feature analysis in clinical proteomics — A case study in renal cancer

In proteomics, useful signal may be unobserved or lost due to the lack of confident peptide-spectral matches. Selection of differential spectra, followed by associative peptide/protein mapping may be a complementary strategy for improving sensitivity and comprehensiveness of analysis (spectra-first paradigm). This approach is complementary to the standard approach where functional analysis is performed only on the finalized protein list assembled from identified peptides from the spectra (protein-first paradigm). Based on a case study of renal cancer, we introduce a simple spectra-binning approach, MZ-bin. We demonstrate that differential spectra feature selection using MZ-bin is class-discriminative and can trace relevant proteins via spectra associative mapping. Moreover, proteins identified in this manner are more biologically coherent than those selected directly from the finalized protein list. Analysis of constituent peptides per protein reveals high expression inconsistency, suggesting that the measured protein expressions are in fact, poor approximations of true protein levels. Moreover, analysis at the level of constituent peptides may provide higher resolution insight into the underlying biology: Via MZ-bin, we identified for the first time differential splice forms for the known renal cancer marker MAPT. We conclude that the spectra-first analysis paradigm is a complementary strategy to the traditional protein-first paradigm and can provide deeper level insight.

A Bayesian algorithm for detecting differentially expressed proteins and its application in breast cancer research

Presence of considerable noise and missing data points make analysis of mass-spectrometry (MS) based proteomic data a challenging task. The missing values in MS data are caused by the inability of MS machines to reliably detect proteins whose abundances fall below the detection limit. We developed a Bayesian algorithm that exploits this knowledge and uses missing data points as a complementary source of information to the observed protein intensities in order to find differentially expressed proteins by analysing MS based proteomic data. We compared its accuracy with many other methods using several simulated datasets. It consistently outperformed other methods. We then used it to analyse proteomic screens of a breast cancer (BC) patient cohort. It revealed large differences between the proteomic landscapes of triple negative and Luminal A, which are the most and least aggressive types of BC. Unexpectedly, majority of these differences could be attributed to the direct transcriptional activity of only seven transcription factors some of which are known to be inactive in triple negative BC. We also identified two new proteins which significantly correlated with the survival of BC patients, and therefore may have potential diagnostic/prognostic values.

Boolean modeling techniques for protein co-expression networks in systems medicine

INTRODUCTION

Application of systems biology/systems medicine approaches is promising for proteomics/biomedical research, but requires selection of an adequate modeling type.

AREAS COVERED

This article reviews the existing Boolean network modeling approaches, which provide in comparison with alternative modeling techniques several advantages for the processing of proteomics data. Application of methods for inference, reduction and validation of protein co-expression networks that are derived from quantitative high-throughput proteomics measurements is presented. It's also shown how Boolean models can be used to derive system-theoretic characteristics that describe both the dynamical behavior of such networks as a whole and the properties of different cell states (e.g. healthy or diseased cell states). Furthermore, application of methods derived from control theory is proposed in order to simulate the effects of therapeutic interventions on such networks, which is a promising approach for the computer-assisted discovery of biomarkers and drug targets. Finally, the clinical application of Boolean modeling analyses is discussed. Expert commentary: Boolean modeling of proteomics data is still in its infancy. Progress in this field strongly depends on provision of a repository with public access to relevant reference models. Also required are community supported standards that facilitate input of both proteomics and patient related data (e.g. age, gender, laboratory results, etc.).