The 2012/2013 ABRF Proteomic Research Group Study: Assessing Longitudinal Intralaboratory Variability in Routine Peptide Liquid Chromatography Tandem Mass Spectrometry Analyses

Multicenter Longitudinal Quality Assessment of MS-Based Proteomics in Plasma and Serum

Advancing MS-based proteomics toward clinical applications evolves around developing standardized start-to-finish and fit-for-purpose workflows for clinical specimens. Steps along the method design involve the determination and optimization of several bioanalytical parameters such as selectivity, sensitivity, accuracy, and precision. In a joint effort, eight proteomics laboratories belonging to the MSCoreSys initiative including the CLINSPECT-M, MSTARS, DIASyM, and SMART-CARE consortia performed a longitudinal round-robin study to assess the analysis performance of plasma and serum as clinically relevant samples. A variety of LC-MS/MS setups including mass spectrometer models from ThermoFisher and Bruker as well as LC systems from ThermoFisher, Evosep, and Waters Corporation were used in this study. As key performance indicators, sensitivity, precision, and reproducibility were monitored over time. Protein identifications range between 300 and 400 IDs across different state-of-the-art MS instruments, with timsTOF Pro, Orbitrap Exploris 480, and Q Exactive HF-X being among the top performers. Overall, 71 proteins are reproducibly detectable in all setups in both serum and plasma samples, and 22 of these proteins are FDA-approved biomarkers, which are reproducibly quantified (CV < 20% with label-free quantification). In total, the round-robin study highlights a promising baseline for bringing MS-based measurements of serum and plasma samples closer to clinical utility.

A Multiyear Longitudinal Harmonization Study of Quality Controls in Mass Spectrometry Proteomics Core Facilities

Quality control procedures play a pivotal role in ensuring the reliability and consistency of data generated in mass spectrometry-based proteomics laboratories. However, the lack of standardized quality control practices across laboratories poses challenges for data comparability and reproducibility. In response, we conducted a harmonization study within proteomics laboratories of the Core for Life alliance with the aim of establishing a common quality control framework, which facilitates comprehensive quality assessment and identification of potential sources of performance drift. Through collaborative efforts, we developed a consensus quality control standard for longitudinal assessment and adopted common processing software. We generated a 4-year longitudinal data set from multiple instruments and laboratories, which enabled us to assess intra- and interlaboratory variability, to identify causes of performance drift, and to establish community reference values for several quality control parameters. Our study enhances data comparability and reliability and fosters a culture of collaboration and continuous improvement within the proteomics community to ensure the integrity of proteomics data.

A uniform data processing pipeline enables harmonized nanoparticle protein corona analysis across proteomics core facilities

Protein corona, a layer of biomolecules primarily comprising proteins, forms dynamically on nanoparticles in biological fluids and is crucial for predicting nanomedicine safety and efficacy. The protein composition of the corona layer is typically analyzed using liquid chromatography-mass spectrometry (LC-MS/MS). Our recent study, involving identical samples analyzed by 17 proteomics facilities, highlighted significant data variability, with only 1.8% of proteins consistently identified across these centers. Here, we implement an aggregated database search unifying parameters such as variable modifications, enzyme specificity, number of allowed missed cleavages and a stringent 1% false discovery rate at the protein and peptide levels. Such uniform search dramatically harmonizes the proteomics data, increasing the reproducibility and the percentage of consistency-identified unique proteins across distinct cores. Specifically, out of the 717 quantified proteins, 253 (35.3%) are shared among the top 5 facilities (and 16.2% among top 11 facilities). Furthermore, we note that reduction and alkylation are important steps in protein corona sample processing and as expected, omitting these steps reduces the number of total quantified peptides by around 20%. These findings underscore the need for standardized procedures in protein corona analysis, which is vital for advancing clinical applications of nanoscale biotechnologies.

Multicenter Collaborative Study to Optimize Mass Spectrometry Workflows of Clinical Specimens

The foundation for integrating mass spectrometry (MS)-based proteomics into systems medicine is the development of standardized start-to-finish and fit-for-purpose workflows for clinical specimens. An essential step in this pursuit is to highlight the common ground in a diverse landscape of different sample preparation techniques and liquid chromatography-mass spectrometry (LC-MS) setups. With the aim to benchmark and improve the current best practices among the proteomics MS laboratories of the CLINSPECT-M consortium, we performed two consecutive round-robin studies with full freedom to operate in terms of sample preparation and MS measurements. The six study partners were provided with two clinically relevant sample matrices: plasma and cerebrospinal fluid (CSF). In the first round, each laboratory applied their current best practice protocol for the respective matrix. Based on the achieved results and following a transparent exchange of all lab-specific protocols within the consortium, each laboratory could advance their methods before measuring the same samples in the second acquisition round. Both time points are compared with respect to identifications (IDs), data completeness, and precision, as well as reproducibility. As a result, the individual performances of participating study centers were improved in the second measurement, emphasizing the effect and importance of the expert-driven exchange of best practices for direct practical improvements.

Comparing Top-Down Proteoform Identification: Deconvolution, PrSM Overlap, and PTM Detection

Generating top-down tandem mass spectra (MS/MS) from complex mixtures of proteoforms benefits from improvements in fractionation, separation, fragmentation, and mass analysis. The algorithms to match MS/MS to sequences have undergone a parallel evolution, with both spectral alignment and match-counting approaches producing high-quality proteoform-spectrum matches (PrSMs). This study assesses state-of-the-art algorithms for top-down identification (ProSight PD, TopPIC, MSPathFinderT, and pTop) in their yield of PrSMs while controlling false discovery rate. We evaluated deconvolution engines (ThermoFisher Xtract, Bruker AutoMSn, Matrix Science Mascot Distiller, TopFD, and FLASHDeconv) in both ThermoFisher Orbitrap-class and Bruker maXis Q-TOF data (PXD033208) to produce consistent precursor charges and mass determinations. Finally, we sought post-translational modifications (PTMs) in proteoforms from bovine milk (PXD031744) and human ovarian tissue. Contemporary identification workflows produce excellent PrSM yields, although approximately half of all identified proteoforms from these four pipelines were specific to only one workflow. Deconvolution algorithms disagree on precursor masses and charges, contributing to identification variability. Detection of PTMs is inconsistent among algorithms. In bovine milk, 18% of PrSMs produced by pTop and TopMG were singly phosphorylated, but this percentage fell to 1% for one algorithm. Applying multiple search engines produces more comprehensive assessments of experiments. Top-down algorithms would benefit from greater interoperability.

New Peak Detection Performance Metrics from the MAM Consortium Interlaboratory Study

The Multi-Attribute Method (MAM) Consortium was initially formed as a venue to harmonize best practices, share experiences, and generate innovative methodologies to facilitate widespread integration of the MAM platform, which is an emerging ultra-high-performance liquid chromatography-mass spectrometry application. Successful implementation of MAM as a purity-indicating assay requires new peak detection (NPD) of potential process- and/or product-related impurities. The NPD interlaboratory study described herein was carried out by the MAM Consortium to report on the industry-wide performance of NPD using predigested samples of the NISTmAb Reference Material 8671. Results from 28 participating laboratories show that the NPD parameters being utilized across the industry are representative of high-resolution MS performance capabilities. Certain elements of NPD, including common sources of variability in the number of new peaks detected, that are critical to the performance of the purity function of MAM were identified in this study and are reported here as a means to further refine the methodology and accelerate adoption into manufacturer-specific protein therapeutic product life cycles.

A Review of the Scientific Rigor, Reproducibility, and Transparency Studies Conducted by the ABRF Research Groups

Shared research resource facilities, also known as core laboratories (Cores), are responsible for generating a significant and growing portion of the research data in academic biomedical research institutions. Cores represent a central repository for institutional knowledge management, with deep expertise in the strengths and limitations of technology and its applications. They inherently support transparency and scientific reproducibility by protecting against cognitive bias in research design and data analysis, and they have institutional responsibility for the conduct of research (research ethics, regulatory compliance, and financial accountability) performed in their Cores. The Association of Biomolecular Resource Facilities (ABRF) is a FASEB-member scientific society whose members are scientists and administrators that manage or support Cores. The ABRF Research Groups (RGs), representing expertise for an array of cutting-edge and established technology platforms, perform multicenter research studies to determine and communicate best practices and community-based standards. This review provides a summary of the contributions of the ABRF RGs to promote scientific rigor and reproducibility in Cores from the published literature, ABRF meetings, and ABRF RGs communications.

Quality control in mass spectrometry-based proteomics

Mass spectrometry is a highly complex analytical technique and mass spectrometry-based proteomics experiments can be subject to a large variability, which forms an obstacle to obtaining accurate and reproducible results. Therefore, a comprehensive and systematic approach to quality control is an essential requirement to inspire confidence in the generated results. A typical mass spectrometry experiment consists of multiple different phases including the sample preparation, liquid chromatography, mass spectrometry, and bioinformatics stages. We review potential sources of variability that can impact the results of a mass spectrometry experiment occurring in all of these steps, and we discuss how to monitor and remedy the negative influences on the experimental results. Furthermore, we describe how specialized quality control samples of varying sample complexity can be incorporated into the experimental workflow and how they can be used to rigorously assess detailed aspects of the instrument performance.

Harmonizing lipidomics: NIST interlaboratory comparison exercise for lipidomics using SRM 1950-Metabolites in Frozen Human Plasma

As the lipidomics field continues to advance, self-evaluation within the community is critical. Here, we performed an interlaboratory comparison exercise for lipidomics using Standard Reference Material (SRM) 1950-Metabolites in Frozen Human Plasma, a commercially available reference material. The interlaboratory study comprised 31 diverse laboratories, with each laboratory using a different lipidomics workflow. A total of 1,527 unique lipids were measured across all laboratories and consensus location estimates and associated uncertainties were determined for 339 of these lipids measured at the sum composition level by five or more participating laboratories. These evaluated lipids detected in SRM 1950 serve as community-wide benchmarks for intra- and interlaboratory quality control and method validation. These analyses were performed using nonstandardized laboratory-independent workflows. The consensus locations were also compared with a previous examination of SRM 1950 by the LIPID MAPS consortium. While the central theme of the interlaboratory study was to provide values to help harmonize lipids, lipid mediators, and precursor measurements across the community, it was also initiated to stimulate a discussion regarding areas in need of improvement.

Quality Assessments of Long-Term Quantitative Proteomic Analysis of Breast Cancer Xenograft Tissues

Clinical proteomics requires large-scale analysis of human specimens to achieve statistical significance. We evaluated the long-term reproducibility of an iTRAQ (isobaric tags for relative and absolute quantification)-based quantitative proteomics strategy using one channel for reference across all samples in different iTRAQ sets. A total of 148 liquid chromatography tandem mass spectrometric (LC-MS/MS) analyses were completed, generating six 2D LC-MS/MS data sets for human-in-mouse breast cancer xenograft tissues representative of basal and luminal subtypes. Such large-scale studies require the implementation of robust metrics to assess the contributions of technical and biological variability in the qualitative and quantitative data. Accordingly, we derived a quantification confidence score based on the quality of each peptide-spectrum match to remove quantification outliers from each analysis. After combining confidence score filtering and statistical analysis, reproducible protein identification and quantitative results were achieved from LC-MS/MS data sets collected over a 7-month period. This study provides the first quality assessment on long-term stability and technical considerations for study design of a large-scale clinical proteomics project.

MSstatsQC: Longitudinal System Suitability Monitoring and Quality Control for Targeted Proteomic Experiments

Selected Reaction Monitoring (SRM) is a powerful tool for targeted detection and quantification of peptides in complex matrices. An important objective of SRM is to obtain peptide quantifications that are (1) suitable for the investigation, and (2) reproducible across laboratories and runs. The first objective is achieved by system suitability tests (SST), which verify that mass spectrometric instrumentation performs as specified. The second objective is achieved by quality control (QC), which provides in-process quality assurance of the sample profile. A common aspect of SST and QC is the longitudinal nature of the data. Although SST and QC have received a lot of attention in the proteomic community, the currently used statistical methods are limited. This manuscript improves upon the statistical methodology for SST and QC that is currently used in proteomics. It adapts the modern methods of longitudinal statistical process control, such as simultaneous and time weighted control charts and change point analysis, to SST and QC of SRM experiments, discusses their advantages, and provides practical guidelines. Evaluations on simulated data sets, and on data sets from the Clinical Proteomics Technology Assessment for Cancer (CPTAC) consortium, demonstrated that these methods substantially improve our ability of real time monitoring, early detection and prevention of chromatographic and instrumental problems. We implemented the methods in an open-source R-based software package MSstatsQC and its web-based graphical user interface. They are available for use stand-alone, or for integration with automated pipelines. Although the examples focus on targeted proteomics, the statistical methods in this manuscript apply more generally to quantitative proteomics.

Variations of Histone Modification Patterns: Contributions of Inter-plant Variability and Technical Factors

Inter-individual variability of conspecific plants is governed by differences in their genetically determined growth and development traits, environmental conditions, and adaptive responses under epigenetic control involving histone post-translational modifications. The apparent variability in histone modifications among plants might be increased by technical variation introduced in sample processing during epigenetic analyses. Thus, to detect true variations in epigenetic histone patterns associated with given factors, the basal variability among samples that is not associated with them must be estimated. To improve knowledge of relative contribution of biological and technical variation, mass spectrometry was used to examine histone modification patterns (acetylation and methylation) among Arabidopsis thaliana plants of ecotypes Columbia 0 (Col-0) and Wassilewskija (Ws) homogenized by two techniques (grinding in a cryomill or with a mortar and pestle). We found little difference in histone modification profiles between the ecotypes. However, in comparison of the biological and technical components of variability, we found consistently higher inter-individual variability in histone mark levels among Ws plants than among Col-0 plants (grown from seeds collected either from single plants or sets of plants). Thus, more replicates of Ws would be needed for rigorous analysis of epigenetic marks. Regarding technical variability, the cryomill introduced detectably more heterogeneity in the data than the mortar and pestle treatment, but mass spectrometric analyses had minor apparent effects. Our study shows that it is essential to consider inter-sample variance and estimate suitable numbers of biological replicates for statistical analysis for each studied organism when investigating changes in epigenetic histone profiles.

Unsupervised Quality Assessment of Mass Spectrometry Proteomics Experiments by Multivariate Quality Control Metrics

Despite many technological and computational advances, the results of a mass spectrometry proteomics experiment are still subject to a large variability. For the understanding and evaluation of how technical variability affects the results of an experiment, several computationally derived quality control metrics have been introduced. However, despite the availability of these metrics, a systematic approach to quality control is often still lacking because the metrics are not fully understood and are hard to interpret. Here, we present a toolkit of powerful techniques to analyze and interpret multivariate quality control metrics to assess the quality of mass spectrometry proteomics experiments. We show how unsupervised techniques applied to these quality control metrics can provide an initial discrimination between low-quality experiments and high-quality experiments prior to manual investigation. Furthermore, we provide a technique to obtain detailed information on the quality control metrics that are related to the decreased performance, which can be used as actionable information to improve the experimental setup. Our toolkit is released as open-source and can be downloaded from https://bitbucket.org/proteinspector/qc_analysis/ .