Data formats and standards for opportunistic rainfall sensors

Opportunistic sensors are increasingly used for rainfall measurement. However, their raw data are collected by a variety of systems that are often not primarily intended for rainfall monitoring, resulting in a plethora of different data formats and a lack of common standards. This hinders the sharing of opportunistic sensing (OS) data, their automated processing, and, at the end, their practical usage and integration into standard observation systems. This paper summarises the experiences of the more than 100 members of the OpenSense Cost Action involved in the OS of rainfall. We review the current practice of collecting and storing precipitation OS data and corresponding metadata, and propose new common guidelines describing the requirements on data and metadata collection, harmonising naming conventions, and defining human-readable and machine readable file formats for data and metadata storage. We focus on three sensors identified by the OpenSense community as prominent representatives of the OS of precipitation: Commercial microwave links (CML): fixed point-to-point radio links mainly used as backhauling connections in telecommunication networks Satellite microwave links (SML): radio links between geostationary Earth orbit (GEO) satellites and ground user terminals. Personal weather stations (PWS): non-professional meteorological sensors owned by citizens. The conventions presented in this paper are primarily designed for storing, handling, and sharing historical time series and do not consider specific requirements for using OS data in real time for operational purposes. The conventions are already now accepted by the ever growing OpenSense community and represent an important step towards automated processing of OS raw data and community development of joint OS software packages.

A standard data format for 3DED/MicroED

Electron diffraction from three dimensional crystals, as a technique for solving molecular structures, is rapidly increasing in popularity. The development of methodology and software has borrowed, to great effect, from macromolecular X-ray crystallography. However, standardization lags behind the development of the technique, and practitioners are forced to work with inadequate data formats that are unable to capture a full description of their experiments. This creates obstacles that are increasingly difficult to overcome as experiments become ever faster and the need for data autoprocessing becomes more pressing. We present a data format standard based on best practice from macromolecular crystallography and demonstrate how the adoption of this standard enabled autoprocessing of datasets collected with a high-throughput detector system.

Introduction to the shared near infrared spectroscopy format

Significance

Functional near-infrared spectroscopy (fNIRS) is a popular neuroimaging technique with proliferating hardware platforms, analysis approaches, and software tools. There has not been a standardized file format for storing fNIRS data, which has hindered the sharing of data as well as the adoption and development of software tools.

Aim

We endeavored to design a file format to facilitate the analysis and sharing of fNIRS data that is flexible enough to meet the community's needs and sufficiently defined to be implemented consistently across various hardware and software platforms.

Approach

The shared NIRS format (SNIRF) specification was developed in consultation with the academic and commercial fNIRS community and the Society for functional Near Infrared Spectroscopy.

Results

The SNIRF specification defines a format for fNIRS data acquired using continuous wave, frequency domain, time domain, and diffuse correlation spectroscopy devices.

Conclusions

We present the SNIRF along with validation software and example datasets. Support for reading and writing SNIRF data has been implemented by major hardware and software platforms, and the format has found widespread use in the fNIRS community.

Closing the gap between formats for storing layout information in systems biology

The understanding of complex biological networks often relies on both a dedicated layout and a topology. Currently, there are three major competing layout-aware systems biology formats, but there are no software tools or software libraries supporting all of them. This complicates the management of molecular network layouts and hinders their reuse and extension. In this paper, we present a high-level overview of the layout formats in systems biology, focusing on their commonalities and differences, review their support in existing software tools, libraries and repositories and finally introduce a new conversion module within the MINERVA platform. The module is available via a REST API and offers, besides the ability to convert between layout-aware systems biology formats, the possibility to export layouts into several graphical formats. The module enables conversion of very large networks with thousands of elements, such as disease maps or metabolic reconstructions, rendering it widely applicable in systems biology.

NMReDATA, a standard to report the NMR assignment and parameters of organic compounds

Even though NMR has found countless applications in the field of small molecule characterization, there is no standard file format available for the NMR data relevant to structure characterization of small molecules. A new format is therefore introduced to associate the NMR parameters extracted from 1D and 2D spectra of organic compounds to the proposed chemical structure. These NMR parameters, which we shall call NMReDATA (for nuclear magnetic resonance extracted data), include chemical shift values, signal integrals, intensities, multiplicities, scalar coupling constants, lists of 2D correlations, relaxation times, and diffusion rates. The file format is an extension of the existing Structure Data Format, which is compatible with the commonly used MOL format. The association of an NMReDATA file with the raw and spectral data from which it originates constitutes an NMR record. This format is easily readable by humans and computers and provides a simple and efficient way for disseminating results of structural chemistry investigations, allowing automatic verification of published results, and for assisting the constitution of highly needed open-source structural databases.

ReadTools: A universal toolkit for handling sequence data from different sequencing platforms

Sequencing whole genomes has become a standard research tool in many disciplines including Molecular Ecology, but the rapid technological advances in combination with several competing platforms have resulted in a confusing diversity of formats. This lack of standard formats causes several problems, such as undocumented preprocessing steps or the loss of information in downstream software tools, which do not account for the specifics of the different available formats. ReadTools is an open-source Java toolkit designed to standardize and preprocess read data from different platforms. It manages FASTQ- and SAM-formatted inputs while dealing with platform-specific peculiarities and provides a standard SAM compliant output. The code and executable are available at https://github.com/magicDGS/ReadTools.

MINC 2.0: A Flexible Format for Multi-Modal Images

It is often useful that an imaging data format can afford rich metadata, be flexible, scale to very large file sizes, support multi-modal data, and have strong inbuilt mechanisms for data provenance. Beginning in 1992, MINC was developed as a system for flexible, self-documenting representation of neuroscientific imaging data with arbitrary orientation and dimensionality. The MINC system incorporates three broad components: a file format specification, a programming library, and a growing set of tools. In the early 2000's the MINC developers created MINC 2.0, which added support for 64-bit file sizes, internal compression, and a number of other modern features. Because of its extensible design, it has been easy to incorporate details of provenance in the header metadata, including an explicit processing history, unique identifiers, and vendor-specific scanner settings. This makes MINC ideal for use in large scale imaging studies and databases. It also makes it easy to adapt to new scanning sequences and modalities.

COMBINE archive and OMEX format: one file to share all information to reproduce a modeling project

BACKGROUND

With the ever increasing use of computational models in the biosciences, the need to share models and reproduce the results of published studies efficiently and easily is becoming more important. To this end, various standards have been proposed that can be used to describe models, simulations, data or other essential information in a consistent fashion. These constitute various separate components required to reproduce a given published scientific result.

RESULTS

We describe the Open Modeling EXchange format (OMEX). Together with the use of other standard formats from the Computational Modeling in Biology Network (COMBINE), OMEX is the basis of the COMBINE Archive, a single file that supports the exchange of all the information necessary for a modeling and simulation experiment in biology. An OMEX file is a ZIP container that includes a manifest file, listing the content of the archive, an optional metadata file adding information about the archive and its content, and the files describing the model. The content of a COMBINE Archive consists of files encoded in COMBINE standards whenever possible, but may include additional files defined by an Internet Media Type. Several tools that support the COMBINE Archive are available, either as independent libraries or embedded in modeling software.

CONCLUSIONS

The COMBINE Archive facilitates the reproduction of modeling and simulation experiments in biology by embedding all the relevant information in one file. Having all the information stored and exchanged at once also helps in building activity logs and audit trails. We anticipate that the COMBINE Archive will become a significant help for modellers, as the domain moves to larger, more complex experiments such as multi-scale models of organs, digital organisms, and bioengineering.

Fingerprints of a message: integrating positional information on the transcriptome

The recent explosion of high-throughput sequencing methods applied to RNA molecules is allowing us to go beyond the description of sequence variants and their relative abundances, as measured by RNA-seq. We can now probe for RNA engagement in polysomes, for ribosomes, RNA binding proteins and microRNAs binding sites, for RNA secondary structure and for RNA methylation. These descriptors produce a steadily growing multidimensional array of positional information on RNA sequences, whose effective integration only would bring to decipher the regulatory interplay occurring between proteins, RNAs and their modifications on the transcriptome. This interplay ultimately dictates the degree of mRNA availability to translation, and thus the occurrence of cell phenotypes. However, several issues in data presentation are slowing down effective integration. A standardization effort for new dataset types produced should be urgently undertaken to solve these issues. Providing uniformed experimental details along with datasets processed to be directly usable and employing shared formats would greatly simplify integration efforts, strengthening hypotheses stemming from correlative observations and eventually bringing to mechanistic understanding.

Data standards for Omics data: the basis of data sharing and reuse

To facilitate sharing of Omics data, many groups of scientists have been working to establish the relevant data standards. The main components of data sharing standards are experiment description standards, data exchange standards, terminology standards, and experiment execution standards. Here we provide a survey of existing and emerging standards that are intended to assist the free and open exchange of large-format data.