Using computer vision on herbarium specimen images to discriminate among closely related horsetails (Equisetum)

Decoding family-level features for modern and fossil leaves from computer-vision heat maps

PREMISE

Angiosperm leaves present a classic identification problem due to their morphological complexity. Computer-vision algorithms can identify diagnostic regions in images, and heat map outputs illustrate those regions for identification, providing novel insights through visual feedback. We investigate the potential of analyzing leaf heat maps to reveal novel, human-friendly botanical information with applications for extant- and fossil-leaf identification.

METHODS

We developed a manual scoring system for hotspot locations on published computer-vision heat maps of cleared leaves that showed diagnostic regions for family identification. Heat maps of 3114 cleared leaves of 930 genera in 14 angiosperm families were analyzed. The top-5 and top-1 hotspot regions of highest diagnostic value were scored for 21 leaf locations. The resulting data were viewed using box plots and analyzed using cluster and principal component analyses. We manually identified similar features in fossil leaves to informally demonstrate potential fossil applications.

RESULTS

The method successfully mapped machine strategy using standard botanical language, and distinctive patterns emerged for each family. Hotspots were concentrated on secondary veins (Salicaceae, Myrtaceae, Anacardiaceae), tooth apices (Betulaceae, Rosaceae), and on the little-studied margins of untoothed leaves (Rubiaceae, Annonaceae, Ericaceae). Similar features drove the results from multivariate analyses. The results echo many traditional observations, while also showing that most diagnostic leaf features remain undescribed.

CONCLUSIONS

Machine-derived heat maps that initially appear to be dominated by noise can be translated into human-interpretable knowledge, highlighting paths forward for botanists and paleobotanists to discover new diagnostic botanical characters.

The Specimen Data Refinery: A Canonical Workflow Framework and FAIR Digital Object Approach to Speeding up Digital Mobilisation of Natural History Collections

A key limiting factor in organising and using information from physical specimens curated in natural science collections is making that information computable, with institutional digitization tending to focus more on imaging the specimens themselves than on efficiently capturing computable data about them. Label data are traditionally manually transcribed today with high cost and low throughput, rendering such a task constrained for many collection-holding institutions at current funding levels. We show how computer vision, optical character recognition, handwriting recognition, named entity recognition and language translation technologies can be implemented into canonical workflow component libraries with findable, accessible, interoperable, and reusable (FAIR) characteristics. These libraries are being developed in a cloud- based workflow platform—the ‘Specimen Data Refinery’ (SDR)—founded on Galaxy workflow engine, Common Workflow Language, Research Object Crates (RO-Crate) and WorkflowHub technologies. The SDR can be applied to specimens’ labels and other artefacts, offering the prospect of greatly accelerated and more accurate data capture in computable form. Two kinds of FAIR Digital Objects (FDO) are created by packaging outputs of SDR workflows and workflow components as digital objects with metadata, a persistent identifier, and a specific type definition. The first kind of FDO are computable Digital Specimen (DS) objects that can be consumed/produced by workflows, and other applications. A single DS is the input data structure submitted to a workflow that is modified by each workflow component in turn to produce a refined DS at the end. The Specimen Data Refinery provides a library of such components that can be used individually, or in series. To cofunction, each library component describes the fields it requires from the DS and the fields it will in turn populate or enrich. The second kind of FDO, RO-Crates gather and archive the diverse set of digital and real-world resources, configurations, and actions (the provenance) contributing to a unit of research work, allowing that work to be faithfully recorded and reproduced. Here we describe the Specimen Data Refinery with its motivating requirements, focusing on what is essential in the creation of canonical workflow component libraries and its conformance with the requirements of an emerging FDO Core Specification being developed by the FDO Forum.

The Herbarium 2021 Half-Earth Challenge Dataset and Machine Learning Competition

Herbarium sheets present a unique view of the world's botanical history, evolution, and biodiversity. This makes them an all-important data source for botanical research. With the increased digitization of herbaria worldwide and advances in the domain of fine-grained visual classification which can facilitate automatic identification of herbarium specimen images, there are many opportunities for supporting and expanding research in this field. However, existing datasets are either too small, or not diverse enough, in terms of represented taxa, geographic distribution, and imaging protocols. Furthermore, aggregating datasets is difficult as taxa are recognized under a multitude of names and must be aligned to a common reference. We introduce the Herbarium 2021 Half-Earth dataset: the largest and most diverse dataset of herbarium specimen images, to date, for automatic taxon recognition. We also present the results of the Herbarium 2021 Half-Earth challenge, a competition that was part of the Eighth Workshop on Fine-Grained Visual Categorization (FGVC8) and hosted by Kaggle to encourage the development of models to automatically identify taxa from herbarium sheet images.

Reversing extinction trends: new uses of (old) herbarium specimens to accelerate conservation action on threatened species

Although often not collected specifically for the purposes of conservation, herbarium specimens offer sufficient information to reconstruct parameters that are needed to designate a species as 'at-risk' of extinction. While such designations should prompt quick and efficient legal action towards species recovery, such action often lags far behind and is mired in bureaucratic procedure. The increase in online digitization of natural history collections has now led to a surge in the number new studies on the uses of machine learning. These repositories of species occurrences are now equipped with advances that allow for the identification of rare species. The increase in attention devoted to estimating the scope and severity of the threats that lead to the decline of such species will increase our ability to mitigate these threats and reverse the declines, overcoming a current barrier to the recovery of many threatened plant species. Thus far, collected specimens have been used to fill gaps in systematics, range extent, and past genetic diversity. We find that they also offer material with which it is possible to foster species recovery, ecosystem restoration, and de-extinction, and these elements should be used in conjunction with machine learning and citizen science initiatives to mobilize as large a force as possible to counter current extinction trends.

An algorithm competition for automatic species identification from herbarium specimens

PREMISE

Plant biodiversity is threatened, yet many species remain undescribed. It is estimated that >50% of undescribed species have already been collected and are awaiting discovery in herbaria. Robust automatic species identification algorithms using machine learning could accelerate species discovery.

METHODS

To encourage the development of an automatic species identification algorithm, we submitted our Herbarium 2019 data set to the Fine-Grained Visual Categorization sub-competition (FGVC6) hosted on the Kaggle platform. We chose to focus on the flowering plant family Melastomataceae because we have a large collection of imaged herbarium specimens (46,469 specimens representing 683 species) and taxonomic expertise in the family. As is common for herbarium collections, some species in this data set are represented by few specimens and others by many.

RESULTS

In less than three months, the FGVC6 Herbarium 2019 Challenge drew 22 teams who entered 254 models for Melastomataceae species identification. The four best algorithms identified species with >88% accuracy.

DISCUSSION

The FGVC competitions provide a unique opportunity for computer vision and machine learning experts to address difficult species-recognition problems. The Herbarium 2019 Challenge brought together a novel combination of collections resources, taxonomic expertise, and collaboration between botanists and computer scientists.

A new fine-grained method for automated visual analysis of herbarium specimens: A case study for phenological data extraction

PREMISE

Herbarium specimens represent an outstanding source of material with which to study plant phenological changes in response to climate change. The fine-scale phenological annotation of such specimens is nevertheless highly time consuming and requires substantial human investment and expertise, which are difficult to rapidly mobilize.

METHODS

We trained and evaluated new deep learning models to automate the detection, segmentation, and classification of four reproductive structures of Streptanthus tortuosus (flower buds, flowers, immature fruits, and mature fruits). We used a training data set of 21 digitized herbarium sheets for which the position and outlines of 1036 reproductive structures were annotated manually. We adjusted the hyperparameters of a mask R-CNN (regional convolutional neural network) to this specific task and evaluated the resulting trained models for their ability to count reproductive structures and estimate their size.

RESULTS

The main outcome of our study is that the performance of detection and segmentation can vary significantly with: (i) the type of annotations used for training, (ii) the type of reproductive structures, and (iii) the size of the reproductive structures. In the case of Streptanthus tortuosus, the method can provide quite accurate estimates (77.9% of cases) of the number of reproductive structures, which is better estimated for flowers than for immature fruits and buds. The size estimation results are also encouraging, showing a difference of only a few millimeters between the predicted and actual sizes of buds and flowers.

DISCUSSION

This method has great potential for automating the analysis of reproductive structures in high-resolution images of herbarium sheets. Deeper investigations regarding the taxonomic scalability of this approach and its potential improvement will be conducted in future work.