A New Method for Counting Reproductive Structures in Digitized Herbarium Specimens Using Mask R-CNN

The promise of digital herbarium specimens in large-scale phenology research

The online mobilization of herbaria has made tens of millions of specimens digitally available, revolutionizing investigations of phenology and plant responses to climate change. We identify two main themes associated with this growing body of research and highlight a selection of recent publications exemplifying: investigating phenology at large spatial and temporal scales and in understudied locations and testing long-standing theories and novel questions in ecology and evolution that were not previously answerable. We explore strengths and limitations of using digitized herbarium specimens in phenology research, including: issues of sampling; reliability, transferability, and biases; and ethical and social justice considerations. This field will see further breakthroughs as herbaria around the world continue to mobilize and digitally interlink their collections. New developments will likely come from advances in technology, international collaborations, and including understudied plant taxa and regions such as the Arctic and the tropics. Advances in technology are already improving digitization workflows and speeding the collection of phenology data from digital specimens.

Long-term trends in global flowering phenology

Flowering phenology is an indicator of the impact of climate change on natural systems. Anthropogenic climate change has progressed over more than two centuries, but ecological studies are mostly short in comparison. Here we harness the large-scale digitization of herbaria specimens to investigate temporal trends in flowering phenology at a global scale. We trained a convolutional neural network model to classify images of angiosperm herbarium specimens as being in flower or not in flower. This model was used to infer flowering across 8 million specimens spanning a century and global scales. We investigated temporal trends in mean flowering date and flowering season duration within ecoregions. We found high diversity of temporal trends in flowering seasonality across ecoregions with a median absolute shift of 2.5 d per decade in flowering date and 1.4 d per decade in flowering season duration. Variability in temporal trends in phenology was higher at low latitudes than at high latitudes. Our study demonstrates the value of digitized herbarium specimens for understanding natural dynamics in a time of change. The higher variability in phenological trends at low latitudes likely reflects the effects of a combination of shifts in temperature and precipitation seasonality, together with lower photoperiodic constraints to flowering.

mvh: An R tool to assemble and organize virtual herbaria from openly available specimen images

Premise

Recent advances in imaging herbarium specimens have enhanced their use in biodiversity studies. However, user-friendly tools that facilitate the assembly of customized sets of herbarium specimen images on personal devices are still lacking.

Methods and Results

Here we present the R package mvh ("my virtual herbarium"), which includes functions designed to search and download metadata and openly available images associated with herbarium specimens based on taxon or geography. We tested the functionalities of mvh by searching metadata associated with five sets of 10 vascular plant species and five sets of 10 terrestrial coordinates. The download function had a success rate of 99%, downloading 291 out of the 293 images found in the search. Possible reasons for download failure are discussed.

Conclusions

As long as an internet connection is available, mvh simplifies the assembly and organization of virtual herbaria, thereby facilitating the investigation of novel empirical questions as well as trends in digitization efforts.

Medicinal plants meet modern biodiversity science

Plants have been an essential source of human medicine for millennia. In this review, we argue that a holistic, interdisciplinary approach to the study of medicinal plants that combines methods and insights from three key disciplines - evolutionary ecology, molecular biology/biochemistry, and ethnopharmacology - is poised to facilitate new breakthroughs in science, including pharmacological discoveries and rapid advancements in human health and well-being. Such interdisciplinary research leverages data and methods spanning space, time, and species associated with medicinal plant species evolution, ecology, genomics, and metabolomic trait diversity, all of which build heavily on traditional Indigenous knowledge. Such an interdisciplinary approach contrasts sharply with most well-funded and successful medicinal plant research during the last half-century, which, despite notable advancements, has greatly oversimplified the dynamic relationships between plants and humans, kept hidden the larger human narratives about these relationships, and overlooked potentially important research and discoveries into life-saving medicines. We suggest that medicinal plants and people should be viewed as partners whose relationship involves a complicated and poorly explored set of (socio-)ecological interactions including not only domestication but also commensalisms and mutualisms. In short, medicinal plant species are not just chemical factories for extraction and exploitation. Rather, they may be symbiotic partners that have shaped modern societies, improved human health, and extended human lifespans.

Sex-dependent phenological responses to climate vary across species' ranges

Anthropogenic climate change has significantly altered the flowering times (i.e., phenology) of plants worldwide, affecting their reproduction, survival, and interactions. Recent studies utilizing herbarium specimens have uncovered significant intra- and inter-specific variation in flowering phenology and its response to changes in climate but have mostly been limited to animal-pollinated species. Thus, despite their economic and ecological importance, variation in phenological responses to climate remain largely unexplored among and within wind-pollinated dioecious species and across their sexes. Using both herbarium specimens and volunteer observations of cottonwood (Populus) species, we examined how phenological sensitivity to climate varies across species, their ranges, sexes, and phenophases. The timing of flowering varied significantly across and within species, as did their sensitivity to spring temperature. In particular, male flowering generally happened earlier in the season and was more sensitive to warming than female flowering. Further, the onset of flowering was more sensitive to changes in temperature than leaf out. Increased temporal gaps between male and female flowering time and between the first open flower date and leaf out date were predicted for the future under two climate change scenarios. These shifts will impact the efficacy of sexual reproduction and gene flow among species. Our study demonstrates significant inter- and intra-specific variation in phenology and its responses to environmental cues, across species' ranges, phenophases, and sex, in wind-pollinated species. These variations need to be considered to predict accurately the effects of climate change and assess their ecological and evolutionary consequences.

From leaves to labels: Building modular machine learning networks for rapid herbarium specimen analysis with LeafMachine2

Premise

Quantitative plant traits play a crucial role in biological research. However, traditional methods for measuring plant morphology are time consuming and have limited scalability. We present LeafMachine2, a suite of modular machine learning and computer vision tools that can automatically extract a base set of leaf traits from digital plant data sets.

Methods

LeafMachine2 was trained on 494,766 manually prepared annotations from 5648 herbarium images obtained from 288 institutions and representing 2663 species; it employs a set of plant component detection and segmentation algorithms to isolate individual leaves, petioles, fruits, flowers, wood samples, buds, and roots. Our landmarking network automatically identifies and measures nine pseudo-landmarks that occur on most broadleaf taxa. Text labels and barcodes are automatically identified by an archival component detector and are prepared for optical character recognition methods or natural language processing algorithms.

Results

LeafMachine2 can extract trait data from at least 245 angiosperm families and calculate pixel-to-metric conversion factors for 26 commonly used ruler types.

Discussion

LeafMachine2 is a highly efficient tool for generating large quantities of plant trait data, even from occluded or overlapping leaves, field images, and non-archival data sets. Our project, along with similar initiatives, has made significant progress in removing the bottleneck in plant trait data acquisition from herbarium specimens and shifted the focus toward the crucial task of data revision and quality control.

The herbarium of the future

The ~400 million specimens deposited across ~3000 herbaria are essential for: (i) understanding where plants have lived in the past, (ii) forecasting where they may live in the future, and (iii) delineating their conservation status. An open access 'global metaherbarium' is emerging as these specimens are digitized, mobilized, and interlinked online. This virtual biodiversity resource is attracting new users who are accelerating traditional applications of herbaria and generating basic and applied scientific innovations, including e-monographs and floras produced by diverse, interdisciplinary, and inclusive teams; robust machine-learning algorithms for species identification and phenotyping; collection and synthesis of ecological trait data at large spatiotemporal and phylogenetic scales; and exhibitions and installations that convey the beauty of plants and the value of herbaria in addressing broader societal issues.

The ecological implications of intra- and inter-species variation in phenological sensitivity

Plant-pollinator mutualisms rely upon the synchrony of interacting taxa. Climate change can disrupt this synchrony as phenological responses to climate vary within and across species. However, intra- and interspecific variation in phenological responses is seldom considered simultaneously, limiting our understanding of climate change impacts on interactions among taxa across their ranges. We investigated how variation in phenological sensitivity to climate can alter ecological interactions simultaneously within and among species using natural history collections and citizen science data. We focus on a unique system, comprising a wide-ranged spring ephemeral with varying color morphs (Claytonia virginica) and its specialist bee pollinator (Andrena erigeniae). We found strongly opposing trends in the phenological sensitivities of plants vs their pollinators. Flowering phenology was more sensitive to temperature in warmer regions, whereas bee phenology was more responsive in colder regions. Phenological sensitivity varied across flower color morphs. Temporal synchrony between flowering and pollinator activity was predicted to change heterogeneously across the species' ranges in the future. Our work demonstrates the complexity and fragility of ecological interactions in time and the necessity of incorporating variation in phenological responses across multiple axes to understand how such interactions will change in the future.

New directions in tropical phenology

Earth's most speciose biomes are in the tropics, yet tropical plant phenology remains poorly understood. Tropical phenological data are comparatively scarce and viewed through the lens of a 'temperate phenological paradigm' expecting phenological traits to respond to strong, predictably annual shifts in climate (e.g., between subfreezing and frost-free periods). Digitized herbarium data greatly expand existing phenological data for tropical plants; and circular data, statistics, and models are more appropriate for analyzing tropical (and temperate) phenological datasets. Phylogenetic information, which remains seldom applied in phenological investigations, provides new insights into phenological responses of large groups of related species to climate. Consistent combined use of herbarium data, circular statistical distributions, and robust phylogenies will rapidly advance our understanding of tropical - and temperate - phenology.

Earbox, an open tool for high-throughput measurement of the spatial organization of maize ears and inference of novel traits

BACKGROUND

Characterizing plant genetic resources and their response to the environment through accurate measurement of relevant traits is crucial to genetics and breeding. Spatial organization of the maize ear provides insights into the response of grain yield to environmental conditions. Current automated methods for phenotyping the maize ear do not capture these spatial features.

RESULTS

We developed EARBOX, a low-cost, open-source system for automated phenotyping of maize ears. EARBOX integrates open-source technologies for both software and hardware that facilitate its deployment and improvement for specific research questions. The imaging platform consists of a customized box in which ears are repeatedly imaged as they rotate via motorized rollers. With deep learning based on convolutional neural networks, the image analysis algorithm uses a two-step procedure: ear-specific grain masks are first created and subsequently used to extract a range of trait data per ear, including ear shape and dimensions, the number of grains and their spatial organisation, and the distribution of grain dimensions along the ear. The reliability of each trait was validated against ground-truth data from manual measurements. Moreover, EARBOX derives novel traits, inaccessible through conventional methods, especially the distribution of grain dimensions along grain cohorts, relevant for ear morphogenesis, and the distribution of abortion frequency along the ear, relevant for plant response to stress, especially soil water deficit.

CONCLUSIONS

The proposed system provides robust and accurate measurements of maize ear traits including spatial features. Future developments include grain type and colour categorisation. This method opens avenues for high-throughput genetic or functional studies in the context of plant adaptation to a changing environment.

Batch-Mask: Automated image segmentation for organisms with limbless or non-standard body forms

Efficient comparisons of biological color patterns are critical for understanding the mechanisms by which organisms evolve in nature, including sexual selection, predator–prey interactions, and thermoregulation. However, limbless, elongate, or spiral-shaped organisms do not conform to the standard orientation and photographic techniques required for many automated analyses. Currently, large-scale color analysis of elongate animals requires time-consuming manual landmarking, which reduces their representation in coloration research despite their ecological importance. We present Batch-Mask: an automated, customizable workflow to automatically analyze large photographic datasets to isolate non-standard biological organisms from the background. Batch-Mask is completely open-source and does not depend on any proprietary software. We also present a user guide for fine-tuning weights to a custom dataset and incorporating existing manual visual analysis tools (e.g., micaToolbox) into a single automated workflow for comparing color patterns across images. Batch-Mask was 60x faster than manual landmarking and produced masks that correctly identified 96% of all snake pixels. To validate our approach, we used micaToolbox to compare pattern energy in a sample set of snake photographs segmented by Batch-Mask and humans and found no significant difference in the output results. The fine-tuned weights, user guide, and automated workflow substantially decrease the amount of time and attention required to quantitatively analyze non-standard biological subjects. With these tools, biologists can compare color, pattern, and shape differences in large datasets that include significant morphological variation in elongate body forms. This advance is especially valuable for comparative analyses of natural history collections across a broad range of morphologies. Through landmark-free automation, Batch-Mask can greatly expand the scale of space, time, or taxonomic breadth across which color variation can be quantitatively examined.

Estimating Alpha, Beta, and Gamma Diversity Through Deep Learning

The reliable mapping of species richness is a crucial step for the identification of areas of high conservation priority, alongside other value and threat considerations. This is commonly done by overlapping range maps of individual species, which requires dense availability of occurrence data or relies on assumptions about the presence of species in unsampled areas deemed suitable by environmental niche models. Here, we present a deep learning approach that directly estimates species richness, skipping the step of estimating individual species ranges. We train a neural network model based on species lists from inventory plots, which provide ground truth data for supervised machine learning. The model learns to predict species richness based on spatially associated variables, including climatic and geographic predictors, as well as counts of available species records from online databases. We assess the empirical utility of our approach by producing independently verifiable maps of alpha, beta, and gamma plant diversity at high spatial resolutions for Australia, a continent with highly heterogeneous diversity patterns. Our deep learning framework provides a powerful and flexible new approach for estimating biodiversity patterns, constituting a step forward toward automated biodiversity assessments.

Deep Learning in Plant Phenological Research: A Systematic Literature Review

Climate change represents one of the most critical threats to biodiversity with far-reaching consequences for species interactions, the functioning of ecosystems, or the assembly of biotic communities. Plant phenology research has gained increasing attention as the timing of periodic events in plants is strongly affected by seasonal and interannual climate variation. Recent technological development allowed us to gather invaluable data at a variety of spatial and ecological scales. The feasibility of phenological monitoring today and in the future depends heavily on developing tools capable of efficiently analyzing these enormous amounts of data. Deep Neural Networks learn representations from data with impressive accuracy and lead to significant breakthroughs in, e.g., image processing. This article is the first systematic literature review aiming to thoroughly analyze all primary studies on deep learning approaches in plant phenology research. In a multi-stage process, we selected 24 peer-reviewed studies published in the last five years (2016-2021). After carefully analyzing these studies, we describe the applied methods categorized according to the studied phenological stages, vegetation type, spatial scale, data acquisition- and deep learning methods. Furthermore, we identify and discuss research trends and highlight promising future directions. We present a systematic overview of previously applied methods on different tasks that can guide this emerging complex research field.

Can Artificial Intelligence Help in the Study of Vegetative Growth Patterns from Herbarium Collections? An Evaluation of the Tropical Flora of the French Guiana Forest

A better knowledge of tree vegetative growth phenology and its relationship to environmental variables is crucial to understanding forest growth dynamics and how climate change may affect it. Less studied than reproductive structures, vegetative growth phenology focuses primarily on the analysis of growing shoots, from buds to leaf fall. In temperate regions, low winter temperatures impose a cessation of vegetative growth shoots and lead to a well-known annual growth cycle pattern for most species. The humid tropics, on the other hand, have less seasonality and contain many more tree species, leading to a diversity of patterns that is still poorly known and understood. The work in this study aims to advance knowledge in this area, focusing specifically on herbarium scans, as herbariums offer the promise of tracking phenology over long periods of time. However, such a study requires a large number of shoots to be able to draw statistically relevant conclusions. We propose to investigate the extent to which the use of deep learning can help detect and type-classify these relatively rare vegetative structures in herbarium collections. Our results demonstrate the relevance of using herbarium data in vegetative phenology research as well as the potential of deep learning approaches for growing shoot detection.

The Herbarium 2021 Half-Earth Challenge Dataset and Machine Learning Competition

Herbarium sheets present a unique view of the world's botanical history, evolution, and biodiversity. This makes them an all-important data source for botanical research. With the increased digitization of herbaria worldwide and advances in the domain of fine-grained visual classification which can facilitate automatic identification of herbarium specimen images, there are many opportunities for supporting and expanding research in this field. However, existing datasets are either too small, or not diverse enough, in terms of represented taxa, geographic distribution, and imaging protocols. Furthermore, aggregating datasets is difficult as taxa are recognized under a multitude of names and must be aligned to a common reference. We introduce the Herbarium 2021 Half-Earth dataset: the largest and most diverse dataset of herbarium specimen images, to date, for automatic taxon recognition. We also present the results of the Herbarium 2021 Half-Earth challenge, a competition that was part of the Eighth Workshop on Fine-Grained Visual Categorization (FGVC8) and hosted by Kaggle to encourage the development of models to automatically identify taxa from herbarium sheet images.

Using Convolutional Neural Networks to Efficiently Extract Immense Phenological Data From Community Science Images

Community science image libraries offer a massive, but largely untapped, source of observational data for phenological research. The iNaturalist platform offers a particularly rich archive, containing more than 49 million verifiable, georeferenced, open access images, encompassing seven continents and over 278,000 species. A critical limitation preventing scientists from taking full advantage of this rich data source is labor. Each image must be manually inspected and categorized by phenophase, which is both time-intensive and costly. Consequently, researchers may only be able to use a subset of the total number of images available in the database. While iNaturalist has the potential to yield enough data for high-resolution and spatially extensive studies, it requires more efficient tools for phenological data extraction. A promising solution is automation of the image annotation process using deep learning. Recent innovations in deep learning have made these open-source tools accessible to a general research audience. However, it is unknown whether deep learning tools can accurately and efficiently annotate phenophases in community science images. Here, we train a convolutional neural network (CNN) to annotate images of Alliaria petiolata into distinct phenophases from iNaturalist and compare the performance of the model with non-expert human annotators. We demonstrate that researchers can successfully employ deep learning techniques to extract phenological information from community science images. A CNN classified two-stage phenology (flowering and non-flowering) with 95.9% accuracy and classified four-stage phenology (vegetative, budding, flowering, and fruiting) with 86.4% accuracy. The overall accuracy of the CNN did not differ from humans (p = 0.383), although performance varied across phenophases. We found that a primary challenge of using deep learning for image annotation was not related to the model itself, but instead in the quality of the community science images. Up to 4% of A. petiolata images in iNaturalist were taken from an improper distance, were physically manipulated, or were digitally altered, which limited both human and machine annotators in accurately classifying phenology. Thus, we provide a list of photography guidelines that could be included in community science platforms to inform community scientists in the best practices for creating images that facilitate phenological analysis.

Machine Learning Undercounts Reproductive Organs on Herbarium Specimens but Accurately Derives Their Quantitative Phenological Status: A Case Study of Streptanthus tortuosus

Machine learning (ML) can accelerate the extraction of phenological data from herbarium specimens; however, no studies have assessed whether ML-derived phenological data can be used reliably to evaluate ecological patterns. In this study, 709 herbarium specimens representing a widespread annual herb, Streptanthus tortuosus, were scored both manually by human observers and by a mask R-CNN object detection model to (1) evaluate the concordance between ML and manually-derived phenological data and (2) determine whether ML-derived data can be used to reliably assess phenological patterns. The ML model generally underestimated the number of reproductive structures present on each specimen; however, when these counts were used to provide a quantitative estimate of the phenological stage of plants on a given sheet (i.e., the phenological index or PI), the ML and manually-derived PI's were highly concordant. Moreover, herbarium specimen age had no effect on the estimated PI of a given sheet. Finally, including ML-derived PIs as predictor variables in phenological models produced estimates of the phenological sensitivity of this species to climate, temporal shifts in flowering time, and the rate of phenological progression that are indistinguishable from those produced by models based on data provided by human observers. This study demonstrates that phenological data extracted using machine learning can be used reliably to estimate the phenological stage of herbarium specimens and to detect phenological patterns.

Scale gaps in landscape phenology: challenges and opportunities

Phenology, or the timing of life history events, can be heterogeneous across biological communities and landscapes and can vary across a wide variety of spatiotemporal scales. Here, we synthesize information from landscape phenology studies across different scales of measurement around a set of core concepts. We highlight why phenology is scale dependent and identify gaps in the spatiotemporal scales of phenological observations and inferences. We discuss the consequences of these gaps and describe opportunities to address the inherent sensitivities of phenological metrics to measurement scale. Although most studies we review and discuss are focused on plants, our work provides a broadly relevant overview of the role of observation scale in landscape phenology and a general approach for measuring and reporting scale dependence.

Reversing extinction trends: new uses of (old) herbarium specimens to accelerate conservation action on threatened species

Although often not collected specifically for the purposes of conservation, herbarium specimens offer sufficient information to reconstruct parameters that are needed to designate a species as 'at-risk' of extinction. While such designations should prompt quick and efficient legal action towards species recovery, such action often lags far behind and is mired in bureaucratic procedure. The increase in online digitization of natural history collections has now led to a surge in the number new studies on the uses of machine learning. These repositories of species occurrences are now equipped with advances that allow for the identification of rare species. The increase in attention devoted to estimating the scope and severity of the threats that lead to the decline of such species will increase our ability to mitigate these threats and reverse the declines, overcoming a current barrier to the recovery of many threatened plant species. Thus far, collected specimens have been used to fill gaps in systematics, range extent, and past genetic diversity. We find that they also offer material with which it is possible to foster species recovery, ecosystem restoration, and de-extinction, and these elements should be used in conjunction with machine learning and citizen science initiatives to mobilize as large a force as possible to counter current extinction trends.

UAV-Based Heating Requirement Determination for Frost Management in Apple Orchard

Frost is a natural disaster that can cause catastrophic damages in agriculture, while traditional temperature monitoring in orchards has disadvantages such as being imprecise and laborious, which can lead to inadequate or wasteful frost protection treatments. In this article, we presented a heating requirement assessment methodology for frost protection in an apple orchard utilizing unmanned aerial vehicle (UAV)-based thermal and RGB cameras. A thermal image stitching algorithm using the BRISK feature was developed for creating georeferenced orchard temperature maps, which attained a sub-centimeter map resolution and a stitching speed of 100 thermal images within 30 s. YOLOv4 classifiers for six apple flower bud growth stages in various network sizes were trained based on 5040 RGB images, and the best model achieved a 71.57% mAP for a test dataset consisted of 360 images. A flower bud mapping algorithm was developed to map classifier detection results into dense growth stage maps utilizing RGB image geoinformation. Heating requirement maps were created using artificial flower bud critical temperatures to simulate orchard heating demands during frost events. The results demonstrated the feasibility of the proposed orchard heating requirement determination methodology, which has the potential to be a critical component of an autonomous, precise frost management system in future studies.