The maximum entropy formalism and the idiosyncratic theory of biodiversity

The powerbend distribution provides a unified model for the species abundance distribution across animals, plants and microbes

Remarkably, almost every ecological community investigated to date is composed of many rare species and a few abundant species. While the precise nature of this species abundance distribution is believed to reflect fundamental ecological principles underlying community assembly, ecologists have yet to identify a single model that comprehensively explains all species abundance distributions. Recent studies using large datasets have suggested that the logseries distribution best describes animal and plant communities, while the Poisson lognormal distribution is the best model for microbes, thereby challenging the notion of a unifying species abundance distribution model across the tree of life. Here, using a large dataset of ~30,000 globally distributed communities spanning animals, plants and microbes from diverse environments, we show that the powerbend distribution, predicted by a maximum information entropy-based theory of ecology, emerges as a unifying model that accurately captures species abundance distributions of all life forms, habitats and abundance scales. Our findings challenge the notion of pure neutrality, suggesting instead that community assembly is driven by a combination of random fluctuations and deterministic mechanisms shaped by interspecific trait variation.

The Equilibrium Theory of Biodiversity Dynamics: A General Framework for Scaling Species Richness and Community Abundance along Environmental Gradients

AbstractLarge-scale temporal and spatial biodiversity patterns have traditionally been explained by multitudinous particular factors and a few theories. However, these theories lack sufficient generality and do not address fundamental interrelationships and coupled dynamics among resource availability, community abundance, and species richness. We propose the equilibrium theory of biodiversity dynamics (ETBD) to address these linkages. According to the theory, equilibrium levels of species richness and community abundance emerge at large spatial scales because of the population size dependence of speciation and/or extinction rates, modulated by resource availability and the species abundance distribution. In contrast to other theories, ETBD includes the effect of biodiversity on community abundance and thus addresses phenomena such as niche complementarity, facilitation, and ecosystem engineering. It reveals how alternative stable states in both diversity and community abundance emerge from these nonlinear biodiversity effects. The theory predicts how the strength of these effects alters scaling relationships among species richness, (meta)community abundance, and resource availability along different environmental gradients. Using data on global-scale variation in tree species richness, we show how the general framework is useful for clarifying the role of speciation, extinction, and resource availability in driving macroecological patterns in biodiversity and community abundance, such as the latitudinal diversity gradient.

Deviation from neutral species abundance distributions unveils geographical differences in the structure of diatom communities

In recent years, the application of metagenomics techniques has advanced our understanding of plankton communities and their global distribution. Despite this progress, the relationship between the abundance distribution of diatom species and varying marine environmental conditions remains poorly understood. This study, leveraging data from the Tara Oceans expedition, tests the hypothesis that diatoms in sampled stations display a consistent species abundance distribution structure, as though they were sampled from a single ocean-wide metacommunity. Using a neutral sampling theory, we thus develop a framework to estimate the structure and diversity of diatom communities at each sampling station given the shape of the species abundance distribution of the metacommunity and the information of a reference station. Our analysis reveals a substantial temperature gradient in the discrepancies between predicted and observed biodiversity across the sampled stations. These findings challenge the hypothesis of a single neutral metacommunity, indicating that environmental differences substantially influence both the composition and structure of diatom communities.

Aqueous habitats and carbon inputs shape the microscale geography and interaction ranges of soil bacteria

Earth's diverse soil microbiomes host bacteria within dynamic and fragmented aqueous habitats that occupy complex pore spaces and restrict the spatial range of ecological interactions. Yet, the spatial distributions of bacterial cells in soil communities remain underexplored. Here, we propose a modelling framework representing submillimeter-scale distributions of soil bacteria based on physical constraints supported by individual-based model results and direct observations. The spatial distribution of bacterial cell clusters modulates various metabolic interactions and soil microbiome functioning. Dry soils with long diffusion times limit localized interactions of the sparse communities. Frequently wet soils enable long-range trophic interactions between dense cell clusters through connected aqueous pathways. Biomes with high carbon inputs promote large and dense cell clusters where anoxic microsites form even in aerated soils. Micro-geographic considerations of difficult-to-observe microbial processes can improve the interpretation of data from bulk soil samples.

The chosen few-variations in common and rare soil bacteria across biomes

Soil bacterial communities are dominated by a few abundant species, while their richness is associated with rare species with largely unknown ecological roles and biogeography. Analyses of previously published soil bacterial community data using a novel classification of common and rare bacteria indicate that only 0.4% of bacterial species can be considered common and are prevalent across biomes. The remaining bacterial species designated as rare are endemic with low relative abundances. Observations coupled with mechanistic models highlight the central role of soil wetness in shaping bacterial rarity. An individual-based model reveals systematic shifts in community composition induced by low carbon inputs in drier soils that deprive common species of exhibiting physiological advantages relative to other species. We find that only a "chosen few" common species shape bacterial communities across biomes; however, their contributions are curtailed in resource-limited environments where a larger number of rare species constitutes the soil microbiome.

Emergent Rarity Properties in Carabid Communities From Chinese Steppes With Different Climatic Conditions

Species abundance distributions (SADs) are increasingly used to investigate how species community structure changes in response to environmental variations. SAD models depict the relative abundance of species recorded in a community and express fundamental aspects of the community structure, namely patterns of commonness and rarity. However, the influence of differences in environmental conditions on SAD characteristics is still poorly understood. In this study we used SAD models of carabid beetles (Coleoptera: Carabidae) in three grassland ecosystems (desert, typical, and meadow steppes) in China. These ecosystems are characterized by different aridity conditions, thus offering an opportunity to investigate how SADs are influenced by differences in environmental conditions (mainly aridity and vegetation cover, and hence productivity). We used various SAD models, including the meta-community zero sum multinomial (mZSM), the lognormal (PLN) and Fisher’s logseries (LS), and uni- and multimodal gambin models. Analyses were done at the level of steppe type (coarse scale) and for different sectors within the same steppe (fine scale). We found that the mZSM model provided, in general, the best fit at both analysis scales. Model parameters were influenced by the scale of analysis. Moreover, the LS was the best fit in desert steppe SAD. If abundances are rarefied to the smallest sample, results are similar to those without rarefaction, but differences in models estimates become more evident. Gambin unimodal provided the best fit with the lowest α-value observed in desert steppe and higher values in typical and meadow steppes, with results which were strongly affected by the scale of analysis and the use of rarefaction. Our results indicate that all investigated communities are adequately modeled by two similar distributions, the mZSM and the LS, at both scales of analyses. This indicates (1) that all communities are characterized by a relatively small number of species, most of which are rare, and (2) that the meta-communities at the large scale maintain the basic SAD shape of the local communities. The gambin multimodal models produced exaggerated α-values, which indicates that they overfit simple communities. Overall, Fisher’s α, mZSM θ, and gambin α-values were substantially lower in the desert steppe and higher in the typical and meadow steppes, which implies a decreasing influence of environmental harshness (aridity) from the desert steppe to the typical and meadow steppes.

The Shape of Species Abundance Distributions Across Spatial Scales

Species abundance distributions (SADs) describe community structure and are a key component of biodiversity theory and research. Although different distributions have been proposed to represent SADs at different scales, a systematic empirical assessment of how SAD shape varies across wide scale gradients is lacking. Here, we examined 11 empirical large-scale datasets for a wide range of taxa and used maximum likelihood methods to compare the fit of the logseries, lognormal, and multimodal (i.e., with multiple modes of abundance) models to SADs across a scale gradient spanning several orders of magnitude. Overall, there was a higher prevalence of multimodality for larger spatial extents, whereas the logseries was exclusively selected as best fit for smaller areas. For many communities the shape of the SAD at the largest spatial extent (either lognormal or multimodal) was conserved across the scale gradient, despite steep declines in area and taxonomic diversity sampled. Additionally, SAD shape was affected by species richness, but we did not detect a systematic effect of the total number of individuals. Our results reveal clear departures from the predictions of two major macroecological theories of biodiversity for SAD shape. Specifically, neither the Neutral Theory of Biodiversity (NTB) nor the Maximum Entropy Theory of Ecology (METE) are able to accommodate the variability in SAD shape we encountered. This is highlighted by the inadequacy of the logseries distribution at larger scales, contrary to predictions of the NTB, and by departures from METE expectation across scales. Importantly, neither theory accounts for multiple modes in SADs. We suggest our results are underpinned by both inter- and intraspecific spatial aggregation patterns, highlighting the importance of spatial distributions as determinants of biodiversity patterns. Critical developments for macroecological biodiversity theories remain in incorporating the effect of spatial scale, ecological heterogeneity and spatial aggregation patterns in determining SAD shape.

Ensembles of Atoms, Ensembles of Species: Comparative Statistical Mechanics

The methods of statistical physics are exemplified in the classical perfect gas—each atom is a single dynamical entity. Such methods can be applied in ecology to the distribution of cosmopolitan species over many sites. The analogue of an atom is a class of species distinguished by the number of sites at which it occurs, hardly a material entity; yet, the methods of statistical physics nonetheless seem applicable. This paper compares the application of statistical mechanics to the distribution of atoms and to the vastly different problem of distribution of cosmopolitan species. A number of different approaches show that these distributed entities must be in some sense equivalent; the dynamics must be controlled by interaction between species and the global environment rather than between species and many uncorrelated local environments.

Invariance in ecological pattern

Background: The abundance of different species in a community often follows the log series distribution. Other ecological patterns also have simple forms. Why does the complexity and variability of ecological systems reduce to such simplicity? Common answers include maximum entropy, neutrality, and convergent outcome from different underlying biological processes.  Methods: This article proposes a more general answer based on the concept of invariance, the property by which a pattern remains the same after transformation. Invariance has a long tradition in physics. For example, general relativity emphasizes the need for the equations describing the laws of physics to have the same form in all frames of reference.  Results: By bringing this unifying invariance approach into ecology, we show that the log series pattern dominates when the consequences of processes acting on abundance are invariant to the addition or multiplication of abundance by a constant. The lognormal pattern dominates when the processes acting on net species growth rate obey rotational invariance (symmetry) with respect to the summing up of the individual component processes. Conclusions: Recognizing how these invariances connect pattern to process leads to a synthesis of previous approaches. First, invariance provides a simpler and more fundamental maximum entropy derivation of the log series distribution. Second, invariance provides a simple derivation of the key result from neutral theory: the log series at the metacommunity scale and a clearer form of the skewed lognormal at the local community scale. The invariance expressions are easy to understand because they uniquely describe the basic underlying components that shape pattern.

Habitat suitability model with maximum entropy approach for European roe deer (Capreolus capreolus) in the Black Sea Region

Evaluating the relationships between wildlife species and their habitats helps to predict effects of habitat change for present and future management of wild animal populations. Building ecological models are good ways to understand and manage wildlife populations and to predict various environmental scenarios. Recently, management of ungulates is becoming more important in Europe due to a high demand of hunting and their role in biodiversity. European roe deer (Capreolus capreolus) is the smallest species of cervids and has a widespread distribution in Turkey. In this study, two habitat suitability models of roe deers, living in the Black Sea Region in Turkey, were created by using a maximum entropy (MaxEnt) approach. Two wildlife development areas, which have widely different habitat types, were selected as study sites. As a result of this study, area under the curve (AUC) values were found to be above 0.80. According to the modeling results, in two different habitat types, ecological variables are quite similar in general. This study is the first study on modeling European roe deers in Turkey.

The Introduction of Entropy and Information Methods to Ecology by Ramon Margalef

In ecology and evolution, entropic methods are now used widely and increasingly frequently. Their use can be traced back to Ramon Margalef's first attempt 70 years ago to use log-series to quantify ecological diversity, including searching for ecologically meaningful groupings within a large assemblage, which we now call the gamma level. The same year, Shannon and Weaver published a generally accessible form of Shannon's work on information theory, including the measure that we now call Shannon-Wiener entropy. Margalef seized on that measure and soon proposed that ecologists should use the Shannon-Weiner index to evaluate diversity, including assessing local (alpha) diversity and differentiation between localities (beta). He also discussed relating this measure to environmental variables and ecosystem processes such as succession. Over the subsequent decades, he enthusiastically expanded upon his initial suggestions. Finally, 2019 also would have been Margalef's 100th birthday.

Derivations of the Core Functions of the Maximum Entropy Theory of Ecology

The Maximum Entropy Theory of Ecology (METE), is a theoretical framework of macroecology that makes a variety of realistic ecological predictions about how species richness, abundance of species, metabolic rate distributions, and spatial aggregation of species interrelate in a given region. In the METE framework, "ecological state variables" (representing total area, total species richness, total abundance, and total metabolic energy) describe macroecological properties of an ecosystem. METE incorporates these state variables into constraints on underlying probability distributions. The method of Lagrange multipliers and maximization of information entropy (MaxEnt) lead to predicted functional forms of distributions of interest. We demonstrate how information entropy is maximized for the general case of a distribution, which has empirical information that provides constraints on the overall predictions. We then show how METE's two core functions are derived. These functions, called the "Spatial Structure Function" and the "Ecosystem Structure Function" are the core pieces of the theory, from which all the predictions of METE follow (including the Species Area Relationship, the Species Abundance Distribution, and various metabolic distributions). Primarily, we consider the discrete distributions predicted by METE. We also explore the parameter space defined by the METE's state variables and Lagrange multipliers. We aim to provide a comprehensive resource for ecologists who want to understand the derivations and assumptions of the basic mathematical structure of METE.

Endemics and Cosmopolitans: Application of Statistical Mechanics to the Dry Forests of Mexico

Data on the seasonally dry tropical forests of Mexico have been examined in the light of statistical mechanics. The results suggest a division into two classes of species. There are drifting populations of a cosmopolitan class capable of existing in most dry forest sites; these have a statistical distribution previously only observed (globally) for populations of alien species. We infer that a high proportion of species found only at a single site are specialists, endemics, and that these prefer sites comparatively low in species richness.

Role of Modelling in International Crop Research: Overview and Some Case Studies

Crop modelling has the potential to contribute to global food and nutrition security. This paper briefly examines the history of crop modelling by international crop research centres of the CGIAR (formerly Consultative Group on International Agricultural Research but now known simply as CGIAR), whose primary focus is on less developed countries. Basic principles of crop modelling building up to a Genotype × Environment × Management × Socioeconomic (G × E × M × S) paradigm, are explained. Modelling has contributed to better understanding of crop performance and yield gaps, better prediction of pest and insect outbreaks, and improving the efficiency of crop management including irrigation systems and optimization of planting dates. New developments include, for example, use of remote sensed data and mobile phone technology linked to crop management decision support models, data sharing in the new era of big data, and the use of genomic selection and crop simulation models linked to environmental data to help make crop breeding decisions. Socio-economic applications include foresight analysis of agricultural systems under global change scenarios, and the consequences of potential food system shocks are also described. These approaches are discussed in this paper which also calls for closer collaboration among disciplines in order to better serve the crop research and development communities by providing model based recommendations ranging from policy development at the level of governmental agencies to direct crop management support for resource poor farmers.

Community-level species' correlated distribution can be scale-independent and related to the evenness of abundance

The spatial distribution of species is not random; instead, individuals tend to gather, resulting in a non-random pattern. Previous studies used the independent negative binomial distribution (NBD) to model the distributional aggregation of a single species, in which the independence of the distribution of individuals of a species in different quadrats had been assumed. This way of analyzing aggregation will result in the scale-dependent estimation of the aggregation or shape parameter. However, because non-random (and therefore non-independent) distribution of individuals of a species in a finite area can be caused by either correlated or clumped distribution of individuals of a species between neighboring sites, an alternative model would assume that the distribution of individuals of a species over different sampling areas is multinomial. Here, we showed that, by assuming that regional species abundance followed a NBD while using a multinomial distribution to assign individuals of species in different non-overlapped sampling quadrats that are from a partition of the entire region (quantifying positive correlation or synchrony), the estimation of the shape parameter in this probabilistic model, which is the negative multinomial distribution (NMD), was scale-invariant (i.e., the estimated shape parameter is identical across different partitions of the study region). Accordingly, the estimation of the shape parameter was related to regional species distribution alone. This implied that, the shape parameter at the community level, using the NMD model, reflected the evenness of interspecific abundance. As a comparison, if the distribution of individuals of a single species followed independent NBDs as studied previously, the shape parameter would measure the evenness of intraspecific abundance (quantifying single-species' distributional aggregation). Moreover, our study highlighted the necessity for adjusting the model for the effects of unsampled species when studying community-level distributional patterns. Collectively, as long as a target area is partitioned into non-overlapping quadrats (no matter how their sizes vary), the proposed NMD model in this study, along with the independent NBDs model, can be jointly formulated as a framework to reconcile the scale-dependent debate on the shape parameter, unifying the relationship between inter- or intraspecific abundance and distributional patterns.

Maximum Entropy Theory of Ecology: A Reply to Harte

In a paper published in this journal, I addressed the following problem: under which conditions will two scientists, observing the same system and sharing the same initial information, reach the same probabilistic description upon the application of the Maximum Entropy inference principle (MaxEnt) independent of the probability distribution chosen to set up the MaxEnt procedure. This is a minimal objectivity requirement which is generally asked for scientific investigation. In the same paper, I applied the findings to a critical examination of the application of MaxEnt made in Harte's Maximum Entropy Theory of Ecology (METE). Prof. Harte published a comment to my paper and this is my reply. For the sake of the reader who may be unaware of the content of the papers, I have tried to make this reply self-contained and to skip technical details. However, I invite the interested reader to consult the previously published papers.

The more-individuals hypothesis revisited: the role of community abundance in species richness regulation and the productivity-diversity relationship

Species richness increases with energy availability, yet there is little consensus as to the exact processes driving this species-energy relationship. The most straightforward explanation is the more-individuals hypothesis (MIH). It states that higher energy availability promotes a higher total number of individuals in a community, which consequently increases species richness by allowing for a greater number of species with viable populations. Empirical support for the MIH is mixed, partially due to the lack of proper formalisation of the MIH and consequent confusion as to its exact predictions. Here, we review the evidence of the MIH and evaluate the reliability of various predictions that have been tested. There is only limited evidence that spatial variation in species richness is driven by variation in the total number of individuals. There are also problems with measures of energy availability, with scale-dependence, and with the direction of causality, as the total number of individuals may sometimes itself be driven by the number of species. However, even in such a case the total number of individuals may be involved in diversity regulation. We propose a formal theory that encompasses these processes, clarifying how the different factors affecting diversity dynamics can be disentangled.

An introduction to the maximum entropy approach and its application to inference problems in biology

A cornerstone of statistical inference, the maximum entropy framework is being increasingly applied to construct descriptive and predictive models of biological systems, especially complex biological networks, from large experimental data sets. Both its broad applicability and the success it obtained in different contexts hinge upon its conceptual simplicity and mathematical soundness. Here we try to concisely review the basic elements of the maximum entropy principle, starting from the notion of 'entropy', and describe its usefulness for the analysis of biological systems. As examples, we focus specifically on the problem of reconstructing gene interaction networks from expression data and on recent work attempting to expand our system-level understanding of bacterial metabolism. Finally, we highlight some extensions and potential limitations of the maximum entropy approach, and point to more recent developments that are likely to play a key role in the upcoming challenges of extracting structures and information from increasingly rich, high-throughput biological data.

Remarks on the Maximum Entropy Principle with Application to the Maximum Entropy Theory of Ecology

In the first part of the paper we work out the consequences of the fact that Jaynes’ Maximum Entropy Principle, when translated in mathematical terms, is a constrained extremum problem for an entropy function H(p) expressing the uncertainty associated with the probability distribution p. Consequently, if two observers use different independent variables p or g(p), the associated entropy functions have to be defined accordingly and they are different in the general case. In the second part we apply our findings to an analysis of the foundations of the Maximum Entropy Theory of Ecology (M.E.T.E.) a purely statistical model of an ecological community. Since the theory has received considerable attention by the scientific community, we hope to give a useful contribution to the same community by showing that the procedure of application of MEP, in the light of the theory developed in the first part, suffers from some incongruences. We exhibit an alternative formulation which is free from these limitations and that gives different results.

Reinterpreting maximum entropy in ecology: a null hypothesis constrained by ecological mechanism

Simplified mechanistic models in ecology have been criticised for the fact that a good fit to data does not imply the mechanism is true: pattern does not equal process. In parallel, the maximum entropy principle (MaxEnt) has been applied in ecology to make predictions constrained by just a handful of state variables, like total abundance or species richness. But an outstanding question remains: what principle tells us which state variables to constrain? Here we attempt to solve both problems simultaneously, by translating a given set of mechanisms into the state variables to be used in MaxEnt, and then using this MaxEnt theory as a null model against which to compare mechanistic predictions. In particular, we identify the sufficient statistics needed to parametrise a given mechanistic model from data and use them as MaxEnt constraints. Our approach isolates exactly what mechanism is telling us over and above the state variables alone.

An extensive comparison of species-abundance distribution models

A number of different models have been proposed as descriptions of the species-abundance distribution (SAD). Most evaluations of these models use only one or two models, focus on only a single ecosystem or taxonomic group, or fail to use appropriate statistical methods. We use likelihood and AIC to compare the fit of four of the most widely used models to data on over 16,000 communities from a diverse array of taxonomic groups and ecosystems. Across all datasets combined the log-series, Poisson lognormal, and negative binomial all yield similar overall fits to the data. Therefore, when correcting for differences in the number of parameters the log-series generally provides the best fit to data. Within individual datasets some other distributions performed nearly as well as the log-series even after correcting for the number of parameters. The Zipf distribution is generally a poor characterization of the SAD.

Comparing process-based and constraint-based approaches for modeling macroecological patterns

Ecological patterns arise from the interplay of many different processes, and yet the emergence of consistent phenomena across a diverse range of ecological systems suggests that many patterns may in part be determined by statistical or numerical constraints. Differentiating the extent to which patterns in a given system are determined statistically, and where it requires explicit ecological processes, has been difficult. We tackled this challenge by directly comparing models from a constraint-based theory, the Maximum Entropy Theory of Ecology (METE) and models from a process-based theory, the size-structured neutral theory (SSNT). Models from both theories were capable of characterizing the distribution of individuals among species and the distribution of body size among individuals across 76 forest communities. However, the SSNT models consistently yielded higher overall likelihood, as well as more realistic characterizations of the relationship between species abundance and average body size of conspecific individuals. This suggests that the details of the biological processes contain additional information for understanding community structure that are not fully captured by the METE constraints in these systems. Our approach provides a first step towards differentiating between process- and constraint-based models of ecological systems and a general methodology for comparing ecological models that make predictions for multiple patterns.

The shape of terrestrial abundance distributions

Ecologists widely accept that the distribution of abundances in most communities is fairly flat but heavily dominated by a few species. The reason for this is that species abundances are thought to follow certain theoretical distributions that predict such a pattern. However, previous studies have focused on either a few theoretical distributions or a few empirical distributions. I illustrate abundance patterns in 1055 samples of trees, bats, small terrestrial mammals, birds, lizards, frogs, ants, dung beetles, butterflies, and odonates. Five existing theoretical distributions make inaccurate predictions about the frequencies of the most common species and of the average species, and most of them fit the overall patterns poorly, according to the maximum likelihood-related Kullback-Leibler divergence statistic. Instead, the data support a low-dominance distribution here called the "double geometric." Depending on the value of its two governing parameters, it may resemble either the geometric series distribution or the lognormal series distribution. However, unlike any other model, it assumes both that richness is finite and that species compete unequally for resources in a two-dimensional niche landscape, which implies that niche breadths are variable and that trait distributions are neither arrayed along a single dimension nor randomly associated. The hypothesis that niche space is multidimensional helps to explain how numerous species can coexist despite interacting strongly.

When can species abundance data reveal non-neutrality?

Species abundance distributions (SAD) are probably ecology's most well-known empirical pattern, and over the last decades many models have been proposed to explain their shape. There is no consensus over which model is correct, because the degree to which different processes can be discerned from SAD patterns has not yet been rigorously quantified. We present a power calculation to quantify our ability to detect deviations from neutrality using species abundance data. We study non-neutral stochastic community models, and show that the presence of non-neutral processes is detectable if sample size is large enough and/or the amplitude of the effect is strong enough. Our framework can be used for any candidate community model that can be simulated on a computer, and determines both the sampling effort required to distinguish between alternative processes, and a range for the strength of non-neutral processes in communities whose patterns are statistically consistent with neutral theory. We find that even data sets of the scale of the 50 Ha forest plot on Barro Colorado Island, Panama, are unlikely to be large enough to detect deviations from neutrality caused by competitive interactions alone, though the presence of multiple non-neutral processes with contrasting effects on abundance distributions may be detectable.

A mean field model for competition: from neutral ecology to the Red Queen

Individual species are distributed inhomogeneously over space and time, yet, within large communities of species, aggregated patterns of biodiversity seem to display nearly universal behaviour. Neutral models assume that an individual's demographic prospects are independent of its species identity. They have successfully predicted certain static, time-independent patterns. But they have generally failed to predict temporal patterns, such as species ages or population dynamics. We construct a new, multispecies framework incorporating competitive differences between species, and assess the impact of this competition on static and dynamic patterns of biodiversity. We solve this model exactly for the special case of a Red Queen hypothesis, where fitter species are continually arising. The model predicts more realistic species ages than neutral models, without greatly changing predictions for static species abundance distributions. Our modelling approach may allow users to incorporate a broad range of ecological mechanisms.

Maximum information entropy: a foundation for ecological theory

The maximum information entropy (MaxEnt) principle is a successful method of statistical inference that has recently been applied to ecology. Here, we show how MaxEnt can accurately predict patterns such as species-area relationships (SARs) and abundance distributions in macroecology and be a foundation for ecological theory. We discuss the conceptual foundation of the principle, why it often produces accurate predictions of probability distributions in science despite not incorporating explicit mechanisms, and how mismatches between predictions and data can shed light on driving mechanisms in ecology. We also review possible future extensions of the maximum entropy theory of ecology (METE), a potentially important foundation for future developments in ecological theory.

The microbial contribution to macroecology

There has been a recent explosion of research within the field of microbial ecology that has been fueled, in part, by methodological improvements that make it feasible to characterize microbial communities to an extent that was inconceivable only a few years ago. Furthermore, there is increasing recognition within the field of ecology that microorganisms play a critical role in the health of organisms and ecosystems. Despite these developments, an important gap still persists between the theoretical framework of macroecology and microbial ecology. We highlight two idiosyncrasies of microorganisms that are fundamental to understanding macroecological patterns and their mechanistic drivers. First, high dispersal rates provide novel opportunities to test the relative importance of niche, stochastic, and historical processes in structuring biological communities. Second, high speciation rates potentially lead to the convergence of ecological and evolutionary time scales. After reviewing these unique aspects, we discuss strategies for improving the conceptual integration of microbes into macroecology. As examples, we discuss the use of phylogenetic ecology as an integrative approach to explore patterns across the tree of life. Then we demonstrate how two general theories of biodiversity (i.e., the recently developed theory of stochastic geometry and the neutral theory) can be adapted to microorganisms. We demonstrate how conceptual models that integrate evolutionary and ecological mechanisms can contribute to the unification of microbial ecology and macroecology.

Neutral theory and the species abundance distribution: recent developments and prospects for unifying niche and neutral perspectives

Published in 2001, The Unified Neutral Theory of Biodiversity and Biogeography (UNTB) emphasizes the importance of stochastic processes in ecological community structure, and has challenged the traditional niche-based view of ecology. While neutral models have since been applied to a broad range of ecological and macroecological phenomena, the majority of research relating to neutral theory has focused exclusively on the species abundance distribution (SAD). Here, we synthesize the large body of work on neutral theory in the context of the species abundance distribution, with a particular focus on integrating ideas from neutral theory with traditional niche theory. First, we summarize the basic tenets of neutral theory; both in general and in the context of SADs. Second, we explore the issues associated with neutral theory and the SAD, such as complications with fitting and model comparison, the underlying assumptions of neutral models, and the difficultly of linking pattern to process. Third, we highlight the advances in understanding of SADs that have resulted from neutral theory and models. Finally, we focus consideration on recent developments aimed at unifying neutral- and niche-based approaches to ecology, with a particular emphasis on what this means for SAD theory, embracing, for instance, ideas of emergent neutrality and stochastic niche theory. We put forward the argument that the prospect of the unification of niche and neutral perspectives represents one of the most promising future avenues of neutral theory research.

Statistical thermodynamics concepts and mathematical tools for a multi-agent ecosystem

Finding the distribution of systems over their possible states is a mathematical problem. One possible solution is the method of the most probable distribution developed by Boltzmann. This method has been instrumental in developing statistical mechanics and explaining the origin of many thermodynamics concepts, like entropy or temperature, but is also applicable in many other fields like ecology or economics. Artificial ecosystems have many features in common with ecological or economic systems, but surprisingly the method does not appear to have been very successful in this field of application. The hypothesis of this article is that this failure is due to the incorrect interpretation of the method's concepts and mathematical tools. We propose to review and reinterpret the method so that it can be correctly applied and all its potential exploited in order to study and characterize the global behavior of an artificial multi-agent ecosystem.

Species abundance distributions, statistical mechanics and the priors of MaxEnt

The methods of Maximum Entropy have been deployed for some years to address the problem of species abundance distributions. In this approach, it is important to identify the correct weighting factors, or priors, to be applied before maximising the entropy function subject to constraints. The forms of such priors depend not only on the exact problem but can also depend on the way it is set up; priors are determined by the underlying dynamics of the complex system under consideration. The problem is one of statistical mechanics and it is the properties of the system that yield the correct MaxEnt priors, appropriate to the way the problem is framed. Here I calculate, in several different ways, the species abundance distribution resulting when individuals in a community are born and die independently. In the usual formulation the prior distribution for the number of species over the number of individuals is 1/n; the problem can be reformulated in terms of the distribution of individuals over species classes, with a uniform prior. Results are obtained using master equations for the dynamics and separately through the combinatoric methods of elementary statistical mechanics; the MaxEnt priors then emerge a posteriori. The first object is to establish the log series species abundance distribution as the outcome of per capita guild dynamics. The second is to clarify the true nature and origin of priors in the language of MaxEnt. Finally, I consider how it may come about that the distribution is similar to log series in the event that filled niches dominate species abundance. For the general ecologist, there are two messages. First, that species abundance distributions are determined largely by population sorting through fractional processes (resulting in the 1/n factor) and secondly that useful information is likely to be found only in departures from the log series. For the MaxEnt practitioner, the message is that the prior with respect to which the entropy is to be maximised is determined by the nature of the problem and the way in which it is formulated.

Advances in modeling trait-based plant community assembly

In this review, we examine two new trait-based models of community assembly that predict the relative abundance of species from a regional species pool. The models use fundamentally different mathematical approaches and the predictions can differ considerably. Maxent obtains the most even probability distribution subject to community-weighted mean trait constraints. Traitspace predicts low probabilities for any species whose trait distribution does not pass through the environmental filter. Neither model maximizes functional diversity because of the emphasis on environmental filtering over limiting similarity. Traitspace can test for the effects of limiting similarity by explicitly incorporating intraspecific trait variation. The range of solutions in both models could be used to define the range of natural variability of community composition in restoration projects.

How species richness and total abundance constrain the distribution of abundance

The species abundance distribution (SAD) is one of the most intensively studied distributions in ecology and its hollow-curve shape is one of ecology's most general patterns. We examine the SAD in the context of all possible forms having the same richness (S) and total abundance (N), i.e. the feasible set. We find that feasible sets are dominated by similarly shaped hollow curves, most of which are highly correlated with empirical SADs (most R(2) values > 75%), revealing a strong influence of N and S on the form of the SAD and an a priori explanation for the ubiquitous hollow curve. Empirical SADs are often more hollow and less variable than the majority of the feasible set, revealing exceptional unevenness and relatively low natural variability among ecological communities. We discuss the importance of the feasible set in understanding how general constraints determine observable variation and influence the forms of predicted and empirical patterns.