Maximum Entropy Principle in Statistical Inference: Case for Non-Shannonian Entropies

Superstatistics from a dynamical perspective: Entropy and relaxation

Distributions that deviate from equilibrium predictions are commonly observed across a broad spectrum of systems, ranging from laboratory experiments to astronomical phenomena. These distributions are generally regarded as a manifestation of a quasiequilibrium state and can very often be represented as a superposition of statistics, i.e., superstatistics. The underlying idea in this methodology is that the nonequilibrium system consists of a collection of smaller subsystems that remain infinitely close to equilibrium. This procedure has been effectively implemented in a kinetic setting, but thus far, only in the collisionless regime, limiting its scope of application. In this paper, we address the effect of collisions on the relaxation process and time evolution of superstatistical systems. After confronting the superstatistical distributions with experimental and simulation data, relevant to our analysis, we first study the effect of superstatistics on entropy. We explicitly show that, in the absence of long-range interactions, the extensivity of entropy is preserved, albeit influenced by the specific class of temperature fluctuations. Then, we examine how collisions drive the system towards a global equilibrium state, characterized by a maximum entropy, by employing the relaxation time approximation. This allows us to define a dynamical version of superstatistics, characterized by a singular time-varying parameter q(t), which undergoes a continuous evolution towards equilibrium. We show how this approach enables the determination of the evolution of the underlying temperature distribution under the influence of collisions, which act as stochastic forces, gradually narrowing the temperature distribution over time.

Entropy defect in thermodynamics

This paper describes the physical foundations of the newly discovered "entropy defect" as a basic concept of thermodynamics. The entropy defect quantifies the change in entropy caused by the order induced in a system through the additional correlations among its constituents when two or more subsystems are assembled. This defect is closely analogous to the mass defect that arises when nuclear particle systems are assembled. The entropy defect determines how the entropy of the system compares to its constituent's entropies and stands on three fundamental properties: each constituent's entropy must be (i) separable, (ii) symmetric, and (iii) bounded. We show that these properties provide a solid foundation for the entropy defect and for generalizing thermodynamics to describe systems residing out of the classical thermal equilibrium, both in stationary and nonstationary states. In stationary states, the consequent thermodynamics generalizes the classical framework, which was based on the Boltzmann-Gibbs entropy and Maxwell-Boltzmann canonical distribution of particle velocities, into the respective entropy and canonical distribution associated with kappa distributions. In nonstationary states, the entropy defect similarly acts as a negative feedback, or reduction of the increase of entropy, preventing its unbounded growth toward infinity.

Senses along Which the Entropy Sq Is Unique

The Boltzmann-Gibbs-von Neumann-Shannon additive entropy SBG=-k∑ipilnpi as well as its continuous and quantum counterparts, constitute the grounding concept on which the BG statistical mechanics is constructed. This magnificent theory has produced, and will most probably keep producing in the future, successes in vast classes of classical and quantum systems. However, recent decades have seen a proliferation of natural, artificial and social complex systems which defy its bases and make it inapplicable. This paradigmatic theory has been generalized in 1988 into the nonextensive statistical mechanics-as currently referred to-grounded on the nonadditive entropy Sq=k1-∑ipiqq-1 as well as its corresponding continuous and quantum counterparts. In the literature, there exist nowadays over fifty mathematically well defined entropic functionals. Sq plays a special role among them. Indeed, it constitutes the pillar of a great variety of theoretical, experimental, observational and computational validations in the area of complexity-plectics, as Murray Gell-Mann used to call it. Then, a question emerges naturally, namely In what senses is entropy Sq unique? The present effort is dedicated to a-surely non exhaustive-mathematical answer to this basic question.

Geometric Structures Induced by Deformations of the Legendre Transform

The recent link discovered between generalized Legendre transforms and non-dually flat statistical manifolds suggests a fundamental reason behind the ubiquity of Rényi's divergence and entropy in a wide range of physical phenomena. However, these early findings still provide little intuition on the nature of this relationship and its implications for physical systems. Here we shed new light on the Legendre transform by revealing the consequences of its deformation via symplectic geometry and complexification. These findings reveal a novel common framework that leads to a principled and unified understanding of physical systems that are not well-described by classic information-theoretic quantities.

Brief Review on the Connection between the Micro-Canonical Ensemble and the Sq-Canonical Probability Distribution

Non-standard thermostatistical formalisms derived from generalizations of the Boltzmann-Gibbs entropy have attracted considerable attention recently. Among the various proposals, the one that has been most intensively studied, and most successfully applied to concrete problems in physics and other areas, is the one associated with the Sq non-additive entropies. The Sq-based thermostatistics exhibits a number of peculiar features that distinguish it from other generalizations of the Boltzmann-Gibbs theory. In particular, there is a close connection between the Sq-canonical distributions and the micro-canonical ensemble. The connection, first pointed out in 1994, has been subsequently explored by several researchers, who elaborated this facet of the Sq-thermo-statistics in a number of interesting directions. In the present work, we provide a brief review of some highlights within this line of inquiry, focusing on micro-canonical scenarios leading to Sq-canonical distributions. We consider works on the micro-canonical ensemble, including historical ones, where the Sq-canonical distributions, although present, were not identified as such, and also more resent works by researchers who explicitly investigated the Sq-micro-canonical connection.

The Typical Set and Entropy in Stochastic Systems with Arbitrary Phase Space Growth

The existence of the typical set is key for data compression strategies and for the emergence of robust statistical observables in macroscopic physical systems. Standard approaches derive its existence from a restricted set of dynamical constraints. However, given its central role underlying the emergence of stable, almost deterministic statistical patterns, a question arises whether typical sets exist in much more general scenarios. We demonstrate here that the typical set can be defined and characterized from general forms of entropy for a much wider class of stochastic processes than was previously thought. This includes processes showing arbitrary path dependence, long range correlations or dynamic sampling spaces, suggesting that typicality is a generic property of stochastic processes, regardless of their complexity. We argue that the potential emergence of robust properties in complex stochastic systems provided by the existence of typical sets has special relevance to biological systems.

Entropy Optimization, Generalized Logarithms, and Duality Relations

Several generalizations or extensions of the Boltzmann-Gibbs thermostatistics, based on non-standard entropies, have been the focus of considerable research activity in recent years. Among these, the power-law, non-additive entropies Sq≡k1-∑ipiqq-1(q∈R;S1=SBG≡-k∑ipilnpi) have harvested the largest number of successful applications. The specific structural features of the Sq thermostatistics, therefore, are worthy of close scrutiny. In the present work, we analyze one of these features, according to which the q-logarithm function lnqx≡x1-q-11-q(ln1x=lnx) associated with the Sq entropy is linked, via a duality relation, to the q-exponential function characterizing the maximum-entropy probability distributions. We enquire into which entropic functionals lead to this or similar structures, and investigate the corresponding duality relations.

Information Shift Dynamics Described by Tsallis q = 3 Entropy on a Compact Phase Space

Recent mathematical investigations have shown that under very general conditions, exponential mixing implies the Bernoulli property. As a concrete example of statistical mechanics that are exponentially mixing we consider the Bernoulli shift dynamics by Chebyshev maps of arbitrary order N≥2, which maximizes Tsallis q=3 entropy rather than the ordinary q=1 Boltzmann-Gibbs entropy. Such an information shift dynamics may be relevant in a pre-universe before ordinary space-time is created. We discuss symmetry properties of the coupled Chebyshev systems, which are different for even and odd N. We show that the value of the fine structure constant αel=1/137 is distinguished as a coupling constant in this context, leading to uncorrelated behaviour in the spatial direction of the corresponding coupled map lattice for N=3.

Permutation group entropy: A new route to complexity for real-valued processes

This is a review of group entropy and its application to permutation complexity. Specifically, we revisit a new approach to the notion of complexity in the time series analysis based on both permutation entropy and group entropy. As a result, the permutation entropy rate can be extended from deterministic dynamics to random processes. More generally, our approach provides a unified framework to discuss chaotic and random behaviors.

Spatial heterogeneity of air pollution statistics in Europe

Air pollution is one of the leading causes of death globally, and continues to have a detrimental effect on our health. In light of these impacts, an extensive range of statistical modelling approaches has been devised in order to better understand air pollution statistics. However, the time-varying statistics of different types of air pollutants are far from being fully understood. The observed probability density functions (PDFs) of concentrations depend very much on the spatial location and on the pollutant substance. In this paper, we analyse a large variety of data from 3544 different European monitoring sites and show that the PDFs of nitric oxide (NO), nitrogen dioxide ([Formula: see text]) and particulate matter ([Formula: see text] and [Formula: see text]) concentrations generically exhibit heavy tails and are asymptotically well approximated by q-exponential distributions with a given width parameter [Formula: see text]. We observe that the power-law parameter q and the width parameter [Formula: see text] vary widely for the different spatial locations. For each substance, we find different patterns of parameter clouds in the [Formula: see text] plane. These depend on the type of pollutants and on the environmental characteristics (urban/suburban/rural/traffic/industrial/background). This means the effective statistical physics description of air pollution exhibits a strong degree of spatial heterogeneity.

Causal Inference in Time Series in Terms of Rényi Transfer Entropy

Uncovering causal interdependencies from observational data is one of the great challenges of a nonlinear time series analysis. In this paper, we discuss this topic with the help of an information-theoretic concept known as Rényi’s information measure. In particular, we tackle the directional information flow between bivariate time series in terms of Rényi’s transfer entropy. We show that by choosing Rényi’s parameter α, we can appropriately control information that is transferred only between selected parts of the underlying distributions. This, in turn, is a particularly potent tool for quantifying causal interdependencies in time series, where the knowledge of “black swan” events, such as spikes or sudden jumps, are of key importance. In this connection, we first prove that for Gaussian variables, Granger causality and Rényi transfer entropy are entirely equivalent. Moreover, we also partially extend these results to heavy-tailed α-Gaussian variables. These results allow establishing a connection between autoregressive and Rényi entropy-based information-theoretic approaches to data-driven causal inference. To aid our intuition, we employed the Leonenko et al. entropy estimator and analyzed Rényi’s information flow between bivariate time series generated from two unidirectionally coupled Rössler systems. Notably, we find that Rényi’s transfer entropy not only allows us to detect a threshold of synchronization but it also provides non-trivial insight into the structure of a transient regime that exists between the region of chaotic correlations and synchronization threshold. In addition, from Rényi’s transfer entropy, we could reliably infer the direction of coupling and, hence, causality, only for coupling strengths smaller than the onset value of the transient regime, i.e., when two Rössler systems are coupled but have not yet entered synchronization.

Generalized statistical mechanics of stellar systems

The observed distributions of stellar parameters, in particular, rotational and radial velocities, often depart from the Maxwellian (Gaussian) distribution. In the absence of a consistent statistical framework, these distributions are, in general, accounted for phenomenologically by employing power-law distributions, such as Tsallis or Kaniadakis distributions. Here we argue that the observed distributions correspond to locally Gaussian distributions, whose characteristic width is regarded as a statistical variable, in accordance with common knowledge that this parameter is mass dependent. The distributions arising within this picture correspond to superstatistics-a formalism emerging naturally in the context of self-gravitating media. We discuss in detail the distributions arising within this formalism and confront them with observational data of open clusters. We compute their moments and show that the Chandrasekhar-Münch relation remains invariant in this scenario. We also address the effect of these distributions on the thermalization of a massive body, e.g., a supermassive black hole, immersed in a stellar gas. We further discuss how the superstatistical picture clarifies certain ambiguities while offering a whole family of distributions (of which asymptotic power laws represent a special case), opening possibilities for fitting observational data.

Entropy, Information, and the Updating of Probabilities

This paper is a review of a particular approach to the method of maximum entropy as a general framework for inference. The discussion emphasizes pragmatic elements in the derivation. An epistemic notion of information is defined in terms of its relation to the Bayesian beliefs of ideally rational agents. The method of updating from a prior to posterior probability distribution is designed through an eliminative induction process. The logarithmic relative entropy is singled out as a unique tool for updating (a) that is of universal applicability, (b) that recognizes the value of prior information, and (c) that recognizes the privileged role played by the notion of independence in science. The resulting framework-the ME method-can handle arbitrary priors and arbitrary constraints. It includes the MaxEnt and Bayes' rules as special cases and, therefore, unifies entropic and Bayesian methods into a single general inference scheme. The ME method goes beyond the mere selection of a single posterior, and also addresses the question of how much less probable other distributions might be, which provides a direct bridge to the theories of fluctuations and large deviations.

Reply to Pessoa, P.; Arderucio Costa, B. Comment on "Tsallis, C. Black Hole Entropy: A Closer Look. Entropy 2020, 22, 17"

In the present Reply we restrict our focus only onto the main erroneous claims by Pessoa and Costa in their recent Comment (Entropy 2020, 22, 1110).

From Rényi Entropy Power to Information Scan of Quantum States

In this paper, we generalize the notion of Shannon's entropy power to the Rényi-entropy setting. With this, we propose generalizations of the de Bruijn identity, isoperimetric inequality, or Stam inequality. This framework not only allows for finding new estimation inequalities, but it also provides a convenient technical framework for the derivation of a one-parameter family of Rényi-entropy-power-based quantum-mechanical uncertainty relations. To illustrate the usefulness of the Rényi entropy power obtained, we show how the information probability distribution associated with a quantum state can be reconstructed in a process that is akin to quantum-state tomography. We illustrate the inner workings of this with the so-called "cat states", which are of fundamental interest and practical use in schemes such as quantum metrology. Salient issues, including the extension of the notion of entropy power to Tsallis entropy and ensuing implications in estimation theory, are also briefly discussed.

Thermodynamics of structure-forming systems

Structure-forming systems are ubiquitous in nature, ranging from atoms building molecules to self-assembly of colloidal amphibolic particles. The understanding of the underlying thermodynamics of such systems remains an important problem. Here, we derive the entropy for structure-forming systems that differs from Boltzmann-Gibbs entropy by a term that explicitly captures clustered states. For large systems and low concentrations the approach is equivalent to the grand-canonical ensemble; for small systems we find significant deviations. We derive the detailed fluctuation theorem and Crooks' work fluctuation theorem for structure-forming systems. The connection to the theory of particle self-assembly is discussed. We apply the results to several physical systems. We present the phase diagram for patchy particles described by the Kern-Frenkel potential. We show that the Curie-Weiss model with molecule structures exhibits a first-order phase transition.

Calibration Invariance of the MaxEnt Distribution in the Maximum Entropy Principle

The maximum entropy principle consists of two steps: The first step is to find the distribution which maximizes entropy under given constraints. The second step is to calculate the corresponding thermodynamic quantities. The second part is determined by Lagrange multipliers’ relation to the measurable physical quantities as temperature or Helmholtz free energy/free entropy. We show that for a given MaxEnt distribution, the whole class of entropies and constraints leads to the same distribution but generally different thermodynamics. Two simple classes of transformations that preserve the MaxEnt distributions are studied: The first case is a transform of the entropy to an arbitrary increasing function of that entropy. The second case is the transform of the energetic constraint to a combination of the normalization and energetic constraints. We derive group transformations of the Lagrange multipliers corresponding to these transformations and determine their connections to thermodynamic quantities. For each case, we provide a simple example of this transformation.

Multivariate group entropies, super-exponentially growing complex systems, and functional equations

We define the class of multivariate group entropies as a novel set of information-theoretical measures, which extends significantly the family of group entropies. We propose new examples related to the "super-exponential" universality class of complex systems; in particular, we introduce a general entropy, representing a suitable information measure for this class. We also show that the group-theoretical structure associated with our multivariate entropies can be used to define a large family of exactly solvable discrete dynamical models. The natural mathematical framework allowing us to formulate this correspondence is offered by the theory of formal groups and rings.

Thermodynamics from first principles: Correlations and nonextensivity

The standard formulation of thermostatistics, being based on the Boltzmann-Gibbs distribution and logarithmic Shannon entropy, describes idealized uncorrelated systems with extensive energies and short-range interactions. In this Rapid Communication, we use the fundamental principles of ergodicity (via Liouville's theorem), the self-similarity of correlations, and the existence of the thermodynamic limit to derive generalized forms of the equilibrium distribution for long-range-interacting systems. Significantly, our formalism provides a justification for the well-studied nonextensive thermostatistics characterized by the Tsallis distribution, which it includes as a special case. We also give the complementary maximum entropy derivation of the same distributions by constrained maximization of the Gibbs-Shannon entropy. The consistency between the ergodic and maximum entropy approaches clarifies the use of the latter in the study of correlations and nonextensive thermodynamics.

Entropic Analysis of Votes Expressed in Italian Elections between 1948 and 2018

In Italy, the elections occur often, indeed almost every year the citizens are involved in a democratic choice for deciding leaders of different administrative entities. Sometimes the citizens are called to vote for filling more than one office in more than one administrative body. This phenomenon has occurred 35 times after 1948; it creates the peculiar condition of having the same sample of people expressing decisions on political bases at the same time. Therefore, the Italian contemporaneous ballots constitute the occasion to measure coherence and chaos in the way of expressing political opinion. In this paper, we address all the Italian elections that occurred between 1948 and 2018. We collect the number of votes per party at each administrative level and we treat each election as a manifestation of a complex system. Then, we use the Shannon entropy and the Gini Index to study the degree of disorder manifested during different types of elections at the municipality level. A particular focus is devoted to the contemporaneous elections. Such cases implicate different disorder dynamics in the contemporaneous ballots, when different administrative level are involved. Furthermore, some features that characterize different entropic regimes have emerged.

Gradient and GENERIC time evolution towards reduced dynamics

Reduction of a mesoscopic dynamical theory to equilibrium thermodynamics brings to the latter theory the fundamental thermodynamic relation (i.e. entropy as a function of the thermodynamic state variables). The reduction is made by following the mesoscopic time evolution to its conclusion, i.e. to fixed points at which the time evolution ceases to continue. The approach to fixed points is driven by entropy, that, if evaluated at the fixed points, becomes the thermodynamic entropy. Since the fixed points are parametrized by the thermodynamic state variables (by constants of motion), the thermodynamic entropy arises as a function of the thermodynamic state variables and thus the final outcome of the reduction is the fundamental thermodynamic relation. This reduction process extends also to reductions in which the reduced theory still involves the time evolution (e.g. reduction of kinetic theory to hydrodynamics). The essence of the extension is the replacement of the mesoscopic time evolution of the state variables with the corresponding mesoscopic time evolution of the vector field (i.e. of the fluxes). The fixed point in this flux time evolution is the vector field generating the reduced mesoscopic time evolution. The flux-entropy driving the flux time evolution becomes, if evaluated at the fixed point, the flux fundamental thermodynamic relation in the reduced dynamical theory. We show that the flux-entropy is a potential related to the entropy production. This article is part of the theme issue 'Fundamental aspects of nonequilibrium thermodynamics'.

When Shannon and Khinchin meet Shore and Johnson: Equivalence of information theory and statistical inference axiomatics

We propose a unified framework for both Shannon-Khinchin and Shore-Johnson axiomatic systems. We do it by rephrasing Shannon-Khinchine axioms in terms of generalized arithmetics of Kolmogorov and Nagumo. We prove that the two axiomatic schemes yield identical classes of entropic functionals-the Uffink class of entropies. This allows to re-establish the entropic parallelism between information theory and statistical inference that has seemed to be "broken" by the use of non-Shannonian entropies.

Tsallis meets Boltzmann: q-index for a finite ideal gas and its thermodynamic limit

Nonadditive Tsallis q-statistics has successfully been applied for a plethora of systems in natural sciences and other branches of knowledge. Nevertheless, its foundations have been severely criticized by some authors based on the standard additive Boltzmann-Gibbs approach, thereby remaining a quite controversial subject. In order to clarify some polemical concepts, the distribution function for an ideal gas with a finite number of point particles and its q-index are analytically determined. The two-particle correlation function is also derived. The degree of correlation diminishes continuously with the growth of the number of particles. The ideal finite gas system is usually correlated, becomes less correlated when the number of particles grows, and is finally fully uncorrelated when the molecular chaos regime is reached. It is also advocated that both approaches can be confronted through a careful kinetic spectroscopic experiment. The analytical results derived here suggest that Tsallis q-statistics may play a physical role more fundamental than usually discussed in the literature.

Generalizing Information to the Evolution of Rational Belief

Information theory provides a mathematical foundation to measure uncertainty in belief. Belief is represented by a probability distribution that captures our understanding of an outcome’s plausibility. Information measures based on Shannon’s concept of entropy include realization information, Kullback–Leibler divergence, Lindley’s information in experiment, cross entropy, and mutual information. We derive a general theory of information from first principles that accounts for evolving belief and recovers all of these measures. Rather than simply gauging uncertainty, information is understood in this theory to measure change in belief. We may then regard entropy as the information we expect to gain upon realization of a discrete latent random variable. This theory of information is compatible with the Bayesian paradigm in which rational belief is updated as evidence becomes available. Furthermore, this theory admits novel measures of information with well-defined properties, which we explored in both analysis and experiment. This view of information illuminates the study of machine learning by allowing us to quantify information captured by a predictive model and distinguish it from residual information contained in training data. We gain related insights regarding feature selection, anomaly detection, and novel Bayesian approaches.

(Generalized) Maximum Cumulative Direct, Residual, and Paired Φ Entropy Approach

A distribution that maximizes an entropy can be found by applying two different principles. On the one hand, Jaynes (1957a,b) formulated the maximum entropy principle (MaxEnt) as the search for a distribution maximizing a given entropy under some given constraints. On the other hand, Kapur (1994) and Kesavan and Kapur (1989) introduced the generalized maximum entropy principle (GMaxEnt) as the derivation of an entropy for which a given distribution has the maximum entropy property under some given constraints. In this paper, both principles were considered for cumulative entropies. Such entropies depend either on the distribution function (direct), on the survival function (residual) or on both (paired). We incorporate cumulative direct, residual, and paired entropies in one approach called cumulative Φ entropies. Maximizing this entropy without any constraints produces an extremely U-shaped (=bipolar) distribution. Maximizing the cumulative entropy under the constraints of fixed mean and variance tries to transform a distribution in the direction of a bipolar distribution, as far as it is allowed by the constraints. A bipolar distribution represents so-called contradictory information, which is in contrast to minimum or no information. In the literature, to date, only a few maximum entropy distributions for cumulative entropies have been derived. In this paper, we extended the results to well known flexible distributions (like the generalized logistic distribution) and derived some special distributions (like the skewed logistic, the skewed Tukey λ and the extended Burr XII distribution). The generalized maximum entropy principle was applied to the generalized Tukey λ distribution and the Fechner family of skewed distributions. Finally, cumulative entropies were estimated such that the data was drawn from a maximum entropy distribution. This estimator will be applied to the daily S&P500 returns and time durations between mine explosions.

Equivalence between four versions of thermostatistics based on strongly pseudoadditive entropies

The class of strongly pseudoadditive (SPA) entropies, which can be represented as an increasing continuous transformation of Shannon and Rényi entropies, have intensively been studied in previous decades. Although their mathematical structure has thoroughly been explored and established by generalized Shannon-Khinchin axioms, the analysis of their thermostatistical properties have mostly been limited to special cases which belong to two parameter Sharma-Mittal entropy class, such as Tsallis, Renyi and Gaussian entropies. In this paper we present a general analysis of the strongly pseudoadditive entropies thermostatistics by taking into account both linear and escort constraints on internal energy. We develop two types of dualities between the thermostatistics formalisms. By the first one, the formalism of Rényi entropy is transformed in the formalism of SPA entropy under general energy constraint and, by the second one, the generalized thermostatistics which corresponds to the linear constraint is transformed into the one which corresponds to the escort constraint. Thus, we establish the equivalence between four different thermostatistics formalisms based on Rényi and SPA entropies coupled with linear and escort constraints and we provide the transformation formulas. In this way we obtain a general framework which is applicable to the wide class of entropies and constraints previously discussed in the literature. As an example, we rederive maximum entropy distributions for Sharma-Mittal entropy and we establish new relationships between the corresponding thermodynamic potentials. We obtain, as special cases, previously developed expressions for maximum entropy distributions and thermodynamic quantities for Tsallis, Rényi, and Gaussian entropies. In addition, the results are applied for derivation of thermostatistical relationships for supraextensive entropy, which has not previously been considered.

Cumulative Tsallis entropy based on power spectrum of financial time series

The complexity of financial time series is an important issue for nonlinear dynamic systems. Generalized power spectrum cumulative Tsallis entropy (PSCTE) is a newly proposed model for measuring dissimilarities between different time series. It solves the problem of traditional Shannon entropy inconsistency. In addition, the power spectrum is used to calculate the probability in the algorithm. In this paper, PSCTE is applied to simulation data sets, and financial time series are used to verify PSCTE reliability. The results show that PSCTE can be worked as an effective tool to measure dissimilarities and help identify signal patterns. Finally, we also obtain the geographical division of the stock market.

Reply to "Comment on 'Rényi entropy yields artificial biases not in the data and incorrect updating due to the finite-size data' "

We reply to the preceding Comment by Jizba and Korbel [Jizba and Korbel, Phys. Rev. E 100, 026101 (2019)10.1103/PhysRevE.100.026101] by first pointing out that the Schur concavity proposed by them falls short of identifying the correct intervals of normalization for the optimum probability distribution even though normalization is a necessary ingredient in the entropy maximization procedure. Second, their treatment of the subset independence axiom requires a modification of the Lagrange multipliers one begins with, thereby rendering the optimization less trustworthy. We also explicitly demonstrate that the Rényi entropy violates the subset independence axiom and compare it with the Shannon entropy. Third, the composition rule offered by Jizba and Korbel is shown to yield probability distributions even without a need for the entropy maximization procedure at the expense of creating artificial bias in the data.

Comment on "Rényi entropy yields artificial biases not in the data and incorrect updating due to the finite-size data"

In their recent paper [Phys. Rev. E 99, 032134 (2019)2470-004510.1103/PhysRevE.99.032134], Oikonomou and Bagci have argued that Rényi entropy is ill suited for inference purposes because it is not consistent with the Shore-Johnson axioms of statistical estimation theory. In this Comment we seek to clarify the latter statement by showing that there are several issues in Oikonomou's and Bagci's reasonings which lead to erroneous conclusions. When all these issues are properly accounted for, no violation of Shore-Johnson axioms is found.

Dynamic Maximum Entropy Reduction

Any physical system can be regarded on different levels of description varying by how detailed the description is. We propose a method called Dynamic MaxEnt (DynMaxEnt) that provides a passage from the more detailed evolution equations to equations for the less detailed state variables. The method is based on explicit recognition of the state and conjugate variables, which can relax towards the respective quasi-equilibria in different ways. Detailed state variables are reduced using the usual principle of maximum entropy (MaxEnt), whereas relaxation of conjugate variables guarantees that the reduced equations are closed. Moreover, an infinite chain of consecutive DynMaxEnt approximations can be constructed. The method is demonstrated on a particle with friction, complex fluids (equipped with conformation and Reynolds stress tensors), hyperbolic heat conduction and magnetohydrodynamics.

Beyond Boltzmann-Gibbs-Shannon in Physics and Elsewhere

The pillars of contemporary theoretical physics are classical mechanics, Maxwell electromagnetism, relativity, quantum mechanics, and Boltzmann–Gibbs (BG) statistical mechanics –including its connection with thermodynamics. The BG theory describes amazingly well the thermal equilibrium of a plethora of so-called simple systems. However, BG statistical mechanics and its basic additive entropy SBG started, in recent decades, to exhibit failures or inadequacies in an increasing number of complex systems. The emergence of such intriguing features became apparent in quantum systems as well, such as black holes and other area-law-like scenarios for the von Neumann entropy. In a different arena, the efficiency of the Shannon entropy—as the BG functional is currently called in engineering and communication theory—started to be perceived as not necessarily optimal in the processing of images (e.g., medical ones) and time series (e.g., economic ones). Such is the case in the presence of generic long-range space correlations, long memory, sub-exponential sensitivity to the initial conditions (hence vanishing largest Lyapunov exponents), and similar features. Finally, we witnessed, during the last two decades, an explosion of asymptotically scale-free complex networks. This wide range of important systems eventually gave support, since 1988, to the generalization of the BG theory. Nonadditive entropies generalizing the BG one and their consequences have been introduced and intensively studied worldwide. The present review focuses on these concepts and their predictions, verifications, and applications in physics and elsewhere. Some selected examples (in quantum information, high- and low-energy physics, low-dimensional nonlinear dynamical systems, earthquakes, turbulence, long-range interacting systems, and scale-free networks) illustrate successful applications. The grounding thermodynamical framework is briefly described as well.