Inland recreational fisheries contribute nutritional benefits and economic value but are vulnerable to climate change

Inland recreational fishing is primarily considered a leisure-driven activity in freshwaters, yet its harvest can contribute to food systems. Here we estimate that the harvest from inland recreational fishing equates to just over one-tenth of all reported inland fisheries catch globally. The estimated total consumptive use value of inland recreational fish destined for human consumption may reach US$9.95 billion annually. We identify Austria, Canada, Germany and Slovakia as countries above the third quantile for nutrition, economic value and climate vulnerability. These results have important implications for populations dependent on inland recreational fishing for food. Our findings can inform climate adaptation planning for inland recreational fisheries, particularly those not currently managed as food fisheries.

Nutrient supply from marine small-scale fisheries

Over 2 billion people are unable to access safe, nutritious and sufficient food year-round. While global fisheries are considered key in providing essential nutrients to hundreds of millions of people around the globe, the specific contribution of small-scale fisheries to the nutrient supply given other available food supplies is unknown. Here, we combined multiple global databases to quantify the importance of marine small-scale fisheries to national-level nutrient supply of coastal populations. We found that, on average across assessed nutrients (iron, zinc, calcium, DHA + EPA and vitamins A and B12), small-scale fisheries contributed about 32% of overall global seafood nutrient supply, 17% of the nutrient supply from animal-sourced foods and 10% of nutrient supply from all foods. These global averages, however, underrepresent some key roles of ocean-based foods. Combining nutrient supply estimates with global estimates of inadequate nutrient intake, we found that about half of coastal countries that have a mean inadequate intake of at least 50% across assessed nutrients (iron, zinc, calcium, DHA + EPA and vitamins A and B12) rely on small scale fisheries for at least 15% of mean nutrient supply, and many rely on small scale fisheries for more than 30% of mean nutrient supply. Catch from small-scale fisheries is particularly important for the supply of vitamin B12, calcium and DHA + EPA, representing up to 100% of supply in selected countries. Our study demonstrates the significance of small-scale fisheries for nutritionally vulnerable coastal populations, emphasizing how effective fisheries management can contribute to public health.

A Decision Framework for Selecting Critically Important Nutrients from Aquatic Foods

PURPOSE OF REVIEW

Aquatic foods are increasingly being recognized as a diverse, bioavailable source of nutrients, highlighting the importance of fisheries and aquaculture for human nutrition. However, studies focusing on the nutrient supply of aquatic foods often differ in the nutrients they examine, potentially biasing their contribution to nutrition security and leading to ineffective policies or management decisions.

RECENT FINDINGS

We create a decision framework to effectively select nutrients in aquatic food research based on three key domains: human physiological importance, nutritional needs of the target population (demand), and nutrient availability in aquatic foods compared to other accessible dietary sources (supply). We highlight 41 nutrients that are physiologically important, exemplify the importance of aquatic foods relative to other food groups in the food system in terms of concentration per 100 g and apparent consumption, and provide future research pathways that we consider of high importance for aquatic food nutrition. Overall, our study provides a framework to select focal nutrients in aquatic food research and ensures a methodical approach to quantifying the importance of aquatic foods for nutrition security and public health.

Estimating national and subnational nutrient intake distributions of global diets

BACKGROUND

Access to high-quality dietary intake data is central to many nutrition, epidemiology, economic, environmental, and policy applications. When data on individual nutrient intakes are available, they have not been consistently disaggregated by sex and age groups, and their parameters and full distributions are often not publicly available.

OBJECTIVES

We sought to derive usual intake distributions for as many nutrients and population subgroups as possible, use these distributions to estimate nutrient intake inadequacy, compare these distributions and evaluate the implications of their shapes on the estimation of inadequacy, and make these distributions publicly available.

METHODS

We compiled dietary data sets from 31 geographically diverse countries, modeled usual intake distributions for 32 micronutrients and 21 macronutrients, and disaggregated these distributions by sex and age groups. We compared the variability and skewness of the distributions and evaluated their similarity across countries, sex, and age groups. We estimated intake inadequacy for 16 nutrients based on a harmonized set of nutrient requirements and bioavailability estimates. Last, we created an R package-nutriR-to make these distributions freely available for users to apply in their own analyses.

RESULTS

Usual intake distributions were rarely symmetric and differed widely in variability and skewness across nutrients and countries. Vitamin intake distributions were more variable and skewed and exhibited less similarity among countries than other nutrients. Inadequate intakes were high and geographically concentrated, as well as generally higher for females than males. We found that the shape of usual intake distributions strongly affects estimates of the prevalence of inadequate intakes.

CONCLUSIONS

The shape of nutrient intake distributions differs based on nutrient and subgroup and strongly influences estimates of nutrient intake inadequacy. This research represents an important contribution to the availability and application of dietary intake data for diverse subpopulations around the world.

Optimising traceability in trade for live animals and animal products with digital technologies

Sound animal traceability systems and supply chain management rely on data and information to respond to outcomes that will both protect animal and human health and facilitate trade. Digital technologies present opportunities and new methods for identifying and tracking animals, collecting more data, integrating communication flows, sharing data securely in supply chains, and analysing data to inform decisions and predict outcomes. Together, these technologies drive more efficient, productive and traceable supply chains, which can help to build more effective animal traceability systems. In addition, they can improve monitoring of, and response to, animal disease, food safety risks and food fraud risks; ensure compliance with animal health and food safety standards; simplify border procedures; facilitate trade with less friction; and raise consumer awareness. As the cost of these technologies decline and they become more accessible, the implementation of a digitally enabled animal traceability system will require an increase in supply chain capacity, improvements in digital infrastructure, and the development of a regulatory framework of standards and policies. Ensuring that these requirements are met will require strong commitment from governments, intergovernmental organisations and the wider animal health community.

The Global Nutrient Database: availability of macronutrients and micronutrients in 195 countries from 1980 to 2013

BACKGROUND

Few data are available on the supply and consumption of nutrients at the country level. To address this data gap, we aimed to create a database that provides information on availability (ie, supply) of 156 nutrients across 195 countries and territories from 1980 to 2013.

METHODS

We matched 394 food and agricultural commodities from the Food and Agriculture Organization of the United Nations Supply and Utilization Accounts (SUAs) to food items in the United States Department of Agriculture Food Composition Database and obtained data on nutrient composition of the SUAs' food items. Then, after adjusting for inedible portion of each food item, we added the contributions of individual food items to the availability of each nutrient and estimated the national availability of macronutrients and micronutrients in each year. We validated our estimates by comparing our results with those of national nutrition surveys from three countries (the USA, South Korea, and Ecuador). Using dietary consumption data from the Global Burden of Disease study and two popular machine learning algorithms (Random Forest and XGBoost [extreme gradient boosting]), we developed predictive models to estimate the consumption of each nutrient based on their national availability.

FINDINGS

Globally 2710 kcal (95% UI 2660-2770) were available per person per day in 2013. Carbohydrates were the major contributor to energy availability (70·5%), followed by fats (17·4%), and protein (10·5%). The energy availability and the contribution of macronutrients to total energy widely varied across levels of development. Countries at the higher level of development (high Socio-demographic Index countries) had more energy available per person per day (3270 kcal, 3220-3310); greater contributions from fats (26·0%) and proteins (11·9%) to total energy availability; and lower contributions from carbohydrate (54·8%). During 1980-2013, energy availability and the contributions of protein and fats to energy availability have increased globally and across levels of development while the contribution of carbohydrates to total energy availability has decreased. The supply of the micronutrients has also increased during the same period globally and across levels of development. Our validation analysis showed that, after accounting for waste at the retail and household level, our estimates of macronutrient availability were very close to the consumption data in nationally representative surveys. Our machine-learning models closely predicted the observed intake of nutrients with the out-of-sample correlation of greater than 0·8 between predicted and observed intake for the nutrients included in the analysis.

INTERPRETATION

Our global nutrient database provides a picture of the supply of various nutrients at the country level and can be useful to assess the performance of national food systems in addressing the nutritional needs of their population.

FUNDING

Bill & Melinda Gates Foundation.

Health Effects of Overweight and Obesity in 195 Countries over 25 Years

BACKGROUND

Although the rising pandemic of obesity has received major attention in many countries, the effects of this attention on trends and the disease burden of obesity remain uncertain.

METHODS

We analyzed data from 68.5 million persons to assess the trends in the prevalence of overweight and obesity among children and adults between 1980 and 2015. Using the Global Burden of Disease study data and methods, we also quantified the burden of disease related to high body-mass index (BMI), according to age, sex, cause, and BMI in 195 countries between 1990 and 2015.

RESULTS

In 2015, a total of 107.7 million children and 603.7 million adults were obese. Since 1980, the prevalence of obesity has doubled in more than 70 countries and has continuously increased in most other countries. Although the prevalence of obesity among children has been lower than that among adults, the rate of increase in childhood obesity in many countries has been greater than the rate of increase in adult obesity. High BMI accounted for 4.0 million deaths globally, nearly 40% of which occurred in persons who were not obese. More than two thirds of deaths related to high BMI were due to cardiovascular disease. The disease burden related to high BMI has increased since 1990; however, the rate of this increase has been attenuated owing to decreases in underlying rates of death from cardiovascular disease.

CONCLUSIONS

The rapid increase in the prevalence and disease burden of elevated BMI highlights the need for continued focus on surveillance of BMI and identification, implementation, and evaluation of evidence-based interventions to address this problem. (Funded by the Bill and Melinda Gates Foundation.).

Measuring the health-related Sustainable Development Goals in 188 countries: a baseline analysis from the Global Burden of Disease Study 2015

BACKGROUND

In September, 2015, the UN General Assembly established the Sustainable Development Goals (SDGs). The SDGs specify 17 universal goals, 169 targets, and 230 indicators leading up to 2030. We provide an analysis of 33 health-related SDG indicators based on the Global Burden of Diseases, Injuries, and Risk Factors Study 2015 (GBD 2015).

METHODS

We applied statistical methods to systematically compiled data to estimate the performance of 33 health-related SDG indicators for 188 countries from 1990 to 2015. We rescaled each indicator on a scale from 0 (worst observed value between 1990 and 2015) to 100 (best observed). Indices representing all 33 health-related SDG indicators (health-related SDG index), health-related SDG indicators included in the Millennium Development Goals (MDG index), and health-related indicators not included in the MDGs (non-MDG index) were computed as the geometric mean of the rescaled indicators by SDG target. We used spline regressions to examine the relations between the Socio-demographic Index (SDI, a summary measure based on average income per person, educational attainment, and total fertility rate) and each of the health-related SDG indicators and indices.

FINDINGS

In 2015, the median health-related SDG index was 59·3 (95% uncertainty interval 56·8-61·8) and varied widely by country, ranging from 85·5 (84·2-86·5) in Iceland to 20·4 (15·4-24·9) in Central African Republic. SDI was a good predictor of the health-related SDG index (r2=0·88) and the MDG index (r2=0·92), whereas the non-MDG index had a weaker relation with SDI (r2=0·79). Between 2000 and 2015, the health-related SDG index improved by a median of 7·9 (IQR 5·0-10·4), and gains on the MDG index (a median change of 10·0 [6·7-13·1]) exceeded that of the non-MDG index (a median change of 5·5 [2·1-8·9]). Since 2000, pronounced progress occurred for indicators such as met need with modern contraception, under-5 mortality, and neonatal mortality, as well as the indicator for universal health coverage tracer interventions. Moderate improvements were found for indicators such as HIV and tuberculosis incidence, minimal changes for hepatitis B incidence took place, and childhood overweight considerably worsened.

INTERPRETATION

GBD provides an independent, comparable avenue for monitoring progress towards the health-related SDGs. Our analysis not only highlights the importance of income, education, and fertility as drivers of health improvement but also emphasises that investments in these areas alone will not be sufficient. Although considerable progress on the health-related MDG indicators has been made, these gains will need to be sustained and, in many cases, accelerated to achieve the ambitious SDG targets. The minimal improvement in or worsening of health-related indicators beyond the MDGs highlight the need for additional resources to effectively address the expanded scope of the health-related SDGs.

FUNDING

Bill & Melinda Gates Foundation.

Global, regional, and national comparative risk assessment of 79 behavioural, environmental and occupational, and metabolic risks or clusters of risks, 1990-2015: a systematic analysis for the Global Burden of Disease Study 2015

BACKGROUND

The Global Burden of Diseases, Injuries, and Risk Factors Study 2015 provides an up-to-date synthesis of the evidence for risk factor exposure and the attributable burden of disease. By providing national and subnational assessments spanning the past 25 years, this study can inform debates on the importance of addressing risks in context.

METHODS

We used the comparative risk assessment framework developed for previous iterations of the Global Burden of Disease Study to estimate attributable deaths, disability-adjusted life-years (DALYs), and trends in exposure by age group, sex, year, and geography for 79 behavioural, environmental and occupational, and metabolic risks or clusters of risks from 1990 to 2015. This study included 388 risk-outcome pairs that met World Cancer Research Fund-defined criteria for convincing or probable evidence. We extracted relative risk and exposure estimates from randomised controlled trials, cohorts, pooled cohorts, household surveys, census data, satellite data, and other sources. We used statistical models to pool data, adjust for bias, and incorporate covariates. We developed a metric that allows comparisons of exposure across risk factors-the summary exposure value. Using the counterfactual scenario of theoretical minimum risk level, we estimated the portion of deaths and DALYs that could be attributed to a given risk. We decomposed trends in attributable burden into contributions from population growth, population age structure, risk exposure, and risk-deleted cause-specific DALY rates. We characterised risk exposure in relation to a Socio-demographic Index (SDI).

FINDINGS

Between 1990 and 2015, global exposure to unsafe sanitation, household air pollution, childhood underweight, childhood stunting, and smoking each decreased by more than 25%. Global exposure for several occupational risks, high body-mass index (BMI), and drug use increased by more than 25% over the same period. All risks jointly evaluated in 2015 accounted for 57·8% (95% CI 56·6-58·8) of global deaths and 41·2% (39·8-42·8) of DALYs. In 2015, the ten largest contributors to global DALYs among Level 3 risks were high systolic blood pressure (211·8 million [192·7 million to 231·1 million] global DALYs), smoking (148·6 million [134·2 million to 163·1 million]), high fasting plasma glucose (143·1 million [125·1 million to 163·5 million]), high BMI (120·1 million [83·8 million to 158·4 million]), childhood undernutrition (113·3 million [103·9 million to 123·4 million]), ambient particulate matter (103·1 million [90·8 million to 115·1 million]), high total cholesterol (88·7 million [74·6 million to 105·7 million]), household air pollution (85·6 million [66·7 million to 106·1 million]), alcohol use (85·0 million [77·2 million to 93·0 million]), and diets high in sodium (83·0 million [49·3 million to 127·5 million]). From 1990 to 2015, attributable DALYs declined for micronutrient deficiencies, childhood undernutrition, unsafe sanitation and water, and household air pollution; reductions in risk-deleted DALY rates rather than reductions in exposure drove these declines. Rising exposure contributed to notable increases in attributable DALYs from high BMI, high fasting plasma glucose, occupational carcinogens, and drug use. Environmental risks and childhood undernutrition declined steadily with SDI; low physical activity, high BMI, and high fasting plasma glucose increased with SDI. In 119 countries, metabolic risks, such as high BMI and fasting plasma glucose, contributed the most attributable DALYs in 2015. Regionally, smoking still ranked among the leading five risk factors for attributable DALYs in 109 countries; childhood underweight and unsafe sex remained primary drivers of early death and disability in much of sub-Saharan Africa.

INTERPRETATION

Declines in some key environmental risks have contributed to declines in critical infectious diseases. Some risks appear to be invariant to SDI. Increasing risks, including high BMI, high fasting plasma glucose, drug use, and some occupational exposures, contribute to rising burden from some conditions, but also provide opportunities for intervention. Some highly preventable risks, such as smoking, remain major causes of attributable DALYs, even as exposure is declining. Public policy makers need to pay attention to the risks that are increasingly major contributors to global burden.

FUNDING

Bill & Melinda Gates Foundation.

The Contribution of Agriculture, Forestry and other Land Use activities to Global Warming, 1990-2012

We refine the information available through the IPCC AR5 with regard to recent trends in global GHG emissions from agriculture, forestry and other land uses (AFOLU), including global emission updates to 2012. Using all three available AFOLU datasets employed for analysis in the IPCC AR5, rather than just one as done in the IPCC AR5 WGIII Summary for Policy Makers, our analyses point to a down-revision of global AFOLU shares of total anthropogenic emissions, while providing important additional information on subsectoral trends. Our findings confirm that the share of AFOLU emissions to the anthropogenic total declined over time. They indicate a decadal average of 28.7 ± 1.5% in the 1990s and 23.6 ± 2.1% in the 2000s and an annual value of 21.2 ± 1.5% in 2010. The IPCC AR5 had indicated a 24% share in 2010. In contrast to previous decades, when emissions from land use (land use, land use change and forestry, including deforestation) were significantly larger than those from agriculture (crop and livestock production), in 2010 agriculture was the larger component, contributing 11.2 ± 0.4% of total GHG emissions, compared to 10.0 ± 1.2% of the land use sector. Deforestation was responsible for only 8% of total anthropogenic emissions in 2010, compared to 12% in the 1990s. Since 2010, the last year assessed by the IPCC AR5, new FAO estimates indicate that land use emissions have remained stable, at about 4.8 Gt CO₂ eq yr^-1 in 2012. Emissions minus removals have also remained stable, at 3.2 Gt CO₂ eq yr^-1 in 2012. By contrast, agriculture emissions have continued to grow, at roughly 1% annually, and remained larger than the land use sector, reaching 5.4 Gt CO₂ eq yr^-1 in 2012. These results are useful to further inform the current climate policy debate on land use, suggesting that more efforts and resources should be directed to further explore options for mitigation in agriculture, much in line with the large efforts devoted to REDD+ in the past decade.

Global food security under climate change

This article reviews the potential impacts of climate change on food security. It is found that of the four main elements of food security, i.e., availability, stability, utilization, and access, only the first is routinely addressed in simulation studies. To this end, published results indicate that the impacts of climate change are significant, however, with a wide projected range (between 5 million and 170 million additional people at risk of hunger by 2080) strongly depending on assumed socio-economic development. The likely impacts of climate change on the other important dimensions of food security are discussed qualitatively, indicating the potential for further negative impacts beyond those currently assessed with models. Finally, strengths and weaknesses of current assessment studies are discussed, suggesting improvements and proposing avenues for new analyses.

Introductory lecture the epidemiology and determinants of obesity in developed and developing countries

Obesity is recognized as a serious problem in the industrialized and developed countries of the world. However, little attention is paid to the fact that obesity is becoming an increasing problem in developing countries too, with some countries showing increasing rates of obesity in the midst of the persisting occurrence of childhood malnutrition and stunting. As developing countries embrace the dominant western economic ways of development, industrialization and urbanization they contribute to improvements in living standards, with consequent dramatic changes in diets and lifestyles leading to weight gain and obesity which in turn poses a growing threat to the health. Overweight and obesity is associated with an increased likelihood of non-insulin dependent diabetes mellitus, hypertension, hyper-lipidaemia, and cardiovascular disease. It is also associated with increased rates of breast, colo-rectal and uterine cancer. Obesity is thus an important factor in the increasing morbidity and mortality due to chronic, non-communicable diseases (NCDs) and thereby contributes to premature mortality in the population. Thus, while the problem of undernutrition persists in much of the developing world, overweight and obesity and its related co-morbidities are posing an increasingly important public health problem both in the developed and developing world.

The changing structure of diets in the European Union in relation to healthy eating guidelines

Objective

Our objective in this paper is to assess diets in the European Union (EU) in relation to the recommendations of the recent World Health Organization/Food and Agriculture Organization expert consultation and to show how diets have changed between 1961 and 2001.

Data and methods

Computations make use of FAOSTAT data on food availability at country level linked to a food composition database to convert foods to nutrients. We further explore the growing similarity of diets in the EU by making use of a consumption similarity index. The index provides a single number measure of dietary overlap between countries.

Results

The data confirm the excessive consumption by almost all countries of saturated fats, cholesterol and sugars, and the convergence of nutrient intakes across the EU. Whereas in 1961 diets in several European countries were more similar to US diets than to those of other European countries, this is no longer the case; moreover, while EU diets have become more homogeneous, the EU as a whole and the USA have become less similar over time.

Conclusions

Although the dominant cause of greater similarity in EU diets over the period studied is increased intakes in Mediterranean countries of saturated fats, cholesterol and sugar, also important are reductions in saturated fat and sugar in some Northern European countries. This suggests that healthy eating messages are finally having an impact on diets; a distinctly European diet may also be emerging.

Learning nonregular languages: a comparison of simple recurrent networks and LSTM

In response to Rodriguez's recent article (2001), we compare the performance of simple recurrent nets and long short-term memory recurrent nets on context-free and context-sensitive languages.

Learning to forget: continual prediction with LSTM

Long short-term memory (LSTM; Hochreiter & Schmidhuber, 1997) can solve numerous tasks not solvable by previous learning algorithms for recurrent neural networks (RNNs). We identify a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset. Without resets, the state may grow indefinitely and eventually cause the network to break down. Our remedy is a novel, adaptive "forget gate" that enables an LSTM cell to learn to reset itself at appropriate times, thus releasing internal resources. We review illustrative benchmark problems on which standard LSTM outperforms other RNN algorithms. All algorithms (including LSTM) fail to solve continual versions of these problems. LSTM with forget gates, however, easily solves them, and in an elegant way.

Processing images by semi-linear predictability minimization

In the predictability minimization approach, input patterns are fed into a system consisting of adaptive, initially unstructured feature detectors. There are also adaptive predictors constantly trying to predict current feature detector outputs from other feature detector outputs. Simultaneously, however, the feature detectors try to become as unpredictable as possible, resulting in a co-evolution of predictors and feature detectors. This paper describes the implementation of a visual processing system trained by semi-linear predictability minimization, and presents many experiments that examine its response to artificial and real-world images. In particular, we observe that under a wide variety of conditions, predictability minimization results in the development of well-known visual feature detectors.

Feature extraction through LOCOCODE

Low-complexity coding and decoding (LOCOCODE) is a novel approach to sensory coding and unsupervised learning. Unlike previous methods, it explicitly takes into account the information-theoretic complexity of the code generator. It computes lococodes that convey information about the input data and can be computed and decoded by low-complexity mappings. We implement LOCOCODE by training autoassociators with flat minimum search, a recent, general method for discovering low-complexity neural nets. It turns out that this approach can unmix an unknown number of independent data sources by extracting a minimal number of low-complexity features necessary for representing the data. Experiments show that unlike codes obtained with standard autoencoders, lococodes are based on feature detectors, never unstructured, usually sparse, and sometimes factorial or local (depending on statistical properties of the data). Although LOCOCODE is not explicitly designed to enforce sparse or factorial codes, it extracts optimal codes for difficult versions of the "bars" benchmark problem, whereas independent component analysis (ICA) and principal component analysis (PCA) do not. It produces familiar, biologically plausible feature detectors when applied to real-world images and codes with fewer bits per pixel than ICA and PCA. Unlike ICA, it does not need to know the number of independent sources. As a preprocessor for a vowel recognition benchmark problem, it sets the stage for excellent classification performance. Our results reveal an interesting, previously ignored connection between two important fields: regularizer research and ICA-related research. They may represent a first step toward unification of regularization and unsupervised learning.

Long short-term memory

Learning to store information over extended time intervals by recurrent backpropagation takes a very long time, mostly because of insufficient, decaying error backflow. We briefly review Hochreiter's (1991) analysis of this problem, then address it by introducing a novel, efficient, gradient-based method called long short-term memory (LSTM). Truncating the gradient where this does not do harm, LSTM can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units. Multiplicative gate units learn to open and close access to the constant error flow. LSTM is local in space and time; its computational complexity per time step and weight is O(1). Our experiments with artificial data involve local, distributed, real-valued, and noisy pattern representations. In comparisons with real-time recurrent learning, back propagation through time, recurrent cascade correlation, Elman nets, and neural sequence chunking, LSTM leads to many more successful runs, and learns much faster. LSTM also solves complex, artificial long-time-lag tasks that have never been solved by previous recurrent network algorithms.

Long short-term memory

Learning to store information over extended time intervals by recurrent backpropagation takes a very long time, mostly because of insufficient, decaying error backflow. We briefly review Hochreiter's (1991) analysis of this problem, then address it by introducing a novel, efficient, gradient-based method called long short-term memory (LSTM). Truncating the gradient where this does not do harm, LSTM can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units. Multiplicative gate units learn to open and close access to the constant error flow. LSTM is local in space and time; its computational complexity per time step and weight is O(1). Our experiments with artificial data involve local, distributed, real-valued, and noisy pattern representations. In comparisons with real-time recurrent learning, back propagation through time, recurrent cascade correlation, Elman nets, and neural sequence chunking, LSTM leads to many more successful runs, and learns much faster. LSTM also solves complex, artificial long-time-lag tasks that have never been solved by previous recurrent network algorithms.

Probabilistic incremental program evolution

Probabilistic incremental program evolution (PIPE) is a novel technique for automatic program synthesis. We combine probability vector coding of program instructions, population-based incremental learning, and tree-coded programs like those used in some variants of genetic programming (GP). PIPE iteratively generates successive populations of functional programs according to an adaptive probability distribution over all possible programs. Each iteration, it uses the best program to refine the distribution. Thus, it stochastically generates better and better programs. Since distribution refinements depend only on the best program of the current population, PIPE can evaluate program populations efficiently when the goal is to discover a program with minimal runtime. We compare PIPE to GP on a function regression problem and the 6-bit parity problem. We also use PIPE to solve tasks in partially observable mazes, where the best programs have minimal runtime.

Flat minima

We present a new algorithm for finding low-complexity neural networks with high generalization capability. The algorithm searches for a "flat" minimum of the error function. A flat minimum is a large connected region in weight space where the error remains approximately constant. An MDL-based, Bayesian argument suggests that flat minima correspond to "simple" networks and low expected overfitting. The argument is based on a Gibbs algorithm variant and a novel way of splitting generalization error into underfitting and overfitting error. Unlike many previous approaches, ours does not require gaussian assumptions and does not depend on a "good" weight prior. Instead we have a prior over input-output functions, thus taking into account net architecture and training set. Although our algorithm requires the computation of second-order derivatives, it has backpropagation's order of complexity. Automatically, it effectively prunes units, weights, and input lines. Various experiments with feedforward and recurrent nets are described. In an application to stock market prediction, flat minimum search outperforms conventional backprop, weight decay, and "optimal brain surgeon/optimal brain damage".