Replications in Comparative Cognition: What Should We Expect and How Can We Improve?

No immediate attentional bias towards or choice bias for male secondary sexual characteristics in Bornean orang-utans (Pongo pygmaeus)

Primate faces provide information about a range of variant and invariant traits, including some that are relevant for mate choice. For example, faces of males may convey information about their health or genetic quality through symmetry or facial masculinity. Because perceiving and processing such information may have bearing on the reproductive success of an individual, cognitive systems are expected to be sensitive to facial cues of mate quality. However, few studies have investigated this topic in non-human primate species. Orang-utans are an interesting species to test mate-relevant cognitive biases, because they are characterised by male bimaturism: some adult males are fully developed and bear conspicuous flanges on the side of their face, while other males look relatively similar to females. Here, we describe two non-invasive computerised experiments with Bornean orang-utans (Pongo pygmaeus), testing (i) immediate attention towards large flanges and symmetrical faces using a dot-probe task (N = 3 individuals; 2F) and (ii) choice bias for pictures of flanged males over unflanged males using a preference test (N = 6 individuals; 4F). In contrast with our expectations, we found no immediate attentional bias towards either large flanges or symmetrical faces. In addition, individuals did not show a choice bias for stimuli of flanged males. We did find exploratory evidence for a colour bias and energy efficiency trade-offs in the preference task. We discuss our null results and exploratory results in the context of the evolutionary history of Bornean orang-utans, and provide suggestions for a more biocentric approach to the study of orang-utan cognition.

Repeatability and heritability of inhibitory control performance in wild toutouwai (Petroica longipes)

Despite increasing interest in the evolution of inhibitory control, few studies have examined the validity of widespread testing paradigms, the long-term repeatability and the heritability of this cognitive ability in the wild. We investigated these aspects in the inhibitory control performance of wild toutouwai (North Island robin; Petroica longipes), using detour and reversal learning tasks. We assessed convergent validity by testing whether individual performance correlated across detour and reversal learning tasks. We then further evaluated task validity by examining whether individual performance was confounded by non-cognitive factors. We tested a subset of subjects twice in each task to estimate the repeatability of performance across a 1-year period. Finally, we used a population pedigree to estimate the heritability of task performance. Individual performance was unrelated across detour and reversal learning tasks, indicating that these measured different cognitive abilities. Task performance was not influenced by body condition, boldness or prior experience, and showed moderate between-year repeatability. Yet despite this individual consistency, we found no evidence that task performance was heritable. Our findings suggest that detour and reversal learning tasks measure consistent individual differences in distinct forms of inhibitory control in toutouwai, but this variation may be environmentally determined rather than genetic.

A Touchscreen-Based, Multiple-Choice Approach to Cognitive Enrichment of Captive Rhesus Macaques (Macaca mulatta)

Research on the psychological and physiological well-being of captive animals has focused on investigating different types of social and structural enrichment. Consequently, cognitive enrichment has been understudied, despite the promising external validity, comparability, and applicability. As we aim to fill this gap, we developed an interactive, multiple-choice interface for cage-mounted touchscreen devices that rhesus monkeys (Macaca mulatta) can freely interact with, from within their home enclosure at the Cognitive Neuroscience Laboratory of the German Primate Center. The multiple-choice interface offers interchangeable activities that animals can choose and switch between. We found that all 16 captive rhesus macaques tested consistently engaged with the multiple-choice interface across 6 weekly sessions, with 11 of them exhibiting clear task preferences, and displaying proficiency in performing the selected tasks. Our approach does not require social separation or dietary restriction and is intended to increase animals' sense of competence and agency by providing them with more control over their environment. Thanks to the high level of automation, our multiple-choice interface can be easily incorporated as a standard cognitive enrichment practice across different facilities and institutes working with captive animals, particularly non-human primates. We believe that the multiple-choice interface is a sustainable, scalable, and pragmatic protocol for enhancing cognitive well-being and animal welfare in captivity.

Great ape cognition is structured by stable cognitive abilities and predicted by developmental conditions

Great ape cognition is used as a reference point to specify the evolutionary origins of complex cognitive abilities, including in humans. This research often assumes that great ape cognition consists of cognitive abilities (traits) that account for stable differences between individuals, which change and develop in response to experience. Here, we test the validity of these assumptions by assessing repeatability of cognitive performance among captive great apes (Gorilla gorilla, Pongo abelii, Pan paniscus, Pan troglodytes) in five tasks covering a range of cognitive domains. We examine whether individual characteristics (age, group, test experience) or transient situational factors (life events, testing arrangements or sociality) influence cognitive performance. Our results show that task-level performance is generally stable over time; four of the five tasks were reliable measurement tools. Performance in the tasks was best explained by stable differences in cognitive abilities (traits) between individuals. Cognitive abilities were further correlated, suggesting shared cognitive processes. Finally, when predicting cognitive performance, we found stable individual characteristics to be more important than variables capturing transient experience. Taken together, this study shows that great ape cognition is structured by stable cognitive abilities that respond to different developmental conditions.

Teaching open and reproducible scholarship: a critical review of the evidence base for current pedagogical methods and their outcomes

In recent years, the scientific community has called for improvements in the credibility, robustness and reproducibility of research, characterized by increased interest and promotion of open and transparent research practices. While progress has been positive, there is a lack of consideration about how this approach can be embedded into undergraduate and postgraduate research training. Specifically, a critical overview of the literature which investigates how integrating open and reproducible science may influence student outcomes is needed. In this paper, we provide the first critical review of literature surrounding the integration of open and reproducible scholarship into teaching and learning and its associated outcomes in students. Our review highlighted how embedding open and reproducible scholarship appears to be associated with (i) students' scientific literacies (i.e. students' understanding of open research, consumption of science and the development of transferable skills); (ii) student engagement (i.e. motivation and engagement with learning, collaboration and engagement in open research) and (iii) students' attitudes towards science (i.e. trust in science and confidence in research findings). However, our review also identified a need for more robust and rigorous methods within pedagogical research, including more interventional and experimental evaluations of teaching practice. We discuss implications for teaching and learning scholarship.

A theoretical approach to improving interspecies welfare comparisons

The number of animals bred, raised, and slaughtered each year is on the rise, resulting in increasing impacts to welfare. Farmed animals are also becoming more diverse, ranging from pigs to bees. The diversity and number of species farmed invite questions about how best to allocate currently limited resources towards safeguarding and improving welfare. This is of the utmost concern to animal welfare funders and effective altruism advocates, who are responsible for targeting the areas most likely to cause harm. For example, is tail docking worse for pigs than beak trimming is for chickens in terms of their pain, suffering, and general experience? Or are the welfare impacts equal? Answering these questions requires making an interspecies welfare comparison; a judgment about how good or bad different species fare relative to one another. Here, we outline and discuss an empirical methodology that aims to improve our ability to make interspecies welfare comparisons by investigating welfare range, which refers to how good or bad animals can fare. Beginning with a theory of welfare, we operationalize that theory by identifying metrics that are defensible proxies for measuring welfare, including cognitive, affective, behavioral, and neuro-biological measures. Differential weights are assigned to those proxies that reflect their evidential value for the determinants of welfare, such as the Delphi structured deliberation method with a panel of experts. The evidence should then be reviewed and its quality scored to ascertain whether particular taxa may possess the proxies in question to construct a taxon-level welfare range profile. Finally, using a Monte Carlo simulation, an overall estimate of comparative welfare range relative to a hypothetical index species can be generated. Interspecies welfare comparisons will help facilitate empirically informed decision-making to streamline the allocation of resources and ultimately better prioritize and improve animal welfare.

Reporting and interpreting non-significant results in animal cognition research

How statistically non-significant results are reported and interpreted following null hypothesis significance testing is often criticized. This issue is important for animal cognition research because studies in the field are often underpowered to detect theoretically meaningful effect sizes, i.e., often produce non-significant p-values even when the null hypothesis is incorrect. Thus, we manually extracted and classified how researchers report and interpret non-significant p-values and examined the p-value distribution of these non-significant results across published articles in animal cognition and related fields. We found a large amount of heterogeneity in how researchers report statistically non-significant p-values in the result sections of articles, and how they interpret them in the titles and abstracts. Reporting of the non-significant results as "No Effect" was common in the titles (84%), abstracts (64%), and results sections (41%) of papers, whereas reporting of the results as "Non-Significant" was less common in the titles (0%) and abstracts (26%), but was present in the results (52%). Discussions of effect sizes were rare (<5% of articles). A p-value distribution analysis was consistent with research being performed with low power of statistical tests to detect effect sizes of interest. These findings suggest that researchers in animal cognition should pay close attention to the evidence used to support claims of absence of effects in the literature, and-in their own work-report statistically non-significant results clearly and formally correct, as well as use more formal methods of assessing evidence against theoretical predictions.

Learning with lacertids: Studying the link between ecology and cognition within a comparative framework

Cognition is an essential tool for animals to deal with environmental challenges. Nonetheless, the ecological forces driving the evolution of cognition throughout the animal kingdom remain enigmatic. Large-scale comparative studies on multiple species and cognitive traits have been advanced as the best way to facilitate our understanding of cognitive evolution, but such studies are rare. Here, we tested 13 species of lacertid lizards (Reptilia: Lacertidae) using a battery of cognitive tests measuring inhibitory control, problem-solving, and spatial and reversal learning. Next, we tested the relationship between species' performance and (a) resource availability (temperature and precipitation), habitat complexity (Normalized Difference Vegetation Index), and habitat variability (seasonality) in their natural habitat and (b) their life history (size at hatching and maturity, clutch size, and frequency). Although species differed markedly in their cognitive abilities, such variation was mostly unrelated to their ecology and life history. Yet, species living in more variable environments exhibited lower behavioral flexibility, likely due to energetic constrains in such habitats. Our standardized protocols provide opportunities for collaborative research, allowing increased sample sizes and replication, essential for moving forward in the field of comparative cognition. Follow-up studies could include more detailed measures of habitat structure and look at other potential selective drivers such as predation.

Episodic-like memory in common bottlenose dolphins

Episodic memory involves the conscious recollection of personally experienced events, which has often been argued to be a uniquely human ability.^1-5 However, evidence for conscious episodic recall in humans is centered around language-based reports. With no agreed upon non-linguistic behavioral makers of consciousness,⁶ episodic-like memory⁷ therefore represents the behavioral characteristics of human episodic memory, in the absence of evidence for subjective experience during recall. Here, we provide compelling evidence for episodic-like memory in common bottlenose dolphins (Tursiops truncatus), based on the incidental encoding and unexpected question paradigm.⁸ This methodology aims to capture the incidental encoding characteristic of human episodic memory, in that when we recall an experience, we remember information that was trivial at the time of encoding, but was encoded automatically.⁹ We show that dolphins are able to use incidentally encoded spatial ("where") and social ("who") information to solve an unexpected memory task, using only a single test trial per test type, which ensured that the dolphins did not have the opportunity to semantically learn "rules" to pass the test. All participating dolphins made correct choices in both the "where" and "who" tests. These results suggest that dolphins are capable of encoding, recalling, and accessing incidental information within remembered events, which is an ability indicative of episodic memory in humans. We argue that the complex socio-ecological background of dolphins may have selected for the ability to recall both spatial and social information in an episodic-like manner.

Learning and motor inhibitory control in crows and domestic chickens

Cognitive abilities allow animals to navigate through complex, fluctuating environments. In the present study, we tested the performance of a captive group of eight crows, Corvus corone and 10 domestic chickens, Gallus gallus domesticus, in the cylinder task, as a test of motor inhibitory control and reversal learning as a measure of learning ability and behavioural flexibility. Four crows and nine chickens completed the cylinder task, eight crows and six chickens completed the reversal learning experiment. Crows performed better in the cylinder task compared with chickens. In the reversal learning experiment, species did not significantly differ in the number of trials until the learning criterion was reached. The performance in the reversal learning experiment did not correlate with performance in the cylinder task in chickens. Our results suggest crows to possess better motor inhibitory control compared with chickens. By contrast, learning performance in a reversal learning task did not differ between the species, indicating similar levels of behavioural flexibility. Interestingly, we describe notable individual differences in performance. We stress the importance not only to compare cognitive performance between species but also between individuals of the same species when investigating the evolution of cognitive skills.

Little evidence that Eurasian jays protect their caches by responding to cues about a conspecific's desire and visual perspective

Eurasian jays have been reported to protect their caches by responding to cues about either the visual perspective or current desire of an observing conspecific, similarly to other corvids. Here, we used established paradigms to test whether these birds can - like humans - integrate multiple cues about different mental states and perform an optimal response accordingly. Across five experiments, which also include replications of previous work, we found little evidence that our jays adjusted their caching behaviour in line with the visual perspective and current desire of another agent, neither by integrating these social cues nor by responding to only one type of cue independently. These results raise questions about the reliability of the previously reported effects and highlight several key issues affecting reliability in comparative cognition research.

The hidden side of animal cognition research: Scientists' attitudes toward bias, replicability and scientific practice

Animal cognition research aims to understand animal minds by using a diverse range of methods across an equally diverse range of species. Throughout its history, the field has sought to mitigate various biases that occur when studying animal minds, from experimenter effects to anthropomorphism. Recently, there has also been a focus on how common scientific practices might affect the reliability and validity of published research. Usually, these issues are discussed in the literature by a small group of scholars with a specific interest in the topics. This study aimed to survey a wider range of animal cognition researchers to ask about their attitudes towards classic and contemporary issues facing the field. Two-hundred and ten active animal cognition researchers completed our survey, and provided answers on questions relating to bias, replicability, statistics, publication, and belief in animal cognition. Collectively, researchers were wary of bias in the research field, but less so in their own work. Over 70% of researchers endorsed Morgan’s canon as a useful principle but many caveated this in their free-text responses. Researchers self-reported that most of their studies had been published, however they often reported that studies went unpublished because they had negative or inconclusive results, or results that questioned “preferred” theories. Researchers rarely reported having performed questionable research practices themselves—however they thought that other researchers sometimes (52.7% of responses) or often (27.9% of responses) perform them. Researchers near unanimously agreed that replication studies are important but too infrequently performed in animal cognition research, 73.0% of respondents suggested areas of animal cognition research could experience a ‘replication crisis’ if replication studies were performed. Consistently, participants’ free-text responses provided a nuanced picture of the challenges animal cognition research faces, which are available as part of an open dataset. However, many researchers appeared concerned with how to interpret negative results, publication bias, theoretical bias and reliability in areas of animal cognition research. Collectively, these data provide a candid overview of barriers to progress in animal cognition and can inform debates on how individual researchers, as well as organizations and journals, can facilitate robust scientific research in animal cognition.

Where Is Ethology Heading? An Invitation for Collective Metadisciplinary Discussion

Many factors can impact the advancement of scientific disciplines. In the study of animal behavior and cognition (i.e., Ethology), a lack of consensus about definitions or the emergence of some current events and inventions, among other aspects, may challenge the discipline's grounds within the next decades. A collective metadisciplinary discussion may help in envisioning the future to come. For that purpose, I elaborated an online questionnaire about the level of consensus and the researchers' ways of doing in seven areas: Discipline name and concepts, species, Umwelt, technology, data, networking, and the impact of sociocultural and ecological factors. I recruited the opinion of almost a hundred of colleagues worldwide (N = 98), both junior and seniors, working both in the wild and in the lab. While the results were pitted against the literature, general conclusions should be taken with caution and considered as a first attempt in exploring the state of the discipline from the researchers' perspective: There is no unanimity for the discipline's name; 71.4% of the researchers reported there is limited consensus in the definition of relevant concepts (i.e., culture, cognition); primate species still predominate in publications whereas the species selection criteria is sometimes based on fascination, chance, or funding opportunities rather than on biocentric questions; 56.1% of the apparatuses employed do not resemble species' ecological problems, and current tech needs would be solved by fostering collaboration with engineers. Finally, embracing the Open Science paradigm, supporting networking efforts, and promoting diversity in research teams may help in gathering further knowledge in the area. Some suggestions are proposed to overcome the aforementioned problems in this contemporary analysis of our discipline.

Extraordinary claims, extraordinary evidence? A discussion

Roberts (2020, Learning & Behavior, 48[2], 191-192) discussed research claiming honeybees can do arithmetic. Some readers of this research might regard such claims as unlikely. The present authors used this example as a basis for a debate on the criterion that ought to be used for publication of results or conclusions that could be viewed as unlikely by a significant number of readers, editors, or reviewers.

Extending the Reach of Tooling Theory: A Neurocognitive and Phylogenetic Perspective

Tool use research has suffered from a lack of consistent theoretical frameworks. There is a plethora of tool use definitions and the most widespread ones are so inclusive that the behaviors that fall under them arguably do not have much in common. The situation is aggravated by the prevalence of anecdotes, which have played an undue role in the literature. In order to provide a more rigorous foundation for research and to advance our understanding of the interrelation between tool use and cognition, we suggest the adoption of Fragaszy and Mangalam's (2018) tooling framework, which is characterized by the creation of a body-plus-object system that manages a mechanical interface between tool and surface. Tooling is limited to a narrower suite of behaviors than tool use, which might facilitate its neurocognitive investigation. Indeed, evidence in the literature indicates that tooling has distinct neurocognitive underpinnings not shared by other activities typically classified as tool use, at least in primates. In order to understand the extent of tooling incidences in previous research, we systematically surveyed the comprehensive tool use catalog by Shumaker et al. (2011). We identified 201 tool use submodes, of which only 81 could be classified as tooling, and the majority of the tool use examples across species were poorly supported by evidence. Furthermore, tooling appears to be phylogenetically less widespread than tool use, with the greatest variability found in the primate order. However, in order to confirm these findings and to understand the evolution and neurocognitive mechanisms of tooling, more systematic research will be required in the future, particularly with currently underrepresented taxa.

Replications, Comparisons, Sampling and the Problem of Representativeness in Animal Cognition Research

Animal cognition research often involves small and idiosyncratic samples. This can constrain the generalizability and replicability of a study's results and prevent meaningful comparisons between samples. However, there is little consensus about what makes a strong replication or comparison in animal research. We apply a resampling definition of replication to answer these questions in Part 1 of this article, and, in Part 2, we focus on the problem of representativeness in animal research. Through a case study and a simulation study, we highlight how and when representativeness may be an issue in animal behavior and cognition research and show how the representativeness problems can be viewed through the lenses of, i) replicability, ii) generalizability and external validity, iii) pseudoreplication and, iv) theory testing. Next, we discuss when and how researchers can improve their ability to learn from small sample research through, i) increasing heterogeneity in experimental design, ii) increasing homogeneity in experimental design, and, iii) statistically modeling variation. Finally, we describe how the strongest solutions will vary depending on the goals and resources of individual research programs and discuss some barriers towards implementing them.

Validity of Cognitive Tests for Non-human Animals: Pitfalls and Prospects

Comparative psychology assesses cognitive abilities and capacities of non-human animals and humans. Based on performance differences and similarities in various species in cognitive tests, it is inferred how their minds work and reconstructed how cognition might have evolved. Critically, such species comparisons are only valid and meaningful if the tasks truly capture individual and inter-specific variation in cognitive abilities rather than contextual variables that might affect task performance. Unlike in human test psychology, however, cognitive tasks for non-human primates (and most other animals) have been rarely evaluated regarding their measurement validity. We review recent studies that address how non-cognitive factors affect performance in a set of commonly used cognitive tasks, and if cognitive tests truly measure individual variation in cognitive abilities. We find that individual differences in emotional and motivational factors primarily affect performance via attention. Hence, it is crucial to systematically control for attention during cognitive tasks to obtain valid and reliable results. Aspects of test design, however, can also have a substantial effect on cognitive performance. We conclude that non-cognitive factors are a minor source of measurement error if acknowledged and properly controlled for. It is essential, however, to validate and eventually re-design several primate cognition tasks in order to ascertain that they capture the cognitive abilities they were designed to measure. This will provide a more solid base for future cognitive comparisons within primates but also across a wider range of non-human animal species.

Trialling Meta-Research in Comparative Cognition: Claims and Statistical Inference in Animal Physical Cognition

Scientific disciplines face concerns about replicability and statistical inference, and these concerns are also relevant in animal cognition research. This paper presents a first attempt to assess how researchers make and publish claims about animal physical cognition, and the statistical inferences they use to support them. We surveyed 116 published experiments from 63 papers on physical cognition, covering 43 different species. The most common tasks in our sample were trap-tube tasks (14 papers), other tool use tasks (13 papers), means-end understanding and string-pulling tasks (11 papers), object choice and object permanence tasks (9 papers) and access tasks (5 papers). This sample is not representative of the full scope of physical cognition research; however, it does provide data on the types of statistical design and publication decisions researchers have adopted. Across the 116 experiments, the median sample size was 7. Depending on the definitions we used, we estimated that between 44% and 59% of our sample of papers made positive claims about animals' physical cognitive abilities, between 24% and 46% made inconclusive claims, and between 10% and 17% made negative claims. Several failures of animals to pass physical cognition tasks were reported. Although our measures had low inter-observer reliability, these findings show that negative results can and have been published in the field. However, publication bias is still present, and consistent with this, we observed a drop in the frequency of p-values above .05. This suggests that some non-significant results have not been published. More promisingly, we found that researchers are likely making many correct statistical inferences at the individual-level. The strength of evidence of statistical effects at the group-level was weaker, and its p-value distribution was consistent with some effect sizes being overestimated. Studies such as ours can form part of a wider investigation into statistical reliability in comparative cognition. However, future work should focus on developing the validity and reliability of the measurements they use, and we offer some starting points.

Macphail’s Null Hypothesis of Vertebrate Intelligence: Insights From Avian Cognition

Macphail famously criticized two foundational assumptions that underlie the evolutionary approach to comparative psychology: that there are differences in intelligence across species, and that intelligent behavior in animals is based on more than associative learning. Here, we provide evidence from recent work in avian cognition that supports both these assumptions: intelligence across species varies, and animals can perform intelligent behaviors that are not guided solely by associative learning mechanisms. Finally, we reflect on the limitations of comparative psychology that led to Macphail’s claims and suggest strategies researchers can use to make more advances in the field.