When logic and belief collide: Individual differences in reasoning times support a selective processing model

Overconfidence in the Cognitive Reflection Test: Comparing Confidence Resolution for Reasoning vs. General Knowledge

This research examines the metacognitive awareness that people have about their reasoning performance in the Cognitive Reflection Test (CRT). The first two studies compare confidence judgments about the CRT vs. general knowledge (GK) questions. Results show that (1) people are generally able to discriminate between correct and incorrect answers, but this ability is far from perfect, and it is greater for GK questions than for CRT problems. Indeed, and strikingly, (2) incorrect responses to CRT problems are produced with approximately the same level of confidence as correct responses to GK questions. However, (3) even though confidence is high for incorrect responses to CRT problems, it is even higher for correct responses. The results of two additional studies show that these differences in confidence are ultimately related to the conflict that CRT problems pose between intuition and deliberation. These findings have implications for the possibility of implicit error monitoring and dual-process models of overconfidence.

The best game in town: The reemergence of the language-of-thought hypothesis across the cognitive sciences

Mental representations remain the central posits of psychology after many decades of scrutiny. However, there is no consensus about the representational format(s) of biological cognition. This paper provides a survey of evidence from computational cognitive psychology, perceptual psychology, developmental psychology, comparative psychology, and social psychology, and concludes that one type of format that routinely crops up is the language-of-thought (LoT). We outline six core properties of LoTs: (i) discrete constituents; (ii) role-filler independence; (iii) predicate-argument structure; (iv) logical operators; (v) inferential promiscuity; and (vi) abstract content. These properties cluster together throughout cognitive science. Bayesian computational modeling, compositional features of object perception, complex infant and animal reasoning, and automatic, intuitive cognition in adults all implicate LoT-like structures. Instead of regarding LoT as a relic of the previous century, researchers in cognitive science and philosophy-of-mind must take seriously the explanatory breadth of LoT-based architectures. We grant that the mind may harbor many formats and architectures, including iconic and associative structures as well as deep-neural-network-like architectures. However, as computational/representational approaches to the mind continue to advance, classical compositional symbolic structures - that is, LoTs - only prove more flexible and well-supported over time.

The bright homunculus in our head: Individual differences in intuitive sensitivity to logical validity

Classic dual process theories of human reasoning attribute explicit reasoning to effortful, deliberative thinking. According to these models, intuitive processes lack any access to the formal rules of logic and probability and hence rely exclusively on superficial problem features to determine a response. However, in recent years, researchers have demonstrated that reasoners are able to solve simple logical or probabilistic problems relatively automatically, a capability which has been called "logical intuition." In four experiments, we instructed participants to judge the validity (Experiments 1 and 4), likeability (Experiments 1, 2, and 3), and physical brightness (Experiments 2, 3, and 4) of the conclusion to several reasoning problems. Brightness judgements were made by evaluating the font shade brightness of the argument's conclusion. Participants were also asked to complete a range of individual differences measures, drawing on cognitive ability and cognitive style, to evaluate the extent to which "logical intuitions" were linked to measures of deliberative reasoning. The results showed that participants judged the conclusion of logically valid statements to be more valid, more likable, and more physically bright than invalid statements. Participants with higher cognitive ability and unlimited processing time showed greater effects of logical validity in their liking judgements (varied across experiments). However, these effects were absent in the brightness tasks, suggesting that logic effects observed under instructions to judge conclusion brightness are a purer measure of "logical intuition." We discuss the implications of our findings for recent dual process theories of human reasoning.

On Dual- and Single-Process Models of Thinking

Popular dual-process models of thinking have long conceived intuition and deliberation as two qualitatively different processes. Single-process-model proponents claim that the difference is a matter of degree and not of kind. Psychologists have been debating the dual-process/single-process question for at least 30 years. In the present article, I argue that it is time to leave the debate behind. I present a critical evaluation of the key arguments and critiques and show that-contra both dual- and single-model proponents-there is currently no good evidence that allows one to decide the debate. Moreover, I clarify that even if the debate were to be solved, it would be irrelevant for psychologists because it does not advance the understanding of the processing mechanisms underlying human thinking.

Belief Bias Effect in Older Adults: Roles of Working Memory and Need for Cognition

Belief bias is the tendency in syllogistic reasoning to rely on prior beliefs rather than to fully obey logical principles. Few studies have investigated the age effect on belief bias. Although several studies have recently begun to explore this topic, little is known about the psychological mechanisms underlying such an effect. Accordingly, we investigated belief bias in older and young adults and explored the roles of working memory (WM) and need for cognition (NFC) in the relationship between age and reasoning performance. We found that older adults showed a lower accuracy rate compared with young adults when conclusion believability and logical validity were incongruent. However, older adults showed a higher accuracy rate compared with young adults when conclusion believability and logical validity were congruent. The results indicated that in comparison with young adults, prior beliefs hampered logical reasoning more significantly in older adults under incongruent conditions and boosted logical reasoning more significantly under congruent conditions. Moreover, the logic index in older adults was significantly lower than in young adults, and the interaction index of believability and validity in older adults was significantly below zero. Furthermore, NFC mediated the age effect on reasoning performance under the two conditions. By contrast, WM mediated the age effect on reasoning performance only under incongruent conditions and did not act as a mediator under congruent conditions.

Who resists belief-biased inferences? The role of individual differences in reasoning strategies, working memory, and attentional focus

A common explanation for individual differences in the ability to draw rule-based inferences, when a putative conclusion suggests a competing belief-based inference, is that the ability to do so depends on working memory capacity. In the following studies, we examined the hypothesis that the ability to draw rule-based inferences in belief bias tasks can also be explained by individual differences in reasoning strategies and in the related attentional focus. The dual-strategy model differentiates counterexample and statistical strategies that involve different information-processing styles. In the first study (N = 139), participants completed a working memory task (operation span), a strategy diagnostic questionnaire, and a belief bias task. The results showed that individual differences in strategy use predicted performance in the belief bias problems over and above any effects of working memory capacity, with counterexample reasoners producing rule-based inferences more often than statistical reasoners. In the second study (N = 196), an eye-tracking methodology was used as a process-tracing technique to investigate attentional differences between the two strategies. On problems showing a conflict between rule-based and belief-based information, counterexample reasoners demonstrated longer fixation times on the premises than did statistical reasoners, thus providing direct evidence that individual differences in strategy use reflect different processing styles. These results clearly indicate that individual differences in strategy use are an important determinant of the way that people make inferences when rule-based and belief-based cues are both present.

How to Get Rid of the Belief Bias: Boosting Analytical Thinking via Pragmatics

The previous research attempts to reduce the influence of the belief bias on deductive thinking have often been unsuccessful and, when they succeeded, they failed to replicate. In this paper, we propose a new way to see an old problem. Instead of considering the analytical abilities of the respondent, we focus on the communicative characteristics of the experimental task. By changing the pragmatics into play through a subtle manipulation of the instruction of the syllogism problem, we obtained a strong improvement in the accuracy of the performance in both untrained and trained in logic respondents. We suggest that current models of deductive thinking should be broadened to consider also communicative understanding as part of the processing of the problem.

Not all who ponder count costs: Arithmetic reflection predicts utilitarian tendencies, but logical reflection predicts both deontological and utilitarian tendencies

Conventional sacrificial moral dilemmas propose directly causing some harm to prevent greater harm. Theory suggests that accepting such actions (consistent with utilitarian philosophy) involves more reflective reasoning than rejecting such actions (consistent with deontological philosophy). However, past findings do not always replicate, confound different kinds of reflection, and employ conventional sacrificial dilemmas that treat utilitarian and deontological considerations as opposite. In two studies, we examined whether past findings would replicate when employing process dissociation to assess deontological and utilitarian inclinations independently. Findings suggested two categorically different impacts of reflection: measures of arithmetic reflection, such as the Cognitive Reflection Test, predicted only utilitarian, not deontological, response tendencies. However, measures of logical reflection, such as performance on logical syllogisms, positively predicted both utilitarian and deontological tendencies. These studies replicate some findings, clarify others, and reveal opportunity for additional nuance in dual process theorist's claims about the link between reflection and dilemma judgments.

A developmental study of the bat/ball problem of CRT: How to override the bias and its relation to executive functioning

In two experiments, we explored the nature of the bias observed in the bat/ball problem of the cognitive reflection test (Frederick, 2005, J. Econ. Perspect., 19, 25), how to override it, and its relation to executive functioning. Based on the original bat/ball problem, we designed two additional isomorphic items. In Experiment 1, for four age groups, including 7-, 11-, and 15-year-olds and adults, we determined that the bias is related to the System 1 intervention; the performance in this item was not a matter of mathematical ability and it could be facilitated by changing the order in which the problems were presented. In Experiment 2, we determined that for 15-year-olds, good and bad performances in the item were related to executive functioning, particularly response-distractor inhibition, updating information in working memory, and the regulation of attention; however, subtle differences were identified when the problem was performed in a facilitative context compared with a non-facilitative context. The results indicated that cognitive abilities are a necessary but non-sufficient condition to resolve the problem.

Developmental frontal brain activation differences in overcoming heuristic bias

Since reasoning is often biased by intuitive heuristics, the development of sound reasoning has long been postulated to depend on successful bias monitoring and inhibition. The present fMRI study aimed to identify neural correlates of developmental changes in these processes. A group of adults and young adolescents were presented with ratio-bias problems in which an intuitively cued heuristic response could be incongruent (conflict item) or congruent (no-conflict item) with the correct response. Results showed that successfully avoiding biased responding on conflict items across both age groups was associated with increased activation in Anterior Cingulate Cortex (ACC) and the right Lateral Prefrontal Cortex (LPFC) regions of interest. Critically, the right LPFC activation decreased with age. Biased responding did not result in right LPFC or ACC modulation and failed to show any developmental activation changes. We discuss implications for ongoing debates on the nature of heuristic bias and its development.

A Cue for Rational Reasoning: Introducing a Reference Point in Cognitive Reflection Tasks

The dual process framework posits that we reason using the quick System 1, and the deliberate System 2, both of which are part of our “adaptive toolbox”. The Cognitive Reflection Test (CRT) estimates which system was used to solve a reasoning problem. Usually, the CRT tasks are solved incorrectly by using System 1, and correctly through System 2. We have applied the reference point hypothesis to the tasks of the CRT and proposed that this change would facilitate the switch between systems, resulting in better performance on the version of the test with a reference point, compared to the CRT without one. The results confirmed our assumptions, as evidenced by a generally higher score on the CRT with a reference point, albeit with different effects between items.

Second-guess: Testing the specificity of error detection in the bat-and-ball problem

In the last decade conflict detection studies in the reasoning and decision-making field have suggested that biased reasoners who give an intuitive response that conflicts with logico-mathematical principles can often detect that their answer is questionable. In the present studies we introduced a second guess paradigm to test the nature and specificity of this error or conflict signal. Participants solved the bat-and-ball problem and were allowed to make a second guess after they had entered their answer. Three studies in which we used a range of second guess elicitation methods show that biased reasoners predominantly give second guesses that are smaller than the intuitively cued heuristic response ("10 cents"). Findings indicate that although biased reasoners do not know the exact correct answer ("5 cents") they do correctly grasp that the right answer must be smaller than the intuitively cued "10 cents" answer. This suggests that reasoners might be savvier about their errors than traditionally assumed. Implications for the conflict detection and dual process literature are discussed.

Characterizing belief bias in syllogistic reasoning: A hierarchical Bayesian meta-analysis of ROC data

The belief-bias effect is one of the most-studied biases in reasoning. A recent study of the phenomenon using the signal detection theory (SDT) model called into question all theoretical accounts of belief bias by demonstrating that belief-based differences in the ability to discriminate between valid and invalid syllogisms may be an artifact stemming from the use of inappropriate linear measurement models such as analysis of variance (Dube et al., Psychological Review, 117(3), 831-863, 2010). The discrepancy between Dube et al.'s, Psychological Review, 117(3), 831-863 (2010) results and the previous three decades of work, together with former's methodological criticisms suggests the need to revisit earlier results, this time collecting confidence-rating responses. Using a hierarchical Bayesian meta-analysis, we reanalyzed a corpus of 22 confidence-rating studies (N = 993). The results indicated that extensive replications using confidence-rating data are unnecessary as the observed receiver operating characteristic functions are not systematically asymmetric. These results were subsequently corroborated by a novel experimental design based on SDT's generalized area theorem. Although the meta-analysis confirms that believability does not influence discriminability unconditionally, it also confirmed previous results that factors such as individual differences mediate the effect. The main point is that data from previous and future studies can be safely analyzed using appropriate hierarchical methods that do not require confidence ratings. More generally, our results set a new standard for analyzing data and evaluating theories in reasoning. Important methodological and theoretical considerations for future work on belief bias and related domains are discussed.

Conflict Detection and Logical Complexity

Empirical evidence for the capacity to detect conflict between biased reasoning and normative principles has led to the proposal that reasoners have an intuitive grasp of some basic logical principles. In two studies, we investigate the boundary conditions of these logical intuitions by manipulating the logical complexity of problems where logical validity and conclusion believability conflict or not. Results pointed to evidence for successful conflict detection on the basic Modus Ponens (MP) inference, but also showed evidence for such a phenomenon on the more complex Modus Tollens (MT) inference. This suggests that both the MP and the MT inferences are simple enough for reasoners to have an intuitive grasp of their logical structure. The boundaries of logical intuition might thus reside in problems of greater complexity than these inferences. We also observed that on the invalid Affirmation of the Consequent (AC) and Denial of the Antecedent (DA) inferences, participants showed higher accuracy on the inference that was expected to be more complex (DA), and no evidence for successful conflict detection was found on these forms. Implications for the logical intuition framework are discussed.

When fast logic meets slow belief: Evidence for a parallel-processing model of belief bias

Two experiments pitted the default-interventionist account of belief bias against a parallel-processing model. According to the former, belief bias occurs because a fast, belief-based evaluation of the conclusion pre-empts a working-memory demanding logical analysis. In contrast, according to the latter both belief-based and logic-based responding occur in parallel. Participants were given deductive reasoning problems of variable complexity and instructed to decide whether the conclusion was valid on half the trials or to decide whether the conclusion was believable on the other half. When belief and logic conflict, the default-interventionist view predicts that it should take less time to respond on the basis of belief than logic, and that the believability of a conclusion should interfere with judgments of validity, but not the reverse. The parallel-processing view predicts that beliefs should interfere with logic judgments only if the processing required to evaluate the logical structure exceeds that required to evaluate the knowledge necessary to make a belief-based judgment, and vice versa otherwise. Consistent with this latter view, for the simplest reasoning problems (modus ponens), judgments of belief resulted in lower accuracy than judgments of validity, and believability interfered more with judgments of validity than the converse. For problems of moderate complexity (modus tollens and single-model syllogisms), the interference was symmetrical, in that validity interfered with belief judgments to the same degree that believability interfered with validity judgments. For the most complex (three-term multiple-model syllogisms), conclusion believability interfered more with judgments of validity than vice versa, in spite of the significant interference from conclusion validity on judgments of belief.

Slower is not always better: Response-time evidence clarifies the limited role of miserly information processing in the Cognitive Reflection Test

We report a study examining the role of ‘cognitive miserliness’ as a determinant of poor performance on the standard three-item Cognitive Reflection Test (CRT). The cognitive miserliness hypothesis proposes that people often respond incorrectly on CRT items because of an unwillingness to go beyond default, heuristic processing and invest time and effort in analytic, reflective processing. Our analysis (N = 391) focused on people’s response times to CRT items to determine whether predicted associations are evident between miserly thinking and the generation of incorrect, intuitive answers. Evidence indicated only a weak correlation between CRT response times and accuracy. Item-level analyses also failed to demonstrate predicted response-time differences between correct analytic and incorrect intuitive answers for two of the three CRT items. We question whether participants who give incorrect intuitive answers on the CRT can legitimately be termed cognitive misers and whether the three CRT items measure the same general construct.

Interactions between inferential strategies and belief bias

The dual strategy model of reasoning proposed by Verschueren, Schaeken, and d'Ydewalle (Thinking & Reasoning, 11(3), 239-278, 2005a; Memory & Cognition, 33(1), 107-119, 2005b) suggests that people can use either a statistical or a counterexample-based strategy to make deductive inferences. Subsequent studies have supported this distinction and investigated some properties of the two strategies. In the following, we examine the further hypothesis that reasoners using statistical strategies should be more vulnerable to the effects of conclusion belief. In each of three studies, participants were given abstract problems used to determine strategy use and three different forms of syllogism with believable and unbelievable conclusions. Responses, response times, and feeling of rightness (FOR) measures were taken. The results show that participants using a statistical strategy were more prone to the effects of conclusion belief across all three forms of reasoning. In addition, statistical reasoners took less time to make inferences than did counterexample reasoners. Patterns of variation in response times and FOR ratings between believable and unbelievable conclusions were very similar for both strategies, indicating that both statistical and counterexample reasoners were aware of conflict between conclusion belief and premise-based reasoning.

Measures of Bayesian Reasoning Performance on 'Normal' and 'Natural' Frequency Tasks

While the majority of similar studies examining Bayesian reasoning investigate how participants avoid common errors such as base-rate neglect, the current research also examines whether different formats (frequency and probability) lead to a difference in levels of absolute accuracy. In Study One, older (≥60 years) and younger (18 to 29 years) participants completed tasks in probability and normalized frequency formats. In Study 2, participants completed tasks in probability and natural frequency formats. Findings are that frequencies lead to less over-estimation, particularly in natural frequency tasks, which also reveal an interaction between age and task format whereby older adults seem unaffected by format. There was no association found between format and the avoidance of errors such as base-rate neglect. Findings are discussed in the light of dual and multi-process theories of reasoning, having failed to support the theory that frequency formats elicit System 2 reasoning processes.

The Doubting System 1: Evidence for automatic substitution sensitivity

A long prevailing view of human reasoning suggests severe limits on our ability to adhere to simple logical or mathematical prescriptions. A key position assumes these failures arise from insufficient monitoring of rapidly produced intuitions. These faulty intuitions are thought to arise from a proposed substitution process, by which reasoners unknowingly interpret more difficult questions as easier ones. Recent work, however, suggests that reasoners are not blind to this substitution process, but in fact detect that their erroneous responses are not warranted. Using the popular bat-and-ball problem, we investigated whether this substitution sensitivity arises out of an automatic System 1 process or whether it depends on the operation of an executive resource demanding System 2 process. Results showed that accuracy on the bat-and-ball problem clearly declined under cognitive load. However, both reduced response confidence and increased response latencies indicated that biased reasoners remained sensitive to their faulty responses under load. Results suggest that a crucial substitution monitoring process is not only successfully engaged, but that it automatically operates as an autonomous System 1 process. By signaling its doubt along with a biased intuition, it appears System 1 is "smarter" than traditionally assumed.

Rethinking fast and slow based on a critique of reaction-time reverse inference

Do people intuitively favour certain actions over others? In some dual-process research, reaction-time (RT) data have been used to infer that certain choices are intuitive. However, the use of behavioural or biological measures to infer mental function, popularly known as ‘reverse inference', is problematic because it does not take into account other sources of variability in the data, such as discriminability of the choice options. Here we use two example data sets obtained from value-based choice experiments to demonstrate that, after controlling for discriminability (that is, strength-of-preference), there is no evidence that one type of choice is systematically faster than the other. Moreover, using specific variations of a prominent value-based choice experiment, we are able to predictably replicate, eliminate or reverse previously reported correlations between RT and selfishness. Thus, our findings shed crucial light on the use of RT in inferring mental processes and strongly caution against using RT differences as evidence favouring dual-process accounts.

In cognitive neuroscience, it is common practice to use reaction time data to infer whether decisions are intuitive or deliberate. Here the authors demonstrate that they can replicate, eliminate and reverse previously reported correlations between selfishness and reaction time.

The intersection between Descriptivism and Meliorism in reasoning research: further proposals in support of 'soft normativism'

The rationality paradox centers on the observation that people are highly intelligent, yet show evidence of errors and biases in their thinking when measured against normative standards. Elqayam and Evans’ (2011) reject normative standards in the psychological study of thinking, reasoning and deciding in favor of a ‘value-free’ descriptive approach to studying high-level cognition. In reviewing Elqayam and Evans’ (2011) position, we defend an alternative to descriptivism in the form of ‘soft normativism,’ which allows for normative evaluations alongside the pursuit of descriptive research goals. We propose that normative theories have considerable value provided that researchers: (1) are alert to the philosophical quagmire of strong relativism; (2) are mindful of the biases that can arise from utilizing normative benchmarks; and (3) engage in a focused analysis of the processing approach adopted by individual reasoners. We address the controversial ‘is–ought’ inference in this context and appeal to a ‘bridging solution’ to this contested inference that is based on the concept of ‘informal reflective equilibrium.’ Furthermore, we draw on Elqayam and Evans’ (2011) recognition of a role for normative benchmarks in research programs that are devised to enhance reasoning performance and we argue that such Meliorist research programs have a valuable reciprocal relationship with descriptivist accounts of reasoning. In sum, we believe that descriptions of reasoning processes are fundamentally enriched by evaluations of reasoning quality, and argue that if such standards are discarded altogether then our explanations and descriptions of reasoning processes are severely undermined.

Fluency and belief bias in deductive reasoning: new indices for old effects

Models based on signal detection theory (SDT) have occupied a prominent role in domains such as perception, categorization, and memory. Recent work by Dube et al. (2010) suggests that the framework may also offer important insights in the domain of deductive reasoning. Belief bias in reasoning has traditionally been examined using indices based on raw endorsement rates—indices that critics have claimed are highly problematic. We discuss a new set of SDT indices fit for the investigation belief bias and apply them to new data examining the effect of perceptual disfluency on belief bias in syllogisms. In contrast to the traditional approach, the SDT indices do not violate important statistical assumptions, resulting in a decreased Type 1 error rate. Based on analyses using these novel indices we demonstrate that perceptual disfluency leads to decreased reasoning accuracy, contrary to predictions. Disfluency also appears to eliminate the typical link found between cognitive ability and the effect of beliefs on accuracy. Finally, replicating previous work, we demonstrate that cognitive ability leads to an increase in reasoning accuracy and a decrease in the response bias component of belief bias.

Traditional difference-score analyses of reasoning are flawed

Studies of the belief bias effect in syllogistic reasoning have relied on three traditional difference score measures: the logic index, belief index, and interaction index. Dube, Rotello, and Heit (2010, 2011) argued that the interaction index incorrectly assumes a linear receiver operating characteristic (ROC). Here, all three measures are addressed. Simulations indicated that traditional analyses of reasoning experiments are likely to lead to incorrect conclusions. Two new experiments examined the role of instructional manipulations on the belief bias effect. The form of the ROCs violated assumptions of traditional measures. In comparison, signal detection theory (SDT) model-based analyses were a better match for the form of the ROCs, and implied that belief bias and instructional manipulations are predominantly response bias effects. Finally, reanalyses of previous studies of conditional reasoning also showed non-linear ROCs, violating assumptions of traditional analyses. Overall, reasoning research using traditional measures is at risk of drawing incorrect conclusions.

Negative valence can evoke a liberal response bias in syllogistic reasoning

Recently, studies have demonstrated that negative valence reduces the magnitude of the belief-bias effect in syllogistic reasoning. This effect has been localized in the reasoning stage, in the form of increased deliberation on trials where validity and conclusion believability are incongruent. Here, using signal detection theory, we show that the attenuation of belief bias observed when valence was negative can also be evoked by a liberal response bias at the decision stage. Indeed, when valence was negative participants adopted a more liberal criterion for judging syllogisms as "valid," and were overconfident in their judgments. They also displayed less sensitivity in distinguishing between valid and invalid syllogisms. Our findings dovetail with recent evidence from memory research suggesting that negative valence can evoke a liberal response bias without improving performance. Our novel contribution is the demonstration that the attenuating effect of negative valence on belief bias can take multiples routes--by influencing the decision stage as was the case here, the reasoning stage as has been demonstrated elsewhere, and potentially both stages.