Examining the robustness of the relationship between metacognitive efficiency and metacognitive bias

Dynamic modulation of confidence based on the metacognitive skills of collaborators

In collaborative decision-making contexts, people typically share their metacognitive experience of confidence to convey the degree of certainty in their decisions. To reach collective decisions, collaborators' individual beliefs can be aggregated and weighted according to their respective confidence, thereby enhancing group accuracy beyond individual capabilities. Previous joint decision-making studies have shown that individuals tend to adopt the same scale for communicating their levels of confidence. However, confidence judgments vary not only in terms of metacognitive bias, that is whether individuals tend to report generally low or high confidence, but also in terms of metacognitive accuracy, or how well the confidence judgments align with choice accuracy. In the first two experiments, where the metacognitive accuracy of the collaborator was manipulated and explicitly communicated to participants, individuals increased their average confidence levels as the metacognitive accuracy of the collaborator decreased, while their own metacognitive accuracy remained unaffected. Trial-wise analyses showed that participants differentially adapted their confidence after a collaborator made a wrong group decision, depending on the metacognitive accuracy of the collaborator. In two follow up studies, we showed that both manipulations (i.e. manipulating objective differences in the metacognitive accuracies of the collaborators and explicitly communicating these differences) were necessary for these effects to emerge. Our findings shed light on how collaborative decision-making contexts can dynamically affect metacognitive processes.

Product, not process: Metacognitive monitoring of visual performance during sustained attention

The performance of the human visual system exhibits moment-to-moment fluctuations influenced by multiple neurocognitive factors. To deal with this instability of the visual system, introspective awareness of current visual performance (metacognitive monitoring) may be crucial. In this study, we investigate whether and how people can monitor their own visual performance during sustained attention by adopting confidence judgments as indicators of metacognitive monitoring - assuming that if participants can monitor visual performance, confidence judgments will accurately track performance fluctuations. In two experiments (N = 40), we found that participants were able to monitor fluctuations in visual performance during sustained attention. Importantly, metacognitive monitoring largely relied on the quality of target perception, a product of visual processing ("I lack confidence in my performance because I only caught a glimpse of the target"), rather than the states of the visual system during visual processing ("I lack confidence because I was not focusing on the task").

Using machine learning to simultaneously quantify multiple cognitive components of episodic memory

Why do we remember some events but forget others? Previous studies attempting to decode successful vs. unsuccessful brain states to investigate this question have met with limited success, potentially due, in part, to assessing episodic memory as a unidimensional process, despite evidence that multiple domains contribute to episodic encoding. Using a machine learning algorithm known as "transfer learning", we leveraged visual perception, sustained attention, and selective attention brain states to better predict episodic memory performance from trial-to-trial encoding electroencephalography (EEG) activity. We found that this multidimensional treatment of memory decoding improved prediction performance compared to traditional, unidimensional, methods, with each cognitive domain explaining unique variance in decoding of successful encoding-related neural activity. Importantly, this approach could be applied to cognitive domains outside of memory. Overall, this study provides critical insight into the underlying reasons why some events are remembered while others are not.

Metacognitive deficits in major depressive disorder

Objective

We aimed to investigate the metacognition of patients with major depressive disorder (MDD) and its correlation with their condition, as well as explore its diagnostic significance in the early stages of the disease, thereby providing a reference for clinical treatment.

Methods

Using a cross-sectional research design, we selected 66 patients diagnosed with MDD and 99 healthy controls for a mental rotation task; we examined their metacognitive performance using a post-decisional confidence assessment paradigm. We evaluated various aspects, including their performance on first-order tasks (d'), metacognitive bias (average confidence), metacognitive sensitivity (meta-d'), metacognitive efficiency (the M Ratio).

Results

In terms of the first-order task performance (d'), the group with MDD scored significantly lower than the healthy controls (t = -4.274, p < 0.001, respectively). Regarding metacognitive bias(average confidence), metacognitive sensitivity (meta-d'), and metacognitive efficiency (the M ratio), the group with MDD performed significantly worse than the healthy controls (t = -4.280, p < 0.001; t = -3.540, p < 0.001; t = -2.104, p = 0.039, respectively). In addition, the Hamilton Rating Scale for Depression (HAMD-17) scores of the patients with MDD were significantly negatively correlated with their d', average confidence levels, meta-d', and M ratio(r = -0.468, p < 0.001; r = -0.601, p < 0.001;r = -0.457, p < 0.001; r = -0.362, p = 0.003), The average confidence levels of MDD patients are significantly positively correlated with d', meta-d', and M ratio. (r = -0.552, p < 0.001; r = 0.738, p < 0.001;r =0.273, p =0.02).

Conclusion

The metacognitive abilities of patients with MDD were significantly impaired, and the degree of metacognitive impairment was related to the severity of clinical depressive symptoms. Moreover, the impairment of their metacognitive abilities was correlated with negative metacognitive bias.

Clinical Trial Registration

https://www.chictr.org.cn, identifier ChiCTR2400091242.

A comprehensive assessment of current methods for measuring metacognition

One of the most important aspects of research on metacognition is the measurement of metacognitive ability. However, the properties of existing measures of metacognition have been mostly assumed rather than empirically established. Here I perform a comprehensive empirical assessment of 17 measures of metacognition. First, I develop a method of determining the validity and precision of a measure of metacognition and find that all 17 measures are valid and most show similar levels of precision. Second, I examine how measures of metacognition depend on task performance, response bias, and metacognitive bias, finding only weak dependences on response and metacognitive bias but many strong dependencies on task performance. Third, I find that all measures have very high split-half reliabilities, but most have poor test-retest reliabilities. This comprehensive assessment paints a complex picture: no measure of metacognition is perfect and different measures may be preferable in different experimental contexts.

Going 'meta': a systematic review of metacognition and functional neurological disorder

In functional neurological disorder (FND), there is a fundamental disconnect between an apparently intact nervous system and the individuals' ability to consistently perform motor actions, perceive sensory signals and/or access effective cognition. Metacognition, the capacity to self-evaluate cognitive performance, appears highly relevant to FND pathophysiology. Poor metacognition is a potential mechanism via which abnormal models of self and the state of the world could arise and persist unchecked. There is therefore a justified enthusiasm that studies of metacognition may give substance to FND's intangible nature. However, many assume an impairment in metacognition even though experimental studies are still in their infancy. This systematic review provides an analytical checkpoint of the evidence after the first five years of experimental work. We firstly summarize current methods for testing metacognition, prerequisite knowledge that allows readers to independently evaluate the evidence. Using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, we then screened the 21 articles on this topic and review the experimental data of the eight studies that specifically tested metacognition in subjects with FND. Questionnaire metrics used to estimate global metacognition and general confidence in FND revealed a mixed picture of low or normal confidence. Of the five studies that used performance-controlled metrics, the gold-standard to estimate local metacognition in FND, four found metacognition to be equivalent to healthy controls and one paper supported impaired metacognition. We consequently try and broaden the debate and discuss alternative headline scenarios. We review how positive studies may offer insight and debate whether null studies could represent false negatives. However, since most studies find equivalent metacognition to controls, we also discuss whether metacognition could be intact and how this could inform mechanistic models of FND and have potential clinical utility. In summary, this review highlights signal of interest within the data, exposes current limitations and flags the many open questions.

No gender difference in confidence or metacognitive ability in perceptual decision-making

Prior research has found inconsistent results regarding gender differences in confidence and metacognitive ability. Different studies have shown that men are either more or less confident and have either higher or lower metacognitive abilities than women. However, this research has generally not used well-controlled tasks or used performance-independent measures of metacognitive ability. Here, we test for gender differences in performance, confidence, and metacognitive ability using data from 10 studies from the Confidence Database (total N = 1,887, total number of trials = 633,168). We find an absence of strong gender differences in performance and no gender differences in either confidence or metacognitive ability. These results were further confirmed by meta-analyses of the 10 datasets. These findings show that it is unlikely that gender has a strong effect on metacognitive evaluation in low-level perceptual decision-making and suggest that previously observed gender differences in confidence and metacognition are likely domain-specific.

Challenging the Bayesian confidence hypothesis in perceptual decision-making

The Bayesian confidence hypothesis (BCH), which postulates that confidence reflects the posterior probability that a decision is correct, is currently the most prominent theory of confidence. Although several recent studies have found evidence against it in the context of relatively complex tasks, BCH remains dominant for simpler tasks. The major alternative to BCH is the confidence in raw evidence space (CRES) hypothesis, according to which confidence is based directly on the raw sensory evidence without explicit probability computations. Here, we tested these competing hypotheses in the context of perceptual tasks that are assumed to induce Gaussian evidence distributions. We show that providing information about task difficulty gives rise to a basic behavioral signature that distinguishes BCH from CRES models even for simple 2-choice tasks. We examined this signature in three experiments and found that all experiments exhibited behavioral signatures in line with CRES computations but contrary to BCH ones. We further performed an extensive comparison of 16 models that implemented either BCH or CRES confidence computations and systematically differed in their auxiliary assumptions. These model comparisons provided overwhelming support for the CRES models over their BCH counterparts across all model variants and across all three experiments. These observations challenge BCH and instead suggest that humans may make confidence judgments by placing criteria directly in the space of the sensory evidence.

Human-like dissociations between confidence and accuracy in convolutional neural networks

Prior research has shown that manipulating stimulus energy by changing both stimulus contrast and variability results in confidence-accuracy dissociations in humans. Specifically, even when performance is matched, higher stimulus energy leads to higher confidence. The most common explanation for this effect, derived from cognitive modeling, is the positive evidence heuristic where confidence neglects evidence that disconfirms the choice. However, an alternative explanation is the signal-and-variance-increase hypothesis, according to which these dissociations arise from changes in the separation and variance of perceptual representations. Because artificial neural networks lack built-in confidence heuristics, they can serve as a test for the necessity of confidence heuristics in explaining confidence-accuracy dissociations. Therefore, we tested whether confidence-accuracy dissociations induced by stimulus energy manipulations emerge naturally in convolutional neural networks (CNNs). We found that, across three different energy manipulations, CNNs produced confidence-accuracy dissociations similar to those found in humans. This effect was present for a range of CNN architectures from shallow 4-layer networks to very deep ones, such as VGG-19 and ResNet-50 pretrained on ImageNet. Further, we traced back the reason for the confidence-accuracy dissociations in all CNNs to the same signal-and-variance increase that has been proposed for humans: higher stimulus energy increased the separation and variance of evidence distributions in the CNNs' output layer leading to higher confidence even for matched accuracy. These findings cast doubt on the necessity of the positive evidence heuristic to explain human confidence and establish CNNs as promising models for testing cognitive theories of human behavior.

Intention offloading: Domain-general versus task-specific confidence signals

Intention offloading refers to the use of external reminders to help remember delayed intentions (e.g., setting an alert to help you remember when you need to take your medication). Research has found that metacognitive processes influence offloading such that individual differences in confidence predict individual differences in offloading regardless of objective cognitive ability. The current study investigated the cross-domain organization of this relationship. Participants performed two perceptual discrimination tasks where objective accuracy was equalized using a staircase procedure. In a memory task, two measures of intention offloading were collected, (1) the overall likelihood of setting reminders, and (2) the bias in reminder-setting compared to the optimal strategy. It was found that perceptual confidence was associated with the first measure but not the second. It is shown that this is because individual differences in perceptual confidence capture meaningful differences in objective ability despite the staircase procedure. These findings indicate that intention offloading is influenced by both domain-general and task-specific metacognitive signals. They also show that even when task performance is equalized via staircasing, individual differences in confidence cannot be considered a pure measure of metacognitive bias.

Reliable, rapid, and remote measurement of metacognitive bias

Metacognitive biases have been repeatedly associated with transdiagnostic psychiatric dimensions of 'anxious-depression' and 'compulsivity and intrusive thought', cross-sectionally. To progress our understanding of the underlying neurocognitive mechanisms, new methods are required to measure metacognition remotely, within individuals over time. We developed a gamified smartphone task designed to measure visuo-perceptual metacognitive (confidence) bias and investigated its psychometric properties across two studies (N = 3410 unpaid citizen scientists, N = 52 paid participants). We assessed convergent validity, split-half and test-retest reliability, and identified the minimum number of trials required to capture its clinical correlates. Convergent validity of metacognitive bias was moderate (r(50) = 0.64, p < 0.001) and it demonstrated excellent split-half reliability (r(50) = 0.91, p < 0.001). Anxious-depression was associated with decreased confidence (β =  - 0.23, SE = 0.02, p < 0.001), while compulsivity and intrusive thought was associated with greater confidence (β = 0.07, SE = 0.02, p < 0.001). The associations between metacognitive biases and transdiagnostic psychiatry dimensions are evident in as few as 40 trials. Metacognitive biases in decision-making are stable within and across sessions, exhibiting very high test-retest reliability for the 100-trial (ICC = 0.86, N = 110) and 40-trial (ICC = 0.86, N = 120) versions of Meta Mind. Hybrid 'self-report cognition' tasks may be one way to bridge the recently discussed reliability gap in computational psychiatry.

More than the sum of its parts: investigating episodic memory as a multidimensional cognitive process

Why do we remember some events but forget others? Previous studies attempting to decode successful vs. unsuccessful brain states to investigate this question have met with limited success, potentially due, in part, to assessing episodic memory as a unidimensional process, despite evidence that multiple domains contribute to episodic encoding. Using a novel machine learning algorithm known as "transfer learning", we leveraged visual perception, sustained attention, and selective attention brain states to better predict episodic memory performance from trial-to-trial encoding electroencephalography (EEG) activity. We found that this multidimensional treatment of memory decoding improved prediction performance compared to traditional, unidimensional, methods, with each cognitive domain explaining unique variance in decoding of successful encoding-related neural activity. Importantly, this approach could be applied to cognitive domains outside of memory. Overall, this study provides critical insight into the underlying reasons why some events are remembered while others are not.

How do humans give confidence? A comprehensive comparison of process models of perceptual metacognition

Humans have the metacognitive ability to assess the accuracy of their decisions via confidence judgments. Several computational models of confidence have been developed but not enough has been done to compare these models, making it difficult to adjudicate between them. Here, we compare 14 popular models of confidence that make various assumptions, such as confidence being derived from postdecisional evidence, from positive (decision-congruent) evidence, from posterior probability computations, or from a separate decision-making system for metacognitive judgments. We fit all models to three large experiments in which subjects completed a basic perceptual task with confidence ratings. In Experiments 1 and 2, the best-fitting model was the lognormal meta noise (LogN) model, which postulates that confidence is selectively corrupted by signal-dependent noise. However, in Experiment 3, the positive evidence (PE) model provided the best fits. We evaluated a new model combining the two consistently best-performing models-LogN and the weighted evidence and visibility (WEV). The resulting model, which we call logWEV, outperformed its individual counterparts and the PE model across all data sets, offering a better, more generalizable explanation for these data. Parameter and model recovery analyses showed mostly good recoverability but with important exceptions carrying implications for our ability to discriminate between models. Finally, we evaluated each model's ability to explain different patterns in the data, which led to additional insight into their performances. These results comprehensively characterize the relative adequacy of current confidence models to fit data from basic perceptual tasks and highlight the most plausible mechanisms underlying confidence generation. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

The future of metacognition research: Balancing construct breadth with measurement rigor

Foundational work in the psychology of metacognition identified a distinction between metacognitive knowledge (stable beliefs about one's capacities) and metacognitive experiences (local evaluations of performance). More recently, the field has focused on developing tasks and metrics that seek to identify metacognitive capacities from momentary estimates of confidence in performance, and providing precise computational accounts of metacognitive failure. However, this notable progress in formalising models of metacognitive judgments may come at a cost of ignoring broader elements of the psychology of metacognition - such as how stable meta-knowledge is formed, how social cognition and metacognition interact, and how we evaluate affective states that do not have an obvious ground truth. We propose that construct breadth in metacognition research can be restored while maintaining rigour in measurement, and highlight promising avenues for expanding the scope of metacognition research. Such a research programme is well placed to recapture qualitative features of metacognitive knowledge and experience while maintaining the psychophysical rigor that characterises modern research on confidence and performance monitoring.

Metacognition and Confidence: A Review and Synthesis

Determining the psychological, computational, and neural bases of confidence and uncertainty holds promise for understanding foundational aspects of human metacognition. While a neuroscience of confidence has focused on the mechanisms underpinning subpersonal phenomena such as representations of uncertainty in the visual or motor system, metacognition research has been concerned with personal-level beliefs and knowledge about self-performance. I provide a road map for bridging this divide by focusing on a particular class of confidence computation: propositional confidence in one's own (hypothetical) decisions or actions. Propositional confidence is informed by the observer's models of the world and their cognitive system, which may be more or less accurate-thus explaining why metacognitive judgments are inferential and sometimes diverge from task performance. Disparate findings on the neural basis of uncertainty and performance monitoring are integrated into a common framework, and a new understanding of the locus of action of metacognitive interventions is developed.

Common computations for metacognition and meta-metacognition

Recent evidence shows that people have the meta-metacognitive ability to evaluate their metacognitive judgments of confidence. However, it is unclear whether meta-metacognitive judgments are made by a different system and rely on a separate set of computations compared to metacognitive judgments. To address this question, we asked participants (N = 36) to perform a perceptual decision-making task and provide (i) an object-level, Type-1 response about the identity of the stimulus; (ii) a metacognitive, Type-2 response (low/high) regarding their confidence in their Type-1 decision; and (iii) a meta-metacognitive, Type-3 response (low/high) regarding the quality of their Type-2 rating. We found strong evidence for the existence of Type-3, meta-metacognitive ability. In a separate condition, participants performed an identical task with only a Type-1 response followed by a Type-2 response given on a 4-point scale. We found that the two conditions produced equivalent results such that the combination of binary Type-2 and binary Type-3 responses acts similar to a 4-point Type-2 response. Critically, while Type-2 evaluations were subject to metacognitive noise, Type-3 judgments were made at no additional cost. These results suggest that it is unlikely that there is a distinction between Type-2 and Type-3 systems (metacognition and meta-metacognition) in perceptual decision-making and, instead, a single system can be flexibly adapted to produce both Type-2 and Type-3 evaluations recursively.

Metacognitive Information Theory

The capacity that subjects have to rate confidence in their choices is a form of metacognition, and can be assessed according to bias, sensitivity and efficiency. Rich networks of domain-specific and domain-general regions of the brain are involved in the rating, and are associated with its quality and its use for regulating the processes of thinking and acting. Sensitivity and efficiency are often measured by quantities called meta-d' and the M-ratio that are based on reverse engineering the potential accuracy of the original, primary, choice that is implied by the quality of the confidence judgements. Here, we advocate a straightforward measure of sensitivity, called meta-𝓘, which assesses the mutual information between the accuracy of the subject's choices and the confidence reports, and two normalized versions of this measure that quantify efficiency in different regimes. Unlike most other measures, meta-𝓘-based quantities increase with the number of correctly assessed bins with which confidence is reported. We illustrate meta-𝓘 on data from a perceptual decision-making task, and via a simple form of simulated second-order metacognitive observer.

Confidence reflects a noisy decision reliability estimate

Decisions vary in difficulty. Humans know this and typically report more confidence in easy than in difficult decisions. However, confidence reports do not perfectly track decision accuracy, but also reflect response biases and difficulty misjudgements. To isolate the quality of confidence reports, we developed a model of the decision-making process underlying choice-confidence data. In this model, confidence reflects a subject's estimate of the reliability of their decision. The quality of this estimate is limited by the subject's uncertainty about the uncertainty of the variable that informs their decision ('meta-uncertainty'). This model provides an accurate account of choice-confidence data across a broad range of perceptual and cognitive tasks, investigated in six previous studies. We find meta-uncertainty varies across subjects, is stable over time, generalizes across some domains and can be manipulated experimentally. The model offers a parsimonious explanation for the computational processes that underlie and constrain the sense of confidence.

Psychiatrically relevant signatures of domain-general decision-making and metacognition in the general population

Human behaviours are guided by how confident we feel in our abilities. When confidence does not reflect objective performance, this can impact critical adaptive functions and impair life quality. Distorted decision-making and confidence have been associated with mental health problems. Here, utilising advances in computational and transdiagnostic psychiatry, we sought to map relationships between psychopathology and both decision-making and confidence in the general population across two online studies (N's = 344 and 473, respectively). The results revealed dissociable decision-making and confidence signatures related to distinct symptom dimensions. A dimension characterised by compulsivity and intrusive thoughts was found to be associated with reduced objective accuracy but, paradoxically, increased absolute confidence, whereas a dimension characterized by anxiety and depression was associated with systematically low confidence in the absence of impairments in objective accuracy. These relationships replicated across both studies and distinct cognitive domains (perception and general knowledge), suggesting that they are reliable and domain general. Additionally, whereas Big-5 personality traits also predicted objective task performance, only symptom dimensions related to subjective confidence. Domain-general signatures of decision-making and metacognition characterise distinct psychological dispositions and psychopathology in the general population and implicate confidence as a central component of mental health.