How to Assess a Model's Testability and Identifiability

On the Identifiability of 3- and 4-Parameter Item Response Theory Models From the Perspective of Knowledge Space Theory

The present work aims at showing that the identification problems (here meant as both issues of empirical indistinguishability and unidentifiability) of some item response theory models are related to the notion of identifiability in knowledge space theory. Specifically, that the identification problems of the 3- and 4-parameter models are related to the more general issues of forward- and backward-gradedness in all items of the power set, which is the knowledge structure associated with IRT models under the assumption of local independence. As a consequence, the identifiability problem of a 4-parameter model is split into two parts: a first one, which is the result of a trade-off between the left-side added parameters and the remainder of the Item Response Function, e.g., a 2-parameter model, and a second one, which is the already well-known identifiability issue of the 2-parameter model itself. Application of the results to the logistic case appears to provide both a confirmation and a generalization of the current findings in the literature for both fixed- and random-effects IRT logistic models.

A tutorial on fitting joint models of M/EEG and behavior to understand cognition

We present motivation and practical steps necessary to find parameter estimates of joint models of behavior and neural electrophysiological data. This tutorial is written for researchers wishing to build joint models of human behavior and scalp and intracranial electroencephalographic (EEG) or magnetoencephalographic (MEG) data, and more specifically those researchers who seek to understand human cognition. Although these techniques could easily be applied to animal models, the focus of this tutorial is on human participants. Joint modeling of M/EEG and behavior requires some knowledge of existing computational and cognitive theories, M/EEG artifact correction, M/EEG analysis techniques, cognitive modeling, and programming for statistical modeling implementation. This paper seeks to give an introduction to these techniques as they apply to estimating parameters from neurocognitive models of M/EEG and human behavior, and to evaluate model results and compare models. Due to our research and knowledge on the subject matter, our examples in this paper will focus on testing specific hypotheses in human decision-making theory. However, most of the motivation and discussion of this paper applies across many modeling procedures and applications. We provide Python (and linked R) code examples in the tutorial and appendix. Readers are encouraged to try the exercises at the end of the document.

Empirical underidentification in estimating random utility models: The role of choice sets and standardizations

A standard approach to distinguishing people's risk preferences is to estimate a random utility model using a power utility function to characterize the preferences and a logit function to capture choice consistency. We demonstrate that with often-used choice situations, this model suffers from empirical underidentification, meaning that parameters cannot be estimated precisely. With simulations of estimation accuracy and Kullback-Leibler divergence measures we examined factors that potentially mitigate this problem. First, using a choice set that guarantees a switch in the utility order between two risky gambles in the range of plausible values leads to higher estimation accuracy than randomly created choice sets or the purpose-built choice sets common in the literature. Second, parameter estimates are regularly correlated, which contributes to empirical underidentification. Examining standardizations of the utility scale, we show that they mitigate this correlation and additionally improve the estimation accuracy for choice consistency. Yet, they can have detrimental effects on the estimation accuracy of risk preference. Finally, we also show how repeated versus distinct choice sets and an increase in observations affect estimation accuracy. Together, these results should help researchers make informed design choices to estimate parameters in the random utility model more precisely.

Modeling temporal dynamics of face processing in youth and adults

A hierarchical model of temporal dynamics was examined in adults (n = 34) and youth (n = 46) across the stages of face processing during the perception of static and dynamic faces. Three ERP components (P100, N170, N250) and spectral power in the mu range were extracted, corresponding to cognitive stages of face processing: low-level vision processing, structural encoding, higher-order processing, and action understanding. Youth and adults exhibited similar yet distinct patterns of hierarchical temporal dynamics such that earlier cognitive stages predicted later stages, directly and indirectly. However, latent factors indicated unique profiles related to behavioral performance for adults and youth and age as a continuous factor. The application of path analysis to electrophysiological data can yield novel insights into the cortical dynamics of social information processing.

Multinomial Models with Linear Inequality Constraints: Overview and Improvements of Computational Methods for Bayesian Inference

Many psychological theories can be operationalized as linear inequality constraints on the parameters of multinomial distributions (e.g., discrete choice analysis). These constraints can be described in two equivalent ways: Either as the solution set to a system of linear inequalities or as the convex hull of a set of extremal points (vertices). For both representations, we describe a general Gibbs sampler for drawing posterior samples in order to carry out Bayesian analyses. We also summarize alternative sampling methods for estimating Bayes factors for these model representations using the encompassing Bayes factor method. We introduce the R package multinomineq, which provides an easily-accessible interface to a computationally efficient implementation of these techniques.

Characterizing belief bias in syllogistic reasoning: A hierarchical Bayesian meta-analysis of ROC data

The belief-bias effect is one of the most-studied biases in reasoning. A recent study of the phenomenon using the signal detection theory (SDT) model called into question all theoretical accounts of belief bias by demonstrating that belief-based differences in the ability to discriminate between valid and invalid syllogisms may be an artifact stemming from the use of inappropriate linear measurement models such as analysis of variance (Dube et al., Psychological Review, 117(3), 831-863, 2010). The discrepancy between Dube et al.'s, Psychological Review, 117(3), 831-863 (2010) results and the previous three decades of work, together with former's methodological criticisms suggests the need to revisit earlier results, this time collecting confidence-rating responses. Using a hierarchical Bayesian meta-analysis, we reanalyzed a corpus of 22 confidence-rating studies (N = 993). The results indicated that extensive replications using confidence-rating data are unnecessary as the observed receiver operating characteristic functions are not systematically asymmetric. These results were subsequently corroborated by a novel experimental design based on SDT's generalized area theorem. Although the meta-analysis confirms that believability does not influence discriminability unconditionally, it also confirmed previous results that factors such as individual differences mediate the effect. The main point is that data from previous and future studies can be safely analyzed using appropriate hierarchical methods that do not require confidence ratings. More generally, our results set a new standard for analyzing data and evaluating theories in reasoning. Important methodological and theoretical considerations for future work on belief bias and related domains are discussed.

Behavioral and Neural Signatures of Reduced Updating of Alternative Options in Alcohol-Dependent Patients during Flexible Decision-Making

Addicted individuals continue substance use despite the knowledge of harmful consequences and often report having no choice but to consume. Computational psychiatry accounts have linked this clinical observation to difficulties in making flexible and goal-directed decisions in dynamic environments via consideration of potential alternative choices. To probe this in alcohol-dependent patients (n = 43) versus healthy volunteers (n = 35), human participants performed an anticorrelated decision-making task during functional neuroimaging. Via computational modeling, we investigated behavioral and neural signatures of inference regarding the alternative option. While healthy control subjects exploited the anticorrelated structure of the task to guide decision-making, alcohol-dependent patients were relatively better explained by a model-free strategy due to reduced inference on the alternative option after punishment. Whereas model-free prediction error signals were preserved, alcohol-dependent patients exhibited blunted medial prefrontal signatures of inference on the alternative option. This reduction was associated with patients' behavioral deficit in updating the alternative choice option and their obsessive-compulsive drinking habits. All results remained significant when adjusting for potential confounders (e.g., neuropsychological measures and gray matter density). A disturbed integration of alternative choice options implemented by the medial prefrontal cortex appears to be one important explanation for the puzzling question of why addicted individuals continue drug consumption despite negative consequences.

SIGNIFICANCE STATEMENT

In addiction, patients maintain substance use despite devastating consequences and often report having no choice but to consume. These clinical observations have been theoretically linked to disturbed mechanisms of inference, for example, to difficulties when learning statistical regularities of the environmental structure to guide decisions. Using computational modeling, we demonstrate disturbed inference on alternative choice options in alcohol addiction. Patients neglecting "what might have happened" was accompanied by blunted coding of inference regarding alternative choice options in the medial prefrontal cortex. An impaired integration of alternative choice options implemented by the medial prefrontal cortex might contribute to ongoing drug consumption in the face of evident negative consequences.

On the Link between Cognitive Diagnostic Models and Knowledge Space Theory

The present work explores the connections between cognitive diagnostic models (CDM) and knowledge space theory (KST) and shows that these two quite distinct approaches overlap. It is proved that in fact the Multiple Strategy DINA (Deterministic Input Noisy AND-gate) model and the CBLIM, a competence-based extension of the basic local independence model (BLIM), are equivalent. To demonstrate the benefits that arise from integrating the two theoretical perspectives, it is shown that a fairly complete picture on the identifiability of these models emerges by combining results from both camps. The impact of the results is illustrated by an empirical example, and topics for further research are pointed out.

On the Unidentifiability of the Fixed-Effects 3PL Model

The paper offers a general review of the basic concepts of both statistical model and parameter identification, and revisits the conceptual relationships between parameter identification and both parameter interpretability and properties of parameter estimates. All these issues are then exemplified for the 1PL, 2PL, and 1PL-G fixed-effects models. For the 3PL model, however, we provide a theorem proving that the item parameters are not identified, do not have an empirical interpretation and that it is not possible to obtain consistent and unbiased estimates of them.

Automatic and controlled stimulus processing in conflict tasks: Superimposed diffusion processes and delta functions

An elaborated diffusion process model (a Diffusion Model for Conflict Tasks, DMC) is introduced that combines conceptual features of standard diffusion models with the notion of controlled and automatic processes. DMC can account for a variety of distributional properties of reaction time (RT) in conflict tasks (e.g., Eriksen flanker, Simon, Stroop). Specifically, DMC is compatible with all observed shapes of delta functions, including negative-going delta functions that are particularly challenging for the class of standard diffusion models. Basically, DMC assumes that the activations of controlled and automatic processes superimpose to trigger a response. Monte Carlo simulations demonstrate that the unfolding of automatic activation in time largely determines the shape of delta functions. Furthermore, the predictions of DMC are consistent with other phenomena observed in conflict tasks such as error rate patterns. In addition, DMC was successfully fitted to experimental data of the standard Eriksen flanker and the Simon task. Thus, the present paper reconciles the prominent and successful class of diffusion models with the empirical finding of negative-going delta functions.

Revisiting absolute and relative judgments in the WITNESS model

The WITNESS model (Clark in Applied Cognitive Psychology 17:629-654, 2003) provides a theoretical framework with which to investigate the factors that contribute to eyewitness identification decisions. One key factor involves the contributions of absolute versus relative judgments. An absolute contribution is determined by the degree of match between an individual lineup member and memory for the perpetrator; a relative contribution involves the degree to which the best-matching lineup member is a better match to memory than the remaining lineup members. In WITNESS, the proportional contributions of relative versus absolute judgments are governed by the values of the decision weight parameters. We conducted an exploration of the WITNESS model's parameter space to determine the identifiability of these relative/absolute decision weight parameters, and compared the results to a restricted version of the model that does not vary the decision weight parameters. This exploration revealed that the decision weights in WITNESS are difficult to identify: Data often can be fit equally well by setting the decision weights to nearly any value and compensating with a criterion adjustment. Clark, Erickson, and Breneman (Law and Human Behavior 35:364-380, 2011) claimed to demonstrate a theoretical basis for the superiority of lineup decisions that are based on absolute contributions, but the relationship between the decision weights and the criterion weakens this claim. These findings necessitate reconsidering the role of the relative/absolute judgment distinction in eyewitness decision making.

Active data collection for efficient estimation and comparison of nonlinear neural models

The stimulus-response relationship of many sensory neurons is nonlinear, but fully quantifying this relationship by a complex nonlinear model may require too much data to be experimentally tractable. Here we present a theoretical study of a general two-stage computational method that may help to significantly reduce the number of stimuli needed to obtain an accurate mathematical description of nonlinear neural responses. Our method of active data collection first adaptively generates stimuli that are optimal for estimating the parameters of competing nonlinear models and then uses these estimates to generate stimuli online that are optimal for discriminating these models. We applied our method to simple hierarchical circuit models, including nonlinear networks built on the spatiotemporal or spectral-temporal receptive fields, and confirmed that collecting data using our two-stage adaptive algorithm was far more effective for estimating and comparing competing nonlinear sensory processing models than standard nonadaptive methods using random stimuli.

Estimating cerebral oxygen metabolism from fMRI with a dynamic multicompartment Windkessel model

Stimulus evoked changes in cerebral blood flow, volume, and oxygenation arise from responses to underlying neuronally mediated changes in vascular tone and cerebral oxygen metabolism. There is increasing evidence that the magnitude and temporal characteristics of these evoked hemodynamic changes are additionally influenced by the local properties of the vasculature including the levels of baseline cerebral blood flow, volume, and blood oxygenation. In this work, we utilize a physiologically motivated vascular model to describe the temporal characteristics of evoked hemodynamic responses and their expected relationships to the structural and biomechanical properties of the underlying vasculature. We use this model in a temporal curve-fitting analysis of the high-temporal resolution functional MRI data to estimate the underlying cerebral vascular and metabolic responses in the brain. We present evidence for the feasibility of our model-based analysis to estimate transient changes in the cerebral metabolic rate of oxygen (CMRO(2)) in the human motor cortex from combined pulsed arterial spin labeling (ASL) and blood oxygen level dependent (BOLD) MRI. We examine both the numerical characteristics of this model and present experimental evidence to support this model by examining concurrently measured ASL, BOLD, and near-infrared spectroscopy to validate the calculated changes in underlying CMRO(2).

Multinomial Processing Tree Models

Multinomial processing tree (MPT) models have become popular in cognitive psychology in the past two decades. In contrast to general-purpose data analysis techniques, such as log-linear models or other generalized linear models, MPT models are substantively motivated stochastic models for categorical data. They are best described as tools (a) for measuring the cognitive processes that underlie human behavior in various tasks and (b) for testing the psychological assumptions on which these models are based. The present article provides a review of MPT models and their applications in psychology, focusing on recent trends and developments in the past 10 years. Our review is nontechnical in nature and primarily aims at informing readers about the scope and utility of MPT models in different branches of cognitive psychology.

Age-related slowing of movement as basal ganglia dysfunction

Attributions of age-related deficits in motor function to structural changes are compromised once the elderly exhibit lower error rates. This is because performance decrements observed in older adults are attributed to inferred strategic preferences for accuracy over speed. To understand genuine age differences in performance, we argue in the following theoretical paper that research needs to resolve methodological shortcomings and account for them within theoretical models of aging. Accounts of aging need to directly manipulate or control strategic differences in performance while assessing structural deficits. When this is done, age-related changes in motor control resemble the intermittencies of control seen in basal ganglia disorders. Given homologous circuitry in the basal ganglia, such observations could generalize to age-related changes in cognitive and emotional processes.

Quantifying Parsimony in Structural Equation Modeling

Fitting propensity (FP) is defined as a model's average ability to fit diverse data patterns, all else being equal. The relevance of FP to model selection is examined in the context of structural equation modeling (SEM). In SEM it is well known that the number of free model parameters influences FP, but other facets of FP are routinely excluded from consideration. It is shown that models possessing the same number of free parameters but different structures may exhibit different FPs. The consequences of this fact are demonstrated using illustrative examples and models culled from published research. The case is made that further attention should be given to quantifying FP in SEM and considering it in model selection. Practical approaches are suggested.

Modeling unidimensional categorization in monkeys

The categorization performance of monkeys on a unidimensional perceptual categorization task was examined with reference to decision bound and exemplar theories of categorization. Three rhesus monkeys were presented with stimuli varying along a single dimension, the displacement of a target light from a fixation point. Left or right saccade responses were probabilistically reinforced according to one of three functions, two of which were nonmonotonic at one end of the stimulus space. The monkeys all showed a monotonic increase in response probability as a function of target light displacement in this region, consistent with decision bound theory. Fits of a single-boundary model (GRT, Ashby & Gott, 1988) and two exemplar models--one using a probabilistic response function (GCM; Nosofsky, 1986), the other using a deterministic response function (DEM; Ashby & Maddox, 1993)--revealed overall support for the decision bound model. The results suggest that monkeys used a perceptual decision boundary to perform the task.

Predicting true patterns of cognitive performance from noisy data

Starting from the premise that the purpose of cognitive modeling is to gain information about the cognitive processes of individuals, we develop a general theoretical framework for assessment of models on the basis of tests of the models' ability to yield information about the true performance patterns of individual subjects and the processes underlying them. To address the central problem that observed performance is a composite of true performance and error, we present formal derivations concerning inference from noisy data to true performance. Analyses of model fits to simulated data illustrate the usefulness of our approach for coping with difficult issues of model identifiability and testability.

Word frequency of irrelevant speech distractors affects serial recall

In this study, participants memorized frequent or rare target words in silence or while ignoring frequent or rare distractor words. Distractor words impaired recall performance, but low-frequency distractor words caused more impairment than did high-frequency distractor words. We demonstrate how to solve the identifiability problem for Schweickert's (1993) multinomial processing tree model of immediate recall, and then use this model to show that irrelevant speech affected both the probability with which intact target word representations were available for serial recall and the probability of successful reconstruction of item identities based on degraded short-term memory traces. However, the type of irrelevant speech--low-versus high-frequency words--selectively affected the probability of intact target word representations. These results are consistent with an explanation of the irrelevant speech effect within the framework proposed by Cowan (1995), and they pose problems for other explanations of the irrelevant speech effect. The analyses also confirm the validity of Schweickert's process model.

The serial-parallel dilemma: a case study in a linkage of theory and method

The question as to whether humans perceive, remember, or cognize psychological items simultaneously (i.e., in parallel) or sequentially (i.e., serially) has been of interest to philosophers and psychologists since at least the 19th century. The advent of the information-processing approach to cognition in the 1960s reopened the inquiry, initiating a flood of experiments and models in the literature. Surprisingly for so elemental an issue, persuasive experimental tests have, until recently, proven rather elusive. Several decades of theoretical, methodological, and experimental effort, propelled and shaped by a meta-theoretical perspective, are leading to powerful strategies for assessing this and related cognitive issues. The present article reviews the theoretical and empirical history of these inquiries and details situations in which decisive experimental tests are possible.

An early- and late-stage convolution model for disease natural history

The standard convolution model of disease natural history posits an asymptomatic (preclinical) and a symptomatic (clinical) state. An augmented model includes, in both the preclinical and clinical states, an early and late stage of disease. In the case of cancer, the early stage would generally correspond to the organ-confined stages before there is evidence of cancer spread. We compute the number of screen-detected (preclinical) and clinical cases in the early and late stages expected under a given screening program and show how the model can be fit to data from a screening trial using maximum likelihood. We also develop expressions for sojourn time, lead time, and overdiagnosis in the context of the model, where each of the above concepts incorporates disease stage. As an example, we fit the model to data from the Mayo Lung Cancer Screening trial.

Statistical Tests for Comparing Possibly Misspecified and Nonnested Models

Model selection criteria (MSC) involves selecting the model with the best estimated goodness-of-fit to the data generating process. Following the method of Vuong (1989), a large sample Model Selection Test (MST), is introduced that can be used in conjunction with most existing MSC procedures to decide if the estimated goodness-of-fit for one model is significantly different from the estimated goodness-of-fit for another model. The MST extends the classical generalized likelihood ratio test, is valid in the presence of model misspecification, and is applicable to situations involving nonnested probability models. Simulation studies designed to illustrate the concept of the MST and its conservative decision rule (relative to the MSC method) are also presented. Copyright 2000 Academic Press.

Key Concepts in Model Selection: Performance and Generalizability

What is model selection? What are the goals of model selection? What are the methods of model selection and how do they work? Which methods perform better than others and in what circumstances? These questions rest on a number of key concepts in a relatively underdeveloped field. The aim of this paper is to explain some background concepts, to highlight some of the results in this special issue, and to add my own. The standard methods of model selection include classical hypothesis testing, maximum likelihood, Bayes method, minimum description length, cross-validation, and Akaike's information criterion. They all provide an implementation of Occam's razor, in which parsimony or simplicity is balanced against goodness-of-fit. These methods primarily take account of the sampling errors in parameter estimation, although their relative success at this task depends on the circumstances. However, the aim of model selection should also include the ability of a model to generalize to predictions in a different domain. Errors of extrapolation, or generalization, are different from errors of parameter estimation. So, it seems that simplicity and parsimony may be an additional factor in managing these errors, in which case the standard methods of model selection are incomplete implementations of Occam's razor. Copyright 2000 Academic Press.