The Road to Hell Is Paved With Good Intentions: How Common Practices in Scale Construction Hurt Validity

Ant colony optimization for parallel test assembly

Ant colony optimization (ACO) algorithms have previously been used to compile single short scales of psychological constructs. In the present article, we showcase the versatility of the ACO to construct multiple parallel short scales that adhere to several competing and interacting criteria simultaneously. Based on an initial pool of 120 knowledge items, we assembled three 12-item tests that (a) adequately cover the construct at the domain level, (b) follow a unidimensional measurement model, (c) allow reliable and (d) precise measurement of factual knowledge, and (e) are gender-fair. Moreover, we aligned the test characteristic and test information functions of the three tests to establish the equivalence of the tests. We cross-validated the assembled short scales and investigated their association with the full scale and covariates that were not included in the optimization procedure. Finally, we discuss potential extensions to metaheuristic test assembly and the equivalence of parallel knowledge tests in general.

A Novel Approach to Measuring an Old Construct: Aligning the Conceptualisation and Operationalisation of Cognitive Flexibility

A successful adjustment to dynamic changes in one's environment requires contingent adaptive behaviour. Such behaviour is underpinned by cognitive flexibility, which conceptually is part of fluid intelligence. We argue, however, that conventional approaches to measuring fluid intelligence are insufficient in capturing cognitive flexibility. We address the discrepancy between conceptualisation and operationalisation by introducing two newly developed tasks that aim at capturing within-person processes of dealing with novelty. In an exploratory proof-of-concept study, the two flexibility tasks were administered to 307 university students, together with a battery of conventional measures of fluid intelligence. Participants also provided information about their Grade Point Averages obtained in high school and in their first year at university. We tested (1) whether an experimental manipulation of a requirement for cognitive inhibition resulted in systematic differences in difficulty, (2) whether these complexity differences reflect psychometrically differentiable effects, and (3) whether these newly developed flexibility tasks show incremental value in predicting success in the transition from high school to university over conventional operationalisations of fluid intelligence. Our findings support the notion that cognitive flexibility, when conceptualised and operationalised as individual differences in within-person processes of dealing with novelty, more appropriately reflects the dynamics of individuals' behaviour when attempting to cope with changing demands.

Model Specification Searches in Structural Equation Modeling Using Bee Swarm Optimization

Metaheuristics are optimization algorithms that efficiently solve a variety of complex combinatorial problems. In psychological research, metaheuristics have been applied in short-scale construction and model specification search. In the present study, we propose a bee swarm optimization (BSO) algorithm to explore the structure underlying a psychological measurement instrument. The algorithm assigns items to an unknown number of nested factors in a confirmatory bifactor model, while simultaneously selecting items for the final scale. To achieve this, the algorithm follows the biological template of bees' foraging behavior: Scout bees explore new food sources, whereas onlooker bees search in the vicinity of previously explored, promising food sources. Analogously, scout bees in BSO introduce major changes to a model specification (e.g., adding or removing a specific factor), whereas onlooker bees only make minor changes (e.g., adding an item to a factor or swapping items between specific factors). Through this division of labor in an artificial bee colony, the algorithm aims to strike a balance between two opposing strategies diversification (or exploration) versus intensification (or exploitation). We demonstrate the usefulness of the algorithm to find the underlying structure in two empirical data sets (Holzinger-Swineford and short dark triad questionnaire, SDQ3). Furthermore, we illustrate the influence of relevant hyperparameters such as the number of bees in the hive, the percentage of scouts to onlookers, and the number of top solutions to be followed. Finally, useful applications of the new algorithm are discussed, as well as limitations and possible future research opportunities.