Is it time for a new measurement approach? A closer look at the assessment of cognitive adaptability in complex problem solving

Observing the World Through Your Own Lenses - The Role of Perceived Adaptability for Epistemological Beliefs About the Development of Scientific Knowledge

Students are exposed to vast amounts of information and knowledge that is rapidly changing. This exposure requires them to be adaptive, that is, to constantly adjust their thinking, behavior, and even their affect to successfully solve information-rich and knowledge-lean problems. Considering these developments, the purpose of the present study is twofold: First, it is aimed at exploring the link between students’ beliefs about their adaptability in an ever-changing world and their beliefs about the changing nature of scientific knowledge, thus linking two educationally relevant belief systems. Second, this study further explores validity issues related to the well-established and commonly used “Epistemological Beliefs about the Development of Scientific Knowledge (EBDE)” scale. Performing structural equation modeling on a large-scale data set of 1,662 Norwegian tenth-grade students, we estimated the correlations among different aspects of adaptability (i.e., cognitive-behavioral and affective-emotional adaptability) and EBDE. Moving beyond these correlations, we tested whether students’ perceived adaptability had an impact on the functioning of EBDE items by means of moderated factor analysis. Our analyses revealed that adaptability was associated with sophisticated EBDE in science, and the EB scale functioned differently with respect to different adaptability scores. The results of this study indicate that students perceive the development of scientific knowledge through the lenses of their own adaptability. Furthermore, the differential functioning of the EBDE scale challenges its validity.

Impact of Cognitive Abilities and Prior Knowledge on Complex Problem Solving Performance - Empirical Results and a Plea for Ecologically Valid Microworlds

The original aim of complex problem solving (CPS) research was to bring the cognitive demands of complex real-life problems into the lab in order to investigate problem solving behavior and performance under controlled conditions. Up until now, the validity of psychometric intelligence constructs has been scrutinized with regard to its importance for CPS performance. At the same time, different CPS measurement approaches competing for the title of the best way to assess CPS have been developed. In the first part of the paper, we investigate the predictability of CPS performance on the basis of the Berlin Intelligence Structure Model and Cattell's investment theory as well as an elaborated knowledge taxonomy. In the first study, 137 students managed a simulated shirt factory (Tailorshop; i.e., a complex real life-oriented system) twice, while in the second study, 152 students completed a forestry scenario (FSYS; i.e., a complex artificial world system). The results indicate that reasoning - specifically numerical reasoning (Studies 1 and 2) and figural reasoning (Study 2) - are the only relevant predictors among the intelligence constructs. We discuss the results with reference to the Brunswik symmetry principle. Path models suggest that reasoning and prior knowledge influence problem solving performance in the Tailorshop scenario mainly indirectly. In addition, different types of system-specific knowledge independently contribute to predicting CPS performance. The results of Study 2 indicate that working memory capacity, assessed as an additional predictor, has no incremental validity beyond reasoning. We conclude that (1) cognitive abilities and prior knowledge are substantial predictors of CPS performance, and (2) in contrast to former and recent interpretations, there is insufficient evidence to consider CPS a unique ability construct. In the second part of the paper, we discuss our results in light of recent CPS research, which predominantly utilizes the minimally complex systems (MCS) measurement approach. We suggest ecologically valid microworlds as an indispensable tool for future CPS research and applications.

Sometimes Less Is Not Enough: A Commentary on Greiff et al. (2015)

In this commentary, I discuss some critical issues in the study by Greiff, S.; Stadler, M.; Sonnleitner, P.; Wolff, C.; Martin, R., "Sometimes less is more: Comparing the validity of complex problem solving measures", Intelligence 2015, 50, 100-113. I conclude that-counter to the claims made in the original study-the specific study design was not suitable for deriving conclusions about the validity of different complex problem-solving (CPS) measurement approaches. Furthermore, a more elaborate consideration of previous CPS research was found to challenge Greiff et al.'s conclusions even further. Therefore, I argue that researchers should be aware of the differences between several kinds of CPS assessment tools and conceptualizations when the validity of CPS assessment tools is examined in future research.