Measuring Cognitive Abilities in the Wild: Validating a Population-Scale Game-Based Cognitive Assessment

Rapid individual cognitive phenotyping holds the potential to revolutionize domains as wide-ranging as personalized learning, employment practices, and precision psychiatry. Going beyond limitations imposed by traditional lab-based experiments, new efforts have been underway toward greater ecological validity and participant diversity to capture the full range of individual differences in cognitive abilities and behaviors across the general population. Building on this, we developed Skill Lab, a novel game-based tool that simultaneously assesses a broad suite of cognitive abilities while providing an engaging narrative. Skill Lab consists of six mini-games as well as 14 established cognitive ability tasks. Using a popular citizen science platform (N = 10,725), we conducted a comprehensive validation in the wild of a game-based cognitive assessment suite. Based on the game and validation task data, we constructed reliable models to simultaneously predict eight cognitive abilities based on the users' in-game behavior. Follow-up validation tests revealed that the models can discriminate nuances contained within each separate cognitive ability as well as capture a shared main factor of generalized cognitive ability. Our game-based measures are five times faster to complete than the equivalent task-based measures and replicate previous findings on the decline of certain cognitive abilities with age in our large cross-sectional population sample (N = 6369). Taken together, our results demonstrate the feasibility of rapid in-the-wild systematic assessment of cognitive abilities as a promising first step toward population-scale benchmarking and individualized mental health diagnostics.

The Opportunity Cost of Compulsory Research Participation: Why Psychology Departments Should Abolish Involuntary Participant Pools

Psychology departments often require undergraduates to participate in faculty and graduate research as part of their course or face a penalty. Involuntary participant pools (human subject pools) in which students are compulsorily enrolled are objectively coercive. Students have less autonomy than other research participants because they face a costly alternative task or the penalties that accompany failure to meet a course requirement if they choose not to participate. By contrast, other research participants are free to refuse consent without cost or penalty. Some researchers claim that the educational value of participation justifies the requirement. They treat coercion as a cost that can be outweighed by the benefits to students. This paper argues that such an approach is flawed because coercion is not like other costs and that educational value is inherently low relative to personal study or classroom time. The unethical nature of involuntary participation is best demonstrated with an opportunity cost analysis. This shows that students are forced to sacrifice higher value alternatives that they have paid to do and undertake a lower value activity that principally benefits others. Faculty have a conflict of interest as they are the beneficiaries of student coercion in their role as researchers and responsible for student achievement in their role as teachers. Voluntary participant pools can resolve this conflict but at the cost of reducing the supply of participants. A change in departmental research conduct is required to restore the autonomy of students who are competent adults and not legitimate subjects of paternalism when it comes to research participation.

Online volunteer laboratories for human subjects research

Once a fixture of research in the social and behavioral sciences, volunteer subjects are now only rarely used in human subjects research. Yet volunteers are a potentially valuable resource, especially for research conducted online. We argue that online volunteer laboratories are able to produce high-quality data comparable to that from other online pools. The scalability of volunteer labs means that they can produce large volumes of high-quality data for multiple researchers, while imposing little or no financial burden. Using a range of original tests, we show that volunteer and paid respondents have different motivations for participating in research, but have similar descriptive compositions. Furthermore, volunteer samples are able to replicate classic and contemporary social science findings, and produce high levels of overall response quality comparable to paid subjects. Our results suggest that online volunteer labs represent a potentially significant untapped source of human subjects data.

Citizen Social Lab: A digital platform for human behavior experimentation within a citizen science framework

Cooperation is one of the behavioral traits that define human beings, however we are still trying to understand why humans cooperate. Behavioral experiments have been largely conducted to shed light into the mechanisms behind cooperation-and other behavioral traits. However, most of these experiments have been conducted in laboratories with highly controlled experimental protocols but with limitations in terms of subject pool or decisions' context, which limits the reproducibility and the generalization of the results obtained. In an attempt to overcome these limitations, some experimental approaches have moved human behavior experimentation from laboratories to public spaces, where behaviors occur naturally, and have opened the participation to the general public within the citizen science framework. Given the open nature of these environments, it is critical to establish the appropriate data collection protocols to maintain the same data quality that one can obtain in the laboratories. In this article we introduce Citizen Social Lab, a software platform designed to be used in the wild using citizen science practices. The platform allows researchers to collect data in a more realistic context while maintaining the scientific rigor, and it is structured in a modular and scalable way so it can also be easily adapted for online or brick-and-mortar experimental laboratories. Following citizen science guidelines, the platform is designed to motivate a more general population into participation, but also to promote engaging and learning of the scientific research process. We also review the main results of the experiments performed using the platform up to now, and the set of games that each experiment includes. Finally, we evaluate some properties of the platform, such as the heterogeneity of the samples of the experiments, the satisfaction level of participants, or the technical parameters that demonstrate the robustness of the platform and the quality of the data collected.