Pay for performance, satisfaction and retention in longitudinal crowdsourced research

In the social and cognitive sciences, crowdsourcing provides up to half of all research participants. Despite this popularity, researchers typically do not conceptualize participants accurately, as gig-economy worker-participants. Applying theories of employee motivation and the psychological contract between employees and employers, we hypothesized that pay and pay raises would drive worker-participant satisfaction, performance, and retention in a longitudinal study. In an experiment hiring 359 Amazon Mechanical Turk Workers, we found that initial pay, relative increase of pay over time, and overall pay did not have substantial influence on subsequent performance. However, pay significantly predicted participants' perceived choice, justice perceptions, and attrition. Given this, we conclude that worker-participants are particularly vulnerable to exploitation, having relatively low power to negotiate pay. Results of this study suggest that researchers wishing to crowdsource research participants using MTurk might not face practical dangers such as decreased performance as a result of lower pay, but they must recognize an ethical obligation to treat Workers fairly.

Gamifying a situational judgment test with immersion and control game elements

Purpose

Assessment gamification, which refers to the addition of game elements to existing assessments, is commonly implemented to improved applicant reactions to existing psychometric measures. This study aims to understand the effects of gamification on applicant reactions to and measurement quality of situational judgment tests.

Design/methodology/approach

In a 2 × 4 between-subjects experiment, this study randomly assigned 315 people to experience different versions of a gamified situational judgment test, crossing immersive game elements (text, audio, still pictures, video) with control game elements (high and low), measuring applicant reactions and assessing differences in convergent validity between conditions.

Findings

The use of immersive game elements improved perceptions of organizational technological sophistication, but no other reactions outcomes (test attitudes, procedural justice, organizational attractiveness). Convergent validity with cognitive ability was not affected by gamification.

Originality/value

This is the first study to experimentally examine applicant reactions and measurement quality to SJTs based upon the implementation of specific game elements. It demonstrates that small-scale efforts to gamify assessments are likely to lead to only small-scale gains. However, it also demonstrates that such modifications can be done without harming the measurement qualities of the test, making gamification a potentially useful marketing tool for assessment specialists. Thus, this study concludes that utility should be considered carefully and explicitly for any attempt to gamify assessment.

A theory of branched situational judgment tests and their applicant reactions

Purpose

Branched situational judgment tests (BSJTs) are an increasingly common employee selection method, yet there is no theory and very little empirical work explaining the designs and impacts of branching. To encourage additional research on BSJTs, and to provide practitioners with a common language to describe their current and future practices, we sought to develop a theory of BSTJs.

Design/methodology/approach

Given the absence of theory on branching, we utilized a ground theory qualitative research design, conducting interviews with 25 BSJT practitioner subject matter experts.

Findings

Our final theory consists of three components: (1) a taxonomy of BSJT branching features (contingency, parallelism, convergence, and looping) and options within those features (which vary), (2) a causal theoretical model describing impacts of branching in general on applicant reactions via proximal effects on face validity, and (3) a causal theoretical model describing impacts on applicant reactions among branching designs via proximal effects on consistency of administration and opportunity to perform.

Originality/value

Our work provides the first theoretical foundation on which future confirmatory research in the BSJT domain can be built. It also gives both researchers and practitioners a common language for describing branching features and their options. Finally, it reveals BSJTs as the results of a complex set of interrelated design features, discouraging the oversimplified contrasting of “branching” vs “not branching.”