More than random responding: Empirical evidence for the validity of the (Extended) Crosswise Model

The Randomized Response Technique (Warner, Journal of the American Statistical Association, 60, 63-69, 1965) has been developed to control for socially desirable responses in surveys on sensitive attributes. The Crosswise Model (CWM; Yu et al., Metrika, 67, 251-263, 2008) and its extension, the Extended Crosswise Model (ECWM; Heck et al., Behavior Research Methods, 50, 1895-1905, 2018), are advancements of the Randomized Response Technique that have provided promising results in terms of improved validity of the obtained prevalence estimates compared to estimates based on conventional direct questions. However, recent studies have raised the question as to whether these promising results might have been primarily driven by a methodological artifact in terms of random responses rather than a successful control of socially desirable responding. The current study was designed to disentangle the influence of successful control of socially desirable responding and random answer behavior on the validity of (E)CWM estimates. To this end, we orthogonally manipulated the direction of social desirability (undesirable vs. desirable) and the prevalence (high vs. low) of sensitive attributes. Our results generally support the notion that the ECWM successfully controls social desirability bias and is inconsistent with the alternative account that ECWM estimates are distorted by a substantial influence of random responding. The results do not rule out a small proportion of random answers, especially when socially undesirable attributes with high prevalence are studied, or when high randomization probabilities are applied. Our results however do rule out that random responding is a major factor that can account for the findings attesting to the improved validity of (E)CWM as compared with DQ estimates.

Detecting computer-generated random responding in questionnaire-based data: A comparison of seven indices

With the development of online data collection and instruments such as Amazon's Mechanical Turk (MTurk), the appearance of malicious software that generates responses to surveys in order to earn money represents a major issue, for both economic and scientific reasons. Indeed, even if paying one respondent to complete one questionnaire represents a very small cost, the multiplication of botnets providing invalid response sets may ultimately reduce study validity while increasing research costs. Several techniques have been proposed thus far to detect problematic human response sets, but little research has been undertaken to test the extent to which they actually detect nonhuman response sets. Thus, we proposed to conduct an empirical comparison of these indices. Assuming that most botnet programs are based on random uniform distributions of responses, we present and compare seven indices in this study to detect nonhuman response sets. A sample of 1,967 human respondents was mixed with different percentages (i.e., from 5% to 50%) of simulated random response sets. Three of the seven indices (i.e., response coherence, Mahalanobis distance, and person-total correlation) appear to be the best estimators for detecting nonhuman response sets. Given that two of those indices-Mahalanobis distance and person-total correlation-are calculated easily, every researcher working with online questionnaires could use them to screen for the presence of such invalid data.

Schizotypal Personality Questionnaire-Brief: Effect of invalid responding on factor structure analysis and scores of schizotypy

OBJECTIVES

We assessed the effect of invalid responding on factor structure and on scores of schizotypy through the factor analysis of the Schizotypal Personality Questionnaire-Brief (SPQ-B) in a sample of 580 Romanian students using 3 validity items and 5 social desirability items.

METHODS

We examined the factor structure of the SPQ-B, we compared the mean SPQ-B scores between reliable and unreliable responders and between high vs. low social desirability responders, and we re-run the factor analysis restricting the sample to the reliable or low social desirability responders.

RESULTS

Factor analysis resulted in a 3-factor solution: Cognitive-perceptual, Interpersonal and Disorganized dimensions. Unreliable responders had lower scores of positive, negative and total schizotypy. Subjects with high social desirability scores had lower scores of disorganized schizotypy. Factor analyses in the samples of "good" responders showed minor differences in reliable responders, whereas, after taking into account the effect of social desirability, 2 items correctly loaded on expected dimensions.

CONCLUSIONS

Random responding and social desirability could influence scores of schizotypy and factor structure. Simple methods could be used to identify invalid responses. The effect of social desirability could be linked to the phrasing of items.

Detecting random responders with infrequency scales using an error-balancing threshold

Infrequency scales are becoming a popular mode of data screening, due to their availability and ease of implementation. Recent research has indicated that the interpretation and functioning of infrequency items may not be as straightforward as had previously been thought (Curran & Hauser, 2015), yet there are no empirically based guidelines for implementing cutoffs using these items. In the present study, we compared two methods of detecting random responding with infrequency items: a zero-tolerance threshold versus a threshold that balances classification error rates. The results showed that a traditional zero-tolerance approach, on average, screens data that are less indicative of careless responding than those screened by the error-balancing approach. Thus, the de facto standard of applying a "zero-tolerance" approach when screening participants with infrequency scales may be too stringent, so that meaningful responses may also be removed from analyses. Recommendations and future directions are discussed.