Ensuring cross-cultural data comparability by means of anchoring vignettes in heterogeneous refugee samples

BACKGROUND

Configural, metric, and scalar measurement invariance have been indicators of bias-free statistical cross-group comparisons, although they are difficult to verify in the data. Low comparability of translated questionnaires or the different understanding of response formats by respondents might lead to rejection of measurement invariance and point to comparability bias in multi-language surveys. Anchoring vignettes have been proposed as a method to control for the different understanding of response categories by respondents (the latter is referred to as differential item functioning related to response categories or rating scales: RC-DIF). We evaluate the question whether the cross-cultural comparability of data can be assured by means of anchoring vignettes or by considering socio-demographic heterogeneity as an alternative approach.

METHODS

We used the Health System Responsiveness (HSR) questionnaire and collected survey data in English (n = 183) and Arabic (n = 121) in a random sample of refugees in the third largest German federal state. We conducted multiple-group Confirmatory Factor Analyses (MGCFA) to analyse measurement invariance and compared the results when 1) using rescaled data on the basis of anchoring vignettes (non-parametric approach), 2) including information on RC-DIF from the analyses with anchoring vignettes as covariates (parametric approach) and 3) including socio-demographic covariates.

RESULTS

For the HSR, every level of measurement invariance between the Arabic and English languages was rejected. Implementing rescaling or modelling on the basis of anchoring vignettes provided superior results over the initial MGCFA analysis, since configural, metric and - for ordered categorical analyses-scalar invariance could not be rejected. A consideration of socio-demographic variables did not show such an improvement.

CONCLUSIONS

Surveys may consider anchoring vignettes as a method to assess cross-cultural comparability of data, whereas socio-demographic variables cannot be used to improve data comparability as a standalone method. More research on the efficient implementation of anchoring vignettes and further development of methods to incorporate them when modelling measurement invariance is needed.

Fairness and Comparability in Achievement Motivation Items: A Differential Item Functioning Analysis

Achievement motivation is a well-documented predictor of a variety of positive student outcomes. However, given observed group differences in motivation and related outcomes, motivation instruments should be checked for comparable item and scale functioning. Therefore, the purpose of this study was to evaluate measurement scale comparability and differential item functioning (DIF) in PISA 2015 achievement motivation items across gender and ethnicity using pairwise and multiple-group comparisons. In addition, DIF was investigated in relation to a common base group that reflected the sample average. Results indicated DIF between gender groups and between the base group and female students. For ethnicity, DIF was consistently flagged in pairwise comparisons with Black/African American students and Asian students as well as in base group comparisons. However, the identified DIF had little practical implications. Implications from these findings are discussed, and recommendations for future research are made.

Cross-Cultural Variations in Extreme Rejecting and Extreme Affirming Response Styles

Previous research has discussed cultural differences in moderacy vs extremity response styles. The present research found that cultural differences in response styles were more complex than previously speculated. We investigated cross-cultural variations in extreme rejecting versus affirming response biases. Although research has indicated that overall Chinese have less extreme responses than Westerners, the difference may be mainly driven by extreme rejecting responses because respondents consider answering survey questions as a way of interacting with researchers, and extreme rejecting responses may disrupt harmony in relationships, which is valued more in Chinese collectivistic culture than in Western individualistic cultures. Studies 1 and 2 revealed that Chinese had less extreme rejecting response style than did British, whereas they did not differ in extreme affirming response style. Study 2 further revealed that the cross-cultural asymmetry in extreme rejecting versus affirming response styles was partially accounted for by individualism orientation at the individual level. Consistently, Study 3 revealed that at the country level, individualism was positively associated with extreme rejecting response style, but was not associated with extreme affirming response style, suggesting that individualism accounted for the asymmetric cultural variation in extreme rejecting versus affirming response styles.

Cultural Differences in Careless Responding

Careless responding (CR) in surveys has been identified as a serious threat for the validity of survey data. It occurs when participants respond without following the instructions or reading the content of items. The purpose of this article is to assess and explain the amount of CR at the individual and country level. In our study, individual CR (measured by indices using a short Big 5 personality questionnaire) correlated with education level and personality traits. In addition, using a sample of 8,320 participants from 34 countries, CR at the country level was investigated and it was strongly correlated with the cultural dimensions: human development, individualism, gender inequality, and power distance. CR can be seen as a powerful predictor of differences between countries. Finally, a comparison between CR and response styles (extreme and midpoint answers, acquiescence, and socially desirable responses) was conducted. CR and response styles showed some overlap, but loaded on different factors. CR is compatible with extreme answers, midpoint answers, and acquiescence, but different from social desirability. Social desirability is the only response style which requires careful reading of items.

Enhancing the Cross-Cultural Comparability of Self-Reports Using the Overclaiming Technique: An Analysis of Accuracy and Exaggeration in 64 Cultures

The overclaiming technique (OCT) is a novel way of measuring how socially desirable responding influences survey responses. It has the potential to enhance the cross-cultural comparability of survey data. It allows the identification of respondents’ knowledge accuracy and exaggeration by comparing their assessments of familiarity with existing and nonexisting concepts in a particular field of knowledge. We aim to compare the response patterns of countries and world regions based on their OCT accuracy and exaggeration-index values and validate these OCT scores using external variables. We also introduce a general model for the categorization of respondents based on their OCT indices values. We use the Programme for International Student Assessment (PISA) 2012 data from 64 countries ( N = 275,904). We found considerable differences in response patterns across world regions: high accuracy values in East Asia, low accuracy values in Southern and Central America, high exaggeration values in Southern Europe, and low exaggeration values in Western Europe. Furthermore, we show that familiarity with math concepts changes substantially after adjustment using the OCT. The correlation between unadjusted math familiarity and math test scores is weak and nonsignificant (.13) whereas after adjustment the correlation becomes strong and significant (.66). Concerning other indicators such as gross domestic product (GDP), public expenditure in education, and Corruption Perceptions Index (CPI), before adjustment the correlation is negative and even significant for CPI (–.11, –.22, and –.45) whereas after adjustment the correlation becomes, though nonsignificant, positive (.05, .19, and .07). We also discuss the OCT in the context of other methods indicative of culturally preferred scale usage.

Bogdanova O, Kovas Y. Cognition, emotion, and arithmetic in primary school: A cross-cultural investigation

The study investigated cross-cultural differences in variability and average performance in arithmetic, mathematical reasoning, symbolic and non-symbolic magnitude processing, intelligence, spatial ability, and mathematical anxiety in 890 6- to 9-year-old children from the United Kingdom, Russia, and China. Cross-cultural differences explained 28% of the variance in arithmetic and 17.3% of the variance in mathematical reasoning, with Chinese children outperforming the other two groups. No cross-cultural differences were observed for spatial ability and mathematical anxiety. In all samples, symbolic magnitude processing and mathematical reasoning were independently related to early arithmetic. Other factors, such as non-symbolic magnitude processing, mental rotation, intelligence, and mathematical anxiety, produced differential patterns across the populations. The results are discussed in relation to potential influences of parental practice, school readiness, and linguistic factors on individual differences in early mathematics. Statement of contribution What is already known on this subject? Cross-cultural differences in mathematical ability are present in preschool children. Similar mechanisms of mathematical development operate in preschool children from the United Kingdom, Russia, and China. Tasks that require understanding of numbers are best predictors of arithmetic in preschool children. What does this study add? Cross-cultural differences in mathematical ability become greater with age/years of formal education. Similar mechanisms of mathematical development operate in early primary school children from the United Kingdom, Russia, and China. Symbolic number magnitude and mathematical reasoning are the main predictors of arithmetic in all three populations.

On Detecting Systematic Measurement Error in Cross-Cultural Research: A Review and Critical Reflection on Equivalence and Invariance Tests

One major threat to revealing cultural influences on psychological states or processes is the presence of bias (i.e., systematic measurement error). When quantitative measures are not targeting the same construct or they differ in metric across cultures, the validity of inferences about cultural variability (and universality) is in doubt. The objectives of this article are to review what can be done about it and what is being done about it. To date, a multitude of useful techniques and methods to reduce or assess bias in cross-cultural research have been developed. We explore the limits of invariance/equivalence testing and suggest more flexible means of dealing with bias. First, we review currently available established and novel methods that reveal bias in cross-cultural research. Second, we analyze current practices in a systematic content analysis. The content analysis of more than 500 culture-comparative quantitative studies (published from 2008 to 2015 in three outlets in cross-cultural, social, and developmental psychology) aims to gauge current practices and approaches in the assessment of measurement equivalence/invariance. Surprisingly, the analysis revealed a rather low penetration of invariance testing in cross-cultural research. Although a multitude of classical and novel approaches for invariance testing is available, these are employed infrequent rather than habitual. We discuss reasons for this hesitation, and we derive suggestions for creatively assessing and handling biases across different research paradigms and designs.

A Note on Construct Validity of the Anchoring Method in PISA 2012

We examine construct validity of the anchoring method used with 12 noncognitive scales from the Programme for International Student Assessment (PISA) 2012 project. This method combines individuals’ responses to vignettes and self-rated scores based on Likert-type items. It has been reported that the use of anchoring vignettes can reverse country-level correlations between academic achievement scores and noncognitive measures from negative to positive, and therefore align them with the typically reported individual-level correlations. Using the PISA 2012 data, we show that construct validity of this approach may be open to question because the anchored scales produce a different set of latent dimensions in comparison with nonanchored scales, even though both scales were created from the same set of individual responses. We also demonstrate that only one of three vignettes may be responsible for the resolution of the “paradox” highlighting that the choice of vignettes may be more important than what was previously reported.