Criterion Validity is Maintained When Items Are Evaluatively Neutralized: Evidence from A Full–Scale Five–Factor Model Inventory

Less Evaluative Measures of Personality in Job Applicant Contexts: The Effect on Socially Desirable Responding and Criterion Validity

Researchers have long sought to mitigate the detrimental effects of socially desirable responding on personality assessments in high-stakes contexts. This study investigated the effect of reducing the social desirability of personality items on response distortion and criterion validity in a job applicant context. Using a 2 × 2 repeated measures design, participants (n = 584) completed standard (International Personality Item Pool) and less evaluative (Less Evaluative Five Factor Inventory) measures of Big Five personality in a low-stakes context and then several weeks later in a simulated job applicant context. Self-report criteria with objective answers, including university grades, were also obtained. In general, the less evaluative measure showed less response distortion than the standard measure on some metrics, but not on others. Declines in criterion validity in the applicant context were smaller for the less evaluative measure. In the applicant context, however, validities were similar across the two measures. Correlations across contexts for corresponding traits (e.g., low-stakes extraversion with high-stakes extraversion) were also similar for both measures. In summary, reducing socially desirable item content might slightly reduce the substantive content required to predict criteria in low-stakes contexts, but this effect appears to be partly offset by reduced response distortion for less evaluative measures in applicant contexts.

The NB5I

Abstract. Personality is usually measured by means of self-ratings. Despite some drawbacks, the method is here to stay, and improving on it, particularly regarding social desirability, is essential. One way to do this is evaluative neutralization, that is, to rephrase items such that it is less obvious to the respondent what would be a desirable response. We present a 120-item evaluatively neutralized five-factor inventory and compare it to the IPIP-NEO ( Goldberg et al., 2006 ). Psychometric analyses revealed that the new inventory has high factor homogeneity, relatively independent facets with acceptable homogeneity and normally distributed ratings, and relatively evaluatively neutral ratings (as indicated by the level of item popularity). In sum, this new inventory captures the same personality variance as other five-factor inventories but with less influence from individual differences in evaluative responding, resulting in less correlation between factors and a factor structure more in line with the simple structure model than many other five-factor inventories. Evaluatively neutralized inventories should be particularly useful when the factor structure is central to the research question and focuses on discriminant validity, such as identifying theoretically valid relationships between personality traits and other concepts.

Remapping the foundations of morality: Well-fitting structural model of the Moral Foundations Questionnaire

Moral foundations theory posits five moral foundations, however 5-factor models provide poor fit to the data. Here, in five studies, each with large samples (total N = 11,496), we construct and replicate a well-fitting model of the Moral Foundations Questionnaire (MFQ). In study 1 (N = 2,271) we tested previously theorised models, confirming none provide adequate fit. We then developed a well-fitting model of the MFQ. In this model, the fairness/reciprocity and harm/care foundations were preserved intact. The binding foundations, however, divided into five, rather than the original three foundations. Purity/sanctity split into independent foundations of purity and sanctity. Similarly, Ingroup/loyalty divided into independent factors of loyalty to clan and loyalty to country. Authority/respect was re-focussed on hierarchy, losing one item to the new sanctity foundation and another into loyalty to country. In addition to these 7 foundations, higher-level factors of binding and individualizing were supported, along with a general/acquiescence factor. Finally, a "moral tilt" factor corresponding to coordinated left-leaning vs. right-leaning moral patterns was supported. We validated the model in four additional studies, testing replication of the 7-foundation model in data including from US, Australia, and China (total N = 9,225). The model replicated with good fit found in all four samples. These findings demonstrate the first well-fitting replicable model of the MFQ. They also highlight the importance of modelling measurement structure, and reveal important additional foundations, and structure (binding, individualizing, tilt) above the foundations.

Socially Desirable Responding in Experience Sampling

Abstract. Experience sampling often makes use of items that are similar to personality questionnaire items. Arguably, this opens up for item-popularity effects, where some respondents react to the items’ level of evaluative phrasing, causing a separate factor. Gauging the risk of item popularity effects in experience sampling is important since the multifactorial aspect of the responses to the items may cause spurious correlations. We investigate this in one original study and two existing datasets. The results reveal that evaluativeness in experience sampling items creates the same type of problems as in self-rating inventories. We conclude that personality researchers need to be aware that the experience sampling method is not vaccinated against socially desirable responding, and that careful phrasing of items promotes purer personality measures. This allows for more optimal testing of theoretical models of personality, as the fit between data and model will not concern variance in socially desirable responding but in the relevant constructs.

A less evaluative measure of Big Five personality: Comparison of structure and criterion validity

Researchers and practitioners have long been concerned about detrimental effects of socially desirable responding on the structure and criterion validity of personality assessments. The current research examined the effect of reducing evaluative item content of a Big Five personality assessment on test structure and criterion validity. We developed a new public domain measure of the Big Five called the Less Evaluative Five Factor Inventory (LEFFI), adapted from the standard 50-item IPIP NEO, and intended to be less evaluative. Participants ( n = 3164) then completed standard (IPIP) and neutralized (LEFFI) measures of personality. Criteria were also collected, including academic grades, age, sex, smoking, alcohol consumption, exercise, protesting, religious worship, music preferences, dental hygiene, blood donation, other-rated communication styles, other-rated HEXACO personality, and cognitive ability (ICAR). Evaluativeness of items was reduced in the neutralized measure. Cronbach's alpha and test-retest reliability were maintained. Correlations between the Big Five were reduced in the neutralized measure and criterion validity was similar or slightly reduced in the neutralized measure. The large sample size and use of objective criteria extend past research. The study also contributes to debates about whether the general factor of personality and agreement with socially desirable content reflect substance or bias.

Is the General Factor of Personality Really Related to Frequency of Agreeable, Conscientious, Emotionally Stable, Extraverted, and Open Behavior?

Abstract. Proponents of the general factor of personality have suggested that it is related to more instances of agreeable, conscientious, emotionally stable, extraverted, and open behavior. We tested this frequency hypothesis by relating the GFP from personality self-ratings to the number of instances of typical Big Five behaviors in real life. For a period of 18 consecutive days, 108 students provided responses to experience sampling items on their mobile phones. The items described concrete behaviors that were rated by experts of the Big Five as typical of either of the five traits and were phrased in a way that made them easy to interpret and respond to. The items were then validated in study 1 and it was found that the behavior correlated with Big Five ratings. In a second study, we tested the hypothesis that GFP is related to a higher frequency of GFP related behavior. The results provided little support for the frequency hypothesis and are more compatible with an interpretation of the GFP in terms of style rather than substance.

Five-Factor Personality Inventories Have a Competence-Related Higher-Order Factor Due to Item Phrasing

This research examines whether the items of some of the most well-established five-factor inventories refer to competence. Results reveal that both experts and laymen can distinguish between items that refer to how competently a behavior is performed and items that do not (Study 1). Responses to items that refer to competence create a higher-order factor in the personality inventories (Study 2), and the variability in responses to competence-related items in personality self-ratings is best modeled as a general factor rather than as also tied to the specific Big Five factors (Studies 3 and 4). We suggest that a focused debate on what personality items should refer to is likely to have considerable positive consequences for both theory and measurement of personality.

The Properties and Utility of Less Evaluative Personality Scales: Reduction of Social Desirability; Increase of Construct and Discriminant Validity

Evaluative neutralization implies rephrasing items such that it is less clear to the respondent what would be a desirable response in the given population. The current research compares evaluatively neutralized scales measuring the FFM model with standard counterparts. Study 1 reveals that evaluatively neutralized scales are less influenced by social desirability. Study 2 estimates higher-order factor models for neutralized vs. standard five-factor scales. In contrast to standard inventories, there was little support for higher-order factors for neutralized scales. Study 3 demonstrates the convergent and discriminant validity for the neutralized scales, e.g., by less inflated correlations to external measures. It is argued that evaluatively neutralized inventories help researchers come to grips with social desirability in personality measurement, and are particularly useful when the factor structure is central to the research question and there is a focus on discriminant validity.

Can’t Make it Better nor Worse

Abstract. Some of the most popular psychological questionnaires violate general rules of item construction: precise, positively keyed items without negations, multiple aspects of content, absolute statements, or vague quantifiers. To investigate if following these rules results in more desirable psychometric properties, 1,733 participants completed online either the original NEO Five-Factor Inventory, an “improved” version whose items follow the rules of item construction, or a “deteriorated” version whose items strongly violate these rules. We compared reliability estimates, item-total correlations, Confirmatory Factor Analysis (CFA) model fit, and fit to the partial credit model between the three versions. Neither of the manipulations resulted in considerable or consistent effects on any of the psychometric indices. Our results question the ability of standard analyses in test construction to distinguish good items from bad ones, as well as the effectiveness of general rules of item construction. To increase the reproducibility of psychological science, more focus should be laid on improving psychological measures.

Is Reliability Compromised Towards the End of Long Personality Inventories?

Abstract. During very long self-rating sessions there is a risk that respondents will be tired and/or lose interest. Is this a concern for users of long personality inventories, such that the reliability becomes threatened in the latter half when respondents have made hundreds of personality self-ratings? Two thousand three hundred fifty-two volunteers completed long (≈ 500 items) personality inventories on the Internet, where items were presented in a unique random order for each participant. Perhaps counterintuitively, there was no evidence that reliability is threatened as respondents approach the end of a long personality inventory. If anything, the ratings in the second half of the inventories had higher reliability than ratings in the first half. Ratings were quicker towards the end of the inventories, but equally reliable. The criterion validity, estimated using Paunonen’s Behavior Report Form, was maintained too. The current results provide little reason to mistrust responses to items that appear towards the end of long personality inventories.

Quantifying the Association of Self-Enhancement Bias With Self-Ratings of Personality and Life Satisfaction

Kwan, John, Kenny, Bond, and Robins conceptualize self-enhancement as a favorable comparison of self-judgments with judgments of and by others. Applying a modified version of Kwan et al.’s approach to behavior observation data, we show that the resulting measure of self-enhancement bias is highly reliable, predicts self-ratings of intelligence as well as does actual intelligence, interacts with item desirability in predicting responses to questionnaire items, and also predicts general life satisfaction. Consistent with previous research, however, self-ratings of intelligence did not become more valid when controlling for self-enhancement bias. We also show that common personality scales like the Rosenberg Self-Esteem Scale reflect self-enhancement at least as strongly as do scales that were designed particularly for that purpose (i.e., “social desirability scales”). The relevance of these findings in regard to the validity and utility of social desirability scales is discussed.