The Properties and Utility of Less Evaluative Personality Scales: Reduction of Social Desirability; Increase of Construct and Discriminant Validity

Extravert Surgical Resident Applicants Get Higher 360-Degree Evaluations From Coworkers

INTRODUCTION

360-degree evaluations are used as an assessment in order to identify strengths and weaknesses of, or as a continuous evaluation for, residents. The aim of this study was to investigate the relationship between personality and ratings on 360-degree evaluations among surgical residency applicants. A secondary aim was to describe the personality profile of applicants for a surgical residency position.

METHODS

Doctors interviewed for a residency or locum position in general, urology, or pediatric surgery were included. Participants rated their personality on the Neutralized Big Five Inventory. A 360-degree assessment was conducted. Scores from two laparoscopic simulators were used as a measure of technical ability. Univariate analyses were used to assess the results. Student's t-test was used to compare personality and Pearson correlations between 360-degree assessment and personality.

RESULTS

Fifty doctors participated: data were complete for 38. Personality profiles showed higher emotional stability, agreeableness, conscientiousness and openness ratings than the norm. Correlations revealed a significant relationship between extraversion and higher scores on the 360-degree assessments. Significant univariate correlations were found between extraversion and the 360-degree assessments, and two of the correlations held up after adjustment for multiple tests. No correlations with performance when using laparoscopic simulators were found.

CONCLUSIONS

Applicants for surgical residency rated significantly higher than the norm in four personality domains. Extraversion correlated with overall higher scores in 360-degree assessments by coworkers. Higher scores were not related to objective measures of technical skill, highlighting the importance of using objective measures for assessment.

Why Forced-Choice and Likert Items Provide the Same Information on Personality, Including Social Desirability

The forced-choice response format is often considered superior to the standard Likert-type format for controlling social desirability in personality inventories. We performed simulations and found that the trait information based on the two formats converges when the number of items is high and forced-choice items are mixed with regard to positively and negatively keyed items. Given that forced-choice items extract the same personality information as Likert-type items do, including socially desirable responding, other means are needed to counteract social desirability. We propose using evaluatively neutralized items in personality measurement, as they can counteract social desirability regardless of response format.

Less Evaluative Measures of Personality in Job Applicant Contexts: The Effect on Socially Desirable Responding and Criterion Validity

Researchers have long sought to mitigate the detrimental effects of socially desirable responding on personality assessments in high-stakes contexts. This study investigated the effect of reducing the social desirability of personality items on response distortion and criterion validity in a job applicant context. Using a 2 × 2 repeated measures design, participants (n = 584) completed standard (International Personality Item Pool) and less evaluative (Less Evaluative Five Factor Inventory) measures of Big Five personality in a low-stakes context and then several weeks later in a simulated job applicant context. Self-report criteria with objective answers, including university grades, were also obtained. In general, the less evaluative measure showed less response distortion than the standard measure on some metrics, but not on others. Declines in criterion validity in the applicant context were smaller for the less evaluative measure. In the applicant context, however, validities were similar across the two measures. Correlations across contexts for corresponding traits (e.g., low-stakes extraversion with high-stakes extraversion) were also similar for both measures. In summary, reducing socially desirable item content might slightly reduce the substantive content required to predict criteria in low-stakes contexts, but this effect appears to be partly offset by reduced response distortion for less evaluative measures in applicant contexts.

The NB5I

Abstract. Personality is usually measured by means of self-ratings. Despite some drawbacks, the method is here to stay, and improving on it, particularly regarding social desirability, is essential. One way to do this is evaluative neutralization, that is, to rephrase items such that it is less obvious to the respondent what would be a desirable response. We present a 120-item evaluatively neutralized five-factor inventory and compare it to the IPIP-NEO ( Goldberg et al., 2006 ). Psychometric analyses revealed that the new inventory has high factor homogeneity, relatively independent facets with acceptable homogeneity and normally distributed ratings, and relatively evaluatively neutral ratings (as indicated by the level of item popularity). In sum, this new inventory captures the same personality variance as other five-factor inventories but with less influence from individual differences in evaluative responding, resulting in less correlation between factors and a factor structure more in line with the simple structure model than many other five-factor inventories. Evaluatively neutralized inventories should be particularly useful when the factor structure is central to the research question and focuses on discriminant validity, such as identifying theoretically valid relationships between personality traits and other concepts.

Socially Desirable Responding in Experience Sampling

Abstract. Experience sampling often makes use of items that are similar to personality questionnaire items. Arguably, this opens up for item-popularity effects, where some respondents react to the items’ level of evaluative phrasing, causing a separate factor. Gauging the risk of item popularity effects in experience sampling is important since the multifactorial aspect of the responses to the items may cause spurious correlations. We investigate this in one original study and two existing datasets. The results reveal that evaluativeness in experience sampling items creates the same type of problems as in self-rating inventories. We conclude that personality researchers need to be aware that the experience sampling method is not vaccinated against socially desirable responding, and that careful phrasing of items promotes purer personality measures. This allows for more optimal testing of theoretical models of personality, as the fit between data and model will not concern variance in socially desirable responding but in the relevant constructs.

A less evaluative measure of Big Five personality: Comparison of structure and criterion validity

Researchers and practitioners have long been concerned about detrimental effects of socially desirable responding on the structure and criterion validity of personality assessments. The current research examined the effect of reducing evaluative item content of a Big Five personality assessment on test structure and criterion validity. We developed a new public domain measure of the Big Five called the Less Evaluative Five Factor Inventory (LEFFI), adapted from the standard 50-item IPIP NEO, and intended to be less evaluative. Participants ( n = 3164) then completed standard (IPIP) and neutralized (LEFFI) measures of personality. Criteria were also collected, including academic grades, age, sex, smoking, alcohol consumption, exercise, protesting, religious worship, music preferences, dental hygiene, blood donation, other-rated communication styles, other-rated HEXACO personality, and cognitive ability (ICAR). Evaluativeness of items was reduced in the neutralized measure. Cronbach's alpha and test-retest reliability were maintained. Correlations between the Big Five were reduced in the neutralized measure and criterion validity was similar or slightly reduced in the neutralized measure. The large sample size and use of objective criteria extend past research. The study also contributes to debates about whether the general factor of personality and agreement with socially desirable content reflect substance or bias.

Is the General Factor of Personality Really Related to Frequency of Agreeable, Conscientious, Emotionally Stable, Extraverted, and Open Behavior?

Abstract. Proponents of the general factor of personality have suggested that it is related to more instances of agreeable, conscientious, emotionally stable, extraverted, and open behavior. We tested this frequency hypothesis by relating the GFP from personality self-ratings to the number of instances of typical Big Five behaviors in real life. For a period of 18 consecutive days, 108 students provided responses to experience sampling items on their mobile phones. The items described concrete behaviors that were rated by experts of the Big Five as typical of either of the five traits and were phrased in a way that made them easy to interpret and respond to. The items were then validated in study 1 and it was found that the behavior correlated with Big Five ratings. In a second study, we tested the hypothesis that GFP is related to a higher frequency of GFP related behavior. The results provided little support for the frequency hypothesis and are more compatible with an interpretation of the GFP in terms of style rather than substance.