Bias, Fairness, and Validity in Graduate-School Admissions: A Psychometric Perspective

As many schools and departments are considering the removal of the Graduate Record Examination (GRE) from their graduate-school admission processes to enhance equity and diversity in higher education, controversies arise. From a psychometric perspective, we see a critical need for clarifying the meanings of measurement "bias" and "fairness" to create common ground for constructive discussions within the field of psychology, higher education, and beyond. We critically evaluate six major sources of information that are widely used to help inform graduate-school admissions decisions: grade point average, personal statements, resumes/curriculum vitae, letters of recommendation, interviews, and GRE. We review empirical research evidence available to date on the validity, bias, and fairness issues associated with each of these admission measures and identify potential issues that have been overlooked in the literature. We conclude by suggesting several directions for practical steps to improve the current admissions decisions and highlighting areas in which future research would be beneficial.

Rejoinder to Commentaries on Woo et al. (2022)

In this rejoinder, we discuss several areas of agreement as well as some noteworthy divergence in perspectives that are worth exploring further. We also note a few areas where immediate clarifications may be necessary. Next, we discuss practical solutions and challenges for improving the validity and fairness of graduate admissions. We conclude with a call for intellectual humility and openness in further advancing the field's discussions on this critical topic as well as for authenticity and persistence in effecting real changes to the system.

A Conceptual Framework for Investigating and Mitigating Machine-Learning Measurement Bias (MLMB) in Psychological Assessment

Given significant concerns about fairness and bias in the use of artificial intelligence (AI) and machine learning (ML) for psychological assessment, we provide a conceptual framework for investigating and mitigating machine-learning measurement bias (MLMB) from a psychometric perspective. MLMB is defined as differential functioning of the trained ML model between subgroups. MLMB manifests empirically when a trained ML model produces different predicted score levels for different subgroups (e.g., race, gender) despite them having the same ground-truth levels for the underlying construct of interest (e.g., personality) and/or when the model yields differential predictive accuracies across the subgroups. Because the development of ML models involves both data and algorithms, both biased data and algorithm-training bias are potential sources of MLMB. Data bias can occur in the form of nonequivalence between subgroups in the ground truth, platform-based construct, behavioral expression, and/or feature computing. Algorithm-training bias can occur when algorithms are developed with nonequivalence in the relation between extracted features and ground truth (i.e., algorithm features are differentially used, weighted, or transformed between subgroups). We explain how these potential sources of bias may manifest during ML model development and share initial ideas for mitigating them, including recognizing that new statistical and algorithmic procedures need to be developed. We also discuss how this framework clarifies MLMB but does not reduce the complexity of the issue.

Who Likes Artificial Intelligence? Personality Predictors of Attitudes toward Artificial Intelligence

We examined how individuals' personality relates to various attitudes toward artificial intelligence (AI). Attitudes were organized into two dimensions of affective components (positive and negative emotions) and two dimensions of cognitive components (sociality and functionality). For personality, we focused on the Big Five personality traits (extraversion, agreeableness, conscientiousness, neuroticism, openness) and personal innovativeness in information technology. Based on a survey of 1,530 South Korean adults, we found that extraversion was related to negative emotions and low functionality. Agreeableness was associated with both positive and negative emotions, and it was positively associated with sociality and functionality. Conscientiousness was negatively related to negative emotions, and it was associated with high functionality, but also with low sociality. Neuroticism was related to negative emotions, but also to high sociality. Openness was positively linked to functionality, but did not predict other attitudes when other proximal predictors were included (e.g. prior use, personal innovativeness). Personal innovativeness in information technology consistently showed positive attitudes toward AI across all four dimensions. These findings provide mixed support for our hypotheses, and we discuss specific implications for future research and practice.

Automated video interview personality assessments: Reliability, validity, and generalizability investigations

Organizations are increasingly adopting automated video interviews (AVIs) to screen job applicants despite a paucity of research on their reliability, validity, and generalizability. In this study, we address this gap by developing AVIs that use verbal, paraverbal, and nonverbal behaviors extracted from video interviews to assess Big Five personality traits. We developed and validated machine learning models within (using nested cross-validation) and across three separate samples of mock video interviews (total N = 1,073). Also, we examined their test-retest reliability in a fourth sample (N = 99). In general, we found that the AVI personality assessments exhibited stronger evidence of validity when they were trained on interviewer-reports rather than self-reports. When cross-validated in the other samples, AVI personality assessments trained on interviewer-reports had mixed evidence of reliability, exhibited consistent convergent and discriminant relations, used predictors that appear to be conceptually relevant to the focal traits, and predicted academic outcomes. On the other hand, there was little evidence of reliability or validity for the AVIs trained on self-reports. We discuss the implications for future work on AVIs and personality theory, and provide practical recommendations for the vendors marketing such approaches and organizations considering adopting them. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Psychometric and Validity Issues in Machine Learning Approaches to Personality Assessment: A Focus on Social Media Text Mining

In the age of big data, substantial research is now moving toward using digital footprints like social media text data to assess personality. Nevertheless, there are concerns and questions regarding the psychometric and validity evidence of such approaches. We seek to address this issue by focusing on social media text data and (i) conducting a review of psychometric validation efforts in social media text mining (SMTM) for personality assessment and discussing additional work that needs to be done; (ii) considering additional validity issues from the standpoint of reference (i.e. ‘ground truth’) and causality (i.e. how personality determines variations in scores derived from SMTM); and (iii) discussing the unique issues of generalizability when validating SMTM for personality assessment across different social media platforms and populations. In doing so, we explicate the key validity and validation issues that need to be considered as a field to advance SMTM for personality assessment, and, more generally, machine learning personality assessment methods. © 2020 European Association of Personality Psychology

Towards a comprehensive science of situations: On the importance of typicality and the lexical approach

The two commentaries to the CAPTION derivation paper (Parrigon, Woo, Tay, & Wang, 2017) provided insightful points regarding the consistency of emergent dimensional structures in the extant literature (Rauthmann & Sherman, 2018, p. 482) and potential concerns regarding the use of the lexical approach to identify psychologically important situation dimensions (Reis, 2018, p. 489). In this rejoinder, we seek to further these important discussions by (a) emphasizing the importance of Typicality in understanding a broad range of psychological processes, (b) clarifying the utility of broad, dimensional-based situation taxonomies such as the CAPTION model for providing key theoretical and empirical linkages across discrete situations, as well as for capturing a broad range of psychologically meaningful dimensions of situations, and (c) reaffirming the need for cross-cultural/language and facet-level investigations of the CAPTION dimensions in future research. We hope that these discussions continue to advance the field toward a more comprehensive science of situations. (PsycINFO Database Record

Putting the “Person” in the Center

This article provides a review and synthesis of person-centered analytic (i.e., clustering) methods in organizational psychology with the aim of (a) placing them into an organizing framework to facilitate analysis and interpretation and (b) constructing a set of practical recommendations to guide future person-centered research. To do so, we first clarify the terminological and conceptual issues that still cloud person-centered approaches. Next, we organize the diverse kinds of person-centered analyses into two major statistical approaches, algorithmic and latent-variable approaches. We then present a literature review that quantifies how these two approaches have been used within our field, identifying trends over time and typical study characteristics. Out of this review, we construct a unifying taxonomy of the five ways in which clusters are differentiated: (1) construct-based patterns, (2) response-style patterns, (3) predictive relations, (4) growth trajectories, and (5) measurement models. We also provide a set of practical guidelines for researchers and highlight a few remaining questions and/or areas in which future work is needed for further advancing person-centered methodologies.

CAPTION-ing the situation: A lexically-derived taxonomy of psychological situation characteristics

In comparison with personality taxonomic research, there has been much less advancement toward establishing an integrative taxonomy of psychological situation characteristics (similar to personality characteristics for persons). One of the main concerns has been the limited content coverage of the characteristics being used. To address this issue, we present a collection of 4 lexically based studies using the largest-to-date number of situation characteristics to identify the major dimensions of the psychological situation. These studies each implemented a unique sampling and analytic methodology-namely, a qualitative dimensional exploration; the factor analyses of 2, independent samples of large-scale in situ ratings of situations; and the use of lexical-vector representations from neural-network-based models derived from millions of sources of natural-language usage with a total of 146.7 billion words. Across these studies, a clear 7-dimensional structure emerged: Complexity, Adversity, Positive Valence, Typicality, Importance, Humor, and Negative Valence-collectively referred to as the "CAPTION" model, which parsimoniously integrates the diversity of dimensions found in the extant literature. We then introduce both full- and short-form measures of these CAPTION. Data from 2 additional diverse samples of native English speakers suggest that the measures have good psychometric properties, and are able to predict a broad range of important psychological outcomes (e.g., behaviors, affect, motivation, and need satisfaction), even when pitted against extant situation taxonomic frameworks. We conclude by discussing how the CAPTION framework may serve as a useful tool for conceptualizing and measuring a broad range of psychological situations across all areas of psychology. (PsycINFO Database Record

Examining Variability in Values Attributed to Culture

Prior work examining the role of cultural groupings (i.e., national membership) on personal values showed small to moderate amounts of variability attributable to culture, refuting the idea that culture determines values. We extend this research by examining the proportion of variance in values that could be explained by cultural membership. Because there is no definitive level of proportion of variance that would lead to a conclusion that values are culturally determined, personality, which is arguably not culturally determined, was used as a relative benchmark. Language groups were used as an alternate conception of cultural groupings. A large data set of 144,857 workers from across 31 major language groups revealed that language groups explained a significant and non-negligible amount of variance in personal value dimensions (7%-17%). Nevertheless, this was not significantly larger than any single personality dimension (3%-12%). In other words, our data failed to support the notion that personal values are strongly determined by cultural groupings compared with personality.

Cultural and Intellectual Openness Differentially Relate to Social Judgments of Potential Work Partners

OBJECTIVE

This study investigates the differential functioning of cultural and intellectual openness (the two aspects of Openness to Experience) in relation to social cognitive processes by examining how they influence people's perceptions and interpretations of social information when deciding to initiate working relationships.

METHOD

Using a policy-capturing design, 681 adult participants were asked to rate their similarity to and preference to work with potential work partners characterized by varying nationalities and levels of work-related competence. Multilevel moderated mediation was conducted to simultaneously evaluate whether the indirect effects of potential work partners' characteristics (i.e., nationalities and levels of work-related competence) on work partner preference through perceived similarity were moderated by cultural and intellectual openness.

RESULTS

Perceived similarity mediated the relationships between work partner nationality and work-related competence and participants' work partner preferences. Furthermore, the negative indirect effect of work partner nationality on work partner preference via perceived similarity was attenuated by cultural openness, and the positive indirect effect of work partner work-related competence on work partner preference via perceived similarity was strengthened by intellectual openness.

CONCLUSIONS

Cultural and intellectual openness may have distinct functions that influence how people perceive, evaluate, and appreciate social information when making social judgments.

Specificity matters: criterion-related validity of contextualized and facet measures of conscientiousness in predicting college student performance

To enhance the predictive validity of self-report personality measures, 2 distinct ways of increasing specificity of personality measures have been proposed in the literature-contextual specificity (i.e., providing a contextual referent) and content specificity (i.e., focusing on more specific constructs such as the Big Five facets). This study extends this line of research by examining whether there is an optimal way to configure, align, or integrate contextual and content specificity using measures of conscientiousness to predict college student success. A sample of 478 undergraduate students completed 4 measures of conscientiousness that varied in the level of content and contextual specificity. These forms of specificity were crossed to yield 4 distinct measures of conscientiousness. We then evaluated and compared the relative importance and the incremental importance of these different measures in the prediction of academic success. Superior predictive validity was found for both contextualized and facet measures of conscientiousness compared to a measure of global conscientiousness in predicting grade-point average and a broader behavioral criterion of student performance. When contextual and content specificity approaches were compared and combined, we observed the strongest predictive validity when the level of specificity is appropriately matched between predictor and criterion.

Developing and Validating Short Form Protean and Boundaryless Career Attitudes Scales

The Protean and Boundaryless Career Attitudes scales developed by Briscoe and colleagues have facilitated research on career attitudes and mindsets, but they are unnecessarily lengthy and somewhat redundant in their content. To address these concerns, this article presents three studies that develop and validate short forms of the Protean and Boundaryless Career Attitudes scales (i.e., PCA-SF and BCA-SF). Study findings suggest that the PCA-SF and BCA-SF provide a more efficient assessment of the protean and boundaryless career attitudes constructs than the full-length measures and exhibit partial measurement equivalence across U.S. and Korean populations.

A Bayesian Primer for the Organizational Sciences

When first learning Bayesian statistics, the organizational scholar may be confronted by a number of conceptual and practical challenges. The present article seeks to minimize these by first explicating how the Bayesian process can be understood simply as the combination of two complementary sources of information: prior beliefs and data. In turn, we describe how each source is derived from Bayes’s theorem and mathematically formalized, essential knowledge for the Bayesian analyst. However, the beginner can also be undermined by practical difficulties such as software implementation. To this end, we offer a walkthrough of how a Bayesian logistic regression model is coded within BugsXLA, a user-friendly Excel add-in for Bayesian estimation. The data for this example come from a previously published study that identified a subpopulation of “job hobos,” individuals characterized by their frequent voluntary turnover and positive attitudes toward quitting. In the original frequentist analysis, exploring the predictors of hoboism proved to be inefficient and inconclusive. We contrast this standard approach with Bayesian estimation, whose results provide rich and novel insights on the topic.

A Conceptual and Methodological Framework for Psychometric Isomorphism

The conceptual and methodological framework for measurement equivalence procedures has been well established and widely used. Although multilevel theories and methods have been widely used in organizational research, there is no comparable framework for measurement equivalence of multilevel constructs, or psychometric isomorphism. In this article, we present a conceptual and methodological framework for understanding and testing various forms of isomorphism. Within this framework, we explicate (a) the different types of psychometric isomorphism, (b) the conditions where psychometric isomorphism is appropriate and necessary, (c) how psychometric isomorphism corresponds with different composition models and estimation methods, and (d) the analytic procedures that can be used. Using simulated data, we also illustrate how the proposed procedures may be applied via two analytic methods—item response theory and factor analysis. We conclude with a discussion of theoretical and methodological implications provided by the proposed framework of psychometric isomorphism.