1
|
Li M, Zhang B, Mou Y. Though Forced, Still Valid: Examining the Psychometric Performance of Forced-Choice Measurement of Personality in Children and Adolescents. Assessment 2024:10731911241255841. [PMID: 38867477 DOI: 10.1177/10731911241255841] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/14/2024]
Abstract
Unveiling the roles personality plays during childhood and adolescence necessitates its accurate measurement, commonly using traditional Likert-type (LK) scales. However, this format is susceptible to various response biases, which can be particularly prevalent in children and adolescents, thus likely undermining measurement accuracy. Forced-choice (FC) scales appear to be a promising alternative because they are largely free from these biases by design. However, some argue that the FC format may not perform satisfactorily in children and adolescents due to its complexity. Little empirical evidence exists regarding the suitability of the FC format for children and adolescents. As such, the current study examined the psychometric performance of an FC measure of the Big Five personality factors in three children and adolescent samples: 5th to 6th graders (N = 428), 7th to 8th graders (N = 449), and 10th to 11th graders (N = 555). Across the three age groups, the FC scale demonstrated a better fit to the Big Five model and better discriminant validity in comparison to the LK counterpart. Personality scores from the FC scale also converged well with those from the LK scale and demonstrated high reliability as well as sizable criterion-related validity. Furthermore, the FC scale had more invariant statements than its LK counterpart across age groups. Overall, we found good evidence showing that FC measurement of personality is suitable for children and adolescents.
Collapse
Affiliation(s)
- Mengtong Li
- University of Illinois Urbana-Champaign, IL, USA
| | - Bo Zhang
- University of Illinois Urbana-Champaign, IL, USA
| | - Yi Mou
- Sun Yat-Sen University, Guangzhou, China
| |
Collapse
|
2
|
Kreitchmann RS, Sorrel MA, Abad FJ. On Bank Assembly and Block Selection in Multidimensional Forced-Choice Adaptive Assessments. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT 2023; 83:294-321. [PMID: 36866066 PMCID: PMC9972126 DOI: 10.1177/00131644221087986] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
Multidimensional forced-choice (FC) questionnaires have been consistently found to reduce the effects of socially desirable responding and faking in noncognitive assessments. Although FC has been considered problematic for providing ipsative scores under the classical test theory, item response theory (IRT) models enable the estimation of nonipsative scores from FC responses. However, while some authors indicate that blocks composed of opposite-keyed items are necessary to retrieve normative scores, others suggest that these blocks may be less robust to faking, thus impairing the assessment validity. Accordingly, this article presents a simulation study to investigate whether it is possible to retrieve normative scores using only positively keyed items in pairwise FC computerized adaptive testing (CAT). Specifically, a simulation study addressed the effect of (a) different bank assembly (with a randomly assembled bank, an optimally assembled bank, and blocks assembled on-the-fly considering every possible pair of items), and (b) block selection rules (i.e., T, and Bayesian D and A-rules) over the estimate accuracy and ipsativity and overlap rates. Moreover, different questionnaire lengths (30 and 60) and trait structures (independent or positively correlated) were studied, and a nonadaptive questionnaire was included as baseline in each condition. In general, very good trait estimates were retrieved, despite using only positively keyed items. Although the best trait accuracy and lowest ipsativity were found using the Bayesian A-rule with questionnaires assembled on-the-fly, the T-rule under this method led to the worst results. This points out to the importance of considering both aspects when designing FC CAT.
Collapse
|
3
|
Lin Y, Brown A, Williams P. Multidimensional Forced-Choice CAT With Dominance Items: An Empirical Comparison With Optimal Static Testing Under Different Desirability Matching. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT 2023; 83:322-350. [PMID: 36866068 PMCID: PMC9972128 DOI: 10.1177/00131644221077637] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
Several forced-choice (FC) computerized adaptive tests (CATs) have emerged in the field of organizational psychology, all of them employing ideal-point items. However, despite most items developed historically follow dominance response models, research on FC CAT using dominance items is limited. Existing research is heavily dominated by simulations and lacking in empirical deployment. This empirical study trialed a FC CAT with dominance items described by the Thurstonian Item Response Theory model with research participants. This study investigated important practical issues such as the implications of adaptive item selection and social desirability balancing criteria on score distributions, measurement accuracy and participant perceptions. Moreover, nonadaptive but optimal tests of similar design were trialed alongside the CATs to provide a baseline for comparison, helping to quantify the return on investment when converting an otherwise-optimized static assessment into an adaptive one. Although the benefit of adaptive item selection in improving measurement precision was confirmed, results also indicated that at shorter test lengths CAT had no notable advantage compared with optimal static tests. Taking a holistic view incorporating both psychometric and operational considerations, implications for the design and deployment of FC assessments in research and practice are discussed.
Collapse
Affiliation(s)
- Yin Lin
- University of Kent, Canterbury,
UK
- SHL, Thames Ditton, Surrey, UK
| | | | | |
Collapse
|
4
|
Lee H. Reduction of faking with the use of a forced‐choice personality test: Cross‐cultural comparisons between South Korea and the United States. INTERNATIONAL JOURNAL OF SELECTION AND ASSESSMENT 2022. [DOI: 10.1111/ijsa.12408] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Affiliation(s)
- HyeSun Lee
- Psychology Department California State University Channel Islands Camarillo California USA
| |
Collapse
|
5
|
Martínez A, Salgado JF, Lado M. Quasi-ipsative Forced-Choice Personality Inventories and the Control of Faking: The Biasing Effects of Transient Error. REVISTA DE PSICOLOGÍA DEL TRABAJO Y DE LAS ORGANIZACIONES 2022. [DOI: 10.5093/jwop2022a16] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
6
|
Block AE, Seifi H, Hilliges O, Gassert R, Kuchenbecker KJ. In the Arms of a Robot: Designing Autonomous Hugging Robots with Intra-Hug Gestures. ACM TRANSACTIONS ON HUMAN-ROBOT INTERACTION 2022. [DOI: 10.1145/3526110] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/05/2023]
Abstract
Hugs are complex affective interactions that often include gestures like squeezes. We present six new guidelines for designing interactive hugging robots, which we validate through two studies with our custom robot. To achieve autonomy, we investigated robot responses to four human intra-hug gestures: holding, rubbing, patting, and squeezing. Thirty-two users each exchanged and rated sixteen hugs with an experimenter-controlled HuggieBot 2.0. The robot’s inflated torso’s microphone and pressure sensor collected data of the subjects’ demonstrations that were used to develop a perceptual algorithm that classifies user actions with 88% accuracy. Users enjoyed robot squeezes, regardless of their performed action, they valued variety in the robot response, and they appreciated robot-initiated intra-hug gestures. From average user ratings, we created a probabilistic behavior algorithm that chooses robot responses in real time. We implemented improvements to the robot platform to create HuggieBot 3.0 and then validated its gesture perception system and behavior algorithm with sixteen users. The robot’s responses and proactive gestures were greatly enjoyed. Users found the robot more natural, enjoyable, and intelligent in the last phase of the experiment than in the first. After the study, they felt more understood by the robot and thought robots were nicer to hug.
Collapse
Affiliation(s)
- Alexis E. Block
- Max Planck Institute for Intelligent Systems and ETH Zürich, Germany
| | | | | | | | | |
Collapse
|
7
|
Martínez A, Salgado JF. A Meta-Analysis of the Faking Resistance of Forced-Choice Personality Inventories. Front Psychol 2021; 12:732241. [PMID: 34659043 PMCID: PMC8511514 DOI: 10.3389/fpsyg.2021.732241] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2021] [Accepted: 09/01/2021] [Indexed: 11/13/2022] Open
Abstract
This study presents a comprehensive meta-analysis on the faking resistance of forced-choice (FC) inventories. The results showed that (1) FC inventories show resistance to faking behavior; (2) the magnitude of faking is higher in experimental contexts than in real-life selection processes, suggesting that the effects of faking may be, in part, a laboratory phenomenon; and (3) quasi-ipsative FC inventories are more resistant to faking than the other FC formats. Smaller effect sizes were found for conscientiousness when the quasi-ipsative format was used (δ = 0.49 vs. δ = 1.27 for ipsative formats). Also, the effect sizes were smaller for the applicant samples than for the experimental samples. Finally, the contributions and practical implications of these findings are discussed.
Collapse
Affiliation(s)
- Alexandra Martínez
- Department of Political Science and Sociology, Faculty of Labor Relations, University of Santiago de Compostela, Santiago de Compostela, Spain
| | | |
Collapse
|
8
|
A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires. Behav Res Methods 2021; 54:1476-1492. [PMID: 34505277 PMCID: PMC9170671 DOI: 10.3758/s13428-021-01677-4] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/25/2021] [Indexed: 11/26/2022]
Abstract
The use of multidimensional forced-choice questionnaires has been proposed as a means of improving validity in the assessment of non-cognitive attributes in high-stakes scenarios. However, the reduced precision of trait estimates in this questionnaire format is an important drawback. Accordingly, this article presents an optimization procedure for assembling pairwise forced-choice questionnaires while maximizing posterior marginal reliabilities. This procedure is performed through the adaptation of a known genetic algorithm (GA) for combinatorial problems. In a simulation study, the efficiency of the proposed procedure was compared with a quasi-brute-force (BF) search. For this purpose, five-dimensional item pools were simulated to emulate the real problem of generating a forced-choice personality questionnaire under the five-factor model. Three factors were manipulated: (1) the length of the questionnaire, (2) the relative item pool size with respect to the questionnaire’s length, and (3) the true correlations between traits. The recovery of the person parameters for each assembled questionnaire was evaluated through the squared correlation between estimated and true parameters, the root mean square error between the estimated and true parameters, the average difference between the estimated and true inter-trait correlations, and the average standard error for each trait level. The proposed GA offered more accurate trait estimates than the BF search within a reasonable computation time in every simulation condition. Such improvements were especially important when measuring correlated traits and when the relative item pool sizes were higher. A user-friendly online implementation of the algorithm was made available to the users.
Collapse
|
9
|
Otero I, Salgado JF, Moscoso S. Criterion Validity of Cognitive Reflection for Predicting Job Performance and Training Proficiency: A Meta-Analysis. Front Psychol 2021; 12:668592. [PMID: 34135827 PMCID: PMC8200478 DOI: 10.3389/fpsyg.2021.668592] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2021] [Accepted: 04/22/2021] [Indexed: 11/13/2022] Open
Abstract
This article presents a meta-analysis of the validity of cognitive reflection (CR) for predicting job performance and training proficiency. It also examines the incremental validity of CR over cognitive intelligence (CI) for predicting these two occupational criteria. CR proved to be an excellent predictor of job performance and training proficiency, and the magnitude of the true validity was very similar across the two criteria. Results also showed that the type of CR is not a moderator of CR validity. We also found that CR showed incremental variance over CI for the explanation of job performance, although the magnitude of the contribution is small. However, CR shows practically no incremental validity over CI validity in the explanation of training proficiency. Finally, we discuss the implications of these findings for the research and practice of personnel selection.
Collapse
Affiliation(s)
- Inmaculada Otero
- Department of Political Science and Sociology, Faculty of Labor Relations, University of Santiago de Compostela, Santiago de Compostela, Spain
| | - Jesús F Salgado
- Department of Political Science and Sociology, Faculty of Labor Relations, University of Santiago de Compostela, Santiago de Compostela, Spain
| | - Silvia Moscoso
- Department of Political Science and Sociology, Faculty of Labor Relations, University of Santiago de Compostela, Santiago de Compostela, Spain
| |
Collapse
|
10
|
Cárdaba-García RM, Pérez Pérez L, Niño Martín V, Cárdaba-García I, Durantez-Fernández C, Olea E. Evaluation of the Risk of Anxiety and/or Depression during Confinement Due to COVID-19 in Central Spain. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2021; 18:ijerph18115732. [PMID: 34071791 PMCID: PMC8199479 DOI: 10.3390/ijerph18115732] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Received: 04/20/2021] [Revised: 05/20/2021] [Accepted: 05/25/2021] [Indexed: 12/23/2022]
Abstract
(1) Background: The confinement of the population in response to the COVID-19 pandemic was related to an increased risk of suffering from anxiety and/or depression in previous studies with other populations. (2) Methods: descriptive study using surveys (Goldberg Anxiety and Depression Scale) with 808 participants over 18 years of age between 14 and 20 of May 2020 during the confinement due to the SARS-CoV-2 virus in Spain. (3) Results: 63% of the participants were at risk of suffering from anxiety and 64.9% were at risk of depression. Variables reaching statistical significance were: age (t anxiety = -0.139 and t depression = -0.153), gender (t anxiety = -4.152 and t depression = -4.178), marital status (anxiety F = 2.893 and depression F = 3.011), symptoms compatible with COVID-19 (t anxiety = -4.177 and t depression = -3.791), previous need for psychological help (t anxiety = -5.385 and t depression = -7.136) and need for such help at the time of the study (t anxiety = -9.144 and depression = -10.995). In addition, we generated two regression models that estimate the risk of anxiety and depression. (4) Conclusions: more than half of the participants were at risk of suffering from anxiety and/or depression, confirming the negative effect of confinement on the population.
Collapse
Affiliation(s)
- Rosa M Cárdaba-García
- Nursing Department, Faculty of Nursing, University of Valladolid, 47005 Valladolid, Spain; (R.M.C.-G.); (L.P.P.); (V.N.M.); (E.O.)
- Nursing Care Research (GICE), Faculty of Nursing, University of Valladolid, 47005 Valladolid, Spain
- Emergencies Management (SACYL), 40002 Segovia, Spain
| | - Lucia Pérez Pérez
- Nursing Department, Faculty of Nursing, University of Valladolid, 47005 Valladolid, Spain; (R.M.C.-G.); (L.P.P.); (V.N.M.); (E.O.)
- Nursing Care Research (GICE), Faculty of Nursing, University of Valladolid, 47005 Valladolid, Spain
- Primary Care Management Valladolid West (SACYL), 47012 Valladolid, Spain
| | - Virtudes Niño Martín
- Nursing Department, Faculty of Nursing, University of Valladolid, 47005 Valladolid, Spain; (R.M.C.-G.); (L.P.P.); (V.N.M.); (E.O.)
- Nursing Care Research (GICE), Faculty of Nursing, University of Valladolid, 47005 Valladolid, Spain
- Primary Care Management Valladolid East (SACYL), 47010 Valladolid, Spain
| | | | - Carlos Durantez-Fernández
- Faculty of Health Sciences, University of Castilla-La Mancha, 45600 Talavera de la Reina, Spain
- Correspondence: ; Tel.: +34-925-72-1010 (ext. 5678)
| | - Elena Olea
- Nursing Department, Faculty of Nursing, University of Valladolid, 47005 Valladolid, Spain; (R.M.C.-G.); (L.P.P.); (V.N.M.); (E.O.)
- Nursing Care Research (GICE), Faculty of Nursing, University of Valladolid, 47005 Valladolid, Spain
- Department of Biochemistry and Molecular Biology and Physiology, University of Valladolid, 47005 Valladolid, Spain
| |
Collapse
|
11
|
Martínez A, Moscoso S, Lado M. Faking Effects on the Factor Structure of a Quasi-Ipsative Forced-Choice Personality Inventory. REVISTA DE PSICOLOGÍA DEL TRABAJO Y DE LAS ORGANIZACIONES 2021. [DOI: 10.5093/jwop2021a7] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
|
12
|
Otero I, Cuadrado D, Martínez A. Convergent and Predictive Validity of the Big Five Factors Assessed with SingleStimulus and Quasi-Ipsative Questionnaires. REVISTA DE PSICOLOGÍA DEL TRABAJO Y DE LAS ORGANIZACIONES 2020. [DOI: 10.5093/jwop2020a17] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
|
13
|
Kreitchmann RS, Abad FJ, Ponsoda V, Nieto MD, Morillo D. Controlling for Response Biases in Self-Report Scales: Forced-Choice vs. Psychometric Modeling of Likert Items. Front Psychol 2019; 10:2309. [PMID: 31681103 PMCID: PMC6803422 DOI: 10.3389/fpsyg.2019.02309] [Citation(s) in RCA: 46] [Impact Index Per Article: 9.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/28/2019] [Accepted: 09/27/2019] [Indexed: 11/23/2022] Open
Abstract
One important problem in the measurement of non-cognitive characteristics such as personality traits and attitudes is that it has traditionally been made through Likert scales, which are susceptible to response biases such as social desirability (SDR) and acquiescent (ACQ) responding. Given the variability of these response styles in the population, ignoring their possible effects on the scores may compromise the fairness and the validity of the assessments. Also, response-style-induced errors of measurement can affect the reliability estimates and overestimate convergent validity by correlating higher with other Likert-scale-based measures. Conversely, it can attenuate the predictive power over non-Likert-based indicators, given that the scores contain more errors. This study compares the validity of the Big Five personality scores obtained: (1) ignoring the SDR and ACQ in graded-scale items (GSQ), (2) accounting for SDR and ACQ with a compensatory IRT model, and (3) using forced-choice blocks with a multi-unidimensional pairwise preference model (MUPP) variant for dominance items. The overall results suggest that ignoring SDR and ACQ offered the worst validity evidence, with a higher correlation between personality and SDR scores. The two remaining strategies have their own advantages and disadvantages. The results from the empirical reliability and the convergent validity analysis indicate that when modeling social desirability with graded-scale items, the SDR factor apparently captures part of the variance of the Agreeableness factor. On the other hand, the correlation between the corrected GSQ-based Openness to Experience scores, and the University Access Examination grades was higher than the one with the uncorrected GSQ-based scores, and considerably higher than that using the estimates from the forced-choice data. Conversely, the criterion-related validity of the Forced Choice Questionnaire (FCQ) scores was similar to the results found in meta-analytic studies, correlating higher with Conscientiousness. Nonetheless, the FCQ-scores had considerably lower reliabilities and would demand administering more blocks. Finally, the results are discussed, and some notes are provided for the treatment of SDR and ACQ in future studies.
Collapse
Affiliation(s)
- Rodrigo Schames Kreitchmann
- Department of Social Psychology and Methodology, Faculty of Psychology, Universidad Autónoma de Madrid, Madrid, Spain
| | - Francisco J Abad
- Department of Social Psychology and Methodology, Faculty of Psychology, Universidad Autónoma de Madrid, Madrid, Spain
| | - Vicente Ponsoda
- Department of Social Psychology and Methodology, Faculty of Psychology, Universidad Autónoma de Madrid, Madrid, Spain
| | - Maria Dolores Nieto
- Department of Social Psychology and Methodology, Faculty of Psychology, Universidad Autónoma de Madrid, Madrid, Spain
| | | |
Collapse
|
14
|
García-Izquierdo AL, Aguado D, Ponsoda-Gil V. New Insights on Technology and Assessment: Introduction to JWOP Special Issue. REVISTA DE PSICOLOGÍA DEL TRABAJO Y DE LAS ORGANIZACIONES 2019. [DOI: 10.5093/jwop2019a6] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
|