Understanding pretest and posttest reactions to cognitive ability and personality tests

Applicant Reactions to the AAMC Standardized Video Interview During the 2018 Application Cycle

PURPOSE

This study examined applicant reactions to the Association of American Medical Colleges Standardized Video Interview (SVI) during its first year of operational use in emergency medicine (EM) residency program selection to identify strategies to improve applicants' SVI experience and attitudes.

METHOD

Individuals who self-classified as EM applicants applying in the Electronic Residency Application Service 2018 cycle and who completed the SVI in summer 2017 were invited to participate in 2 surveys. Survey 1, which focused on procedural issues, was administered immediately after SVI completion. Survey 2, which focused on applicants' SVI experience, was administered in fall 2017, after SVI scores were released.

RESULTS

The response rates for surveys 1 and 2 were 82.3% (2,906/3,532) and 58.7% (2,074/3,532), respectively. Applicant reactions varied by aspect of the SVI studied and their SVI total scores. Most applicants were satisfied with most procedural aspects of the SVI, but most applicants were not satisfied with the SVI overall or with their total SVI scores. About 20% to 30% of applicants had neutral opinions about most aspects of the SVI. Negative reactions to the SVI were stronger for applicants who scored lower on the SVI.

CONCLUSIONS

Applicants had generally negative reactions to the SVI. Most were skeptical of its ability to assess the target competencies and its potential to add value to the selection process. Applicant acceptance and appreciation of the SVI will be critical to the SVI's acceptance by the graduate medical education community.

Applicant Perceptions of Initial Job Candidate Screening With Asynchronous Job Interviews

Abstract. Applicant fairness perceptions of asynchronous job interviews were assessed among panelists (Study 1, N = 160) and highly educated actual applicants (Study 2, N = 103). Furthermore, we also examined whether personality explained applicants’ perceptions. Participants, particularly actual applicants, had negative perceptions of the fairness and procedural justice of asynchronous job interviews. Extraverted applicants perceived more opportunity to perform with the asynchronous job interview than introverts. A trait interaction between Neuroticism and Extraversion was tested, but no significant results were found. Although the first selection stage is increasingly digitized, this study shows that applicant perceptions of asynchronous job interviews are relatively negative. The influence of personality on these perceptions appears to be limited.

Auswahlverfahren für Masterstudiengänge aus Sicht von Bachelorstudierenden

Zusammenfassung. Eine Studierendenauswahl mit Hilfe prognostisch valider Verfahren, welche von den Bewerberinnen und Bewerbern wertgeschätzt werden, kann die Wettbewerbsfähigkeit von Hochschulen stärken. Diese multimethodale Studie untersuchte bei 394 Bachelorstudierenden die Akzeptanz von elf Verfahren der Studierendenauswahl für Masterstudiengänge. Die Methoden zur Akzeptanzmessung umfassten eine Rangreihung, eine Ratingskala und einen vollständigen Paarvergleich. Methodenübergeifend erhielten studienfachspezifische Fähigkeitstests und Bachelornoten die höchste Akzeptanz. Akzeptanzwerte korrelierten positiv mit der selbsteingeschätzten Leistung in Vorerfahrungen mit den Verfahren; es zeigten sich geringe Zusammenhänge mit den Big-Five-Persönlichkeitsfaktoren und dem akademischen Selbstkonzept. Die Akzeptanz von strukturierten und frei geführten Auswahlgesprächen sowie von Persönlichkeitstests variierte abhängig vom Studienfach der Bachelorstudierenden. Die Ergebnisse können als Orientierung bei der Gestaltung der Studierendenauswahl für Masterstudiengänge dienen.

A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research

The establishment of measurement invariance across groups is a logical prerequisite to conducting substantive cross-group comparisons (e.g., tests of group mean differences, invariance of structural parameter estimates), but measurement invariance is rarely tested in organizational research. In this article, the authors (a) elaborate the importance of conducting tests of measurement invariance across groups, (b) review recommended practices for conducting tests of measurement invariance, (c) review applications of measurement invariance tests in substantive applications, (d) discuss issues involved in tests of various aspects of measurement invariance, (e) present an empirical example of the analysis of longitudinal measurement invariance, and (f) propose an integrative paradigm for conducting sequences of measurement invariance tests.

Situational judgement tests in medical education and training: Research, theory and practice: AMEE Guide No. 100

Why use SJTs? Traditionally, selection into medical education professions has focused primarily upon academic ability alone. This approach has been questioned more recently, as although academic attainment predicts performance early in training, research shows it has less predictive power for demonstrating competence in postgraduate clinical practice. Such evidence, coupled with an increasing focus on individuals working in healthcare roles displaying the core values of compassionate care, benevolence and respect, illustrates that individuals should be selected on attributes other than academic ability alone. Moreover, there are mounting calls to widen access to medicine, to ensure that selection methods do not unfairly disadvantage individuals from specific groups (e.g. regarding ethnicity or socio-economic status), so that the future workforce adequately represents society as a whole. These drivers necessitate a method of assessment that allows individuals to be selected on important non-academic attributes that are desirable in healthcare professionals, in a fair, reliable and valid way. What are SJTs? Situational judgement tests (SJTs) are tests used to assess individuals' reactions to a number of hypothetical role-relevant scenarios, which reflect situations candidates are likely to encounter in the target role. These scenarios are based on a detailed analysis of the role and should be developed in collaboration with subject matter experts, in order to accurately assess the key attributes that are associated with competent performance. From a theoretical perspective, SJTs are believed to measure prosocial Implicit Trait Policies (ITPs), which are shaped by socialisation processes that teach the utility of expressing certain traits in different settings such as agreeable expressions (e.g. helping others in need), or disagreeable actions (e.g. advancing ones own interest at others, expense). Are SJTs reliable, valid and fair? Several studies, including good quality meta-analytic and longitudinal research, consistently show that SJTs used in many different occupational groups are reliable and valid. Although there is over 40 years of research evidence available on SJTs, it is only within the past 10 years that SJTs have been used for recruitment into medicine. Specifically, evidence consistently shows that SJTs used in medical selection have good reliability, and predict performance across a range of medical professions, including performance in general practice, in early years (foundation training as a junior doctor) and for medical school admissions. In addition, SJTs have been found to have significant added value (incremental validity) over and above other selection methods such as knowledge tests, measures of cognitive ability, personality tests and application forms. Regarding differential attainment, generally SJTs have been found to have lower adverse impact compared to other selection methods, such as cognitive ability tests. SJTs have the benefit of being appropriate both for use in selection where candidates are novices (i.e. have no prior role experience or knowledge such as in medical school admissions) as well as settings where candidates have substantial job knowledge and specific experience (as in postgraduate recruitment for more senior roles). An SJT specification (e.g. scenario content, response instructions and format) may differ depending on the level of job knowledge required. Research consistently shows that SJTs are usually found to be positively received by candidates compared to other selection tests such as cognitive ability and personality tests. Practically, SJTs are difficult to design effectively, and significant expertise is required to build a reliable and valid SJT. Once designed however, SJTs are cost efficient to administer to large numbers of candidates compared to other tests of non-academic attributes (e.g. personal statements, structured interviews), as they are standardised and can be computer-delivered and machine-marked.

Applicant reactions and faking in real-life personnel selection

Faking may affect hiring decisions in personnel selection. All the antecedents of faking are still not known. The present study investigates the association between applicants' reactions about the selection procedure and their tendency to fake. The subjects (N = 180) were real-life applicants for a fire and rescue personnel school. After completing the selection process, the applicants filled out a questionnaire about their test reactions (Chan, Schmitt, Sacco & DeSohon, 1998b) and a faking scale, the Balanced Inventory of Desirable Responding (Paulhus, 1991). The results based on Structural Equation Modelling (SEM) indicated that the more positive reactions applicant had about the selection procedure the more impression management they had. The applicant reactions were not associated with self-deception.

Evaluating candidate reactions to selection practices using organisational justice theory

OBJECTIVES

This study aimed to examine candidate reactions to selection practices in postgraduate medical training using organisational justice theory.

METHODS

We carried out three independent cross-sectional studies using samples from three consecutive annual recruitment rounds. Data were gathered from candidates applying for entry into UK general practice (GP) training during 2007, 2008 and 2009. Participants completed an evaluation questionnaire immediately after the short-listing stage and after the selection centre (interview) stage. Participants were doctors applying for GP training in the UK. Main outcome measures were participants' evaluations of the selection methods and perceptions of the overall fairness of each selection stage (short-listing and selection centre).

RESULTS

A total of 23,855 evaluation questionnaires were completed (6893 in 2007, 10,497 in 2008 and 6465 in 2009). Absolute levels of perceptions of fairness of all the selection methods at both the short-listing and selection centre stages were consistently high over the 3years. Similarly, all selection methods were considered to be job-related by candidates. However, in general, candidates considered the selection centre stage to be significantly fairer than the short-listing stage. Of all the selection methods, the simulated patient consultation completed at the selection centre stage was rated as the most job-relevant.

CONCLUSIONS

This is the first study to use a model of organisational justice theory to evaluate candidate reactions during selection into postgraduate specialty training. The high-fidelity selection methods are consistently viewed as more job-relevant and fairer by candidates. This has important implications for the design of recruitment systems for all specialties and, potentially, for medical school admissions. Using this approach, recruiters can systematically compare perceptions of the fairness and job relevance of various selection methods.

Applicant Reactions to Rejection

While applicant reactions to selection (un)fairness have often been investigated, less is known about applicants’ attributions and reactions to specific performance feedback. This paper discusses two studies into the influence of fairness perceptions and attributional processing on well-being and organizational perceptions after rejection. In both studies, distributive fairness and attributional style interactively influenced post-rejection well-being, with optimistically attributing individuals showing higher well-being when fairness perceptions were low. In Study 2, performance feedback negatively influenced rejected individuals’ well-being, and influenced the interaction effect of fairness and attributional style. Distributive fairness positively affected post-rejection organizational perceptions. Finally, performance feedback and attributional style interactively influenced post-rejection organizational perceptions. Implications for future research and practice concerning this issue are considered.

Validität des BIP und des NEO-PI-R

Zusammenfassung. Die vorliegende Studie untersuchte die konkurrente Validität eines explizit berufsbezogenen (BIP) und eines allgemeinen Persönlichkeitsverfahrens (NEO-PI-R) zur Erklärung objektiver und subjektiver Berufserfolgskriterien. Als objektive Kriterien wurden Bruttoeinkommen und Berufsstatus erhoben, stellen- sowie umfeldbezogene Arbeitszufriedenheit und der subjektiv eingeschätzte Berufserfolg dienten hingegen als subjektive Kriterien des beruflichen Erfolges. Anhand einer Stichprobe berufstätiger Erwachsener wurde einerseits die Validität der beiden Persönlichkeitstests separat untersucht, andererseits wurde der inkrementelle Anteil bestimmt, den BIP und NEO-PI-R zur Varianzaufklärung über den jeweils anderen Test hinaus lieferten. Unter Kontrolle verschiedener mit Berufserfolg in Beziehung stehender Variablen (Alter, Geschlecht, Ausbildungsniveau, Durchschnittsnote, Dauer der Tätigkeit) leisteten beide Verfahren bedeutsame Beiträge zur Erklärung der Varianz objektiver und subjektiver Berufserfolgskriterien. Darüber hinaus leisteten beide Inventare vergleichbare Beiträge zur Varianzaufklärung über das jeweils andere Verfahren hinaus. Auch in Bezug auf die von den Teilnehmern beurteilte Akzeptanz unterschieden sich die beiden Verfahren nicht substanziell voneinander. Implikationen dieser Befunde werden in Hinblick auf bisherige Ergebnisse zum Zusammenhang zwischen Persönlichkeitsvariablen und Berufserfolg diskutiert.

Consequences of organizational justice expectations in a selection system

This study examined several consequences of applicants' expectations of organizational justice at multiple stages in a selection process. The authors assessed the justice expectations of 1,832 job applicants prior to their participation in a testing process and examined how these expectations influenced their pretest attitudes and intentions as well as their perceptions of the testing process. Results revealed that applicants with higher expectations of justice reported higher levels of pretest motivation and more positive job acceptance and recommendation intentions. Justice expectations were also positively related to applicants' perceptions of justice in the testing process. Results provided some evidence that justice expectations have a moderating influence, such that justice perceptions have a greater influence on applicants' affective and cognitive states when expectations of justice are high. The theoretical and practical implications of these findings are discussed in the context of research on organizational justice and applicant perceptions.

Traditional tests and job simulations: minority and majority performance and test validities

Performance of 565 minority and 414 majority job applicants on traditional paper-and-pencil tests and on a telephone simulation that measured similar constructs was assessed. Models of the measurement characteristics of these 2 test batteries indicated larger subgroup mean differences on the traditional tests than on the simulation. Correlations between traditional tests were lower in the majority sample than in the minority sample, and the variance of the minority candidates' scores on the traditional tests was much larger than the variance of majority applicants' scores on the same measures. The validity of the simulation was lower than the validity of the traditional tests. This study replicates previous laboratory research that has indicated smaller subgroup differences on simulations than on paper-and-pencil tests and extends this research by providing evidence of the relative validity of these 2 types of measures.

Personnel selection: looking toward the future--remembering the past

This chapter reviews personnel selection research from 1995 through 1999. Areas covered are job analysis; performance criteria; cognitive ability and personality predictors; interview, assessment center, and biodata assessment methods; measurement issues; meta-analysis and validity generalization; evaluation of selection systems in terms of differential prediction, adverse impact, utility, and applicant reactions; emerging topics on team selection and cross-cultural issues; and finally professional, legal, and ethical standards. Three major themes are revealed: (a) Better taxonomies produce better selection decisions; (b) The nature and analyses of work behavior are changing, influencing personnel selection practices; (c) The field of personality research is healthy, as new measurement methods, personality constructs, and compound constructs of well-known traits are being researched and applied to personnel selection.

Development and examination of an expectancy-based measure of test-taking motivation

A 10-item multidimensional measure of test-taking motivation based on expectancy theory, the Valence, Instrumentality, Expectancy Motivation Scale (VIEMS), was developed using a student sample (N = 90) and tested using 2 samples of job applicants in a field setting (N = 296; N = 246). In Field Study 1, the VIEMS was related to test performance. Hierarchical regression analysis showed that the VIEMS explained variance in test score beyond a general measure of test motivation. In a second longitudinal field study, pretest and posttest perceptions of motivation were compared. Results indicated that expectancy was related to actual test performance, and perceived test performance accounted for variance in posttest reports of motivation after controlling for pretest levels of motivation. Test-taking motivation did not account for variance in test performance differences between African Americans and Whites in either field study.

Applicant self-selection: correlates of withdrawal from a multiple hurdle process

The authors examined applicant self-selection from a multiple hurdle hiring process. The relationships of the selection status of 3,550 police applicants (self-selected out prior to 1 of the hurdles, passing, or failing) and perceptions of the organization, commitment to a law enforcement job, expectations regarding the job, employment status, the need to relocate, the opinions of family and friends, and perceptions of the hiring process were examined. Differences between those who stayed in the process and those who self-selected out were observed in most areas, and those who self-selected out at early stages differed from those self-selecting out at later stages. African Americans' and women's perceptions also differed from the majority group, indicating some of the difficulties an organization faces in attempting to diversify.