Video-based versus paper-and-pencil method of assessment in situational judgment tests: subgroup differences in test performance and face validity perceptions

Development of a situational judgment test to supplement current US air force measures of officership

Aptitude requirements for US Air Force officer commissioning include completion of a college degree and minimum scores on the Air Force Officer Qualifying Test (AFOQT) Verbal and Quantitative composites. Although the AFOQT has demonstrated predictive validity for officer training, the Air Force has striven to improve predictive validity and diversity. To this end, a Situational judgment Test (SJT) was added to the AFOQT in 2015. SJT development was consistent with recommendations to broaden the competencies assessed by the AFOQT with the goal of providing incremental validity, while reducing adverse impact for historically underrepresented groups. To ensure content validity and realism, SJT development was based on competencies identified in a large-scale analysis of officership and input from junior officers in scenario and response generation and scoring. Psychometric evaluations have affirmed its potential benefits for inclusion on the AFOQT. An initial study showed the SJT to be perceived as highly face valid regardless of whether it was presented as a paper-and-pencil test (with narrative or scripted scenarios) or in a video-based format. Preliminary studies demonstrated criterion-related validity within small USAF samples, and a larger Army cadet sample. Additionally, operational administration of the SJT since 2015 has demonstrated its potential for improving diversity (i.e., reduced adverse impact relative to the AFOQT Verbal and Quantitative composites). Predictive validation studies with larger Air Force officer accession samples are ongoing to assess the incremental validity of the SJT beyond current AFOQT composites for predicting important outcomes across accession sources.

Identification and evaluation of criterion measurement methods

Criterion measures vary greatly in terms of their psychometric quality and ease of use. This paper serves two purposes. First, it provides a general summary of different approaches to criterion measurement in a military context. Second, it provides an extensive review of 16 specific types of criterion measurement methods (e.g., job performance rating scales, self-report questionnaires, job knowledge tests) on nine psychometric and ease-of-use evaluation factors. Eight criterion measurement experts read a summary of extant research and made ratings to evaluate each measurement method on the evaluation factors. Rater intra-class correlations (ICCs) were high, ranging from .75 to .95 across the evaluation dimensions with a median of .91. Data showed a quality-feasibility tradeoff, where criterion data that are easy to obtain often have technical flaws. Recommendations for military services and future directions in criterion measurement (e.g., applications of machine learning) are discussed.

Designing criterion measures for physically demanding jobs

The Bureau of Labor Statistics reported that 39.1% of the civilian workforce in the United States performs physically demanding jobs that require lifting, carrying, pushing/pulling, kneeling, stooping, crawling, and climbing activities in varied environmental conditions. United States military occupations are similar to those in the civilian sector involving equipment installation, emergency rescues, and maintenance, along with combat arms occupations. This article provides an overview of the types of criterion measures used to assess the physical domain and approaches for designing and evaluating the criteria. It also includes a method for generating criterion measures that are applicable across multiple jobs.

Applicant perceptions of selection methods for health professions education: Rationales and subgroup differences

CONTEXT

Applicant perceptions of selection methods can affect motivation, performance and withdrawal and may therefore be of relevance in the context of widening access. However, it is unknown how applicant subgroups perceive different selection methods.

OBJECTIVES

Using organisational justice theory, the present multi-site study examined applicant perceptions of various selection methods, rationales behind perceptions and subgroup differences.

METHODS

Applicants to five Dutch undergraduate health professions programmes (N = 704) completed an online survey including demographics and a questionnaire on applicant perceptions applied to 11 commonly used selection methods. Applicants rated general favourability and justice dimensions (7-point Likert scale) and could add comments for each method.

RESULTS

Descriptive statistics revealed a preference for selection methods on which applicants feel more 'in control': General favourability ratings were highest for curriculum-sampling tests (mean [M] = 5.32) and skills tests (M = 5.13), while weighted lottery (M = 3.05) and unweighted lottery (M = 2.97) were perceived least favourable. Additionally, applicants preferred to distinguish themselves on methods that assess attributes beyond cognitive abilities. Qualitative content analysis of comments revealed several conflicting preferences, including a desire for multiple selection methods versus concerns of experiencing too much stress. Results from a linear mixed model of general favourability indicated some small subgroup differences in perceptions (based on gender, migration background, prior education and parental education), but practical meaning of these differences was negligible. Nevertheless, concerns were expressed that certain selection methods can hinder equitable admission due to inequal access to resources.

CONCLUSIONS

Our findings illustrate that applicants desire to demonstrate a variety of attributes on a combination of selection tools, but also observe that this can result in multiple drawbacks. The present study can help programmes in deciding which selection methods to include, which more negatively perceived methods should be better justified to applicants, and how to adapt methods to meet applicants' needs.

Game based assessments of cognitive ability in recruitment: Validity, fairness and test-taking experience

Gamification and machine learning are emergent technologies in recruitment, promising to improve the user experience and fairness of assessments. We test this by validating a game based assessment of cognitive ability with a machine learning based scoring algorithm optimised for validity and fairness. We use applied data from 11,574 assessment completions. The assessment has convergent validity (r = 0.5) and test-retest reliability (r = 0.68). It maintains fairness in a separate sample of 3,107 job applicants, showing that fairness-optimised machine learning can improve outcome parity issues with cognitive ability tests in recruitment settings. We show that there are no significant gender differences in test taking anxiety resulting from the games, and that anxiety does not directly predict game performance, supporting the notion that game based assessments help with test taking anxiety. Interactions between anxiety, gender and performance are explored. Feedback from 4,778 job applicants reveals a Net Promoter score of 58, indicating more applicants support than dislike the assessment, and that games deliver a positive applicant experience in practise. Satisfaction with the format is high, but applicants raise face validity concerns over the abstract games. We encourage the use of gamification and machine learning to improve the fairness and user experience of psychometric tests.

Statistical modelling of vignette data in psychology

Vignette methods are widely used in psychology and the social sciences to obtain responses to multi-dimensional scenarios or situations. Where quantitative data are collected this presents challenges to the selection of an appropriate statistical model. This depends on subtle details of the design and allocation of vignettes to participants. A key distinction is between factorial survey experiments where each participant receives a different allocation of vignettes from the full universe of possible vignettes and experimental vignette studies where this restriction is relaxed. The former leads to nested designs with a single random factor and the latter to designs with two crossed random factors. In addition, the allocation of vignettes to participants may lead to fractional or unbalanced designs and a consequent loss of efficiency or aliasing of the effects of interest. Many vignette studies (including some factorial survey experiments) include unmodeled heterogeneity between vignettes leading to potentially serious problems if traditional regression approaches are adopted. These issues are reviewed and recommendations are made for the efficient design of vignette studies including the allocation of vignettes to participants. Multilevel models are proposed as a general approach to handling nested and crossed designs including unbalanced and fractional designs. This is illustrated with a small vignette data set looking at judgements of online and offline bullying and harassment.

Can a Situational Judgment Test Identify Trainees at Risk of Professionalism Issues? A Multi-Institutional, Prospective Cohort Study

PURPOSE

To determine whether overall situational judgment test (SJT) scores are associated with programs' clinical competency committee (CCC) ratings of trainee professionalism, any concerning behavior, and concerning behavior requiring active remediation at 2 time periods.

METHOD

In fall 2019, trainees from 17 U.S. programs (16 residency, 1 fellowship) took an online 15-scenario SJT developed to measure 7 dimensions of professionalism. CCC midyear and year-end (6 months and 1 year following SJT completion, respectively) professionalism scores and concern ratings were gathered for academic year 2019-2020. Analyses were conducted to determine whether overall SJT scores related to overall professionalism ratings, trainees displaying any concerns, and trainees requiring active remediation at both time periods.

RESULTS

Overall SJT scores correlated positively with midyear and year-end overall professionalism ratings ( r = .21 and .14, P < .001 and = .03, respectively). Holding gender and race/ethnicity constant, a 1 standard deviation (SD) increase in overall SJT score was associated with a .20 SD increase in overall professionalism ratings at midyear ( P = .005) and a .22 SD increase at year-end ( P = .001). Holding gender and race/ethnicity constant, a 1 SD increase in overall SJT score decreased the odds of a trainee displaying any concerns by 37% (odds ratio [OR] 95% confidence interval [CI]: [.44, .87], P = .006) at midyear and 34% (OR 95% CI: [.46, .95], P = .025) at year-end and decreased the odds of a trainee requiring active remediation by 51% (OR 95% CI: [.25, .90], P = .02) at midyear.

CONCLUSIONS

Overall SJT scores correlated positively with midyear and year-end overall professionalism ratings and were associated with whether trainees exhibited any concerning behavior at midyear and year-end and whether trainees needed active remediation at midyear. Future research should investigate whether other potential professionalism measures are associated with concerning trainee behavior.

A video-based situational judgement test of medical students' communication competence in patient encounters: Development and first evaluation

OBJECTIVE

We developed and evaluated the Video-Based Assessment of Medical Communication Competence (VA-MeCo), a construct-driven situational judgement test measuring medical students' communication competence in patient encounters.

METHODS

In the construction phase, we conducted two expert studies (n_panel1 = 6, n_panel2 = 13) to ensure curricular and content validity and sufficient expert agreement on the answer key. In the evaluation phase, we conducted a cognitive pre-test (n = 12) and a pilot study (n = 117) with medical students to evaluate test usability and acceptance, item statistics and test reliability depending on the applied scoring method (raw consensus vs. pairwise comparison scoring).

RESULTS

The results of the expert interviews indicated good curricular and content validity. Expert agreement on the answer key was high (ICCs> .86). The pilot study showed favourable usability and acceptance by students. Irrespective of the scoring method, reliability for the complete test (Cronbach's α >.93) and its subscales (α >.83) was high.

CONCLUSION

There is promising evidence that medical communication competence can be validly and reliably measured using a construct-driven and video-based situational judgement test.

PRACTICE IMPLICATIONS

Video-based SJTs allow efficient online assessment of medical communication competence and are well accepted by students and educators.

Using game-like animations of geometric shapes to simulate social interactions: An evaluation of group score differences

This study introduces a novel, game-like method for measuring social intelligence: the Social Shapes Test. Unlike other existing video or game-based tests, the Shapes Test uses animations of abstract shapes to represent social interactions. We explore demographic differences in Shapes Test scores compared to a written situational judgment test. Gender and race/ethnicity only had meaningful effects on written SJT scores while no effects were found for Shapes Test scores. This pattern of results remained after controlling for general mental ability and English language exposure. We also found metric invariance between demographic groups for both tests. Our results demonstrate the potential for using animated shape tasks as an alternative to written SJTs when designing future game-based assessments.

The selection gap in teacher education: Adverse effects of ethnicity, gender, and socio-economic status on situational judgement test performance

Background

Situational judgement tests (SJTs) measure non‐cognitive attributes and have recently drawn attention as a selection method for initial teacher education programmes. To date, very little is known about adverse impact in teacher selection SJT performance.

Aims

This study aimed to shed light on adverse effects of gender, ethnicity, and socio‐economic status (SES) on SJT scores, by exploring both main effects and interactions, and considering both overall SJT performance and separate SJT domain scores (mindset, emotion regulation, and conscientiousness).

Sample

A total of 2,808 prospective teachers from the United Kingdom completed the SJTs as part of the initial stage of selection into a teacher education programme.

Methods

In addition to SJT scores, the variables gender (female vs. male), ethnicity (majority group vs. minority group), and home SES background (higher SES status vs. lower SES status) were used in the analyses. Regression models and moderated regression models were employed.

Results and conclusions

Results from the regression models revealed that gender effects (females scoring higher than males) were restricted to emotion regulation, while ethnicity effects (ethnic majority group members scoring higher than ethnic minority group members) emerged for SJT overall scores and all three domains. Moderated regression modelling results furthermore showed significant interactions (gender and ethnicity) for SJT overall scores and two domains. Considering the importance of reducing subgroup differences in selection test scores to ensure equal access to teacher education, this study’s findings are a critical contribution. The partially differentiated results for overall vs. domain‐specific scores point towards the promise of applying a domain‐level perspective in research on teacher selection SJTs.

Measuring Motivation to Take Low-Stakes Large-Scale Test: New Model Based on Analyses of "Participant-Own-Defined" Missingness

In large-scale low-stake assessment such as the Programme for International Student Assessment (PISA), students may skip items (missingness) which are within their ability to complete. The detection and taking care of these noneffortful responses, as a measure of test-taking motivation, is an important issue in modern psychometric models. Traditional approaches based on questionnaires and item response theory may have different limitations. In the present research, we proposed a new way by directly using "participant-own-defined" missing item information (user missingness) in a zero-inflated Poisson model. An empirical study using the PISA 2015 data (eight representative economies in two cultures) and another simulation study were conducted to validate our new approach. Results indicated that our model could successfully capture test-taking motivation. We also found that the Confucian students had lower user missingness irrespective of item positions as compared with their Western counterparts.

Gamifying a situational judgment test with immersion and control game elements

Purpose

Assessment gamification, which refers to the addition of game elements to existing assessments, is commonly implemented to improved applicant reactions to existing psychometric measures. This study aims to understand the effects of gamification on applicant reactions to and measurement quality of situational judgment tests.

Design/methodology/approach

In a 2 × 4 between-subjects experiment, this study randomly assigned 315 people to experience different versions of a gamified situational judgment test, crossing immersive game elements (text, audio, still pictures, video) with control game elements (high and low), measuring applicant reactions and assessing differences in convergent validity between conditions.

Findings

The use of immersive game elements improved perceptions of organizational technological sophistication, but no other reactions outcomes (test attitudes, procedural justice, organizational attractiveness). Convergent validity with cognitive ability was not affected by gamification.

Originality/value

This is the first study to experimentally examine applicant reactions and measurement quality to SJTs based upon the implementation of specific game elements. It demonstrates that small-scale efforts to gamify assessments are likely to lead to only small-scale gains. However, it also demonstrates that such modifications can be done without harming the measurement qualities of the test, making gamification a potentially useful marketing tool for assessment specialists. Thus, this study concludes that utility should be considered carefully and explicitly for any attempt to gamify assessment.

Experimental research in expatriation and its challenges: A literature review and recommendations

Via a systematic literature review, this article draws attention to the alarming scarcity of experimental studies and the ensuing shortness of evidence for causality in the field of expatriate management. Only 17 articles could be identified, published over more than 20 years, which utilize randomized experiments or quasi-experiments on topics of expatriation. Moreover, these articles show specific patterns, such as dealing exclusively with pre-departure and on-assignment issues, or, in their majority, sampling individuals who interact with expatriates rather than expatriates themselves. This lack of experimental studies is problematic, as it is difficult to establish causality between different variables without conducting experimental studies. Yet many critical issues in expatriation are precisely questions of causality. Hence, in this article, we provide resources to help move the expatriation field toward a more balanced use of different research methodologies and, thus, a greater understanding of the many relationships uncovered in past research. First, we identify four main challenges unique to conducting experimental research in the context of expatriation: Challenging data access, global sample dispersion, restricted manipulability of variables, and cultural boundedness of constructs and interpretations. Second, we provide strategies to overcome these challenges, based on studies included in the review as well as taking ideas from neighboring fields such as cross-cultural psychology. The article concludes with a discussion of how experimental research can take the field of expatriation forward and improve the decision-making process of practitioners managing international assignees.

What Technology Can and Cannot Do to Support Assessment of Non-cognitive Skills

Advances in technology hold great promise for expanding what assessments may achieve across domains. We focus on non-cognitive skills as our domain, but lessons can be extended to other domains for both the advantages and drawbacks of new technological approaches for different types of assessments. We first briefly review the limitations of traditional assessments of non-cognitive skills. Next, we discuss specific examples of technological advances, considering whether and how they can address such limitations, followed by remaining and new challenges introduced by incorporating technology into non-cognitive assessments. We conclude by noting that technology will not always improve assessments over traditional methods and that careful consideration must be given to the advantages and limitations of each type of assessment relative to the goals and needs of the assessor. The domain of non-cognitive assessments in particular remains limited by lack of agreement and clarity on some constructs and their relations to observable behavior (e.g., self-control versus -regulation versus -discipline), and until these theoretical limitations must be overcome to realize the full benefit of incorporating technology into assessments.

Applicant Perceptions of Initial Job Candidate Screening With Asynchronous Job Interviews

Abstract. Applicant fairness perceptions of asynchronous job interviews were assessed among panelists (Study 1, N = 160) and highly educated actual applicants (Study 2, N = 103). Furthermore, we also examined whether personality explained applicants’ perceptions. Participants, particularly actual applicants, had negative perceptions of the fairness and procedural justice of asynchronous job interviews. Extraverted applicants perceived more opportunity to perform with the asynchronous job interview than introverts. A trait interaction between Neuroticism and Extraversion was tested, but no significant results were found. Although the first selection stage is increasingly digitized, this study shows that applicant perceptions of asynchronous job interviews are relatively negative. The influence of personality on these perceptions appears to be limited.

Situational Judgment Tests as a method for measuring personality: Development and validity evidence for a test of Dependability

Situational Judgment Tests (SJTs) are criterion valid low fidelity measures that have gained much popularity as predictors of job performance. A broad variety of SJTs have been studied, but SJTs measuring personality are still rare. Personality traits such as Conscientiousness are valid predictors of many educational, work and life-related outcomes and SJTs are less prone to faking than classical self-report measurements. We developed an SJT measure of Dependability, a core facet of Conscientiousness, by gathering critical incidents in semi-structured interviews using the construct definition of Dependability as a prompt. We examined the psychometric properties of the newly developed SJTs across two studies (N = 546 general population; N = 440 sales professionals). The internal validity of the SJTs was examined by correlating the SJT scores with related self-report measures of Dependability and Conscientiousness, as well as testing the unidimensionality of the measure with CFA. Additionally, we specified a bi-factor model of SJT, self-report and behavioral checklist measures of Dependability accounting for common and specific measurement variance. External validity was examined by correlating the SJT scale and specific factor with work-related outcomes. The results show that the Dependability SJTs with an expert based scoring procedure were psychometrically sound and correlated moderately to highly with traditional self-report measures of Dependability and Conscientiousness. However, a large proportion of SJT variance cannot be accounted for by personality alone. This supports the notion that SJTs measure general domain knowledge about the effectiveness of personality-related behaviors. We conclude that SJT measures of personality can be a promising addition to classical self-report assessments and can be used in a wide variety of applications beyond measurement and selection, for instance as formative assessments of personality.

Influence of response instructions and response format on applicant perceptions of a situational judgement test for medical school selection

BACKGROUND

This study examined the influence of two Situational Judgement Test (SJT) design features (response instructions and response format) on applicant perceptions. Additionally, we investigated demographic subgroup differences in applicant perceptions of an SJT.

METHODS

Medical school applicants (N = 372) responded to an online survey on applicant perceptions, including a description and two example items of an SJT. Respondents randomly received one of four SJT versions (should do-rating, should do-pick-one, would do-rating, would do-pick-one). They rated overall favourability and items on four procedural justice factors (face validity, applicant differentiation, study relatedness and chance to perform) and ease-of-cheating. Additionally, applicant perceptions were compared for subgroups based on gender, ethnic background and first-generation university status.

RESULTS

Applicants rated would-do instructions as easier to cheat than should-do instructions. Rating formats received more favourable judgements than pick-one formats on applicant differentiation, study-relatedness, chance to perform and ease of cheating. No significant main effect for demographic subgroup on applicant perceptions was found, but significant interaction effects showed that certain subgroups might have more pronounced preferences for certain SJT design features. Specifically, ethnic minority applicants - but not ethnic majority applicants - showed greater preference for should-do than would-do instructions. Additionally, first-generation university students - but not non-first-generation university students - were more favourable of rating formats than of pick-one formats.

CONCLUSIONS

Findings indicate that changing SJT design features may positively affect applicant perceptions by promoting procedural justice factors and reducing perceived ease of cheating and that response instructions and response format can increase the attractiveness of SJTs for minority applicants.

Integrity situational judgement test for medical school selection: judging 'what to do' versus 'what not to do'

CONTEXT

Despite their widespread use in medical school selection, there remains a lack of clarity on exactly what situational judgement tests (SJTs) measure.

OBJECTIVES

We aimed to develop an SJT that measures integrity by combining critical incident interviews (inductive approach) with an innovative deductive approach. The deductive approach guided the development of the SJT according to two established theoretical models, of which one was positively related to integrity (honesty-humility [HH]) and one was negatively related to integrity (cognitive distortions [CD]). The Integrity SJT covered desirable (HH-based) and undesirable (CD-based) response options. We examined the convergent and discriminant validity of the Integrity SJT and compared the validity of the HH-based and CD-based subscores.

METHODS

The Integrity SJT was administered to 402 prospective applicants at a Dutch medical school. The Integrity SJT consisted of 57 scenarios, each followed by four response options, of which two represented HH facets and two represented CD categories. Three SJT scores were computed, including a total, an HH-based and a CD-based score. The validity of these scores was examined according to their relationships with external integrity-related measures (convergent validity) and self-efficacy (discriminant validity).

RESULTS

The three SJT scores correlated significantly with all integrity-related measures and not with self-efficacy, indicating convergent and discriminant validity. In addition, the CD-based SJT score correlated significantly more strongly than the HH-based SJT score with two of the four integrity-related measures.

CONCLUSIONS

An SJT that assesses the ability to correctly recognise CD-based response options as inappropriate (i.e. what one should not do) seems to have stronger convergent validity than an SJT that assesses the ability to correctly recognise HH-based response options as appropriate (i.e. what one should do). This finding might be explained by the larger consensus on what is considered inappropriate than on what is considered appropriate in a challenging situation. It may be promising to focus an SJT on the ability to recognise what one should not do.

Development of a Situational Judgment Test as a Predictor of College Student Performance

It has been suggested that tacit knowledge may be a good predictor of performance in college. The purpose of this study was to investigate the extent to which a situational judgment test developed to measure tacit knowledge correlates with predictors and indicators of college performance. This situational judgment test includes eight situations relevant to the life of college (undergraduate) students and is comprised of 211 behavioral strategies. Four hundred forty-eight college students participated in the study. The results of this study suggest that tacit knowledge has small, statistically nonsignificant correlations with cumulative grade point average (GPA), the percentage of the academic requirements passed on the first attempt, cognitive abilities, achievement motivation, and attention. However, tacit knowledge was found to correlate moderately with the personality factor of agreeableness. The findings do not support claims about the importance of tacit knowledge in academic settings and question what tacit knowledge really is and if it is a useful construct for performance prediction.

A Call for Conceptual Models of Technology in I-O Psychology: An Example From Technology-Based Talent Assessment

The rate of technological change is quickly outpacing today's methods for understanding how new advancements are applied within industrial-organizational (I-O) psychology. To further complicate matters, specific attempts to explain observed differences or measurement equivalence across devices are often atheoretical or fail to explain why a technology should (or should not) affect the measured construct. As a typical example, understanding how technology influences construct measurement in personnel testing and assessment is critical for explaining or predicting other practical issues such as accessibility, security, and scoring. Therefore, theory development is needed to guide research hypotheses, manage expectations, and address these issues at this intersection of technology and I-O psychology. This article is an extension of a Society for Industrial and Organizational Psychology (SIOP) 2016 panel session, which (re)introduces conceptual frameworks that can help explain how and why measurement equivalence or nonequivalence is observed in the context of selection and assessment. We outline three potential conceptual frameworks as candidates for further research, evaluation, and application, and argue for a similar conceptual approach for explaining how technology may influence other psychological phenomena.

Watch his deed or examine his words? Exploring the potential of the behavioral experiment method for collecting data to measure culture

Purpose

To address three issues of survey-based methods (i.e. the absence of behaviors, the reference inequivalence, and the lack of cross-cultural interaction), the purpose of this paper is to explore the potential of using the behavioral experiment method to collect cross-cultural data as well as the possibility of measuring culture with the experimental data. Moreover, challenges to this method and possible solutions are elaborated for intriguing further discussion on the use of behavioral experiments in international business/international management (IB/IM) research.

Design/methodology/approach

This paper illustrates the merits and downside of the proposed method with an ultimate-game experiment conducted in a behavioral laboratory. The procedure of designing, implementing, and analyzing the behavioral experiment is delineated in detail.

Findings

The exploratory findings show that the ultimate-game experiment may observe participants’ behaviors with comparable references and allow for cross-cultural interaction. The findings also suggest that the fairness-related cultural value may be calibrated with the horizontal and vertical convergence of cross-cultural behaviors (i.e. people’s deed), and this calibration may be strengthened by incorporating complementary methods such as a background survey to include people’s words.

Originality/value

The behavioral experiment method illustrated and discussed in this study contributes to the IB/IM literature by addressing three methodological issues that are not widely recognized in the IB/IM literature.

Scoring method of a Situational Judgment Test: influence on internal consistency reliability, adverse impact and correlation with personality?

Situational Judgment Tests (SJTs) are increasingly used for medical school selection. Scoring an SJT is more complicated than scoring a knowledge test, because there are no objectively correct answers. The scoring method of an SJT may influence the construct and concurrent validity and the adverse impact with respect to non-traditional students. Previous research has compared only a small number of scoring methods and has not studied the effect of scoring method on internal consistency reliability. This study compared 28 different scoring methods for a rating SJT on internal consistency reliability, adverse impact and correlation with personality. The scoring methods varied on four aspects: the way of controlling for systematic error, and the type of reference group, distance and central tendency statistic. All scoring methods were applied to a previously validated integrity-based SJT, administered to 931 medical school applicants. Internal consistency reliability varied between .33 and .73, which is likely explained by the dependence of coefficient alpha on the total score variance. All scoring methods led to significantly higher scores for the ethnic majority than for the non-Western minorities, with effect sizes ranging from 0.48 to 0.66. Eighteen scoring methods showed a significant small positive correlation with agreeableness. Four scoring methods showed a significant small positive correlation with conscientiousness. The way of controlling for systematic error was the most influential scoring method aspect. These results suggest that the increased use of SJTs for selection into medical school must be accompanied by a thorough examination of the scoring method to be used.

Development and Validation of a Bilingual Stroke Preparedness Assessment Instrument

BACKGROUND AND PURPOSE

Stroke preparedness interventions are limited by the lack of psychometrically sound intermediate end points. We sought to develop and assess the reliability and validity of the video-Stroke Action Test (video-STAT) an English and a Spanish video-based test to assess people's ability to recognize and react to stroke signs.

METHODS

Video-STAT development and testing was divided into 4 phases: (1) video development and community-generated response options, (2) pilot testing in community health centers, (3) administration in a national sample, bilingual sample, and neurologist sample, and (4) administration before and after a stroke preparedness intervention.

RESULTS

The final version of the video-STAT included 8 videos: 4 acute stroke/emergency, 2 prior stroke/nonemergency, 1 nonstroke/emergency, and 1 nonstroke/nonemergency. Acute stroke recognition and action response were queried after each vignette. Video-STAT scoring was based on the acute stroke vignettes only (score range 0-12 best). The national sample consisted of 598 participants, 438 who took the video-STAT in English and 160 who took the video-STAT in Spanish. There was adequate internal consistency (Cronbach α=0.72). The average video-STAT score was 5.6 (SD=3.6), whereas the average neurologist score was 11.4 (SD=1.3). There was no difference in video-STAT scores between the 116 bilingual video-STAT participants who took the video-STAT in English or Spanish. Compared with baseline scores, the video-STAT scores increased after a stroke preparedness intervention (6.2 versus 8.9, P<0.01) among a sample of 101 black adults and youth.

CONCLUSIONS

The video-STAT yields reliable scores that seem to be valid measures of stroke preparedness.

Social emotional information processing in adults: Development and psychometrics of a computerized video assessment in healthy controls and aggressive individuals

A computerized version of an assessment of Social-Emotional Information Processing (SEIP) using audio-video film stimuli instead of written narrative vignettes was developed for use in adult participants. This task allows for an assessment of encoding or relevant/irrelevant social-emotional information, attribution bias, and endorsement of appropriate, physically aggressive, and relationally aggressive responses to aversive social-emotional stimuli. The psychometric properties of this Video-SEIP (V-SEIP) assessment were examined in 75 healthy controls (HC) and in 75 individuals with DSM-5 Intermittent Explosive Disorder (IED) and were also compared with the original questionnaire (SEIP-Q) version of the task (HC=26; IED=26). Internal consistency, inter-rater reliability, and test-retest properties of the V-SEIP were good to excellent. In addition, IED participants displayed reduced encoding of relevant information from the film clips, elevated hostile attribution bias, elevated negative emotional response, and elevated endorsement of physically aggressive and relationally aggressive responses to the ambiguous social-emotional stimuli presented in the V-SEIP. These data indicate that the V-SEIP represents a valid and comprehensive alternative to the paper-and-pencil assessment of social-emotional information processing biases in adults.

Does the UKCAT predict performance on exit from medical school? A national cohort study

OBJECTIVES

Most UK medical programmes use aptitude tests during student selection, but large-scale studies of predictive validity are rare. This study assesses the UK Clinical Aptitude Test (UKCAT: http://www.ukcat.ac.uk), and 4 of its subscales, along with individual and contextual socioeconomic background factors, as predictors of performance during, and on exit from, medical school.

METHODS

This was an observational study of 6294 medical students from 30 UK medical programmes who took the UKCAT from 2006 to 2008, for whom selection data from the UK Foundation Programme (UKFPO), the next stage of UK medical education training, were available in 2013. We included candidate demographics, UKCAT (cognitive domains; total scores), UKFPO Educational Performance Measure (EPM) and national exit situational judgement test (SJT). Multilevel modelling was used to assess relationships between variables, adjusting for confounders.

RESULTS

The UKCAT-as a total score and in terms of the subtest scores-has significant predictive validity for performance on the UKFPO EPM and SJT. UKFPO performance was also affected positively by female gender, maturity, white ethnicity and coming from a higher social class area at the time of application to medical school An inverse pattern was seen for a contextual measure of school, with those attending fee-paying schools performing significantly more weakly on the EPM decile, the EPM total and the total UKFPO score, but not the SJT, than those attending other types of school.

CONCLUSIONS

This large-scale study, the first to link 2 national databases-UKCAT and UKFPO, has shown that UKCAT is a predictor of medical school outcome. The data provide modest supportive evidence for the UKCAT's role in student selection. The conflicting relationships of socioeconomic contextual measures (area and school) with outcome adds to wider debates about the limitations of these measures, and indicates the need for further research.

The Conceptualization and Analysis of Change Over Time: An Integrative Approach Incorporating Longitudinal Mean and Covariance Structures Analysis (LMACS) and Multiple Indicator Latent Growth Modeling (MLGM)

The concept of change over time is fundamental to many phenomena investigated in organizational research. This didactically oriented article proposes an integrative approach incorporating longitudinal mean and covariance structures analysis and multiple indicator latent growth modeling to aid organizational researchers in directly addressing fundamental questions concerning the conceptualization and analysis of change over time. The approach is illustrated using a numerical example involving several organizationally relevant variables. Advantages, limitations, and extensions of the approach are discussed.

Applications of Mean and Covariance Structure Analysis: Integrating Correlational and Experimental Approaches

This article discusses mean and covariance structure (MACS) analysis as a mechanism for testing individual differences and group mean differences within a single integrated framework. The article is most appropriate for those with a basic understanding of confirmatory factor analysis and structural equation modeling but who have not been introduced to mean structures in either model. The article is both a tutorial and an extension of previous research. It is a tutorial because it introduces the MACS framework using more familiar regression terms, proposes a model-testing framework to be followed, provides numerous illustrations and applications, and discusses both conceptual and programming issues (as well as providing the code for the programming). It is also an extension of previous research because it illustrates several novel applications of MACS to organizational research questions, including how to (a) model mean differences across three independent groups, (b) conduct latent pairwise comparisons, (c) conduct latent contrasts, and (d) analyze repeated measures data. As these examples illustrate, applying MACS more frequently in those organizational research designs that call for it could perhaps improve theory testing by helping to integrate experimental and correlational research methods.

A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research

The establishment of measurement invariance across groups is a logical prerequisite to conducting substantive cross-group comparisons (e.g., tests of group mean differences, invariance of structural parameter estimates), but measurement invariance is rarely tested in organizational research. In this article, the authors (a) elaborate the importance of conducting tests of measurement invariance across groups, (b) review recommended practices for conducting tests of measurement invariance, (c) review applications of measurement invariance tests in substantive applications, (d) discuss issues involved in tests of various aspects of measurement invariance, (e) present an empirical example of the analysis of longitudinal measurement invariance, and (f) propose an integrative paradigm for conducting sequences of measurement invariance tests.

Problems With Item Parceling for Confirmatory Factor Analytic Tests of Measurement Invariance

Combining items into parcels in confirmatory factor analysis (CFA) can improve model estimation and fit. Because adequate model fit is imperative for CFA tests of measurement invariance, parcels have frequently been used. However, the use of parcels as indicators in a CFA model can have serious detrimental effects on tests of measurement invariance. Using simulated data with a known lack of invariance, the authors illustrate how models using parcels as indicator variables erroneously indicate that measurement invariance exists much more often than do models using items as indicators. Moreover, item-by-item tests of measurement invariance were often more informative than were tests of the entire parameter matrices.

Current Directions in Personnel Selection Research

For many decades, the focus of personnel selection research was on developing selection tests that maximized prediction of job performance; the approach was typically lacking in theoretical bases. The past two decades saw significant shifts in research to a focus on the nature of constructs and their interrelationships, characterized by an approach that emphasizes theoretical understanding of the phenomena under investigation. This article provides an overview of how a construct-oriented approach underlies major current directions in scientific research on personnel selection. Emerging trends that are likely to constitute issues of enduring importance are discussed.