An RCT of Virtual Reality Job Interview Training for Individuals With Serious Mental Illness in IPS Supported Employment

Scaling Up Evidence-Based Supported Employment in the United States

Implementing evidence-based practices has become a standard policy in U.S. behavioral health care. However, current penetration remains inadequate, and longitudinal studies of scaling up these practices are rare. This article examined the steady, 3-decade growth of individual placement and support (IPS), the evidence-based model of employment services for people with psychiatric disabilities. The number of IPS programs in the United States has increased to >1,000. Nevertheless, coverage remains limited, and challenges to further growth are substantial. Facilitators of IPS growth over the past few decades include client demand, scientific evidence, initial financial support from foundations, subsequent federal agency support, state leadership in behavioral health and vocational rehabilitation, standardized training and quality assurance methods, an international learning community, and the collection of national data to establish benchmarks. Challenges to further expansion of IPS include the complexity of multiple partial funding sources; the increasing use of information technology for training, implementation, and quality assurance; extension to new populations and settings; outreach to historically marginalized groups (e.g., young adults not engaged in the mental health system); employer partnerships; and adaptations related to cultural, workforce, and workplace changes. Developing reliable research methods to assess and compare scaling-up strategies is also critical.

A type 1 hybrid multi-site randomized controlled trial protocol for evaluating virtual interview training among autistic transition-age youth

A number of policies mandate that autistic transition-age youth receive employment services to prepare for the workforce before high school graduation. A key limitation to these services is the job interview component, which relies on non-standardized, resource-intensive, staff-led role-plays to help autistic transition-age youth improve their interview skills. The autism community has called for better job interview preparation. To address this gap in services, our team, collaborated with the autism community to adapt the intervention, Virtual Reality Job Interview Training (VR-JIT; effective among adults with serious mental illness), into Virtual Interview Training for Transition Age Youth (VIT-TAY). This adapted intervention was tailored to meet the needs of autistic transition age youth while maintaining the core components of VR-JIT (i.e., an online job interview simulator with four levels of automated feedback and e-learning content). A pilot randomized controlled trial (RCT) demonstrated VIT-TAY's feasibility and initial effectiveness at improving job interview skills, reducing anxiety, and increasing employment rates within six months when added to transition services or pre-employment transition services (Pre-ETS). Thus, the overarching goal of this Hybrid Type 1 effectiveness-implementation study protocol is to conduct a fully-powered RCT of VIT-TAY across 16 schools in various geographical locations. Our specific aims are to 1) Evaluate whether Pre-ETS (or transition services) with VIT-TAY, as compared to Pre-ETS (or transition services) with an active control intervention (i.e., job interview didactics/e-learning with a series of 3-5 min videos of employed autistic adults talking about their career pathways) enhances employment outcomes; 2) Evaluate mechanisms of employment by nine months post-randomization; and 3) Conduct a multilevel, mixed-method process evaluation of the initial implementation of VIT-TAY across settings (e.g., acceptability, feasibility, and barriers and facilitators of implementation).

Thinking About the Future of Cognitive Remediation Therapy Revisited: What Is Left to Solve Before Patients Have Access?

In our previous paper on the Future of Cognitive Remediation published more than 10 years ago, we envisaged an imminent and wide implementation of cognitive remediation therapies into mental health services. This optimism was misplaced. Despite evidence of the benefits, costs, and savings of this intervention, access is still sparse. The therapy has made its way into some treatment guidance, but these documents weight the same evidence very differently, causing confusion, and do not consider barriers to implementation. This paper revisits our previous agenda and describes how some challenges were overcome but some remain. The scientific community, with its commitment to Open Science, has produced promising sets of empirical data to explore the mechanisms of treatment action. This same community needs to understand the specific and nonspecific effects of cognitive remediation if we are to provide a formulation-based approach that can be widely implemented. In the last 10 years we have learned that cognitive remediation is not "brain training" but is a holistic therapy that involves an active therapist providing motivation support, and who helps to mitigate the impact of cognitive difficulties through metacognition to develop awareness of cognitive approaches to problems. We conclude that, of course, more research is needed but, in addition and perhaps more importantly at this stage, we need more public and health professionals' understanding of the benefits of this therapy to inform and include this approach as part of treatment regimens.

Virtual Job Interview Training: A Dose Response to Improve Employment for Transition-Age Youth With Disabilities

Virtual Interview Training for Transition Age Youth and Virtual Reality Job Interview Training are job interview simulators with demonstrated effectiveness in randomized controlled trials. We evaluated their dose responses via secondary data analysis of 558 transition-age youth with disabilities in 47 schools where the simulators were implemented in quasi-experimental studies. Cut-point analyses determined dosing efficiency and efficacy to optimize competitive employment. The most efficient dose when accounting for the balance between dose and employment was completing nine virtual interviews. The most efficacious dose to maximize the likelihood of successful employment was 38, but varied across race, IQ, IDEA categories and employment history. This study provides a novel approach to inform implementation guidelines for virtual interview training in pre-employment transition services. Limitations and implications for research and practice are discussed.

A Mixed-Methods Implementation Evaluation of Virtual Reality Job Interview Training in IPS Supported Employment

OBJECTIVE

Employment rates among individuals with serious mental illness may be improved by engagement in the individual placement and support (IPS) model of supported employment. Results from a recent randomized controlled trial (RCT) indicate that virtual reality job interview training (VR-JIT) improves employment rates among individuals with serious mental illness who have been actively engaged in IPS for at least 90 days. This study reports on an initial implementation evaluation of VR-JIT during the RCT in a community mental health agency.

METHODS

A sequential, complementary mixed-methods design included use of qualitative data to improve understanding of quantitative findings. Thirteen IPS staff trained to lead VR-JIT implementation completed VR-JIT acceptability, appropriateness, and feasibility surveys. Participants randomly assigned to IPS with VR-JIT completed acceptability (N=42) and usability (N=28) surveys after implementation. The authors also conducted five focus groups with IPS staff (N=11) and VR-JIT recipients (N=13) and semistructured interviews with IPS staff (N=9) and VR-JIT recipients (N=4), followed by an integrated analysis process.

RESULTS

Quantitative results suggest that IPS staff found VR-JIT to be highly acceptable, appropriate for integration with IPS, and feasible for delivery. VR-JIT was highly acceptable to recipients. Qualitative results add important context to the quantitative findings, including benefits of VR-JIT for IPS staff as well as adaptations for delivering technology-based interventions to individuals with serious mental illness.

CONCLUSIONS

These qualitative and quantitative findings are consistent with each other and were influenced by VR-JIT's adaptability and perceived benefits. Tailoring VR-JIT instruction and delivery to individuals with serious mental illness may help optimize VR-JIT implementation within IPS.

Implementation Preparation Costs of Virtual Interview Training in Pre-Employment Transition Services: A Budget Impact Analysis

Virtual Reality Job Interview Training (VR-JIT) and Virtual Interview Training for Transition Age Youth (VIT-TAY) demonstrated initial effectiveness at increasing employment among transition-age youth with disabilities engaged in pre-employment transition services. We characterized activities and estimated the labor and non-labor costs required to prepare schools to implement VR-JIT or VIT-TAY. Implementation preparation and support teams reported labor hours throughout the implementation preparation process. Implementation preparation labor hours at 43 schools cost approximately $1,427 per school, while non-labor costs were $100 per trainee (student). We estimated the replication of implementation preparation labor activities would cost $1,024 per school (range: $841-$1,208). Most costs were spent in delivery planning and teacher training. Given that implementation preparation costs can be barriers to intervention adoption, our results provide critical information for contemplating future implementation of VR-JIT or VIT-TAY.

Implementing Virtual Interview Training for Transition-Age Youth: Practical Strategies for Educators

This column shares strategies for implementing a technology-based intervention within secondary special education transition services. Virtual Interview Training for Transition-Age Youth is a web-based job interview training that uses educational content and virtual hiring managers (Rita and Travis) to help teach job interview skills. Eight strategies were assembled by research staff, teachers, and administrators during two research studies conducted across n=47 schools that delivered the program (commonly called "Rita and Travis") to n=561 students in secondary special education transition programs. Eight support strategies for implementing Rita and Travis are described. Strategies include: prepare for implementation, ensure administrative support, prioritize training and support, address logistical challenges, integrate the program into the curriculum, take advantage of the administrative center, individualize student support, and offer virtual delivery. The purpose of this column is to prepare others for successful implementation of Virtual Interview Training for Transition-Age Youth within special education transition programs.

Psychometric properties of the mock interview rating scale for autistic transition-age youth

Background

Employment is a major contributor to quality of life. However, autistic people are often unemployed and underemployed. One potential barrier to employment is the job interview. However, the availability of psychometrically-evaluated assessments of job interviewing skills is limited for autism services providers and researchers.

Objective

We analyzed the psychometric properties of the Mock Interview Rating Scale that was adapted for research with autistic transition-age youth (A-MIRS; a comprehensive assessment of video-recorded job interview role-play scenarios using anchor-based ratings for 14 scripted job scenarios).

Methods

Eighty-five transition-age youth with autism completed one of two randomized controlled trials to test the effectiveness of two interventions focused on job interview skills. All participants completed a single job interview role-play at pre-test that was scored by raters using the A-MIRS. We analyzed the structure of the A-MIRS using classical test theory, which involved conducting both exploratory and confirmatory factor analyzes, Rasch model analysis and calibration techniques. We then assessed internal consistency, inter-rater reliability, and test-retest reliability. Pearson correlations were used to assess the A-MIRS' construct, convergent, divergent, criterion, and predictive validities by comparing it to demographic, clinical, cognitive, work history measures, and employment outcomes.

Results

Results revealed an 11-item unidimensional construct with strong internal consistency, inter-rater reliability, and test-retest reliability. Construct [pragmatic social skills (r = 0.61, p < 0.001), self-reported interview skills (r = 0.34, p = 0.001)], divergent [e.g., age (r = -0.13, p = 0.26), race (r = 0.02, p = 0.87)], and predictive validities [competitive employment (r = 0.31, p = 0.03)] received initial support via study correlations, while convergent [e.g., intrinsic motivation (r = 0.32, p = 0.007), job interview anxiety (r = -0.19, p = 0.08)] and criterion [e.g., prior employment (r = 0.22, p = 0.046), current employment (r = 0.21, p = 0.054)] validities were limited.

Conclusion

The psychometric properties of the 11-item A-MIRS ranged from strong-to-acceptable, indicating it may have utility as a reliable and valid method for assessing the job interview skills of autistic transition-age youth.

Enhancing pre-employment transition services: A type 1 hybrid randomized controlled trial protocol for evaluating WorkChat: A Virtual Workday among autistic transition-age youth

Autistic transition-age youth experience high rates of unemployment and underemployment, in part due to the social challenges they may face when having conversations in the workplace. In an effort to help enhance conversational abilities in the workplace, our collaborative team partnered to develop WorkChat: A Virtual Workday. Specifically, our team of scientists, community partners, and diversity and inclusion experts participated in a community-engaged process to develop WorkChat using iterative feedback from autistic transition-age youth and their teachers. With initial development complete, this study reports on the protocol that our collaborative team developed, reviewed, and approved to conduct a randomized controlled trial (RCT) to evaluate the real-world effectiveness and initial implementation process outcomes of WorkChat when integrated into post-secondary pre-employment transition services (Pre-ETS). Our aims are to: 1) evaluate whether services-as-usual in combination with WorkChat, compared to services-as-usual with an attention control, enhances social cognition and work-based social ability (between pre- and post-test); reduces anxiety about work-based social encounters (between pre- and post-test), and increases sustained employment by 9-month follow-up; 2) evaluate whether social cognitive ability and work-based social ability mediate the effect of WorkChat on sustained employment; and 3) conduct a multilevel, mixed-method process evaluation of WorkChat implementation.

Psychometric properties of the mock interview rating scale for schizophrenia and other serious mental illnesses

Background

Over the past 10 years, job interview training has emerged as an area of study among adults with schizophrenia and other serious mental illnesses who face significant challenges when navigating job interviews. The field of mental health services research has limited access to assessments of job interview skills with rigorously evaluated psychometric properties.

Objective

We sought to evaluate the initial psychometric properties of a measure assessing job interview skills via role-play performance.

Methods

As part of a randomized controlled trial, 90 adults with schizophrenia or other serious mental illnesses completed a job interview role-play assessment with eight items (and scored using anchors) called the mock interview rating scale (MIRS). A classical test theory analysis was conducted including confirmatory factor analyses, Rasch model analysis and calibration, and differential item functioning; along with inter-rater, internal consistency, and test-retest reliabilities. Pearson correlations were used to evaluate construct, convergent, divergent, criterion, and predictive validity by correlating the MIRS with demographic, clinical, cognitive, work history measures, and employment outcomes.

Results

Our analyses resulted in the removal of a single item (sounding honest) and yielded a unidimensional total score measurement with support for its inter-rater reliability, internal consistency, and test-retest reliability. There was initial support for the construct, convergent, criterion, and predictive validities of the MIRS, as it correlated with measures of social competence, neurocognition, valuing job interview training, and employment outcomes. Meanwhile, the lack of correlations with race, physical health, and substance abuse lent support for divergent validity.

Conclusion

This study presents initial evidence that the seven-item version of the MIRS has acceptable psychometric properties supporting its use to assess job interview skills reliably and validly among adults with schizophrenia and other serious mental illnesses.

Clinical Trial Registration

NCT03049813.

Development and Psychometric Properties of Self-Reported Job Interview Skills and Job Interview Anxiety for Autistic Transition-Age Youth

BACKGROUND

The study of job interview training is an emerging area among transition-age autistic youth who face significant challenges when navigating job interviews. The autism field has limited measures that have undergone rigorous psychometric evaluation.

OBJECTIVE

We sought to evaluate the psychometric properties of adapted self-report measures assessing job interview skills and job interview anxiety.

METHODS

As part of two parent randomized controlled trials, eighty-five transition-age autistic youth completed measures related to the strength of their job interview skills and their level of job interview anxiety. We conducted classical test theory analyses, exploratory and confirmatory factor analyses, and Rasch model analytic and calibration analyses. Pearson correlations were used to establish concurrent, divergent, and criterion validity by correlating these scales with measures of social differences, depressive symptoms, behaviors, neuropsychological functioning, and work history.

RESULTS

Our analyses yielded two brief and reliable scales: Measure of Job Interview Skills (MOJO-iSkills) and Measure of Job Interview Anxiety (MOJO-iAnxiety), which demonstrated initial concurrent, divergent, and criterion validities when correlated with measures of depressive symptoms, social differences, internalizing and externalizing behavior, and work history.

CONCLUSIONS

This study presents initial evidence that MOJO-iSkills and MOJO-iAnxiety have acceptable psychometric properties supporting they can be used to reliably and validly assess job interview skills and interview anxiety.

Virtual Reality Job Interview Training for Adults Receiving Prison-Based Employment Services: A Randomized Controlled Feasibility and Initial Effectiveness Trial

Returning citizens struggle to obtain employment after release from prison, and navigating job interviews is a critical barrier they encounter. Implementing evidence-based interview training is a major gap in prison-based vocational services. We conducted a randomized controlled trial (RCT) to evaluate the feasibility and initial effectiveness of Virtual Reality Job Interview Training within two prisons. Forty-four male returning citizens were randomized to receive service-as-usual (SAU) with VR-JIT (SAU+VR-JIT, n = 28) or SAU (n = 16). Participants reported VR-JIT was highly acceptable and usable. SAU+VR-JIT, as compared to SAU, had significant improvements (with large effect sizes) in interview skills, interview training motivation, and interview anxiety (all p < .05; ηp2 > .15), and greater employment by 6-month follow-up (OR = 7.4, p = .045). VR-JIT can potentially help fill a major gap in prison-based services. Future research is needed to validate VR-JIT effectiveness and evaluate VR-JIT implementation strategies within prisons.

Implementation Preparation Costs of Virtual Reality Job Interview Training in Prisons: A Budget Impact Analysis

Virtual Reality Job Interview Training (VR-JIT) has increased employment rates for returning citizens when added to a successful prison-based employment readiness program. However, implementation preparation cost-expenses prior to offering VR-JIT to intended recipients-is unknown. We estimated the cost of implementation preparation activities (e.g., organizing workflow) for two prisons to deliver VR-JIT. We conducted a budget impact analysis and enumerated the labor costs incurred during this important stage of implementation. Labor costs were approximately $8,847 per prison. Our sensitivity analysis estimated the labor costs to replicate this effort in a new prison to range from $2,877 to $4,306 per prison. Thus, VR-JIT may be an affordable tool for prison-based employment readiness programs to improve gainful employment.

The use of virtual reality and augmented reality in psychosocial rehabilitation for adults with neurodevelopmental disorders: A systematic review

Objectives

Virtual reality and augmented reality have been used in psychosocial rehabilitation for adults with neurodevelopmental disorders in recent years, to provide functional training in a scaffolded and appealing manner. This systematic review attempted to evaluate (1) how virtual reality or augmented reality technology was deployed, when used as an intervention for adults with neurodevelopmental disorders; and (2) how virtual or augmented reality-assisted psychosocial rehabilitation programs impacted on the functional domains of community living, employment and social participation.

Methods

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines was adopted and a search of publications between June 2012 and June 2022 was carried out. The target groups were adults with schizophrenia/schizoaffective disorders, autism spectrum disorder, intellectual disabilities and attention deficit hyperactivity disorder. Interventions that targeted at least one functional domain were included.

Results

The database search generated 1,267 records and 38 studies met the inclusion criteria. Three studies utilized augmented reality while the rest utilized virtual reality. The virtual scenarios were displayed in various ways, such as head-mounted displays, computer screens, mobile devices and cave rooms. A few studies also used features such as speech recognition, eye tracking and motion-capture device to provide real-time feedback to participants during rehabilitation. Eleven studies reported interventions that addressed community living, 15 studies addressed vocational skills and nine studies trained participants in social skills or social cognition. Three studies sought to improve quality of life using virtual scenarios to expose participants to various situations. Majority of these studies reported preliminary promising results, with improvement in the functional domains addressed. However, several studies had small sample sizes and many single-arm pretest-posttest studies were assessed to be of serious or critical risk of bias.

Conclusion

Virtual reality and augmented reality are deployed in various ways to augment psychosocial rehabilitation for adults with neurodevelopmental disorders. Most interventions target skills training or strategy learning in the areas of community living, work and social participation. Preliminary positive findings of their effects on functional performance were reported. Larger and robust studies using ecologically valid outcome measures will be needed to establish their effects on real-world functional outcomes.

Systematic review registration

identifier: CRD42022335443.

Narrative and Behavioral Engagement as Indicators for the Effectiveness of Intentionally Designed Virtual Simulations of Interpersonal Interactions

Interventions involving simulated interactions aimed at mimicking real situations must be engaging to maximize their effectiveness. This study aimed to assess how a sample of middle school girls displayed behavioral and cognitive indicators of engagement when interacting with avatars representing game characters that were controlled by a human digital puppeteer. The simulation game, DRAMA-RAMA, is a component of an intervention intended to reduce at-risk girls' sexual and other risky behaviors. We used verbal/nonverbal behaviors and surveys to assess the game players' cognitive and behavioral involvement (N = 131). Participants perceived the game scenarios and interactions as realistic and the characters as similar to people in real life. Participants' behavior indicated their involvement and interest in interacting with the game characters. Finally, participants tended to be appropriate but not effective when attempting to advise/support the characters. These findings have implications for assessing successful operationalization of communication designs in interactive virtual learning environments.