Evidence of validity for the Norwegian version of the interprofessional collaborative competency attainment survey (ICCAS)

The Balancing Act of Assessment Validity in Interprofessional Healthcare Education: A Qualitative Evaluation Study

CONSTRUCT & BACKGROUND

In order to determine students' level of interprofessional competencies, there is a need for well-considered and thoroughly designed interprofessional assessments. Current literature about interprofessional assessments focuses largely on the development and validation of assessment instruments such as self-assessments or questionnaires to assess students' knowledge or attitudes. Less is known about the design and validity of integral types of assessment in interprofessional education, such as case-based assessments, or performance assessments. The aim of this study is to evaluate the evidence for and threats to the validity of the decisions about students' interprofessional performances based on such integral assessment task. We investigated whether the assessment prototype is a precursor to practice (authenticity) and whether the assessment provides valid information to determine the level of interprofessional competence (scoring).

APPROACH

We used a design-based qualitative research design in which we conducted three group interviews with teachers, students, and interprofessional assessment experts. In semi-structured group interviews, participants evaluated the evidence for and threats to the validity of an interprofessional assessment task, which were analyzed using deductive and inductive content analysis.

FINDINGS

Although both evidence for and threats to validity were mentioned, the threats refuting the assessment's validity prevailed. Evidence for the authenticity aspect was that the assessment task, conducting a team meeting, is common in practice. However, its validity was questioned because the assessment task appeared more structured as compared to practice. The most frequently mentioned threat to the scoring aspect was that the process of interprofessional collaboration between the students could not be evaluated sufficiently by means of this assessment task.

CONCLUSIONS

This study showed that establishing interprofessional assessment validity requires three major balancing acts. The first is the balance between authenticity and complexity. As interprofessional practice and competencies are complex, interprofessional tasks require build-up or guidance toward this complexity and chaotic practice. The second is that between authenticity and scoring, in which optimal authenticity might lead to threats to scoring and vice versa. Simultaneous optimal authenticity and scoring seems impossible, requiring ongoing evaluation and monitoring of interprofessional assessment validity to ensure authentic yet fair assessments for all participating professions. The third balancing act is between team scoring and individual scoring. As interprofessional practice requires collaboration and synthesis of diverse professions, the team process is at the heart of solving interprofessional tasks. However, to stimulate individual accountability, the individual performance should not be neglected.

Variations in measurement of interprofessional core competencies: a systematic review of self-report instruments in undergraduate health professions education

Educating health care professionals for working in interprofessional teams is a key preparation for roles in modern healthcare. Interprofessional teams require members who are competent in their roles. Self-assessment instruments measuring interprofessional competence (IPC) are widely used in educational preparation, but their ability to accurately and reliably measure competence is unknown. We conducted a systematic review to identify variations in the characteristics and use of self-report instruments measuring IPC. Following a systematic search of electronic databases and after applying eligibility criteria, 38 articles were included that describe 8 IPC self-report instruments. A large variation was found in the extent of coverage of IPC core competencies as articulated by the Interprofessional Education Collaborative. Each instrument's strength of evidence, psychometric performance and uses varied. Rather than measuring competency as "behaviours", they measured indirect proxies for competence, such as attitudes towards core interprofessional competencies. Educators and researchers should identify the most appropriate and highest-performing IPC instruments according to the context in which they will be used.Systematic review registration: Open Science Framework (https://archive.org/details/osf-registrations-vrfjn-v1).

Interprofessional collaboration skills and motivation one year after an interprofessional educational intervention for undergraduate medical and nursing students

BACKGROUND

The increasingly complex patient care in the twenty-first century is delivered by interprofessional health care teams. Interprofessional collaboration can be taught during interprofessional education. However, whether a long-term change in collaborative competencies can be achieved by interprofessional education has not been studied sufficiently. Our research questions were: How does motivation for interprofessional collaboration and interprofessional collaborative skills change up to one year after an interprofessional educational intervention? How are they related to each other?

METHODS

During a one-year period, undergraduate medical and nursing students attended four interprofessional (intervention) or uniprofessional (control group) education sessions. Self-determination Theory was used as the theoretical framework. Autonomous and controlled motivation scores for interprofessional collaboration were calculated using the Academic Self-Regulation Questionnaire, before (T1), directly after (T2) and one year post-intervention (T3). At T3, the students also filled out the Interprofessional Collaborative Competencies Attainment Survey (ICCAS), which measured the perceived attainment of collaborative competencies by a retrospective pre-test/post-test design. We used linear mixed effects models to analyse the motivation scores and linear regression for the relation between motivation and competence.

RESULTS

In the interprofessional group, autonomous motivation scores of the participants were significantly lower at T2 vs. T1. Controlled motivation scores were significantly higher at T3 vs. T1. Controlled motivation scores for T2 were significantly higher in the uniprofessional group than in the interprofessional group. Perceived competence was related to higher autonomous motivation scores. At T3 the interprofessional collaborative competencies seemed to have grown more among students in the interprofessional group.

CONCLUSIONS

The perceived growth in interprofessional collaboration competence lasted at least up to one year after the intervention, and was measurable with the ICCAS. The growth was significantly more in the IPE students than in the UPE students. The few differences found in motivation scores for interprofessional collaboration were probably caused by an imbalance of nursing versus medical students over the different time points. This finding indicates that classroom based IPE can contribute to interprofessional collaboration skills of nursing and medical students at least up to one year after an intervention.

Perception of Interprofessional Education among Students Following Pharmacy Studies

INTRODUCTION

Interprofessional Education (IPE) activities are a first experience of real-world patient care practice for students, where collaboration with different professions is appreciated. Methods and timing of inclusion of IPE are not well-defined, and it is interesting to assess students' perception on IPE activities.

OBJECTIVE

To assess changes in pharmacy students' perception of IPE before (t0) and after (t1) an IPE activity.

METHODS

The 'Student Perceptions of Interprofessional Clinical Education-Revised 2' (SPICE-R2) tool was adopted to assess perception of IPE activities in third year pharmacy students, final year pharmacy students and in postgraduate Doctorate in Pharmacy (PharmD) students at t0 and t1.

RESULTS

The SPICE-R2 tool was completed at t0 and t1 by 61 students: 12 third year pharmacy students, 13 final year students and 36 PharmD students. A significant improvement between t0 and t1 (P < .05) was measured in the three groups of students for all three subscales of the tool. The largest improvement was observed in the 'Roles/Responsibilities for Collaborative Practice' subscale in all three groups of students.

CONCLUSION

Perception of IPE was positively increased in all three student groups. The results could be useful to support the design of IPE activities within pharmacy programmes.

Validation of the Interprofessional Collaborative Competency Attainment Survey (ICCAS) retrospective pre-test measures

Establishing the effectiveness of IPE experiences is essential, and current assessment measures may be subject to various types of bias. The Interprofessional Collaborative Competency Attainment Survey (ICCAS) is administered after the learning experience as retrospective pretest and posttest measures of perceived collaborative practice skills. Because the ICCAS does not have an actual pretest, there are potential risks of recall bias and social desirability bias. To address these concerns, Jackson (2017) proposed conducting a true ICCAS pretest before the learning experience and examining the differences between the true and retrospective pretest scores and the differences between these pretest and posttest scores. This study design was implemented for two in-person Interprofessional forums conducted in the Fall (N = 745) and following Spring (N = 599) semesters. Students from 11 (Fall) and 12 (Spring) health professions education programs were included. True ICCAS pretest scores did not significantly differ from the retrospective pretest scores in either sample, although the effect sizes for pretest to posttest gains were slightly smaller for the true pretest scores. These results support and provide confidence for using the standard ICCAS administration methodology by demonstrating the administration methodology is not adversely impacted by recall or social desirability biases.

Exploring the Efficacy of a Virtual First Year Interprofessional Education Event

Interprofessional education (IPE) activities are utilized in health education programs to develop interprofessional collaboration (IPC) competencies. All first-year healthcare students at three postsecondary learning institutions attend a mandatory introductory IPE event annually. During the 2020/2021 academic year, the event was moved from a face-to-face activity to a virtual format due to the COVID-19 pandemic restrictions. This study examined whether the virtual IPE activity was effective in supporting the development of interprofessional competencies for first-year healthcare students. Two hundred and six students attended a synchronous didactic presentation on IPE competencies and discussed a simulated case in interprofessional groups of eight students and two faculty facilitators. The Interprofessional Collaborative Competency Attainment Survey (ICCAS) was used to measure the students’ opinions on interprofessional competencies. Paired t-tests were used to compare the pre- and post-scores. One hundred and nine (52.9% response rate) students completed the survey. Surveys from 99 students with matched pre- and post-scores were included in the study. The ICCAS competencies showed improvements (p < 0.05) in all of the students’ self-reported IPE competencies following the activity compared to before the training. Our findings indicate that the virtual IPE activity is effective in facilitating the development of IPC for first-year healthcare students.

A preliminary simulation-based qualitative study of healthcare students’ experiences of interprofessional primary care scenarios

Background

Introducing interprofessional education (IPE) in healthcare curricula can prepare students for healthcare practices that have become increasingly complex. The use of simulation is promoted to support IPE. This study explores healthcare students’ experiences of participating in common, sub-acute patient scenarios that routinely occur in clinical practice in primary care. More specifically, it looks at how sub-acute patient scenarios from primary care can help develop interprofessional collaborative competence.

Methods

Medical students (N = 10), master’s students in advanced geriatric nursing (N = 8) and bachelor’s students in nursing (N = 9) participated in the simulations. The students were in their last or second-to-last year of education. We conducted five semi-structured focus group interviews with the participants’ directly after the simulation training to elicit experiences related to the scenarios, the simulation and interprofessional collaboration. The transcripts were analysed using systematic text condensation. To supplement the focus group interviews, the students also completed the interprofessional collaborative competency attainment survey (ICCAS), which measures the students’ self-assessed interprofessional competence.

Results

Three main themes emerged from the analysis of the focus group interviews: realism, uncertainty and reflection. The students emphasised the importance of authentic and recognisable scenarios. They said the vague and unspecific patient symptoms created uncertainty in the situation, making it difficult to understand the patient’s diagnosis. Despite that uncertainty, they described the experience as positive. Further, the students expressed that the simulation increased their confidence in interprofessional collaboration and prepared them for future work. The results from the ICCAS questionnaire showed that the students reported a subjective positive change in their interprofessional competence after participating in the scenarios.

Conclusions

This study showed that simulation-based IPE with sub-acute primary care scenarios contributes to develop interprofessional collaborative competence in healthcare education. Sub-acute scenarios can supplement the more common approaches with acute care scenarios and aid in developing the collaborative competence required to work in healthcare teams.

Supplementary Information

The online version contains supplementary material available at 10.1186/s41077-022-00204-5.

Core Competencies for Interprofessional Collaborative Practice Among Teacher Education, Health and Social Care Students in a Large Scaled Blended Learning Course

Introduction

Society’s demands for better coordination of services for children are increasing. Interprofessional learning (IPL) has been suggested to achieve the triple aim of better services, better outcomes and reduced costs. The aims were to assess 1) to what extent students taking teacher education, health and social care programmes agreed that blended learning was a suitable approach in a mandatory IPL course, 2) to what extent they had learnt about the WHO’s core IPL competencies (roles and responsibilities, values and ethics, interprofessional communication, and teams and teamwork), and 3) the students’ ranking of the learning outcomes from different components of the IPL course.

Methods

This was a quantitative cross-sectional study. Students completed an online course evaluation after a two-day combination of online and face-to-face IPL small-group training.

Findings

The response rate was 25.8% (n=363). Among the students, 60.6% strongly agreed that blended learning was suitable, while 8.9% strongly disagreed. Among the respondents, 46.8%, 50.2%, 56.8% and 62.3% gained increased insight into roles and responsibilities, values and ethics, interprofessional communication, and teams and teamwork, respectively. In ascending order, students were most satisfied with the learning outcomes from the supervision (16.0%), the syllabus (28.6%), the submission assignment (42.4%), the digital learning content of Canvas (43.8%), the combination of everything (43.8%), and the IPL group discussions (78.6%). In stratified analyses, ‘teacher education and child welfare students’ were significantly more likely to gain better insight into the WHO competencies than “health and social care students”, and they were also more overall satisfied.

Conclusion

Students agreed that blended learning was a suitable approach, although the learning outcomes from the face-to-face discussions were markedly higher than from other course components. While the majority had learnt something about the WHO competencies, the teacher and child welfare students achieved the best learning outcomes, including new knowledge about the WHO competencies.