Exploring the use of ChatGPT to analyze student course evaluation comments

BACKGROUND

Since the release of ChatGPT, numerous positive applications for this artificial intelligence (AI) tool in higher education have emerged. Faculty can reduce workload by implementing the use of AI. While course evaluations are a common tool used across higher education, the process of identifying useful information from multiple open-ended comments is often time consuming. The purpose of this study was to explore the use of ChatGPT in analyzing course evaluation comments, including the time required to generate themes and the level of agreement between instructor-identified and AI-identified themes.

METHODS

Course instructors independently analyzed open-ended student course evaluation comments. Five prompts were provided to guide the coding process. Instructors were asked to note the time required to complete the analysis, the general process they used, and how they felt during their analysis. Student comments were also analyzed through two independent Open-AI ChatGPT user accounts. Thematic analysis was used to analyze the themes generated by instructors and ChatGPT. Percent agreement between the instructor and ChatGPT themes were calculated for each prompt, along with an overall agreement statistic between the instructor and two ChatGPT themes.

RESULTS

There was high agreement between the instructor and ChatGPT results. The highest agreement was for course-related topics (range 0.71-0.82) and lowest agreement was for weaknesses of the course (range 0.53-0.81). For all prompts except themes related to student experience, the two ChatGPT accounts demonstrated higher agreement with one another than with the instructors. On average, instructors took 27.50 ± 15.00 min to analyze their data (range 20-50). The ChatGPT users took 10.50 ± 1.00 min (range 10-12) and 12.50 ± 2.89 min (range 10-15) to analyze the data. In relation to reviewing and analyzing their own open-ended course evaluations, instructors reported feeling anxiety prior to the process, satisfaction during the process, and frustration related to findings.

CONCLUSIONS

This study offers valuable insights into the potential of ChatGPT as a tool for analyzing open-ended student course evaluation comments in health professions education. However, it is crucial to ensure ChatGPT is used as a tool to assist with the analysis and to avoid relying solely on its outputs for conclusions.

Scoping Review of Programmatic Well-Being Interventions and Outcomes to Support Pharmacy Students

OBJECTIVES

Recent literature highlights various well-being initiatives implemented across pharmacy programs; however, there is much heterogeneity in their implementation and limited studies assessing the impact and success of these initiatives on pharmacy students' well-being. The purpose of this scoping review was to identify the effectiveness of strategies implemented by pharmacy schools to improve the well-being of pharmacy students.

FINDINGS

In total, 23 studies were included, and well-being strategies were categorized into 3 groups: organizational wellness programs, curriculum design and educational course activities, and specific relaxation or meditation activities. Strategies included yoga meditation, structured mindfulness courses, and self-directed mindfulness with digital smartphone applications. The majority of studies assessed outcomes of stress, burnout, and mindfulness, though other well-being domains such as resilience and belonging emerged. Some also assessed perceptions and acceptability of the interventions through qualitative approaches.

SUMMARY

Despite the heterogeneity of interventions and outcome measures, this review provides a comprehensive scope of well-being domains, assessment tools, interventions, and approaches targeted at pharmacy students and highlights the strongest evidence for interventions on reducing stress. Pharmacy programs can benefit from approaches at both the organization level and those fostering individual accountability by exposing students to a variety of self-help well-being strategies that develop protective factors and motivate them to sustain well-being practices themselves as a shared approach. This scoping review addresses a critical gap by gaining an understanding of the current landscape of well-being initiatives and their effectiveness to better guide pharmacy programs on strategies that are most likely to improve student well-being.

Understanding pathways to health-system pharmacy administration: A journey mapping study

PURPOSE

Pharmacy administrators play a critical role in the success of health systems, yet pathways to pharmacy administration are incompletely understood. The purpose of this study was to explore and describe pathways to becoming a health-system pharmacy administrator, including challenges experienced and resources needed.

METHODS

Participants (N = 21) individually completed one or more journey maps detailing the pathway(s) to pharmacy administration and then engaged in small group discussion to identify common themes. A large group debrief was used to share and identify findings across small groups. Thematic analysis was utilized to identify common ideas shared.

RESULTS

There was variation across the maps, illustrating many steps one might follow to become a pharmacy administrator. In the 23 maps completed, pathways most frequently began with pharmacy school (n = 9, 39.1%), high school (n = 6, 26.1%), or undergraduate training (n = 4, 17.4%). Most included residency training or dual master's of science in health-system pharmacy administration/residency training (n = 16, 69.6%). Common challenges included distractions, time constraints, grades/performance, financial constraints, and stress. Common resources needed included exposure to diverse career paths, pipeline and outreach support, mentoring, and leadership. Themes included debate about the definition of a pharmacy administrator, questions regarding the core competencies of the pharmacy administrator, discussion regarding ways to enter pharmacy administration, and reflection on the role of serendipity and networking.

CONCLUSION

Understanding the pathways to pharmacy administration is critical to fostering a sustainable and competent workforce. Work must be done to identify strategies and develop processes for identifying, equipping, and retaining talented health-system pharmacy administrators and leaders.

Assessing Workplace Factors That Influence Burnout and Identifying Recommendations to Support Well-being in Pharmacy Faculty

OBJECTIVE

Creating environments that promote well-being is critical as studies have shown that burnout, and thus poor well-being, results in poor health outcomes and declining workplace retention. While studies have measured the prevalence of burnout in pharmacy faculty, a gap exists regarding workplace factors that impact faculty well-being. The purpose of this study was to assess factors influencing pharmacy faculty burnout and identify recommendations to improve faculty well-being.

METHODS

Full-time pharmacy faculty were invited to participate in this exploratory study. Focus groups were divided by faculty rank (ie, assistant, associate, full professor). A semistructured interview format was used to identify factors contributing to faculty burnout and fulfillment as well as solicit suggestions for strategies to improve well-being. Inductive coding was used to identify themes.

RESULTS

Eleven faculty participated in 4 focus groups: 6 assistant professors in 2 focus groups (n = 3 faculty/focus group), 3 associate professors in 1 focus group, and 2 full professors in 1 focus group. Common factors influencing faculty burnout included overwhelming workload, workplace inefficiencies, and unexpected factors. Factors contributing to faculty fulfillment included impact, having a passion for work, and connecting with others. Recommendations for improving well-being included school-level well-being initiatives from leadership, encouraging work-life balance, and implementing intentional well-being initiatives and resources.

CONCLUSION

Participants identified several workplace factors that contribute to burnout and faculty fulfillment. This study fills a critical gap in the literature regarding factors that influence pharmacy faculty well-being and burnout and can inform strategies for fostering faculty well-being in pharmacy education.

Pharmacy Student Stress and Time Use in Pre-Clinical and Clinical Students

OBJECTIVE

Literature has identified high levels of stress in pharmacy students; however, more information is needed to understand how student stress relates to time use. This study explored the causes of stress and its relationship to time use in pre-clinical and clinical pharmacy students, which were compared, as previous literature has found differences in time management and stress in these groups.

METHODS

In this observational, mixed methods study, pre-Advanced Pharmacy Practice Experience students completed a baseline and final stress assessment, recorded daily time use and stress levels for 1 week, and participated in a semi-structured focus group. Predetermined time use categories were used to collect and analyze time use data. Inductive coding was used to determine themes from focus group transcripts.

RESULTS

Pre-clinical students were found to have overall higher baseline and final stress scores and spent more time on stress-inducing activities (ie, academics) than clinical students. Both groups reported spending more time on activities related to pharmacy school during the week and increased time on activities of daily life and discretionary activities during the weekend. Common sources of stress for both groups included academics, cocurriculars, and ineffective stress management techniques.

CONCLUSIONS

Our findings support the hypothesis that time use and stress are linked. Pharmacy students acknowledged having many responsibilities and too little time to participate in stress-relieving activities. Understanding sources of student stress, including demands on students' time, and the relationship between the two is needed to support student stress management and academic success of both pre-clinical and clinical pharmacy students.

"Put me in Coach!" Engaging Faculty to Form a Winning Assessment Team

This commentary uses the metaphor of an athletic team model to provide guidance when managing a successful assessment committee and assessment processes. To become a winning team, a joint effort must be exerted by players, coaches, and the athletic director. The topics of developing a team of productive members, creating, and implementing an assessment plan, forming a positive culture, and establishing leadership are discussed. Examples and advice are provided to assist with promoting an engaged faculty to form a well-rounded and productive assessment committee with clearly defined roles and responsibilities.

There's no time for no stress! Exploring the relationship between pharmacy student stress and time use

BACKGROUND

Health professions students experience significantly more stress than students 20 years ago. While prior studies have explored student time use and other studies have begun to explore factors influencing student stress, little is known about the relationship between student time use and stress. As more efforts are employed to promote student wellness and better understand student stress, it is imperative to recognize the implications of time as a finite resource. Thus, it is important to understand if and how time use relates to student stress so the two can be better managed.

METHODS

A mixed methods approach exploring the challenge-hindrance stressor framework was used to collect and analyze student stress and time use. First, second, and third year pharmacy students were invited to participate. Participants completed the Perceived Stress Scale (PSS10), a week-long daily time logging activity, and daily stress questionnaire. After the week-long daily time logging, students participated in a semi-structured focus group. Descriptive statistics were used to analyze quantitative data, and inductive coding along with creation of summary reports were created for the qualitative data.

RESULTS

Students reported moderate stress on the PSS10, and spending the majority of their time on activities of daily life and academic activities. Students shared that academics, co-curriculars, and working for pay increased their stress, whereas discretionary activities such as socializing and exercising alleviated stress. Finally, students reported feeling overwhelmed due to too little time to complete all necessary activities daily, including limited time to invest in discretionary activities to support their wellbeing.

CONCLUSIONS

Increased stress levels among students is a concerning trend that affects students' mental health and therefore limits their ability to perform to their greatest potential. Better understanding of the relationship between time use and stress is critical to improve the quality of life of students in the health professions. These findings provide critical insight into factors contributing towards student stress that can inform curricular strategies that support wellness within health professions education.

Implementation and evaluation of pharmaly houses as learning communities to promote community, wellness, and advising

INTRODUCTION

Although medical education has demonstrated learning communities promote wellness and community while supporting advising and mentoring, its application in pharmacy is limited. This study explored faculty and student perceptions of pharmacy learning communities (PLC), including structure, member interactions, and curriculum topics.

METHODS

PLCs involving first-year (PY1) student advisees, third-year (PY3) peer mentors, and faculty coaches were implemented. The PLC model included Pharmaly Houses, which were compromised of multiple Pharmalies, with each Pharmaly consisting of one faculty coach, two PY3 peer mentors, and approximately 10 PY1 advisees. PLC members were invited to participate in semi-structured focus groups, and transcripts were analyzed using inductive coding.

RESULTS

Three major themes and related subthemes emerged: strengths of the PLC structure (e.g., multiple perspectives, community, networking); interactions of Pharmaly members (i.e. faculty coaches, PY3 peer mentors, PY1 advisees); and feedback on PLC curriculum topics (e.g., topic consultation, continued learning). Participants indicated the multiple perspectives provided by Pharmaly members was a strength and highlighted the PLC structure facilitated a sense of community and belonging among Pharmaly members while promoting expanded networking within the House. While the PLC expanded networks, faculty and PY1 advisees desired more one-to-one interactions. Further, students reflected on how current PLC curriculum topics (e.g., wellness) prompted continued learning, reflection, and discussion outside the House or Pharmaly meeting.

CONCLUSIONS

Participants indicated PLCs can support student access to mentoring and advising, promote wellness, and foster a sense of community and belonging. The unique peer mentoring model further promoted community.

Development and implementation of a formative instructional coaching program using the Teaching Practices Inventory within a health professions program

BACKGROUND

A growing body of literature describes teaching practices that are positively associated with student achievement. Observing, characterizing, and providing feedback on these teaching practices is a necessary, yet significant challenge to improving teaching quality. This study describes the design, implementation, and evaluation of an instructional coaching program created to provide formative feedback to instructors based on their use of evidence-based teaching practices.

METHODS

The program was designed for formative purposes utilizing an instrument adapted from the Teaching Practices Inventory. All faculty were invited to participate in the program on a voluntary basis when the program launched in Fall 2019. Program coaches included any School personnel who completed required training. Two rounds of instrument development were conducted with multiple observers and assessed using Krippendorff's Alpha. The program was evaluated using an anonymous post-session survey.

RESULTS

Interrater reliability of the form improved over two rounds of piloting and no differences were found in scoring between trainees and education professionals. Seventeen observations were completed by nine coaches. Instructors indicated that feedback was practical, timely, specific, and collegial, suggesting that including student perspectives (e.g., focus groups, student course evaluations) in the coaching program might be helpful.

CONCLUSIONS

Creating programs that emphasize and foster the use of evidence-based teaching are critical for health professions education. Additional research is needed to further develop coaching programs that ensure teaching practices in the health professions are optimizing student learning.

Identifying health-system pharmacy operational process categories and corresponding tasks across a diverse health system using a modified Delphi process

DISCLAIMER

In an effort to expedite the publication of articles related to the COVID-19 pandemic, AJHP is posting these manuscripts online as soon as possible after acceptance. Accepted manuscripts have been peer-reviewed and copyedited, but are posted online before technical formatting and author proofing. These manuscripts are not the final version of record and will be replaced with the final article (formatted per AJHP style and proofed by the authors) at a later time.

PURPOSE

The purpose of this study was to identify and build consensus on operational tasks that occur within a health-system pharmacy.

METHODS

An expert panel of 8 individuals was invited to participate in a 3-round modified Delphi process. In the first round, the expert panel independently reviewed an initial list and provided feedback. All feedback was incorporated into the second round and then reviewed and discussed as a group. The expert panel reviewed an updated list based on feedback from the second round and reached consensus on a final list of operational processes and corresponding tasks.

RESULTS

All 8 participants agreed to serve on the Delphi expert panel and reviewed an initial list of 9 process categories (hazardous intravenous [IV] medications, nonhazardous IV medications, hazardous oral medications, nonhazardous oral medications, controlled substances, total parenteral nutrition [TPN]/fluid preparations, distribution and delivery, clinical tasks, and miscellaneous operational tasks) and 44 corresponding tasks. Through the Delphi process, 72 new tasks were identified in the first round, while 34 new tasks were identified in the second round. In the third and final round, the expert panel reviewed the updated list of 9 process categories and 150 corresponding tasks, made additional edits, and reached consensus on a final list of 9 processes and 138 corresponding tasks that represented operational work within a health-system pharmacy.

CONCLUSION

The modified Delphi process effectively identified operational processes and corresponding tasks occurring within hospital pharmacies in a diverse health system. This process facilitated consensus building, and the findings may inform development of an operational workload model.

Development of a system-wide pharmacy operational weighted workload model at a large academic health system

PURPOSE

The purpose of this study is to develop a standard operational and distributional weighted workload model that is applicable across an integrated, diverse healthcare system. This model aims to not only demonstrate the operational intensity of pharmacy practice but also to inform opportunities to decrease waste, increase efficiency, facilitate growth, and demonstrate value across operational and distributional pharmacy services.

SUMMARY

Time studies were conducted at 8 hospitals within the UNC Health system to objectively measure time spent within each operational process in order to create a system-wide weighted workload model. Time study results informed the development of a system-wide weighted workload model. Data from December 29, 2019, through December 26, 2020, was then applied to this weighted workload model. With this model, acute care hospital and infusion center operational areas were compared in thousands of combinations within single operational areas and across any and all operational areas by dispense code, weighted work, and ratio of weighted work to total sum of dispenses at each site.

CONCLUSION

The model successfully achieved the objective to develop a standard operational weighted workload model that is applicable across the integrated, diverse care system. This model provides a foundation for UNC Health to further productivity measurement and fills a gap in the literature by offering a novel method of developing a system-level operational workload model that can be used to evaluate and compare operational workloads across health-system sites.

Does the ability to identify the construct on an empathy situational judgment test relate to performance? Exploring a new concept in assessment

INTRODUCTION

The ability to identify criteria (ATIC) refers to an examinee's capacity to distinguish the construct being evaluated. Previous research indicates ATIC can be predictive of performance on some assessments. This exploratory study investigated the relationship between a participant's ability to identify criteria and their performance on an empathy situational judgment test (SJT), an assessment format used to measure social and behavioral attributes.

METHODS

A 12-item empathy SJT was completed by 15 students and 15 pharmacists. During a cognitive interview, participants were asked what they believed each exam question measured. Responses were coded to determine whether participants stated "empathy" (indication of ATIC). The point-biserial correlation coefficient was calculated to explore the relationship of ATIC (correctly or not correctly identifying the item measured empathy) and performance on the SJT (total score).

RESULTS

Participants identified empathy 33.3% of the time, and it was the construct most often identified. Pharmacists (27.5%) identified empathy less often than students (39.2%). When empathy was identified as the construct, it was most often reported for items in a non-healthcare setting (56.3%) rather than a healthcare setting (43.7%) and for questions targeting affective empathy (71.3%) rather than cognitive empathy (28.7%). There were no statistically significant relationships with correctly identifying the construct and performance on individual items and the overall test.

CONCLUSIONS

There is inconclusive evidence that ATIC relates to performance on an empathy SJT. Additional research is needed to evaluate the role of ATIC and assessment performance to corroborate study results.

A Scoping Review of the Pharmacy Curriculum Outcomes Assessment Literature

Objective. To identify themes, gaps, and sources of evidence from the literature regarding the Pharmacy Curriculum Outcomes Assessment (PCOA) to inform practice and additional areas for research in pharmacy education.Findings. Nineteen articles describing the administration and use of PCOA were identified. Since PCOA was made a curricular requirement by the Accreditation Council of Pharmacy Education in 2016, the focus of literature related to the PCOA has shifted from administration practices (four articles published before 2016 vs two articles published since) to determining models that may predict student performance on the assessment (two vs five articles) or how the examination might be used to predict future performance (one vs seven articles), especially on the North American Pharmacist Licensing Examination. While there is a growing body of literature focused on the PCOA's utility for measuring performance, few variables have been consistently used.Summary. This review found no studies with objectives that aligned with the initial intended use of the PCOA as defined by the National Association of Boards of Pharmacy, which included tracking individual student performance throughout the curriculum, benchmarking programs against other programs, and evaluating whether a program was meeting their desired outcomes. Additionally, no consensus across the Academy was found as to the proper use of the PCOA, and a paucity of literature exists regarding how the PCOA informs schools and colleges about the effectiveness of their curriculum. There is a need for the Academy to establish a uniform application for the PCOA in pharmacy schools, assess the resources that programs need to administer this required assessment, and determine the utility of the PCOA to measure curricular effectiveness and/or student performance.

Exploring the Role of Item Scenario Features on Situational Judgment Test Response Selections

Objective. To explore pharmacists' and pharmacy students' perceptions regarding the significance of changing the features of test item scenario (eg, switching from a health care to a non-health care context) on their situational judgment test (SJT) responses.Methods. Fifteen Doctor of Pharmacy students and 15 pharmacists completed a 12-item SJT intended to measure empathy. The test included six scenarios in a health care context and six scenarios in a non-health care context; participants had to rank potential response options in order of appropriateness and no two items could be of equal rank. Qualitative data were collected individually from participants using think-aloud and cognitive interview techniques. During the cognitive interview, participants were asked how they selected their final responses for each item and whether they would have changed their answer if features of the scenario were switched (eg, changed to a non-health care context if the original item was in a health care context). Interviews were transcribed and a thematic analysis was conducted to identify the features of the scenario for each item that were perceived to impact response selections.Results. Participants stated that they would have changed their responses on average 51.3% of the time (range 20%-100%) if the features of the scenario for an item were changed. Qualitative analysis identified four pertinent scenario features that may influence response selections, which included information about the examinee, the actors in the scenario, the relationship between examinee and actors, and details about the situation. There was no discernible pattern linking scenario features to the component of empathy being measured or participant type.Conclusion. Results from this study suggest that the features of the scenario described in an SJT item could influence response selections. These features should be considered in the SJT design process and require further research to determine the extent of their impact on SJT performance.

Challenges to Implementation of the Co-Curriculum in Accredited Pharmacy Programs

Objective. To determine areas of concern, and challenges to implementing and assessing the co-curriculum in accredited Doctor of Pharmacy programs, along with how confident programs are in their ability to meet the co-curriculum requirement as mandated by the Accreditation Council for Pharmacy Education (ACPE).Methods. A survey was administered to all ACPE-accredited pharmacy programs to collect information regarding areas of concern, challenges, and confidence in their ability to meet the co-curriculum requirement. The frequency of responses to items are presented along with comparisons based on characteristics, including institution type, cohort size, most recent ACPE accreditation review, and supporting offices.Results. The most common concerns centered on the documentation and assessment process. The most commonly reported challenges were lack of enthusiasm or buy-in from faculty, staff, and students; lack of a clear definition of co-curriculum; and faculty time and insufficient staff. Overall, programs had a high level of confidence in their ability to meet the requirements for co-curriculum. The only differences found were related to supporting offices and cohort size.Conclusion. The results suggest that having supporting offices may reduce the co-curriculum burden. Similarly, student cohort size may have an impact on the challenges for some programs, particularly those with moderate-sized cohorts reporting challenges related to faculty and staff. Further research is needed to determine how programs address these critical issues, and to explore whether programs report differently on these areas after completing an accreditation review. The study results may be useful to members of the Academy when evaluating co-curriculum.

Exploring career development through a student-directed practicum to provide individualized learning experiences

INTRODUCTION

The primary objective was to assess the impact of an optional student-directed career development program on career and professional development between the second and third year of a doctor of pharmacy curriculum.

METHODS

Students who completed the student-directed practicum (SDP) completed a nine item pre-/post-survey assessing its impact on their career development within the learning constructs of discovery, application, and reflection. Additional open-ended questions assessed why students participated in the SDP and their perceived benefits of doing so. An additional questionnaire regarding the impact of the practicum was conducted at graduation. Quantitative data were analyzed with Wilcoxon signed-rank test for the pre-/post-survey and descriptive statistics for the graduation survey while qualitative data used a two-cycle open coding process.

RESULTS

Statistically significant increases were noted with those responding with "agree" or "strongly agree" to each of the nine statements in the pre-/post-survey. Students participated to further explore career options, and believed doing so resulted in enhanced career direction and future employability. The graduation survey showed those participating in the SDP had a higher rate of: postgraduate placement (86% vs. 77%), receiving ≥ five residency interview offers (81% vs. 69%), and first or second choice residency match (92% vs. 85%) compared to those who completed an internship outside of the SDP. Students commented that the SDP was a valuable part of their education and helped them take the next steps for their career.

CONCLUSIONS

A student-directed career development experience can positively impact student's career direction and professional development.

Qualitative Evaluation of a Junior Faculty Team Mentoring Program

Objective. To investigate the strengths and challenges of a structured junior faculty mentoring program at a public four-year school of pharmacy, identify areas of opportunity to improve the program, and describe the mentoring needs of mid-career faculty.Methods. Focus groups and interviews were conducted to elicit participants' experiences, perceptions, and suggestions for opportunity to improve the program. Stakeholder groups included junior faculty enrolled in the mentoring program, mid-career faculty who had graduated from the program, mid-career faculty who had not participated in the program, internal mentors, external mentors, and division chairs. Thematic coding was used to identify semantic themes, and summaries of participant perceptions were generated. The program was mapped to the PAIRS checklist from the 2014 American Association of Colleges of Pharmacy Joint Council Task Force on Mentoring.Results. Participants described the structure of the program and mentee-mentor relationships as strengths of the program. Challenges included finding time to meet and ensuring mentee-mentor fit. Several areas of opportunity were identified, such as adjusting the topics for large mentee seminars, providing mentors with training, and providing mentoring for mid-career faculty. The mentoring needs of mid-career faculty were described as unique and requiring potentially different strategies than those used for mentoring junior faculty.Conclusion. Mentoring is critical to the professional development of faculty, supporting faculty retention and job satisfaction, and reducing faculty burnout. Scholarly endeavors that explore faculty mentoring, specifically those using qualitative methods, can help the Academy better understand and meet the needs of faculty.

Role of Knowledge and Experience in Situational Judgment Test Responses of Pharmacists and Pharmacy Students

Objective. To describe the role of examinee knowledge and experience in situational judgment test (SJT) response processes.Methods. Thirty participants (15 students and 15 pharmacists) completed a 12-item SJT on empathy. Each participant completed a think-aloud interview followed by a cognitive interview to elicit their understanding of the items and factors that influenced their response selections. Interviews were coded to identify references to general and job-specific knowledge and experiences. Utterances were quantified to explore differences in the occurrence based on the individual item, item setting (ie, health care or non-health care setting), participant type (ie, student or pharmacists), and empathy component being assessed (ie, affective or cognitive empathy).Results. Participants made 480 references to knowledge and experiences: 45.2% were job-specific knowledge or experiences, 27.5% were general knowledge or experiences, 17.9% related to a lack of experience, and 9.4% were nondescript and could not be distinguished. There were significant differences in the reference to general and job-specific knowledge or experiences based on whether the item scenario occurred in a health care or non-health care setting and the component of empathy being assessed. Experience references often included comments about location, actors, task, similarity, specificity, and recency; knowledge references were classified by information, strategies, and skills.Conclusion. Findings from this study suggest general and job-specific knowledge and experiences influence response processes in SJTs.

Situational judgment test validity: an exploratory model of the participant response process using cognitive and think-aloud interviews

BACKGROUND

Situational judgment tests (SJTs) are used in health sciences education to measure examinee knowledge using case-based scenarios. Despite their popularity, there is a significant gap in the validity research on the response process that demonstrates how SJTs measure their intended constructs. A model of SJT response processes has been proposed in the literature by Robert Ployhart; however, few studies have explored and expanded the factors. The purpose of this study was to describe the factors involved in cognitive processes that examinees use as they respond to SJT items in a health professions education context.

METHODS

Thirty participants-15 student pharmacists and 15 practicing pharmacists-completed a 12-item SJT designed to measure empathy. Each participant engaged in a think-aloud interview while completing the SJT, followed by a cognitive interview probing their decision-making processes. Interviews were transcribed and independently coded by three researchers to identify salient factors that contributed to response processes.

RESULTS

The findings suggest SJT response processes include all four stages (comprehension, retrieval, judgment, and response selection) as initially proposed by Ployhart. The study showed factors from other published research were present, including job-specific knowledge and experiences, emotional intelligence, and test-taking. The study also identified new factors not yet described, including identifying a task objective in the scenario, assumptions about the scenario, perceptions about the scenario, and the setting of the item.

CONCLUSIONS

This study provides additional SJT validity evidence by exploring participants' response processes through cognitive and think-aloud interviews. It also confirmed the four-stage model previously described by Ployhart and identified new factors that may influence SJT response processes. This study contributes to the literature with an expanded SJT response process model in a health professions education context and offers an approach to evaluate SJT response processes in the future.

Identifying Priority Student Leadership and Professionalism Attributes Among Faculty, Preceptors, and Students via Modified Delphi

Objective. To identify and build consensus on priority leadership and professionalism attributes for pharmacy student development among faculty, preceptors, and students.Methods. One hundred individuals (27 faculty members, 30 preceptors, 43 students) were invited to participate in a three-round, modified Delphi. Published literature on leadership and professionalism informed the initial attribute list. In the first round, participants reviewed and provided feedback on this list. In the second round, participants prioritized attributes as highly important, important, or less important for pharmacy student development. Leadership and professionalism attributes that achieved an overall consensus (a priori set to ≥80.0%) of being highly important or important for pharmacy student development were retained. In the third round, participants rank ordered priorities for leadership and professionalism attributes.Results. Fifteen leadership and 20 professionalism attributes were included in round one while 21 leadership and 21 professionalism attributes were included in round two. Eleven leadership and 13 professionalism attributes advanced to round three. Consensus was reached on the top four leadership attributes (adaptability, collaboration, communication, integrity) and five professionalism attributes (accountability, communication, honor and integrity, respect for others, trust). Differences were observed for certain attributes between faculty members, preceptors, and/or students.Conclusion. The modified Delphi technique effectively identified and prioritized leadership and professionalism attributes for pharmacy student development. This process facilitated consensus building and identified gaps among stakeholders (ie, faculty, preceptors, students). Identified gaps may represent varying priorities among stakeholders and/or different opportunities for emphasis and development across classroom, experiential, and/or cocurricular settings.

Evaluation of a Council Structure and Meeting Format for Pharmacy Student Government in the Co-Curriculum

Objective. To evaluate student perceptions of a transformed pharmacy student government structure and meeting format. Methods. The structure and general assembly meeting format of the University of North Carolina at Chapel Hill (UNC) Eshelman School of Pharmacy Student Senate (ie, student government) underwent a transformation from that of a forum to that of a council in an effort to address concerns regarding limited information exchange, minimal discussion, and unengaged meeting participants. To evaluate student perceptions of the new senate council format, members who attended at least one council meeting during the academic year were asked to complete a 12-item survey evaluating interorganizational relations, communication, collaboration, and efficiencies. Descriptive statistics were used to summarize students' agreement with 10 survey items. Thematic analysis was employed to identify common themes in students' responses to the two open-ended survey items. Results. The majority of students agreed or strongly agreed that the senate council facilitated interorganizational relations (86%), communication (93%), and collaboration (86%). Ninety-three percent of respondents agreed or strongly agreed that the senate council improved efficiencies, while 96% preferred the new senate council meeting format over the previous senate meeting format. Common qualitative themes included improved engagement during council meetings because of the "no laptop" policy and facilitation of open conversation by having a roundtable format and smaller number of members in attendance. Overall, 93% of students indicated they were satisfied or highly satisfied with the new meeting format. Conclusion. Student perceptions suggest that the new senate council structure and meeting format was effective at improving interorganizational relations, communication, collaboration, and efficiencies within student body government.

Co-Curriculum Implementation and Assessment in Accredited Doctor of Pharmacy Programs

Objective. To determine how accredited Doctor of Pharmacy programs implement and evaluate the co-curriculum requirement as mandated by the Accreditation Council for Pharmacy Education (ACPE). Methods. A survey was administered to all ACPE-accredited pharmacy programs to collect information regarding how co-curriculum models were being implemented, including types of activities, structure, learning outcomes, oversight, and assessment. The frequency of responses to items were presented to describe the general features of co-curriculum models. Results. The types of co-curricular activities reported by programs were generally consistent, with the majority of programs categorizing these activities and allowing students to choose which they would engage in. Most respondents reported that the program mapped co-curricular activities to learning outcomes, primarily ACPE Standards 1-4. The structural oversight of the co-curriculum typically included a co-curriculum committee, subcommittee, or task force, and supporting offices. The most common offices/departments involved in the co-curriculum were assessment, student affairs/services, experiential education, and academic/curricular affairs. The most common assessments were reflections, self-assessment surveys, and checklists. Conclusion. In most programs, implementation of the co-curriculum was a joint effort among various individuals, committees, and offices. Given the developing nature of programs, descriptive studies should be repeated to identify how programs develop and enhance co-curriculum models. The study results may be useful to members of the Academy when evaluating the current state of co-curriculum implementation and potential areas for program development.

Assessing the Co-Curriculum by Mapping Student Organization Involvement to Curricular Outcomes Using Mixed Methods

Objective. To use institution-specific curricular outcomes as a framework to map skill development opportunities available through cocurricular involvement in pharmacy student organizations. Methods. Participants completed a modified Extracurricular Involvement Inventory individually to measure the intensity of their involvement in each student organization. Participants also completed the Co-Curriculum Outcomes Assessment Mapping Survey (COAMS) instrument as a group to indicate what skills (ie, curricular outcomes) they developed through involvement in a student organization and student organization activities, programs, and events, and to provide examples of these skills. Data sources were triangulated to map skill development opportunities in the co-curriculum to curricular outcomes. Results. The COAMS identified all curricular outcomes as skills students have the opportunity to develop through student organization involvement in the co-curriculum. Communication was the most common skill identified. Other common skills included professionalism and ethical behavior, collaboration and influence, and in-depth knowledge and proficient skills. A co-curriculum heat map was used to illustrate the degree to which students reported these skills were emphasized through student organization involvement in the co-curriculum. Conclusion. Evaluation of activities in the context of curricular outcomes can provide a more comprehensive understanding of how the co-curriculum complements the curriculum, thereby complying with accreditation expectations. Cocurricular mapping provides valuable information regarding student skill development opportunities to multiple stakeholders (eg, students, faculty, curriculum leadership). This process can be applied to diverse programs, adapted to measure institution-specific experiences, and measure various constructs of interest.

Identifying and Mapping Skill Development Opportunities Through Pharmacy Student Organization Involvement

Objective. To describe a process for identifying skills that students can gain through co-curricular involvement and to map these skills to curricular outcomes. Methods. This pilot study used a mixed-methods approach involving document reviews, student involvement surveys, and focus groups/interviews to evaluate skill development opportunities in three pharmacy student organizations. Investigators reviewed key documents (eg, student organization websites, annual review forms) to identify skills emphasized by the organization. Student participants completed modified Extracurricular Involvement Inventory surveys to measure the intensity of their student organization involvement. Two student focus groups, one student leader focus group and one general student member focus group, were held for each student organization. Interviews were conducted with each student organization's faculty advisor. Data sources were triangulated to identify and map skill development opportunities to program core competencies. Results. Six of nine core competencies were identified as skills pharmacy students can develop through involvement in the pilot student organizations. All three organizations provided opportunities for members to develop communication skills and in-depth knowledge and proficient skills of the discipline of pharmacy. Two organizations provided opportunities for students to develop collaboration skills. Conclusion. A mixed-methods approach can be used to identify and map skills that students can develop through co-curricular involvement. This approach provides several advantages: objective evaluation and triangulation of skill development opportunities, evaluation of students' involvement, and identification of linkages between the co-curriculum and curriculum in the context of curricular outcomes.

Assessing student academic time use: assumptions, predictions and realities

CONTEXT

In an era of medical education reform and increasing accountability at all levels of higher education, there is a need to understand how the time in which students engage in academic activities can inform evidence-based quality improvement of the curriculum. Time logging provides an opportunity to quantify student use of academic time and guide data-informed decision making in curriculum design.

OBJECTIVES

This study aimed to evaluate faculty staff and student predictions of students' academic time use and to assess students' reported academic time use.

METHODS

Graduate-level professional students engaged in a time use exercise during the first semester of Year 1 (autumn 2015) and second semester of Year 2 (spring 2017) of a redesigned curriculum launched in autumn 2015. This exercise involved three key activities: (i) prediction of time use; (ii) time logging, and (iii) reflection on time use. Key faculty staff predicted students' weekday time use in both semesters.

RESULTS

Students' predictions of academic time use strongly correlated with their reported academic time use during both the first semester of Year 1 and second semester of Year 2 (r = 0.55 and r = 0.53, respectively). Faculty members' predictions of academic time use did not correlate with student academic time use during either semester. Although 63.8% of Year-1 students reported the time use exercise motivated them to change their time use, students reported spending similar amounts of time on academic activities during the first semester of Year 1 (7.8 ± 1.5 hours per weekday) and the second semester of Year 2 (7.9 ± 2.0 hours per weekday). Most students reported that the exercise had been useful and indicated that their logged time accurately reflected their actual time use.

CONCLUSIONS

Although curriculum reform efforts may always require that some assumptions be made, time logging can quantify students' academic use of time. Although students predict their use of time more accurately than do faculty staff, negligible changes in students' academic time use despite reported desires to make changes indicate that students' academic time use may remain inelastic. Educators must consider these findings as they design curricula, identify academic rigour, and establish student expectations of academic time use.

The Report of the 2017-2018 Professional Affairs Standing Committee: The Development of the Preceptor Self-Assessment Tool for Entrustable Professional Activities for New Graduates

The 2017-2018 AACP Professional Affairs Committee addressed the charges of (1) developing a self-reflection/self-assessment tool for pharmacy faculty and preceptors to allow them to assess their capability and confidence with Center for the Advancement of Pharmacy Education (CAPE) Educational Outcomes, the Pharmacists' Patient Care Process (PPCP), and the Entrustable Professional Activities (EPAs) for New Pharmacy Graduates and (2) creation of a plan for AACP to utilize in the development of preceptor continuing education and training programs. This report describes the framework, rationale and process for the development of the Preceptor Self-Assessment Tool for Entrustable Professional Activities for New Pharmacy Graduates (PSAE Tool), the pre-test for the PSAE tool, and the online pilot test and its results for the PSAE Tool. The committee provides recommendations for AACP in the continued development and distribution of the PSAE Tool to the schools of pharmacy. Considerations for AACP and the schools of pharmacy to consider in the continuing professional development (CPD) for all preceptors are also discussed. The committee provides a policy statement, adopted by the AACP House of Delegates, regarding the commitment of AACP regarding the CPD for all preceptors. The committee also provides several recommendations to AACP and suggestions to schools of pharmacy and other stakeholder groups pertaining to the committee charges.

Using the multiple mini interview as an assessment strategy within the first year of a health professions curriculum

BACKGROUND

The multiple mini-interview (MMI) is a common assessment strategy used in student selection. The MMI as an assessment strategy within a health professions curriculum, however, has not been previously studied. This study describes the integration of a 5-station MMI as part of an end-of-year capstone following the first year of a health professions curriculum. The goal of the capstone MMI was to assess professional competencies of students and to offer formative feedback to prepare students for their upcoming clinical practice experiences. The purpose of this study was to evaluate the psychometric properties of an MMI integrated into a health professions curriculum.

METHODS

Five capstone MMI stations were designed to each evaluate a single construct assessed by one rater. A principal component analysis (PCA) was used to evaluate the structure of the model and its ability to distinguish 5 separate constructs. A Multifaceted Rasch Measurement (MFRM) model assessed student performance and estimated the sources of measurement error attributed to 3 facets: student ability, rater stringency, and station difficulty. At the conclusion, students were surveyed about the capstone MMI experience.

RESULTS

The PCA confirmed the MMI reliably assessed 5 unique constructs and performance on each station was not strongly correlated with one another. The 3-facet MFRM analysis explained 58.79% of the total variance in student scores. Specifically, 29.98% of the variance reflected student ability, 20.25% reflected rater stringency, and 8.56% reflected station difficulty. Overall, the data demonstrated an acceptable fit to the MFRM model. The majority of students agreed the MMI allowed them to effectively demonstrate their communication (80.82%), critical thinking (78.77%), and collaboration skills (70.55%).

CONCLUSIONS

The MMI can be a valuable assessment strategy of professional competence within a health professions curriculum. These findings suggest the MMI is well-received by students and can produce reliable results. Future research should explore the impact of using the MMI as a strategy to monitor longitudinal competency development and inform feedback approaches.

Design and evaluation of a two-phase learner-centered new student orientation program

BACKGROUND AND PURPOSE

To identify student perceptions associated with a learner-centered, multi-stage, flipped, new student orientation for a doctor of pharmacy (PharmD) program.

EDUCATIONAL ACTIVITY AND SETTING

First-year PharmD students over three years (2012-2014) were engaged in a two-phase learner-centered orientation process designed to allow students to customize the transition experience. After orientation, students completed a survey to assess programming and their perceptions of readiness to begin the PharmD program. Utilizing student feedback, the orientation program evolved each year to better meet the needs of incoming students.

FINDINGS

In 2014, 78.9% of students felt confident in their ability to transition to the PharmD program after orientation, an increase from 55.7% in 2012 when the changes in orientation were first implemented. Additionally, student perceptions of readiness to transition to the PharmD program improved each year.

DISCUSSION AND SUMMARY

Implementation of a learner-centered, flipped, new student orientation may improve students' perceptions of readiness for study and decrease anxiety of transitioning to a PharmD program. Additionally, it may be an effective tool for increasing orientation program efficiency, while improving student engagement.

Validity and reliability of an application review process using dedicated reviewers in one stage of a multi-stage admissions model

INTRODUCTION

With increased emphasis placed on non-academic skills in the workplace, a need exists to identify an admissions process that evaluates these skills. This study assessed the validity and reliability of an application review process involving three dedicated application reviewers in a multi-stage admissions model.

METHODS

A multi-stage admissions model was utilized during the 2014-2015 admissions cycle. After advancing through the academic review, each application was independently reviewed by two dedicated application reviewers utilizing a six-construct rubric (written communication, extracurricular and community service activities, leadership experience, pharmacy career appreciation, research experience, and resiliency). Rubric scores were extrapolated to a three-tier ranking to select candidates for on-site interviews. Kappa statistics were used to assess interrater reliability. A three-facet Many-Facet Rasch Model (MFRM) determined reviewer severity, candidate suitability, and rubric construct difficulty.

RESULTS

The kappa statistic for candidates' tier rank score (n = 388 candidates) was 0.692 with a perfect agreement frequency of 84.3%. There was substantial interrater reliability between reviewers for the tier ranking (kappa: 0.654-0.710). Highest construct agreement occurred in written communication (kappa: 0.924-0.984). A three-facet MFRM analysis explained 36.9% of variance in the ratings, with 0.06% reflecting application reviewer scoring patterns (i.e., severity or leniency), 22.8% reflecting candidate suitability, and 14.1% reflecting construct difficulty.

DISCUSSION AND CONCLUSIONS

Utilization of dedicated application reviewers and a defined tiered rubric provided a valid and reliable method to effectively evaluate candidates during the application review process. These analyses provide insight into opportunities for improving the application review process among schools and colleges of pharmacy.

Report of the 2016-2017 Student Affairs Standing Committee

The 2016-2017 AACP Student Affairs Standing Committee addressed charges related to recruitment to the profession of pharmacy and a national awareness campaign for pharmacy careers, as well as promotion of student wellness and stress management. The Committee report provides six recommendations to the American Association of Colleges of Pharmacy (AACP) and one proposed policy statement for the AACP House of Delegates related to recruitment to the pharmacy profession. The Committee report also provides three recommendations to AACP and one proposed policy statement for the AACP House of Delegates related to student wellness and stress management. In addition, this report provides recommendations for future AACP Student Affairs Standing Committee work.