Progress examination for assessing students' readiness for advanced pharmacy practice experiences

A scoping review of advanced pharmacy practice experience readiness assessment measures across schools of pharmacy

BACKGROUND

The development and implementation of Advanced Pharmacy Practice Experience (APPE) readiness assessments has been an area of increased interest to the academy since the publication of the Accreditation Council for Pharmacy Education (ACPE) standards in 2016. This scoping literature review aims to provide an updated summary of current APPE readiness assessment practices among ACPE-accredited institutions in the United States (US).

METHODS

A literature search was conducted between 2022 and 2024 using the terms "APPE student readiness," "APPE readiness assessment," "APPE preparedness," "APPE student preparedness," "pharmacy" AND "readiness assessment", "pharmacy" AND "practice readiness," and "pharmacy" AND "student practice readiness" in Pubmed and Embase. The websites for the American Journal of Pharmaceutical Education and Currents in Pharmacy Teaching and Learning were also searched using these terms. Abstracts for all results were reviewed. Abstract only and poster presentations were excluded, as well as articles centered on non-US and/or non-ACPE-accredited institutions. Data was collected regarding the outcomes assessed, type of assessment activity, and how the assessment was implemented. Results were reviewed by a second author to ensure consistency in reporting.

RESULTS

A total of 289 unique abstracts were reviewed by author pairs for inclusion based on relevance to the review objectives. A total of 13 articles were included in the final analysis. Each institution measured different knowledge areas, abilities, and entrustable professional activities (EPAs). Most programs had a summative component, with APPE readiness being primarily assessed in the final didactic year of the pharmacy curriculum. Most programs report at least one source of validity.

IMPLICATIONS

Given the lack of specific APPE readiness assessment requirements in the ACPE Standards 2016, the different assessment methods among programs were not surprising. However, the commonalities identified can be leveraged with the release of the Curricular Outcomes and Entrustable Professional Activities (COEPA) 2022 to promote a standardized definition of APPE readiness. Future research should focus on formative assessment methods embedded throughout the didactic curriculum to identify "at-risk" students prior to a "high-stakes" summative assessment at the end of the didactic curriculum that impedes student progression to the APPE year.

What's been trending with OSCEs in pharmacy education over the last 20 years? A bibliometric review and content analysis

BACKGROUND

Objective structured clinical examinations (OSCEs) remain an integral part of pharmacy education. This study aimed to characterize key researchers, areas, and themes in pharmacy education OSCEs using a bibliometric review with content analysis.

METHODS

A bibliometric review was conducted on literature from over 23 years from January 2000 to May 2023. Articles focusing on any type of OSCE research in pharmacy education in both undergraduate and postgraduate sectors were included. Articles were excluded if they were not original articles or not published in English. A summative content analysis was also conducted to identify key topics.

RESULTS

A total of 192 articles were included in the analysis. There were 242 institutions that contributed to the OSCE literature in pharmacy education, with the leading country being Canada. Most OSCE research came from developed countries and were descriptive studies based on single institution data. The top themes emerging from content analysis were student perceptions on OSCE station styles (n = 98), staff perception (n = 19), grade assessment of OSCEs (n = 145), interprofessional education (n = 11), standardized patients (n = 12), and rubric development and standard setting (n = 8).

IMPLICATIONS

There has been a growth in virtual OSCEs, interprofessional OSCEs, and artificial intelligence OSCEs. Communication rubrics and minimizing assessor variability are still trending research areas. There is scope to conduct more research on evaluating specific types of OSCEs, when best to hold an OSCE, and comparing OSCEs to other assessments.

A Systematic Review of APPE Readiness Assessments Utilized by US Schools of Pharmacy

OBJECTIVE

The 2016 Accreditation Council for Pharmacy Education standard 25.8 requires schools of pharmacy to assess student readiness for Advanced Pharmacy Practice Experiences (APPEs). We performed a systematic review to identify how schools of pharmacy in the United States assess student readiness for APPE rotations in accordance with Accreditation Council for Pharmacy Education accreditation guidelines.

FINDINGS

From a search of 6 databases, we identified 1053 unique studies, of which 11 were eligible for inclusion in our review. The most commonly reported assessment method was the use of a capstone course; however, these courses varied significantly from school to school in duration, resources used, and content. Regardless of the specific approach used, first-time and overall pass rates were high.

SUMMARY

We found that while most studies reported using some type of capstone course for APPE readiness assessment, there was variability in how these courses were structured and the assessment methods used within the courses. The future 2025 standards may dictate a more uniform structure for readiness assessments; however, further research is needed to identify best practices regarding the assessment of APPE readiness.

Design and Evaluation of an APPE Readiness Dashboard Tool

OBJECTIVE

To describe the design and evaluation of a dashboard tool to assess the readiness for advanced pharmacy practice experiences (APPEs).

METHODS

The investigators developed a dashboard tool for the appraisal of APPE readiness consisting of indicators of pharmacy student knowledge, skills, and noncognitive attributes. For the 2018-2019 and 2019-2020 rotation years, a panel of stakeholders used the dashboard to evaluate APPE readiness. Students deemed not ready (5 or more dashboard indicators) had a remediation experience before the APPEs. The investigators monitored the students who were deemed ready but at risk for substandard performance (3 or 4 indicators or completion of remediation experience) and tracked the incidence of substandard performance for all APPE students. The outcomes were presented using descriptive statistics. Performance of the tool between years was compared using the χ2 test.

RESULTS

For 2 APPE rotation cycles, the investigators evaluated readiness in 499 students. A total of 3 students were deemed not ready, and 78 students were considered at risk for substandard performance. The tool predicted 52.1% of students who experienced a substandard performance. It had a negative predictive value of 93.6% and a positive predictive value of 38.5% for substandard performance. The predictive value was similar across the risk categories of knowledge, skills, and noncognitive attributes.

CONCLUSION

A dashboard tool facilitating holistic evaluation of APPE readiness was helpful in identifying students who were not ready for APPEs or at risk for substandard performance. The next steps include refinement of the tool through the evaluation of characteristics of students who experienced substandard performance and were not identified by the tool.

Evolution of a capstone exam for third-year doctor of pharmacy students

BACKGROUND AND PURPOSE

Pharmacy capstone exams are an example of an assessment strategy to evaluate student competency and practice-readiness. Capstone exam processes have been discussed in the literature; however, description of format, evolution of the process, and implementation in the year prior to rotations is lacking. This manuscript discusses the development, implementation, and utilization of a third professional year pharmacy capstone exam to assess advanced pharmacy practice experience (APPE) readiness.

EDUCATIONAL ACTIVITY AND SETTING

A pharmacy year three (PY3) capstone exam was initially developed prior to enrollment of the inaugural graduating class. Since the first iteration, utility has remained the same, but the overall development and implementation process has evolved due to changes in programmatic and student learning outcomes, curricular mapping, student and faculty feedback, and practice expectations.

FINDINGS

The entities responsible for exam implementation perceive faculty and students have a positive perception of the exam evolution and improvements implemented. Specifically, feedback from students and faculty mentioned clear expectations, comfort with the new capstone exam process, and an appreciation for the focus on interrater reliability. As assessment is an iterative process, quality improvement strategies continue to be implemented to address feedback regarding the PY3 capstone exam.

SUMMARY

Purposeful utilization of capstone exams in the third year is one method to assess APPE readiness. Sharing one college's experience with the evolution of such an exam, quality improvement methods, and experience in implementing reliability and validity measures may provide a model for institutions to implement similar assessment methods efficiently.

Objective structured clinical examination as a competency assessment tool of students' readiness for advanced pharmacy practice experiences in South Korea: a pilot study

BACKGROUND

The assessment of pharmacy students' readiness to begin the education of an advanced pharmacy practice experience (APPE) in clinical pharmacy settings continues to gain increasing attention. This study aimed to develop an objective structured clinical examination (OSCE) in the core domains acquired through an introductory pharmacy practice experience (IPPE), for evaluating its appropriateness as a tool of assessing clinical pharmacist competency for APPEs in Korean pharmacy students throughout a pilot study.

METHODS

OSCE's core competency domains and case scenarios were developed through a literature review, ideation by researchers, and external experts' consensus by a Delphi method. A prospective single-arm pilot test was conducted to implement the OSCE for Korean pharmacy students who completed a 60-h course of in-class simulation IPPE. Their competencies were assessed by four assessors in each OSCE station with a pass-fail grading system accompanied by a scoring rubric.

RESULTS

OSCE competency areas including patient counseling, provision of drug information, over-the-counter (OTC) counseling, and pharmaceutical care services were developed with four interactive and one non-interactive cases. Twenty pharmacy students participated in the OSCE pilot test, and their competencies were evaluated by 20 assessors. The performance rate was the lowest in the area of patient counseling for a respiratory inhaler (32.1%) and the highest (79.7%) in OTC counseling for constipation. The students had an average performance rate of 60.4% in their communication skills. Most participants agreed on the appropriateness, necessity, and effectiveness of the OSCE in evaluating pharmacy students' clinical performance and communication skills.

CONCLUSIONS

The OSCE model can be used to assess pharmacy students' readiness for off-campus clinical pharmacy practice experience. Our pilot study suggests the necessity of conducting an OSCE domain-based adjustment of difficulty levels, and strengthening simulation-based IPPE education.

Making progress: A national study on the development and use of progression assessments in United States pharmacy curricula

BACKGROUND

The objective was to determine the utilization, frequency, characteristics, and standard-setting methods of progression assessments in pharmacy education.

METHODS

A survey was sent to 139 United States schools/colleges of pharmacy having an identifiable assessment lead and students enrolled in the doctor of pharmacy program. The survey examined programs' use, frequency, and characteristics of progression assessments within their curriculum. Respondents also reported any changes made due to the COVID-19 pandemic and which, if any, would be maintained in future years. Analysis consisted of descriptive statistics and thematic coding. This research was deemed exempt by the university's institutional review board.

RESULTS

Seventy-eight programs responded to the survey (response rate = 56%). Sixty-seven percent of programs administered at least one progression assessment in 2019-2020. There was some variability in assessment practice, including professional year(s) administered, course(s) involved, and content. Approximately 75% of programs used assessments to ensure student competency in the programs' learning outcomes and to identify individual student learning deficiencies. Diversity was seen in validity and reliability practices, and most programs used pre-determined cut scores without formal standard setting. Because of the pandemic, 75% of programs changed the assessment delivery mode and 20 programs planned to maintain at least one pandemic-related change in future iterations.

CONCLUSIONS

Most pharmacy programs utilize some type of progression assessment within their curriculum. While many schools administer progression assessments, there is little agreement on their purpose, development, and use. The pandemic changed the mode of delivery, which numerous programs will continue with in the future.

Are we ready yet for digital transformation? Virtual versus on-campus OSCE as assessment tools in pharmacy education. A randomized controlled head-to-head comparative assessment

Background

The global COVID-19 pandemic has influenced pharmacy education including learning, assessment, and exams. In the UAE, pharmacy instructors have adapted several innovative teaching methods to strive for quality learning outcomes. The current trial presented a head-to-head comparative assessment between on-campus versus virtual Objective Structured Clinical Examination (OSCE) with examiners' and students' perspectives.

Aim

The main aim was to compare fourth-year students' and examiners' perceptions of the feasibility (time and logistics), stress, performance, and satisfaction between on-campus versus virtual OSCE.

Method

A randomized controlled head-to-head comparative assessment between the On-campus and virtual OSCE was conducted to explore performance and satisfaction of pharmacy students and examiners towards the two OSCE settings. The virtual OSCE was carried out directly after the on-campus -OSCE and the setting was designed in a way that aligned with the on-campus OSCE but in a virtual way. Microsoft Teams® breakout room was used as a virtual stations. Respondus-lockdown-browse and Google Meet® were used for proctoring purposes.

Results

Students who sat for the on-campus assessment were more satisfied with the instructions, the orientation session, the time management, and the overall exam setting, the ability of the exam to assess their communication and clinical skills, professionalism and attitude, and the interactivity of the exam compared to the students who sat for the virtual assessment. Examiners' perceptions for both settings were the same with the exception of interaction with students (p less than 0.05) as the on-campus OSCE was more interactive.

Conclusion

Students still prefer the on-campus OSCE to the virtual OSCE format in many aspects. Nevertheless, virtual OSCE is still a feasible and satisfactory method of assessment when on-campus OSCE is not possible. There is a need of a specialized platform to conduct the virtual OSCE from A to Z rather than maximizing the use of options in the current digital platforms.

A Longitudinal Integrative Course Series to Prepare Students for Advanced Pharmacy Practice Experiences

Background: This paper describes a series of integrative courses intentionally designed to prepare students for Advanced Pharmacy Practice Experiences (APPEs) in a block system curriculum. Innovation: Three integration blocks are interspersed throughout the didactic curriculum to serve as checkpoints to ensure competency as students progress in the curriculum, rather than waiting until the end to determine competency. Complex patient case discussions and a series of high-stakes assessments are used to reinforce and evaluate cumulative retention of knowledge, skills, and attitudes. Findings: Class of 2022 exam results showed that in the cohort of students who failed the high-stakes comprehensive knowledge assessment (CKA) and pharmacy calculations exams during the first integration block (IB), failure rates decreased in subsequent IBs, indicating early detection of knowledge deficiencies and either exam performance improvement in each IB or failure to progress to the next IB. A survey of the same cohort indicated that the final integration block prior to advanced pharmacy practice experiences (APPEs) helped improve confidence in applying key knowledge and skills into practice. Conclusion: The series of integration blocks designed and implemented at WesternU provides opportunities to reinforce knowledge and skills while requiring students to demonstrate maintenance of core competency as they progress through the curriculum.

A classroom-based patient case activity as a potential formative indicator of pre-advanced pharmacy practice experience readiness

BACKGROUND AND PURPOSE

Pharmacy programs must assess student readiness for advanced pharmacy practice experiences (APPEs). This study evaluated: (1) if a relationship exists between student performance on a classroom-based patient case activity (Rx Review) and (2) performance on direct patient care and perceptions of the transferability of Rx Reviews to the APPE setting.

EDUCATIONAL ACTIVITY AND SETTING

Rx Reviews are timed, independent, prospective reviews of a patient's medical record intended to simulate pre-rounding activities common in acute and ambulatory care settings. Students complete 12 Rx Reviews across a skills laboratory course sequence. Rx Review scores were compared to performance on acute care and ambulatory care APPEs for students in the Class of 2019 using descriptive statistics for student scores and bivariate logistic regression for the association between Rx Review scores and performance outcomes on patient care APPEs. Student perceptions of the Rx Reviews were assessed through a survey. Descriptive statistics and qualitative responses were used to analyze responses to the multiple-choice survey questions and open-ended question, respectively.

FINDINGS

One hundred twenty-two students participated in the study. A relationship was found between students' Rx Review scores and performance in certain direct patient care APPEs, with students performing in the bottom 50% on Rx Reviews being more likely to perform poorly on those APPEs. Over 93% of students identified Rx Reviews as at least somewhat helpful in preparing them for APPEs.

SUMMARY

This study indicated that a classroom-based patient case activity could be used as a formative measure of APPE readiness.

Predictors of student pharmacist success on an advanced pharmacy practice experience-readiness performance-based assessment

INTRODUCTION

The study objective was determining the relationship of certain predictor variables with student performance on an advanced pharmacy practice experience (APPE)-readiness performance-based assessment (PBA) in the third professional year (P3).

METHODS

This retrospective study aimed to identify if student predictors and/or curricular-related predictor variables correlate with student performance on a patient interview and documentation activity (PIDA), an APPE-readiness assessment. Student predictors included Pharmacy College Admission Test score, pre-pharmacy cumulative grade point average (GPA), end of first professional year (P1) GPA, end of second professional year GPA, end of fall P3 GPA, and elective Pharmacists' Patient Care Process (PPCP) score. Curricular-related predictor variables comprised specific activities with emphasis on patient care skill development in P1 through P3 years. Adjusted and unadjusted linear regression models assessed correlations between PIDA score and student predictors as well as curricular variables.

RESULTS

Students from the Classes of 2018 through 2020 at Southern Illinois University Edwardsville School of Pharmacy (N = 231) were included. Two student predictors, P3 GPA and PPCP elective score, showed statistically significant correlations with PIDA performance. One curriculum-related variable, P3 activity with an emphasis on medication reconciliation, demonstrated a statistically significant correlation with student performance on the PIDA.

CONCLUSIONS

Student performance on a cumulative PBA may be predictive from prior curricular learning experiences that assess similar patient care skills. The predictive impact of the elective PPCP score on a PBA needs additional research. Recent implementation of a new curriculum may lead to identification of other predictors of student success on PBAs.

Exploring Multiple Perspectives on Pharmacy Students' Readiness for Advanced Pharmacy Practice Experiences

Objective. This study aimed to enrich the Academy's understanding of pharmacy students' readiness for advanced pharmacy practice experiences (APPEs) by exploring the perspectives of three primary stakeholders: APPE students, APPE preceptors, and APPE faculty site directors.Methods. A descriptive qualitative study of APPE readiness was conducted using workplace learning as a guiding conceptual framework. Data were collected between March and September 2019 through semi-structured focus groups and interviews with students (five groups), preceptors (four groups), and faculty site directors (one group, two individual interviews). The data were analyzed using directed content analysis.Results. Participants described APPE readiness as a multifaceted construct comprised of four themes: learner characteristics, participation in workplace activities, relationship-building, and workplace practices to orient and support students. While all participants addressed each category, faculty site directors and preceptors tended to focus on learner characteristics, while students emphasized their participation in the workplace and relationship building.Conclusion. Knowledge is widely recognized as a requirement for APPE readiness. This study identified learner characteristics, workplace participation, and relational skills as additional requirements. Some of these criteria are challenging to assess prior to APPEs, which makes orienting students both prior to and at the start of APPEs particularly important to support readiness. Thus, a comprehensive review of APPE readiness might also include assessing the readiness of workplaces, administrators, and preceptors for APPE students.

Use of Capstone Experiences in Pharmacy Education to Synthesize and Apply Students' Knowledge and Skills

Objective. To investigate capstone experiences in pharmacy education with the broad aim of better understanding how they are designed, used, and sustained.Methods. Capstones were defined as culminating experiences that required Doctor of Pharmacy (PharmD) students to apply their knowledge and skills and demonstrate their ability to integrate, extend, and apply multiple components of their academic experience. This study was conducted in two phases. Phase one was a mapping review, which provided insight into the design of pharmacy capstone experiences, including the purpose they served and their associated outcomes. Phase two consisted of conducting follow-up interviews to explore the current status and sustainability of those capstone experiences. Qualitative methods with multiple coders were used to analyze the data.Results. Five distinct purposes for capstone experiences were identified: preparation for advanced pharmacy practice experiences (APPEs), pharmacotherapy education, research, reflection on pharmacy school experiences, and immersive learning through an external organization. Wide variability was found in the design, purpose, and outcomes of capstone experiences. Study authors described the capstone experiences at their institution in terms of the viability, feasibility, desirability, and challenges faced.Conclusion. Capstones can play an important role in assessing student progress and practice readiness in pharmacy education. The findings of this study can be used to help pharmacy schools design capstone experiences that are effective and sustainable.

Predictors of Student Failure or Poor Performance on Advanced Pharmacy Practice Experiences

Objective. To determine factors predictive of student failure or poor performance on advanced pharmacy practice experiences (APPEs) at a single pharmacy program. Methods. This retrospective cohort evaluated students entering the Doctor of Pharmacy (PharmD) program from 2012-2014 at St. Louis College of Pharmacy. Students who received a grade of F for one or more APPEs (failure group) were compared to all other students (non-failure group). A secondary evaluation compared students with a C or F on one or more APPEs (poor performers) to all other students (non-poor performers). Data were collected on didactic and experiential performance, identifiable professionalism issues from introductory pharmacy practice experiences (IPPEs), and academic honor code violations. Univariable and multivariable logistic regressions were performed to determine factors associated with APPE failure and poor performance. Results. A total of 669 students were analyzed. Twenty-eight students (4.2%) failed one or more APPEs and 81 students (12.1%) were identified as poor performers (grade of C or F). For the primary outcome, professional grade point average (GPA) of less than 2.7, practicum failure, IPPE professionalism issue(s), and pharmacotherapy course failure were identified for inclusion in the multivariable analysis. The IPPE professionalism issue(s) (HR 4.8 [95% CI 1.9-12.4]) and pharmacotherapy course failure (HR 4.2 [95% CI, 1.6-11.1]) were associated with APPE failure on multivariable regression. On the secondary analysis, the same variables were identified for multivariable regression, with professional GPA of less than 2.7 (HR 2.7 [95% CI 1.5-5]), IPPE professionalism issue(s) (HR 3.9 [95% CI 2.2-6.9]), and pharmacotherapy course failure (HR 2.0 [95% CI 1.1-3.7]) associated with poor performance. Conclusion. Poor academic performance and/or identified unprofessional behavior while completing IPPEs are associated with APPE failure and poor performance. Interventions should be aimed at identifying at-risk students and addressing risk factors prior to APPEs.

Lessons from the implementation of developmental progress assessment: A scoping review

OBJECTIVES

Educators and researchers recently implemented developmental progress assessment (DPA) in the context of competency-based education. To reap its anticipated benefits, much still remains to be understood about its implementation. In this study, we aimed to determine the nature and extent of the current evidence on DPA, in an effort to broaden our understanding of the major goals and intended outcomes of DPA as well as the lessons learned from how it has been executed in, or applied across, educational contexts.

METHODS

We conducted a scoping study based on the methodology of Arksey and O'Malley. Our search strategy yielded 2494 articles. These articles were screened for inclusion and exclusion (90% agreement), and numerical and qualitative data were extracted from 56 articles based on a pre-defined set of charting categories. The thematic analysis of the qualitative data was completed with iterative consultations and discussions until consensus was achieved for the interpretation of the results.

RESULTS

Tools used to document DPA include scales, milestones and portfolios. Performances were observed in clinical or standardised contexts. We identified seven major themes in our qualitative thematic analysis: (a) underlying aims of DPA; (b) sources of information; (c) barriers; (d) contextual factors that can act as barriers or facilitators to the implementation of DPA; (e) facilitators; (f) observed outcomes, and (g) documented validity evidences.

CONCLUSIONS

Developmental progress assessment seems to fill a need in the training of future competent health professionals. However, moving forward with a widespread implementation of DPA, factors such as lack of access to user-friendly technology and time to observe performance may render its operationalisation burdensome in the context of competency-based medical education.

An initial environmental scan of APPE readiness assessment

INTRODUCTION

Standards 2016 require schools/colleges of pharmacy (s/cop) to assess students' readiness to enter advanced pharmacy practice experiences (APPEs). However, literature describing how schools are meeting this standard is limited. The purpose of this study was to conduct an environmental scan to describe how s/cop assess student readiness to enter APPEs.

METHODS

A web-based survey was distributed to assessment leads at United States s/cop, regardless of accreditation status. Respondents answered questions related to their current approach to assessing student APPE readiness, existence of intentional assessment plans, competencies used, assessment methods, benchmarks, and remediation strategies. Aggregate data were analyzed using descriptive statistics.

RESULTS

Fifty-two S/COP (36.1%) responded. The majority (90.1%) were fully accredited schools. Most respondents have an intentional APPE readiness plan (73.5%), although the duration since implementation varied. There was no consensus among schools on which competencies informed APPE readiness with 67.3% listing Center for the Advancement of Pharmacy Education (CAPE) 2013 outcomes, 61.2% Guidance for Standards 2016 Appendix A, 53.1% pre-APPE domains (Standards 2007), and 30.6% Entrustable Professional Activities. Twenty-eight S/COP (57.1%) reported having individual student-level data to assess student APPE readiness. The most common methods for validating student APPE readiness were preceptor (48.9%) and student (44.9%) surveys.

CONCLUSIONS

This environmental scan begins to identify trends in how S/COP is approaching the assessment of student readiness to begin APPEs. Further research is needed to identify best practices and practical methods to ensure compliance with current accreditation standards.

A Modeling Exercise to Identify Predictors of Student Readiness for Advanced Pharmacy Practice Experiences

Objective. To model the relationship of common pharmacy education assessment data including student demographics, pre-pharmacy performance, core didactic performance, and external testing measures to identify predictors of student readiness for advanced pharmacy practice experiences (APPEs). Methods. The associations between 23 predictive covariates from 226 graduating students from 2015-2018 (5786 observations) and APPE readiness as measured by midpoint core APPE scores were modeled. Multiple linear and Poisson regression models with backward selection were used. A selection criterion of p >.10 was used for covariate elimination from the model. Three models were evaluated: average of all midpoint core APPE rotation scores; average of midpoint acute care pharmacy practice and ambulatory care APPE rotation scores; and number of midpoint core clerkship failing scores. Results. The average age of the population at admission was 25.4±4.5 years, 47% were female, and 75.2% had prior degrees. Across the three prediction models, knowledge-retention covariates were the strongest predictors. Total score on the Pharmacy Curriculum Outcomes Assessment was a modest yet consistent predictor across the models. All other significant predictors were unique to the various models. Conclusion. This four-year, population-based modeling study of the relationship of common pharmacy education assessment data to APPE midpoint scores shows a modest correlation with knowledge-based measures. There is a need for greater innovation in this area of research.

Developing a comprehensive APPE-readiness plan with a focus on skills, attitudes, and behaviors

BACKGROUND AND PURPOSE

Pharmacy schools must have a framework to ensure students have the necessary knowledge, skills, attitudes, and behaviors to be successful during advanced pharmacy practice experiences (APPEs).

EDUCATIONAL ACTIVITY AND SETTING

Ten summative assessments, called APPE-readiness assessments (ARAs), were developed based on eight competencies encompassing skills, behaviors, and attitudes that must be demonstrated prior to APPEs. All eight competencies were assessed in the course Case Studies in Pharmacotherapy IV, which is offered in the final semester of the didactic curriculum immediately prior to APPEs. A 15-question pre- and post-survey was conducted to assess student confidence in performing each APPE-readiness competency. Cohort data was evaluated by the curriculum and assessment committee to assess curriculum effectiveness and areas for improvement.

FINDINGS

Upon completion of the course, the average first-attempt pass rate across all ARAs was 92.4%. All students who failed on the first attempt passed on the second attempt, thereby demonstrating APPE-readiness. Out of 62 students, 45 and 44 completed the pre- and post-survey, respectively. Prior to the ARAs, the overall average of students who felt (strongly) confident about their ability to perform each competency was 82.2 ± 2.1%. This increased to 92.6 ± 1.6% after the ARAs.

SUMMARY

The development of an APPE-readiness assessment plan focusing on skills, attitudes, and behaviors provides insight into student and cohort performance and allows for continuous quality assurance of the pre-APPE curriculum.

An Objective Structured Clinical Examination to Assess Competency Acquired During an Introductory Pharmacy Practice Experience

Objective. To evaluate the use of an objective structured clinical examinations (OSCE) to assess clinical competency acquired during an off-campus introductory pharmacy practice experience (IPPE). Methods. Third-year pharmacy students completed an IPPE in transitions of care and completed 24 experiential contact hours at one of 17 practice sites. Students were assessed using two OSCEs, the first occurring prior to beginning an off-site IPPE (pre-experience OSCE) and the second occurring after completion of the off-site IPPE (post-experience OSCE). Each OSCE consisted of 10 stations and covered five graded competency domains. The primary outcome was the degree of change in student performance from the pre-experience OSCE to the post-experience OSCE. Secondary outcomes included changes in each graded domain, OSCE pass rate, and failure conversion rate. Results. Of 111 students, 109 completed both the pre- and post-experience OSCE. Significant improvements were observed in overall score and cohort pass rate. Overall scores improved from 80 for the pre-experience OSCE to 87 for the post-experience OSCE. The OSCE pass rate also improved from 47% to 84%. Conclusion. Although preceptor evaluations have traditionally served as the primary summative assessment for IPPE and APPE, this study indicates that OSCEs may be a reliable alternative to assess clinical competency acquired from off-site practice experiences.

Characterization of Use of the Pharmacy Curriculum Outcomes Assessment Across Accredited Colleges of Pharmacy

Objective. To characterize use of the Pharmacy Curriculum Outcomes Assessment (PCOA) in terms of timing, manner of delivery, and application of the results by accredited colleges of pharmacy. Methods. Accredited pharmacy programs were surveyed regarding PCOA administration, perceived benefits, and practical application of score reports. Survey items were comprised of new items developed from a literature review and items from prior studies. The survey addressed five domains: program demographics, administration, student preparation, use of results, and recommendations to improve the utility of the PCOA. Results. Responses were received from 126 of 139 (91%) surveyed programs. The majority of respondent programs administered PCOA in one session on a single campus. Most indicated PCOA results had limited use for individual student assessment. Almost half reported that results were or could be useful in curriculum review and benchmarking. Considerable variability existed in the preparation and incentives for PCOA performance. Differences in some results were found based on prior PCOA experience and between new vs older programs. Open-ended responses provided suggestions to enhance the application and utility of PCOA. Conclusion. The intended uses of PCOA results, such as for student assessment, curricular review, and programmatic benchmarking, are not being implemented across the academy. Streamlining examination logistics, providing additional examination-related data, and clarifying the purpose of the examination to faculty members and students may increase the utility of PCOA results.

National Trends in the Adoption of Pharmacy Curriculum Outcomes Assessment for Student Assessment and Remediation

Objective. To determine and describe the current uses of the Pharmacy Curriculum Outcomes Assessment (PCOA) by US schools and colleges of pharmacy. Methods. Assessment professionals from 135 US schools and colleges of pharmacy were invited to complete a 38-item electronic survey. Survey items were designed to investigate common uses of the PCOA, cut points, and "stakes" assigned to the PCOA, identification of at-risk students, and remediation approaches. Results. The school response rate was 68%. The most common uses of the PCOA included curricular assessment (76%), individual student performance assessment (74%), and cohort performance assessment (71%). The PCOA was most frequently administered to third-year pharmacy (P3) students. The approach for assigning "stakes" to PCOA performance varied among programs depending on the student's professional year in the curriculum. Programs used a variety of approaches to establish the benchmark (or cut point) for PCOA performance. Remediation for at risk students was required by less than 25% of programs. Remediation was most commonly required for P3 students (22%). Conclusion. Survey results indicate wide variability between programs regarding PCOA cut points (benchmarks), stakes, and remediation approaches. In the future, it will be important for pharmacy educators to identify and study best practices for use of PCOA within student assessment and remediation plans.

Developing a Comprehensive First-year Capstone to Assess and Inform Student Learning and Curriculum Effectiveness

Objective. To describe the development and implementation of an innovative, comprehensive, multi-day module focused on assessing and providing feedback on student cognitive and interpersonal skill development and practice readiness after the first year (PY1) of a Doctor of Pharmacy (PharmD) curriculum. Methods. A multi-day capstone assessment was developed to evaluate first-year students' knowledge of course content, ability to find and apply information, and interpersonal skills, including teamwork and adaptability. The PY1 Capstone consisted of four parts. Knowledge was assessed using 130 multiple-choice items on first-year course content and 50 fill-in-the-blank items on Top 200 brand and generic drug names. The ability to find and apply information was assessed using a 45-question open-book test. Interpersonal skills were assessed using a specially designed multiple mini-interview (MMI). The final part of the assessment was a debriefing session that provided rapid-cycle feedback on capstone performance and a bridge between students' recently completed first-year coursework and an upcoming 2-month experiential immersion. Results. The average score on the closed-book and open-book assessments were 75% and 68%, respectively. Most students displayed satisfactory interpersonal skills based on the MMI. Students viewed the assessment positively based on post-assessment survey responses (>75%). Most students (98%) reported not studying for the assessment, indicating that the results should reflect students' retention of knowledge and skills. Conclusion. The capstone assesses students on knowledge and skills and provides students with feedback on areas to focus on during their early immersion. Continued work is needed to ensure the process is transparent and cost-effective.

Examining the Association of GPA and PCAT Scores on Objective Structured Clinical Examination Scores

Objective. To examine the association between certain demographic and admission measures and Objective Structured Clinical Examination (OSCE) performance in a cohort of pharmacy students. Methods. A retrospective review of demographic characteristics, admissions data [cumulative and science admission Grade Point Average (GPA), Pharmacy College Admissions Test (PCAT) scores], and OSCE scores was performed for the Class of 2017 at the University of Tennessee College of Pharmacy. Results. Female students scored significantly higher than male students on the Warfarin OSCE - Standardized Patient (SP) rated General Communication Skills and on the Warfarin OSCE - Faculty rated Patient Interviewing Skills. Age was significantly, inversely correlated with Warfarin OSCE - Faculty rated Therapeutic Knowledge score. Warfarin OSCE - SP rated General Communication Skills score was significantly, positively related to PCAT composite score and PCAT reading comprehension score. PCAT composite score was significantly, inversely correlated to Warfarin OSCE - Faculty rated Patient Interviewing Skills score. Warfarin OSCE - Faculty rated General Communication Skills score was significantly, positively related to cumulative admission GPA and admission science GPA. Conclusion. Eight statistically significant correlations were found between demographic and admissions measures and specific OSCE scores. Regression models were significant but explained a low percentage of the variance in OSCE scores, suggesting other factors not included in the study have a greater effect on scores. Such factors may include knowledge acquired in pharmacy school courses such as the therapeutics course series.

Examination of Factors Relating to Student Performance on the Pharmacy Curriculum Outcomes Assessment

Objective. To examine relationships between students' demographic and academic performance factors and their scores on the Pharmacy Curriculum Outcomes Assessment (PCOA). Methods. Students' PCOA scores and demographics (eg, age, race/ethnicity, sex), preadmission data [eg, cumulative and science grade point average (GPA), Pharmacy College Admissions Test (PCAT)], and academic performance variables (eg, pharmacy GPA, academic standing) were analyzed for one class of third-year pharmacy students (N=159). Independent t-tests and Analysis of Variance (ANOVA) were used to compare scores by demographic variables. Pearson's r correlations were used to assess relationships between PCOA scores and age, PCAT scores, and GPA. Stepwise linear regression was conducted to determine the predictive ability of variables with significant correlations to PCOA performance. Results. Significant correlations were found between sex and PCOA scores with males scoring higher than females. Significant correlations with PCOA scores were also found for PCAT scores, pre-pharmacy science GPA, and pharmacy didactic GPA. Significant differences were found by academic standing, where students in academic difficulty who were allowed to proceed without repeating curricular content scored significantly lower on the PCOA than those who did not experience academic difficulty. Conversely, there were no statistical differences between those who repeated courses and those who never experienced academic difficulty. PCOA performance predictors in the final regression model included PCAT composite score, pharmacy GPA and sex. Conclusion. New findings included differences in PCOA scores by sex and by academic standing, a variable not previously explored in published studies. Findings have implications for remediation decisions in pharmacy curricula.

Design and activity evaluation of an Advanced-Introductory Pharmacy Practice Experience (aIPPE) course for assessment of student APPE-readiness

BACKGROUND AND PURPOSE

The newly implemented Advanced-Introductory Pharmacy Practice Experience (aIPPE) course at the University of Colorado Skaggs School of Pharmacy and Pharmaceutical Sciences was designed to assess student readiness for Advanced Pharmacy Practice Experience (APPE) and provide formative feedback regarding skills and abilities needed to be successful during subsequent coursework and practice experiences.

EDUCATIONAL ACTIVITY AND SETTING

The aIPPE is a full-time, six week (240hours) direct patient care rotation occurring in the spring semester of the third-professional year following a longitudinal integrated IPPE program. Required aIPPE course elements mimic the activities and expectations students should anticipate encountering during APPE training.

FINDINGS

All students participating in the initial aIPPE course offering were described as APPE-Ready. Students and preceptors described the aIPPE course as successful in achieving the primary outcome of preparing students for improved downstream performance.

SUMMARY

The aIPPE provided students opportunities to demonstrate readiness to enter advanced pharmacy practice experience.

A Call for an Integrated Program of Assessment

An integrated curriculum that does not incorporate equally integrated assessment strategies is likely to prove ineffective in achieving the desired educational outcomes. We suggest it is time for colleges and schools of pharmacy to re-engineer their approach to assessment. To build the case, we first discuss the challenges leading to the need for curricular developments in pharmacy education. We then turn to the literature that informs how assessment can influence learning, introduce an approach to learning assessment that is being used by several medical education programs, and provide some examples of this approach in operation. Finally, we identify some of the challenges faced in adopting such an integrated approach to assessment and suggest that this is an area ripe with research opportunities for pharmacy educators.

An analysis of the use of Pharmacy Curriculum Outcomes Assessment (PCOA) scores within one professional program

OBJECTIVE

The Pharmacy Curriculum Outcomes Assessment (PCOA) is a recent assessment requirement for US pharmacy professional programs. This study analyses PCOA scores for uses described in the 2016 Standards with data from one professional program.

METHODS

PCOA data were analyzed for two consecutive classes (n=215) of pharmacy students at the end of their didactic curriculum to explore relationships among PCOA scores, grade point average (GPA), and North American Pharmacist Licensure Examination (NAPLEX) scores utilizing regression analyses.

RESULTS

Decisions about student learning based on PCOA scores and GPA indicated remediation would have been prescribed for approximately 7% of students. In comparison, NAPLEX scores revealed a 1% failure rate among the study sample. Relationships between PCOA scores and GPA (r=0.47) and NAPLEX (r=0.51) were moderate to large, respectively. GPA explained a larger portion of unique variance (14%) than PCOA (8%) in NAPLEX scores.

CONCLUSIONS

In this sample of students, academic decisions would have varied depending upon the learning assessment, which is consistent with a moderate correlation between GPA and PCOA scores. Although PCOA scores correlate with GPA and NAPLEX, PCOA scores explained a smaller portion of unique variance in NAPLEX scores than GPA. The ongoing establishment of validity evidence of PCOA scores is important for meaningful interpretation of scores for the intended uses.

Development of a direct observation of procedural skills rubric for fourth-year pharmacy students in ambulatory care rotations

PURPOSE

The development and implementation of a pilot program requiring direct observation of procedural skills (DOPS) assessments of fourth-year pharmacy students during ambulatory care rotations are described.

METHODS

All fourth-year pharmacy students at the University of Texas at Austin College of Pharmacy who engaged in a required advanced pharmacy practice experience (APPE) in ambulatory care during the period January-May 2015 participated in a DOPS pilot program. The DOPS process required students to select a patient for a medication-focused "workup," evaluate the selected patient case, present the results to a preceptor, and conduct a preceptor-supervised patient interview. Preceptors used a DOPS rubric to rate students' performance in 12 domains. At the time of submission of DOPS evaluation forms, program participants were invited to complete online surveys soliciting feedback on the effectiveness of the DOPS evaluation process and other aspects of the program.

RESULTS

A total of 81 students and preceptors participated in the DOPS pilot program, with 47 DOPS evaluation forms submitted; the median ± S.D. score was 90.4% ± 29.7%. Results of the online surveys indicated that the overall perception of the DOPS program was positive, with majorities of both students and preceptors supporting DOPS incorporation into the curriculum for fourth-year pharmacy students.

CONCLUSION

The DOPS rubric was a useful tool for evaluating clinical skills of APPE students on ambulatory care rotations.

Evidence of Criterion Validity for One Pharmacy School's Progress Examination Program

Objective. To provide evidence that the progress examination program accurately assesses student failure to demonstrate competence. Methods. A progress examination program aligned with each grade level was locally developed and administered annually to 289 PharmD students in the spring of their first year - fourth year. Correlations and linear regressions were performed to compare the examination scores to performance on national licensing examinations and cumulative didactic grade point average (GPA). Odds ratio analysis was run to determine the ability of the passing scores of the progress examination to identify students at increased risk of failing to graduate on time, earn a GPA below 3.0, and fail the licensing examinations on their first attempt. Results. Progress examination scores were strongly correlated to GPA and national licensing examination scores and weakly correlated to jurisprudence examination scores. Regression analysis indicated a significant linear relationship between examination scores and both GPA and the licensing examinations. Students who performed poorly on the progress examinations were more likely fail the national licensing examination, more likely to fail to graduate on time, and more likely to earn a cumulative didactic GPA below 3.0. Conclusions. The second-year examination program strongly predicts students at risk for failure to graduate on time or achieve a GPA below 3.0, while all four examinations identify students at risk of failing the national licensing examination on their first attempt.

Pharmacy Student Learning During Advanced Pharmacy Practice Experiences in Relation to the CAPE 2013 Outcomes

Outcomes from The Center for Advancement of Pharmacy Education (CAPE) are intended to represent the terminal knowledge, skills, and attitudes pharmacy students should possess and have guided delivery of pharmacy education for more than two decades. Advanced pharmacy practice experiences (APPEs) are the endpoint of pharmacy curricula where demonstration and assessment of terminal learning occurs. This review examines published literature in relation to the most recent CAPE outcomes to determine the extent to which they have been addressed during APPEs since 1996. Details related to the APPE focus, intervention(s)/learning setting(s), and assessments are summarized according to the 15 CAPE outcomes. Further, the assessments are categorized according to the level of learning achieved using an available method. Common CAPE outcomes are highlighted, as well as those for which published reports are lacking for APPEs. The range and quality of assessments are discussed and emphasize the need for continuous improvement of scholarly design and assessment.

The Validation of a Case-Based, Cumulative Assessment and Progressions Examination

OBJECTIVE

To assess content and criterion validity, as well as reliability of an internally developed, case-based, cumulative, high-stakes third-year Annual Student Assessment and Progression Examination (P3 ASAP Exam).

METHODS

Content validity was assessed through the writing-reviewing process. Criterion validity was assessed by comparing student scores on the P3 ASAP Exam with the nationally validated Pharmacy Curriculum Outcomes Assessment (PCOA). Reliability was assessed with psychometric analysis comparing student performance over four years.

RESULTS

The P3 ASAP Exam showed content validity through representation of didactic courses and professional outcomes. Similar scores on the P3 ASAP Exam and PCOA with Pearson correlation coefficient established criterion validity. Consistent student performance using Kuder-Richardson coefficient (KR-20) since 2012 reflected reliability of the examination.

CONCLUSION

Pharmacy schools can implement internally developed, high-stakes, cumulative progression examinations that are valid and reliable using a robust writing-reviewing process and psychometric analyses.

Implementation and Use of the Pharmacy Curriculum Outcomes Assessment at US Schools of Pharmacy

OBJECTIVE

To describe how schools and colleges of pharmacy use the Pharmacy Curriculum Outcomes Assessment (PCOA) in relation to student assessment and curricular feedback.

METHODS

A survey was distributed to all programs that have implemented the PCOA. The survey was designed to assess 3 domains regarding the use of the PCOA: rationale for use, logistics of administration, and performance data review and distribution.

RESULTS

A 79% response rate (41/52) was obtained. The mix of responses was 93% current PCOA users and 7% past users. The most common reasons for PCOA use were for programmatic assessment and benchmarking. The examination was most frequently administered during the P3 year, with minimal stakes attached to performance. Significant differences in responses based on public vs private institution were seen with respect to length of accreditation of current PCOA users, messaging to students regarding performance, inclusion of results in student advising, and distribution of results to stakeholders.

CONCLUSION

Programs were using the PCOA primarily as an assessment in the P3 year for reasons related to programmatic and curricular assessment. Some differences existed between public and private institutional PCOA use and examination-related processes and results distribution.

Limited Predictive Utility of Admissions Scores and Objective Structured Clinical Examinations for APPE Performance

OBJECTIVE

To examine the relationship between admissions, objective structured clinical examination (OSCE), and advanced pharmacy practice experience (APPE) scores.

METHODS

Admissions, OSCE, and APPE scores were collected for students who graduated from the doctor of pharmacy (PharmD) program in spring of 2012 and spring of 2013 (n=289). Pearson correlation was used to examine relationships between variables, and independent t test was used to compare mean scores between groups.

RESULTS

All relationships among admissions data (undergraduate grade point average, composite PCAT scores, and interview scores) and OSCE and APPE scores were weak, with the strongest association found between the final OSCE and ambulatory care APPEs. Students with low scores on the final OSCE performed lower than others on the acute care, ambulatory care, and community APPEs.

CONCLUSION

This study highlights the complexities of assessing student development of noncognitive professional skills over the course of a curriculum.

Use of Cumulative Assessments in U.S. Schools and Colleges of Pharmacy

The Accreditation Council of Pharmacy Education (ACPE) has taken a strong stance on assessment in pharmacy education. One available assessment tool is cumulative assessments, which may be administered at various points in the curriculum. This article presents the results of a survey of U.S. schools of pharmacy regarding the use of cumulative assessments within their curriculum. A 20-question survey tool was emailed to 125 schools of pharmacy. A total of 105 out of 125 schools participated (response rate 84%). Of these, 52 schools currently have a cumulative assessment program; 18 have one cumulative exam prior to advanced pharmacy practice experiences (APPEs); 19 have a cumulative exam every didactic year; and seven have accumulative exams every semester, except during APPEs (n = 44). Increased faculty workload emerged as the top challenge faced by schools that have implemented a cumulative assessment program. Eighteen schools indicated that no outcomes are measured to determine the utility of the cumulative assessment. From these results, it appears that almost half of participating U.S. schools have implemented a cumulative assessment plan. However, it is apparent that more research needs to be done to determine which outcomes are expected to improve with the implementation of such an assessment plan.

Reliability of a minimal competency score for an annual skills mastery assessment

OBJECTIVE

To determine whether the modified Angoff process can be used to calculate a reliable minimal competency ("cut") score for the Annual Skills Mastery Assessment (ASMA).

METHODS

Three panels of pharmacy faculty members used a modified Angoff method to create a minimal competency score for 60 previously used test items. The panels did not know which items were included. Data were analyzed to determine differences between rating sessions, faculty type, item difficulty, and rater scoring bias.

RESULTS

The cut score generated was not significantly different by session or faculty type. The range of cut scores varied by less than 3% per examination. Faculty panelists correctly predicted student performance on items grouped as easy, medium, and hard.

CONCLUSION

A properly constructed faculty panel can determine a reliable cut score and accurately rank relative test item difficulty using the modified Angoff process.

An objective structured clinical examination to assess problem-based learning

OBJECTIVES

To compare pharmacy students' performance on an objective structured clinical examination (OSCE) to their performance on a written examination for the assessment of problem-based learning (PBL); and to determine students' and faculty members' perceptions of OSCEs for PBL evaluations.

DESIGN

Four OSCEs were added to the written examination to assess 4 PBL cases in a third-year pharmacotherapy course. OSCE scores were compared to written examination scores. Faculty members evaluated student performance.

ASSESSMENT

OSCE performance did not correlate with the written-examination scores. Most students (≥ 75%) agreed that OSCEs reflected their learning from PBL and measured knowledge, communication, and clinical skills. A majority of faculty members (≥75%) agreed that OSCEs should be part of PBL assessment.

CONCLUSIONS

Addition of an OSCE to written examinations was valued and provided a more comprehensive assessment of the PBL experience.

An electronic NAPLEX review program for longitudinal assessment of pharmacy students' knowledge

OBJECTIVES

To track pharmacy student knowledge over time using a proprietary software program in an accelerated program for curricular assessment.

METHODS

All students were required to complete a computerized comprehensive diagnostic examination 3 times during the doctor of pharmacy (PharmD) program: at the beginning of the second year, and near the end of the second and third years. The examination was comprised of 100 questions in 3 content areas: pharmacotherapy, preparation and dispensing of medications, and providing health care information. Within-subject differences in mean area and total percent scores were compared.

RESULTS

Based on 123 students' data, mean scores for pharmacotherapy and total percent scores for examination 1 were significantly different from examinations 2 and 3.

CONCLUSION

The computer-based comprehensive diagnostic examination shows promise for use as a component of a comprehensive assessment plan.

Impact of previous pharmacy work experience on pharmacy school academic performance

OBJECTIVES

To determine whether students' previous pharmacy-related work experience was associated with their pharmacy school performance (academic and clinical).

METHODS

The following measures of student academic performance were examined: pharmacy grade point average (GPA), scores on cumulative high-stakes examinations, and advanced pharmacy practice experience (APPE) grades. The quantity and type of pharmacy-related work experience each student performed prior to matriculation was solicited through a student survey instrument. Survey responses were correlated with academic measures, and demographic-based stratified analyses were conducted.

RESULTS

No significant difference in academic or clinical performance between those students with prior pharmacy experience and those without was identified. Subanalyses by work setting, position type, and substantial pharmacy work experience did not reveal any association with student performance. A relationship was found, however, between age and work experience, ie, older students tended to have more work experience than younger students.

CONCLUSIONS

Prior pharmacy work experience did not affect students' overall academic or clinical performance in pharmacy school. The lack of significant findings may have been due to the inherent practice limitations of nonpharmacist positions, changes in pharmacy education, and the limitations of survey responses.