Grading trends and evaluation of student performance across advanced pharmacy practice experiences (APPE) in the Big Ten Academic Alliance (The GRAPPES study)

The Divergence between Self- and Preceptor-Assessments of Student Performance during Advanced Pharmacy Practice Experiences

(1) Objectives: A divergence in self- and preceptor-evaluations of clinical skills has been noted during Advanced Pharmacy Practice Experiences (APPEs). The goal of this study was to determine the domains of overestimation of clinical skills by students during their APPE rotations. (2) Methods: Preceptor-assigned grades for APPE rotations from 2017-2022 were analyzed to identify instances of letter grade B or lower. The self- and preceptor-evaluations of APPE rotation were compared to determine the domains of divergence in evaluation between students and preceptors. (3) Results: Between 2017 and 2022, 305 student APPE rotations were graded as B or lower (~14%) by the preceptors. A statistically significant difference was noted between self- and preceptor-assigned letter grades across all practice settings including ambulatory patient care, community pharmacy, general medicine patient care, hospital/health system pharmacy, and special population patient care APPE rotations. In addition, examining the self- and preceptor evaluation rubric for these rotations revealed a statistically significant overestimation of clinical skills by students in all 9 domains of APPE evaluation. Finally, the divergence in the rating of clinical skills between student- and preceptor evaluation was found to be highest in the domains of planning and follow-up of patient care, disease knowledge, and communication with patients. (4) Conclusions: Students who fail to exhibit exemplary practice readiness during APPEs tend to overestimate their clinical skills in all domains of APPE evaluation. The results from our study support the need for additional avenues to assist in the identification of deficits in student learning before APPEs to increase their self-awareness (metacognition).

Relationship between advanced pharmacy practice experience grading schemes and pharmacy residency match rates

DISCLAIMER

In an effort to expedite the publication of articles, AJHP is posting manuscripts online as soon as possible after acceptance. Accepted manuscripts have been peer-reviewed and copyedited, but are posted online before technical formatting and author proofing. These manuscripts are not the final version of record and will be replaced with the final article (formatted per AJHP style and proofed by the authors) at a later time.

PURPOSE

To determine the relationship of advanced pharmacy practice experience (APPE) grading schemes and other pharmacy program variables (ie, program age and funding) with pharmacy residency match rates.

SUMMARY

A 12-question survey was disseminated to experiential administrators of pharmacy programs in October 2018. Respondents identified their program's APPE grading scheme (pass/fail, letter grades, or other) and associated pros and cons. Responding programs were categorized by age and funding status. Survey responses were correlated with the American Society of Health-System Pharmacists residency match rates for 2016 through 2018. Data were analyzed using descriptive statistics and logistic regression models as well as by attributes via thematic analysis. Most pharmacy programs (62%) reported using letter grades for APPEs compared to pass/fail (30%) or other (8%) schemes. Pharmacy programs using pass/fail grading were more likely to have students match to postgraduate year 1 (PGY1) (P < 0.001) and postgraduate year 2 (PGY2) (P = 0.0074) residencies. Older pharmacy programs for each grading scheme were more likely to have higher match rates; however, for PGY1 match rates, older programs utilizing letter grades correlated to lower match rates than those utilizing pass/fail grading (P < 0.0001). Likewise, both public and private pharmacy programs using pass/fail grading had higher PGY1 match rates than those using letter grades (P = 0.0006 and P = 0.0014).

CONCLUSION

Pass/fail grading in APPEs does not hinder PGY1 or PGY2 residency placement compared to other grading schemes both overall and in combination with certain pharmacy program variables. Grading scheme strengths and weaknesses should be considered when deciding on optimal assessment strategies for APPEs and when evaluating candidates for residencies.

Use of an objective structured clinical examination (OSCE) to assess intern performance in an advanced pharmacy practice experiences (APPE) Ambulatory Care rotation

Purpose

Intern assessment during advanced pharmacy practice experiences (APPEs) are generally based upon each individual preceptor's perceptions without an objective measurement of intern understanding and performance. Therefore, we sought to determine whether a pre- and post-OSCE could be used to confirm that interns achieved the goals and objectives of the Ambulatory Care rotation. The aims of this study were to determine whether a pre-rotation OSCE can help pharmacy interns self-assess their clinical strengths and weaknesses and assess whether their knowledge and skills improved after completing a post-rotation OSCE.

Methods

Pharmacy interns undergoing APPE Ambulatory Care rotations from September 2018 to March 2020 participated in a pre- and post-rotation OSCE to assess their knowledge of various chronic disease states. Interns completed pre- and post-OSCE surveys to assess their perceptions about their knowledge and the OSCE experience.

Results

Pharmacy intern knowledge about diabetes, hypertension, dyslipidemia, and atrial fibrillation significantly improved post-OSCE compared to their pre-OSCE scores (p < 0.001). The mean post-OSCE scores for diabetes (p < 0.001), dyslipidemia (P = 0.046), anticoagulation (P = 0.006), and the overall mean post-OSCE scores (P = 0.005) were significantly higher compared to interns' pre-OSCE scores. Students believed that the post-OSCE significantly highlighted their strengths and weaknesses in skills and knowledge compared to the pre-OSCE (P = 0.008).

Conclusion

Pre- and post-APPE OSCE assessments are important tools that can provide interns and preceptors with objective evaluations of student performance. OSCEs can either be used as an alternative to perception-based assessments or integrated into existing preceptor evaluations. Furthermore, OSCEs can help preceptors identify areas that require more emphasis in their rotations.

Objective structured clinical examination vs traditional clinical examination to evaluate students' clinical competence: A systematic review of nursing faculty and students' perceptions and experiences

BACKGROUND

Assessment of clinical competence of nursing students is an essential requirement in professional nursing education. This article summarizes the current published evidence indicating the nursing faculty and students' perceptions and experiences on benefits of OSCE over TCE.

METHODS

A systematic review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) recommendations. Electronic databases including Scopus, Medline, Science Direct, CINAHL, EBSCO, PsychINFO, and Pubmed Central were used to identify relevant articles. The studies published between January 1, 2010 and December 31, 2020 and fourteen full-text articles that met all the inclusion criteria were included in the review.

RESULTS

The review identified five themes namely: a) Student's perception of OSCE; b) student's satisfaction regarding OSCE as an examination approach; c) students' perception of TCE as an examination approach; d) student's level of stress and anxiety towards OSCE vs TCE; and e) faculty member's perception and experience of OSCE.

CONCLUSION

We conclude that OSCE is a more credible assessment format to evaluate the clinical competence of undergraduate nursing students compared to the TCE method.

Predictors of Student Failure or Poor Performance on Advanced Pharmacy Practice Experiences

Objective. To determine factors predictive of student failure or poor performance on advanced pharmacy practice experiences (APPEs) at a single pharmacy program. Methods. This retrospective cohort evaluated students entering the Doctor of Pharmacy (PharmD) program from 2012-2014 at St. Louis College of Pharmacy. Students who received a grade of F for one or more APPEs (failure group) were compared to all other students (non-failure group). A secondary evaluation compared students with a C or F on one or more APPEs (poor performers) to all other students (non-poor performers). Data were collected on didactic and experiential performance, identifiable professionalism issues from introductory pharmacy practice experiences (IPPEs), and academic honor code violations. Univariable and multivariable logistic regressions were performed to determine factors associated with APPE failure and poor performance. Results. A total of 669 students were analyzed. Twenty-eight students (4.2%) failed one or more APPEs and 81 students (12.1%) were identified as poor performers (grade of C or F). For the primary outcome, professional grade point average (GPA) of less than 2.7, practicum failure, IPPE professionalism issue(s), and pharmacotherapy course failure were identified for inclusion in the multivariable analysis. The IPPE professionalism issue(s) (HR 4.8 [95% CI 1.9-12.4]) and pharmacotherapy course failure (HR 4.2 [95% CI, 1.6-11.1]) were associated with APPE failure on multivariable regression. On the secondary analysis, the same variables were identified for multivariable regression, with professional GPA of less than 2.7 (HR 2.7 [95% CI 1.5-5]), IPPE professionalism issue(s) (HR 3.9 [95% CI 2.2-6.9]), and pharmacotherapy course failure (HR 2.0 [95% CI 1.1-3.7]) associated with poor performance. Conclusion. Poor academic performance and/or identified unprofessional behavior while completing IPPEs are associated with APPE failure and poor performance. Interventions should be aimed at identifying at-risk students and addressing risk factors prior to APPEs.

An Objective Structured Clinical Examination to Assess Competency Acquired During an Introductory Pharmacy Practice Experience

Objective. To evaluate the use of an objective structured clinical examinations (OSCE) to assess clinical competency acquired during an off-campus introductory pharmacy practice experience (IPPE). Methods. Third-year pharmacy students completed an IPPE in transitions of care and completed 24 experiential contact hours at one of 17 practice sites. Students were assessed using two OSCEs, the first occurring prior to beginning an off-site IPPE (pre-experience OSCE) and the second occurring after completion of the off-site IPPE (post-experience OSCE). Each OSCE consisted of 10 stations and covered five graded competency domains. The primary outcome was the degree of change in student performance from the pre-experience OSCE to the post-experience OSCE. Secondary outcomes included changes in each graded domain, OSCE pass rate, and failure conversion rate. Results. Of 111 students, 109 completed both the pre- and post-experience OSCE. Significant improvements were observed in overall score and cohort pass rate. Overall scores improved from 80 for the pre-experience OSCE to 87 for the post-experience OSCE. The OSCE pass rate also improved from 47% to 84%. Conclusion. Although preceptor evaluations have traditionally served as the primary summative assessment for IPPE and APPE, this study indicates that OSCEs may be a reliable alternative to assess clinical competency acquired from off-site practice experiences.