Inter-rater Reliability of a Clinical Documentation Rubric Within Pharmacotherapy Problem-Based Learning Courses

Determination of Interrater Reliability of a Universal Evaluator Rubric to Assess Student Pharmacist Communication Skills

Objective. To evaluate the interrater reliability of a universal evaluator rubric used to assess student pharmacist communication skills during patient education sessions.Methods. Six US schools and colleges of pharmacy each submitted 10 student videos of a simulated community pharmacy patient education session and recruited two raters in each of the five rater groups (faculty, standardized patients, postgraduate year one residents, student pharmacists, and pharmacy preceptors). Raters used a rubric containing 20 items and a global assessment to evaluate student communication of 12 videos. Agreement was computed for individual items and overall rubric score within each rater group, and for each item across all rater groups. Average overall rubric agreement scores were compared between rater groups. Agreement coefficient scores were categorized as no to minimal, weak, moderate, strong, or almost perfect agreement.Results. Fifty-five raters representing five rater groups and six pharmacy schools evaluated student communication. Item agreement analysis for all raters revealed five items with no to minimal or weak agreement, 10 items with moderate agreement, one item with strong agreement, and five items with almost perfect agreement. Overall average agreement across all rater groups was 0.73 (95% CI, 0.66-0.81). The preceptor rater group exhibited the lowest agreement score of 0.68 (95% CI, 0.58-0.78), which significantly deviated from the overall average.Conclusion. While strong or almost perfect agreement scores were not observed for all rubric items, overall average interrater reliability results support the use of this rubric in a variety of raters to assess student pharmacist communication skills during patient education sessions.

Feedback for Learning in Pharmacy Education: A Scoping Review

Feedback is an effective pedagogy aimed to create cognitive dissonance and reinforce learning as a key component of clinical training programs. Pharmacy learners receive constant feedback. However, there is limited understanding of how feedback is utilized in pharmacy education. This scoping review sought to summarize the breadth and depth of the use of feedback within pharmacy education and identify areas for future research. PubMed, Embase, Scopus, and Web of Science were searched for English articles since January 2000 to identify studies related to feedback in pharmacy education. Sixty-four articles were included for analysis, stratified by moderate and major theory talk, where moderate theory talk explicitly included feedback into study design and major theory talk included feedback into both study design and analysis. Feedback was provided in Bachelor (14%), Master (15.6%), Doctor of Pharmacy (67.2%) and post-graduate programs (4.7%) on a variety of curricular objectives including communication and patient work up in didactic, objective structured clinical examination (OSCE), and experiential settings, and career/interview preparation in the co-curriculum. Feedback comments were mostly written in didactic courses, and both written and verbal in OSCE, experiential, and co-curricular settings. The pharmacy education feedback literature lacks depth beyond student perceptions, especially with respect to assessing the effectiveness and quality of feedback for learning. While feedback has been utilized throughout pharmacy education across myriad outcomes, several areas for inquiry exist which can inform the design of faculty and preceptor development programs, ensuring provision of effective, quality feedback to pharmacy learners.