An Objective Structured Clinical Examination to Assess Competency Acquired During an Introductory Pharmacy Practice Experience

PharmD students experiences on end year clerkship evaluation using objective structured clinical exam (OSCE) method at JSS College of Pharmacy, Mysuru, India

Objectives

To assess the PharmD student's experiences about end year clerkship evaluation conducted using Objective Structured Clinical Exam (OSCE) format at JSS College of Pharmacy, Mysuru, India.

Methods

The student's experiences were captured using a newly developed, 14-item, 5-point Likert's scale feedback form. The results were analyzed and presented descriptively. The Mann-Whitney U test was used to compare the Likert's scale responses between the sex, entry level for PharmD and performance in the end year exam, whereas the Spearman's rank correlation coefficient test was used to measure the strength of association between the Likert's' scale responses and these variables. A p value of <0.05 was considered statistically significant.

Results

Thirty-seven students of fifth year PharmD attended the end year clerkship evaluation and provided their feedback. Out of the 14-items, the most frequent response in eleven items was strongly agree and in three items it was agree. The Mann-Whitney U test revealed statistically significant differences between regular and postbaccalaureate students with respect to Likert's scale responses in all the domains (p < 0.02). The Spearman's rank correlation test revealed no association between the students' performance and their experiences with OSCE as an assessment tool for the end year clerkship exam.

Conclusions

The study results demonstrate that OSCE is an alternative and preferred method of evaluating the clinical skills and competencies of fifth year PharmD students in their end year clerkship exam in India.

Innovation

For the first time in India, the JSS College of Pharmacy, Mysuru, had successfully implemented the OSCE method for evaluating PharmD students' clerkship in their end year exam and had assessed their experiences about OSCE as an assessment tool.

A Cycle of Reinforcing Challenges and Ideas for Action in Experiential Settings

Evidence suggests that both pharmacy students and preceptors are struggling in the experiential setting. Underlying this phenomenon is a potential interconnected and cyclic set of behaviors being reinforced between students and preceptors. These behaviors can contribute to or are the result of higher levels of burnout and a decrease in the development of student clinical skills and subsequent performance on rotation. In this review, the authors investigate various challenges commonly encountered in the experiential environment. These challenges can range from an observed decrease in student engagement, motivation, and critical thinking skills to an increase in preceptor burnout and culture shifts in the clinical practice environments. These factors all ultimately impact patient care and overall student performance. For each challenge identified, strategies will be presented that can be implemented by students, preceptors, and pharmacy programs to break the cyclic pattern identified.

Entrustable professional activities-based objective structured clinical examinations in a pharmacy curriculum

BACKGROUND

The integration of Objective Structured Clinical Examinations (OSCEs) within the professional pharmacy program, contributes to assessing the readiness of pharmacy students for Advanced Pharmacy Practice Experiences (APPEs) and real-world practice.

METHODS

In a study conducted at an Accreditation Council for Pharmacy Education (ACPE)-accredited Doctor of Pharmacy professional degree program, 69 students in their second professional year (P2) were engaged in OSCEs. These comprised 3 stations: best possible medication history, patient education, and healthcare provider communication. These stations were aligned with Entrustable Professional Activities (EPAs) and Ability Statements (AS). The assessment aimed to evaluate pharmacy students' competencies in key areas such as ethical and legal behaviors, general communication skills, and interprofessional collaboration.

RESULTS

The formulation of the OSCE stations highlighted the importance of aligning the learning objectives of the different stations with EPAs and AS. The evaluation of students' ethical and legal behaviors, the interprofessional general communication, and collaboration showed average scores of 82.6%, 88.3%, 89.3%, respectively. Student performance on communication-related statements exceeded 80% in all 3 stations. A significant difference (p < 0.0001) was found between the scores of the observer and the SP evaluator in stations 1 and 2 while comparable results (p = 0.426) were shown between the observer and the HCP evaluator in station 3. Additionally, a discrepancy among the observers' assessments was detected across the 3 stations. The study shed light on challenges encountered during OSCEs implementation, including faculty involvement, resource constraints, and the necessity for consistent evaluation criteria.

CONCLUSIONS

This study highlights the importance of refining OSCEs to align with EPAs and AS, ensuring a reliable assessment of pharmacy students' clinical competencies and their preparedness for professional practice. It emphasizes the ongoing efforts needed to enhance the structure, content, and delivery of OSCEs in pharmacy education. The findings serve as a catalyst for addressing identified challenges and advancing the effectiveness of OSCEs in accurately evaluating students' clinical readiness.

Shifting to Authentic Assessments? A Systematic Review of Student Perceptions of High-Fidelity Assessments in Pharmacy

OBJECTIVE

Despite becoming increasingly popular, there is no comprehensive review on high-fidelity assessments in pharmacy education that has a focus on the perceptions and experiences of students. This systematic review investigates the student acceptability of high-fidelity simulation for summative assessments in pharmacy education and provides recommendations regarding high-fidelity simulation practices.

FINDINGS

The search yielded 37 studies in total. The articles fell into 3 distinct categories: objective structured clinical examinations (N = 25); face-to-face simulation assessments (N = 9), and augmented reality assessments (N = 3). Most high-fidelity assessments were well received with most students agreeing they had a vital role in assessing the application of clinical knowledge, even though they are stressful. Students prefer high-fidelity assessments to be face-to-face rather than online and also prefer when simulated patients whom they are unfamiliar with are used. Students also expressed a need to be well-prepared for the assessment regarding the logistics of the exam and technology use.

SUMMARY

High-fidelity simulation is likely to be increasingly important in the assessment of the knowledge and skills of pharmacy students and student perception is an important factor to consider when developing such assessments. Reducing stress associated with high-fidelity assessments could include familiarizing students with task logistics or technology used prior to the assessment, using external simulated patients, and having face-to-face assessments and practice sessions.

A Comparison of the Checklist Scoring Systems, Global Rating Systems, and Borderline Regression Method for an Objective Structured Clinical Examination for a Small Cohort in a Saudi Medical School

BACKGROUND

This study aims to compare the effectiveness of using the checklist and global rating scores to evaluate the clinical competency of medical students in Objective Structured Clinical Examinations (OSCEs). Additionally, the study assesses the appropriateness of using the borderline regression method to set standards for small-scale OSCE exams and determines if the estimated passing marks differ significantly from the university's prefixed passing score of 70%. The study also examines whether the university should utilize the borderline regression method to determine passing scores for each OSCE exam instead of a set passing score.

METHODS

The study analyzed medical students' grades in 11 OSCE exams in the 2022-2023 academic year at Alfaisal University, Riyadh, Saudi Arabia. Students received family medicine clerkship rotations, and after each rotation, they took an OSCE exam consisting of three stations that family medicine consultants graded. The exam included a checklist of 30 tasks and a five-level global rank scale. The study collected all the checklist marks and global rank grades and analyzed them using IBM® Statistical Package for Social Sciences (SPSS® Statistics) software. The statistical tests used were descriptive statistics, the T-test, chi-square tests, Fisher's exact test, and Pearson correlation.

RESULTS

The study showed that students were more likely to pass when using the global rating system than the checklist scoring system. Additionally, students had a significantly lower passing rate when using the higher cut-off passing score estimated using the borderline regression method compared to the pre-set passing score of 70% established by the university (with a p-value of 0.00).

CONCLUSION

Each scoring system has advantages and disadvantages, but they complement each other. Combining scoring systems can produce a more comprehensive and precise evaluation of a candidate's performance. The study also emphasizes the importance of carefully selecting and validating cut-off points in OSCE exams to ensure fairness and consistency in assessment.

Development, Validation, and Reliability of a P1 Objective Structured Clinical Examination Assessing the National EPAs

OBJECTIVE

To document the performance of first-year pharmacy students on a revised objective structured clinical examination (OSCE) based on national entrustable professional activities, identify risk factors for poor performance, and assess its validity and reliability.

METHODS

A working group developed the OSCE to verify students' progress toward readiness for advanced pharmacy practice experiences at the L1 level of entrustment (ready for thoughtful observation) on the national entrustable professional activities, with stations cross-mapped to the Accreditation Council for Pharmacy Education educational outcomes. Baseline characteristics and academic performance were used to investigate risk factors for poor performance and validity, respectively, by comparing students who were successful on the first attempt with those who were not. Reliability was evaluated using re-grading by a blinded, independent grader, and analyzed using Cohen's kappa.

RESULTS

A total of 65 students completed the OSCE. Of these, 33 (50.8%) successfully completed all stations on first attempt, and 32 (49.2%) had to re-attempt at least 1 station. Successful students had higher Health Sciences Reasoning Test scores (mean difference 5, 95% CI 2-9). First professional year grade point average was higher for students who passed all stations on first attempt (mean difference 0.4 on a 4-point scale, 95% CI 0.1-0.7). When evaluated in a multiple logistic regression, no differences were statistically significant between groups. Most kappa values were above 0.4 (range 0.404-0.708), suggesting moderate to substantial reliability.

CONCLUSION

Though predictors of poor performance were not identified when accounting for covariates, the OSCE was found to have good validity and reliability.

Objective structured clinical examination as a competency assessment tool of students' readiness for advanced pharmacy practice experiences in South Korea: a pilot study

BACKGROUND

The assessment of pharmacy students' readiness to begin the education of an advanced pharmacy practice experience (APPE) in clinical pharmacy settings continues to gain increasing attention. This study aimed to develop an objective structured clinical examination (OSCE) in the core domains acquired through an introductory pharmacy practice experience (IPPE), for evaluating its appropriateness as a tool of assessing clinical pharmacist competency for APPEs in Korean pharmacy students throughout a pilot study.

METHODS

OSCE's core competency domains and case scenarios were developed through a literature review, ideation by researchers, and external experts' consensus by a Delphi method. A prospective single-arm pilot test was conducted to implement the OSCE for Korean pharmacy students who completed a 60-h course of in-class simulation IPPE. Their competencies were assessed by four assessors in each OSCE station with a pass-fail grading system accompanied by a scoring rubric.

RESULTS

OSCE competency areas including patient counseling, provision of drug information, over-the-counter (OTC) counseling, and pharmaceutical care services were developed with four interactive and one non-interactive cases. Twenty pharmacy students participated in the OSCE pilot test, and their competencies were evaluated by 20 assessors. The performance rate was the lowest in the area of patient counseling for a respiratory inhaler (32.1%) and the highest (79.7%) in OTC counseling for constipation. The students had an average performance rate of 60.4% in their communication skills. Most participants agreed on the appropriateness, necessity, and effectiveness of the OSCE in evaluating pharmacy students' clinical performance and communication skills.

CONCLUSIONS

The OSCE model can be used to assess pharmacy students' readiness for off-campus clinical pharmacy practice experience. Our pilot study suggests the necessity of conducting an OSCE domain-based adjustment of difficulty levels, and strengthening simulation-based IPPE education.

Implementation of a patient-teaching workshop to improve pharmacy students' competencies in patient-centered communication: a case report

BACKGROUND

The pharmacist-patient relationship has evolved over recent decades and the development of clinical pharmacy requires pharmacists to take patient-centered responsibilities. This requires a specific set of skills, such as patient-centered communication. Evaluation of students' competencies in patient-centered communication is challenging in academic settings and complementary assessment methods may be designed in order to overcome the limits of traditional preceptors' ratings or objective structured clinical examination (OSCE). There is increasing interest in a more active patient role in healthcare professional education and there are very few reports about patient-led education in pharmacies. Thus, the objective of this work was to implement a patient-teaching workshop and to assess its impact on pharmacy students' competencies in patient-centered communication.

METHODS

The workshop was developed in collaboration between four patients, a senior clinical pharmacist and a lecturer in education sciences and implemented in the hospital pharmacy residency program. The main course objective was acquiring the three competencies of the Calgary-Cambridge guide to the medical interview: (i) building a relationship, (ii) conducting structured interview and (iii) gathering information. The learning process integrated: working on participants' perception of pharmacists-patient communication, a first simulated interview, didactic learning and a second simulated interview. After simulated interviews, patients and peer residents assessed learner's performance with a competency chart and provided individual feedback. Assessment methods included comparisons between the first and second interview scores and an anonymous post-course survey.

RESULTS

Forty-seven residents and 19 patient teachers attended the session. Competency scores were higher after the second interview in all three competencies as rated by both patients (+ 25%) and peer residents (+ 29%). Residents expressed a high satisfaction and reported learning about conducting interviews and soft skills contributing to the development of a relationship with patients. "The involvement of patients" was expressed as most appreciated in the majority of the evaluation charts (87%) and the residents valued the importance of collaborative and interprofessional learning during the workshop. Three themes emerged: (1) patients' expertise, (2) reliability and (3) relationship, which underlined that the students estimated the patients were credible sources of information in this pedagogical context.

CONCLUSION

This patient-teaching approach improved patient-centered competencies of pharmacy residents and promoted partnership between patients and pharmacy students.

The impact of assessment and feedback practice on the student learning experiences in higher education

Sound knowledge of the learning process and connectedness of assessment in higher education is imperative. Assessment is essential for both learners and teachers to ensure, and to provide evidence, that learning outcomes/objectives against set competency levels have been achieved for the chosen programme of the study. However, many coming into higher education are often unaware of the rules of the game. In this perspective, we will highlight the reasons why assessment is important, how assessment and feedback affect students' learning process and why the process of assessment can often affect students' mental well-being. We will appraise the different methods of assessment with a specific example (Objective Structured Clinical Examination, OSCE) and highlight why it is important that we adopt a holistic approach towards fostering assessment know-how and student well-being.

The influence of OSPE and PBL on competency-based pharmacy student self-assessment

BACKGROUND

In addition to vocational education and experience, the pharmaceutical profession's proper pursuit requires acquiring and continuously improving professional competencies. In recent years, the need has been increasingly highlighted for developing a medical education system based on helping students develop their competencies. It is necessary to adapt the tools and methods for assessing competencies during formal education. It will enable students to know the directions of further personal or professional development.

OBJECTIVE

The study aimed to compare pharmacy students' self-assessment outcomes before and after the Objective Structured Practical Examination (OSPE), which finished the Pharmaceutical Care course (PCc). The study's purpose was also to compare the outcomes of the self-assessment of competencies between the students of two academic years for whom classes on the PCc were provided by different methods.

METHODS

The study was conducted over two academic years (2018/2019 and 2019/2020) among 5th-year students enrolled at the Faculty of Pharmacy of the Jagiellonian University Medical College (JUMC) at the end of the Pharmaceutical Care course. Different teaching methods were used in the delivery of the course in these academic years. The students self-assessed their competencies using a questionnaire consisting of a list of personal and patient care competencies. The students completed the questionnaire before and after the OSPE, which followed the completion of the PCc.

RESULTS

Students' professional competencies as self-assessed after the exam were higher than those assessed before the exam. Differences were observed in both personal and patient care competencies. Students taking the course in the 2019/2020 academic year set their pre-OSPE competencies higher than students taking the PCc in 2018/2019.

CONCLUSION

The self-assessment scores increased for most competencies included in the study following the OSPE. This may suggest that taking part in the exam, involvement in patient's case simulations, and self-assessment of performance at individual stages of the exam contributed to increased subjective assessment of professional competencies.

Use of an objective structured clinical examination (OSCE) to assess intern performance in an advanced pharmacy practice experiences (APPE) Ambulatory Care rotation

Purpose

Intern assessment during advanced pharmacy practice experiences (APPEs) are generally based upon each individual preceptor's perceptions without an objective measurement of intern understanding and performance. Therefore, we sought to determine whether a pre- and post-OSCE could be used to confirm that interns achieved the goals and objectives of the Ambulatory Care rotation. The aims of this study were to determine whether a pre-rotation OSCE can help pharmacy interns self-assess their clinical strengths and weaknesses and assess whether their knowledge and skills improved after completing a post-rotation OSCE.

Methods

Pharmacy interns undergoing APPE Ambulatory Care rotations from September 2018 to March 2020 participated in a pre- and post-rotation OSCE to assess their knowledge of various chronic disease states. Interns completed pre- and post-OSCE surveys to assess their perceptions about their knowledge and the OSCE experience.

Results

Pharmacy intern knowledge about diabetes, hypertension, dyslipidemia, and atrial fibrillation significantly improved post-OSCE compared to their pre-OSCE scores (p < 0.001). The mean post-OSCE scores for diabetes (p < 0.001), dyslipidemia (P = 0.046), anticoagulation (P = 0.006), and the overall mean post-OSCE scores (P = 0.005) were significantly higher compared to interns' pre-OSCE scores. Students believed that the post-OSCE significantly highlighted their strengths and weaknesses in skills and knowledge compared to the pre-OSCE (P = 0.008).

Conclusion

Pre- and post-APPE OSCE assessments are important tools that can provide interns and preceptors with objective evaluations of student performance. OSCEs can either be used as an alternative to perception-based assessments or integrated into existing preceptor evaluations. Furthermore, OSCEs can help preceptors identify areas that require more emphasis in their rotations.

Decision-Making and Relationship Competence When Reporting Suspected Physical Abuse and Child Neglect: An Objective Structured Clinical Evaluation

Mandatory reporting of suspected child abuse and neglect highlights the challenges between the ethical and legal obligations of social workers and the need to maintain the therapeutic relationship with the client. The ability to bridge this tension is paramount to ensure continued psychosocial treatment and the well-being of children. This paper discusses a study to determine the decision-making factors of social work students and practitioners when facing a suspicion of child abuse and neglect, how they justify their decision to report or not report to child protection services, and the current and future relationship repair strategies used with simulated clients during an objective structured clinical evaluation (OSCE). Nineteen BSW, MSW, and experienced practitioners (N = 19) underwent an OSCE with one of two child maltreatment vignettes, physical abuse or neglect. Fisher's exact test was used to examine participants' historical and current reporting behaviors. Independent samples T-tests, Cohen's D, and qualitative content analysis was used to examine participants' decision making and relationship repair strategies when faced with suspected child abuse and neglect. Results showed that six participants discussed the duty to report during the OSCE while 13 participants did not. Participants' who discussed and did not discuss the duty to report during the OSCE articulated clear reasons for their decision and identified relationship repair strategies in working with the client. A sub-group of participants who identified the child maltreatment but did not discuss the duty to report, provided more tentative and complex reasons for their inaction and next steps in working with the client. All participants demonstrated a degree of competence and critical reflection in the OSCE, with integration for future learning. These findings are discussed and implications for future practice are offered.