Assessment of Learner Metacognition in a Professional Pharmacy Elective Course

Competency in medical training: current concepts, assessment modalities, and practical challenges

In the past two decades, competency-based medical education (CBME) has rapidly become the cornerstone of medical training and accreditation programmes worldwide. It has increasingly replaced traditional time-based educational approaches which were often rigid, fragmented, and overly emphasized clinical content knowledge over practical skillsets and attitudes. CBME adoption was in the hope of better preparing medical graduates for the demands and responsibilities of real-world clinical practice. For all the supposed merits of CBME, there hitherto remains difficulties in arriving at comprehensive and practical 'competency' definitions, and actual challenges with implementation of clinical competency assessment modalities pertaining to construct validity, reliability, and applicability with the use and interpretation of evaluation metrics. Therefore, in this article, we describe the various conceptualizations of 'competency' in medical education literature and attempt to refine its usage in practice to meet the evolving needs and expectations of healthcare stakeholders, as well as incorporate emerging concepts in the medical education discourse. We herein propose that clinical 'competencies' should be defined as multi-domain clinical expertise, comprising medical knowledge, skills, attitudes and metacognitive capabilities that reflects the prevailing needs of healthcare stakeholders, and is inferred from performance evaluations of medical trainees. In order to attain 'competence', there must then be a process of integrating multi-domain competencies into meaningful professional identity formation that is commensurate with the context and stage of medical training. In addition, we review the current competency assessment modalities, including common pitfalls with their usage, and sought to provide practical strategies to mitigate the identified challenges.

'Insight' in medical training: what, why, and how?

The term 'insight' is generically defined in English language as the ability to perceive deeper truths about people and situations. In clinical practice, patient insight is known to have important implications in treatment compliance and clinical outcomes, and can be assessed clinically by looking for the presence of illness awareness, correct attribution of symptoms to underlying condition, and acceptance of treatment. In this article, we suggest that cultivating insight is actually a highly important, yet often overlooked, component of medical training, which may explain why some consistently learn well, communicate effectively, and quickly attain clinical competency, while others struggle throughout their clinical training and may even be difficult to remediate. We herein define 'insight' in the context of medical training as having an astute perception of personal cognitive processes, motivations, emotions, and ability (strengths, weaknesses, and limitations) that should drive self-improvement and effective behavioural regulation. We then describe the utility of cultivating 'insight' in medical training through three lenses of (i) promoting self-regulated, lifelong clinical learning, (ii) improving clinical competencies and person-centred care, and (iii) enhancing physician mental health and well-being. In addition, we review educational pedagogies that are helpful to create a medical eco-system that promotes the cultivation of insight among its trainees and practitioners. Finally, we highlight several tell-tale signs of poor insight and discuss psychological and non-psychological interventions that may help those severely lacking in insight to become more amenable to change and remediation.

Examining Competency-Based Education Through the Lens of Implementation Science: A Scoping Review

OBJECTIVES

This study aimed to identify evidence for the implementation and assessment of competency-based education (CBE) in health professions curricula using an implementation science framework.

FINDINGS

Using the PRISMA framework, a systematic review of the literature applying a prespecified and piloted search strategy from 2017 to the present in PubMed and CINAHL was performed. References identified from the search strategy were imported into Covidence for title and abstract screening and full-text review by 2 researchers. A third researcher resolved discrepancies. Data were extracted and synthesized to identify key elements from the article related to implementation science, with a quality appraisal. A total of 25 studies out of 304 initially identified records were included. The studies covered a broad range of health professions and countries. Key findings were limited use of implementation science elements, including variability in CBE implementation, limited fidelity assessment, and partial examination of the process continuum. Programs with a more robust implementation approach have a team-based strategy to lead, implement, and support CBE. Motivation and training of faculty are also key components of successful CBE implementation.

SUMMARY

Competency-based education is implemented differently across institutions, with variation among programs in their choice of elements of implementation science used. Further research is needed to examine CBE from an implementation science perspective and address remaining questions.

Pre- and post-examination reflections of first-year medical students in an integrated medical anatomy course

Due to the rigor and pace of undergraduate medical anatomy courses, it is not uncommon for students to struggle and fail initially. However, repetition of coursework places an additional burden on the student, instructor, and institution. The purpose of this study was to compare the exam preparation strategies of repeating and non-repeating students to identify areas where struggling students can be supported prior to course failure. As part of their integrated anatomy course, first-year medical students at Indiana University completed a metacognitive Practice-Based Learning and Improvement (PBLI) assignment prior to and after their first exam. In the PBLIs, students were asked to reflect on their exam preparation strategies, confidence, and satisfaction, as well as their predicted and actual exam performance. PBLI responses from non-repeating and repeating students were then analyzed quantitatively and qualitatively. A total of 1802 medical students were included in this study, including 1751 non-repeating and 51 repeating students. Based on their PBLI responses, non-repeating students were appropriately confident, somewhat satisfied, and more accurate when predicting their exam performance. Repeating students were overconfident, dissatisfied, and inaccurate when predicting their first exam performance on their initial, unsuccessful attempt but were more successful on their second, repeat attempt. Qualitative analysis revealed that repeating students aimed to improve their studying by modifying their existing study strategies and managing their time more effectively. In conjunction with other known risk factors, these insights into repeater and non-repeater exam preparation practices can help anatomy educators better identify and support potential struggling students.

Metacognition in oral health education: A pedagogy worthy of further exploration

AIM

This study aimed to investigate the perceptions of dental students and teachers about introducing metacognition pedagogy within an established clinical professional curriculum to provide primary data informing its feasibility.

METHODOLOGY AND METHOD

A qualitative study using phenomenography methodology was undertaken as part of a master's dissertation. Semi-structured interviews were conducted on 16 participants which included 9 clinical teachers and 7 dental students.

FINDINGS

Metacognition pedagogy was positively perceived by most of the participants as being beneficial to students' learning in oral health education. A few reported some negativity. All participants identified some challenges to be addressed if a metacognition pedagogy was to be implemented in the undergraduate curriculum.

CONCLUSION

According to the perceptions of most participants in the study, metacognition emerged as a potential factor in improving student learning and exam performance, and facilitating the development of critical thinking, professionalism, and clinical skills. In the context of rigorous, demanding, and challenging courses, and recognising the complexities and uncertainties inherent in health professional working environments, metacognition emerges as a valuable tool, fostering self-awareness, regulation, and adaptability. Ultimately, metacognition has the capability to shape more adept learners and clinicians, yielding benefits for students, teachers, and patients alike.

Promoting metacognition in an allied health anatomy course

Metacognition, the ability to self-regulate one's learning and performance, has been shown to improve student outcomes. Anatomy is recognized as one of the toughest courses in allied health curricula, and students could benefit from metacognitive activities. The purpose of this study was to explore the changes in metacognition of allied health students in an anatomy course and identify which groups need support with this skill. First-year physician assistant (MPAS), physical therapy (DPT), and occupational therapy (OTD) students (n = 129) were invited to participate. At the beginning and end of the course, students completed a questionnaire including the metacognitive awareness inventory (MAI) that assesses metacognition. Students were also asked to reflect on their examination performances using a modified Likert scale and participated in reflective discussion boards to encourage development of metacognitive skills, which were thematically analyzed. Paired metacognition scores had increased significantly by the end of the course. However, middle-performers anticipated high grades and were less satisfied with their grade, indicating a disconnect in their metacognition compared to high- and low-performers. Students' receptiveness to modifying study strategies to improve performance declined throughout the course; by mid-way through, they relied more on existing strategies. Increasing time constraints were frequently cited as a major factor when considering study strategies and modification of such strategies. To maximize the effectiveness of metacognitive activities, they should be positioned early in the course when students are most receptive. In addition, middle performers may benefit from additional support to improve metacognition.

Self-Assessment of Therapeutic Decision-Making Skills in Pharmacy Students

Objective. The ability of pharmacy students to self-regulate and reflect on knowledge and skills is important as proficient self-assessment skills guide learning strategies and prompt students to bridge their knowledge gaps. The objectives for this study were to determine how well third-year pharmacy students self-assess, explore the rationales behind their self-assessments and determine whether there is a correlation between self-assessment accuracy and academic performance.Methods. A quasi-experimental one-group pre-/post-test design was conducted with third-year pharmacy students. Examiner grades, student self-assessment grades, comparative reports, and end-of-semester grades were collected. Students were categorized into tertiles based on academic performance for data analysis. Paired t tests, Pearson r and percentage agreements were conducted to investigate self-assessment accuracy. Correlational statistical tests were implemented to examine the relationships between self-assessment accuracy and academic performance.Results. One hundred sixty-two third-year pharmacy students were included. On average, students demonstrated poor self-evaluation skills and underestimated themselves by 4.9%. Lower performing students were generally overconfident in evaluating their performance. There was no significant correlation between students' self-assessment accuracy and academic performance on the subsequent end-of-semester examination questions.Conclusion. Overall, students tended to underestimate their academic performance. Further research on self-assessment is needed to better understand how students think about their performance, which may help to improve education methods, such as inclusion of reflective practices after case-based activities.

Nailed it! Doctor of pharmacy learners' self-awareness of performance on authentic assessments

INTRODUCTION

The primary objective of this pilot study was to determine the reliability and validity of an objective structured clinical examination (OSCE) self-awareness survey. A secondary objective was to assess learner self-awareness in OSCE performance.

METHODS

A retrospective review of OSCE data from a single cohort in a three-year accelerated doctor of pharmacy program occurred. The cohort completed four OSCEs with a total of 23 discrete cases. At the end of each case, before receiving feedback, learners completed an evaluation of their skills, communication, overall performance, and identified if they believed the patient/provider would return ("nailed or failed"). These self-evaluations were then compared with respective learner performance scores for each case. Content validity was assessed for the self-awareness survey via the Lawshe method. Reliability of the OSCE cases and self-awareness survey were conducted utilizing Cronbach's alpha and the Spearman-Brown formula, respectively.

RESULTS

The OSCE self-awareness survey was found to be face valid and reliable for the overall survey. The subsections of return to pharmacist and overall learner performance were also found to be valid and reliable. A statistical relationship was found between learner self-assessment of pass/fail and their actual grade for overall performance and return to pharmacist.

CONCLUSIONS

Results from this study may guide remediation plans related to OSCE performance and development of other self-awareness activities in practice settings. Further studies are needed to evaluate the broader application and use of self-awareness tools.

Knowledge and Behavior in Rational Drug Use Among College Students in Zunyi City

Objective

To investigate the current status of knowledge and behavior about rational drug use in college students in different colleges and universities in a city in southwest China, providing reference for students’ education of rational drug use in colleges and universities.

Methods

A questionnaire survey on knowledge and behavior in rational drug use was carried out on the students recruited by occasional sampling method in 6 colleges and universities in Zunyi, China. Statistical analyses on demographic information and answers to questionnaire questions were carried out with SPSS 18.0.

Results

A total of 865 valid questionnaires were recovered from 923 questionnaires sent out. Some knowledge and behaviors of the students on drug use were irrational. There was statistically significant difference in some specific questions of the knowledge in rational drug use between medical and non-medical students (P<0.05); the average score of rational drug use behavior of medical students was lower than that of non-medical students (P<0.05); the average score of the rational drug use behavior of female students was lower than that of male (P<0.05); the students’ major types had significantly different influence on their behavior in rational drug use (P<0.05). The school-carried pharmacy education can effectively improve students’ rational medication; the majority of college students believed that it is necessary to popularize the knowledge of rational drug use on campus; and students’ favorite way to acquire knowledge about rational drug use was to attend related lectures or elective courses.

Conclusion

The knowledge and behavior of rational drug use among college students need to be improved. Professional medical education may exert a positive impact on rational drug use among college students. Thus, it is necessary to popularize the knowledge of rational drug use among college students, especially in non-medical colleges and universities.

A Concept Mapping Activity to Enhance Pharmacy Students' Metacognition and Comprehension of Fundamental Disease State Knowledge

Objective. To examine the impact of pre-class concept mapping activities on pharmacy students' ability to self-assess their degree of foundational disease state knowledge and predict their pre-class quiz performance.Methods. Second year pharmacy students in a problem-based learning course were responsible for self-directed learning of foundational knowledge for 14 disease states. After completing their independent pre-class reading, students worked in groups to create concept maps for which feedback was provided for four laboratory sessions, worked in groups to create concept maps but received no formal feedback for three laboratory sessions, and did not engage in any formal group activity for seven laboratory sessions. The day following each session, prior to the formal in-class discussion, students were asked to predict the number of questions they could answer correctly on a quiz covering foundational knowledge and then completed the quiz. Quiz performance was compared based on the three conditions, and bias and absolute bias were calculated to evaluate students' metacognitive skills.Results. There was no difference in pharmacy students' metacognition based on the conditions, as reflected by inaccuracy between predicted and actual quiz scores. However, when students had engaged in concept mapping the previous day, their quiz performance was significantly higher than when they had not.Conclusion. Concept mapping did not improve pharmacy students' metacognitive skills but did have a small effect on their quiz performance. More research is needed to tease apart the roles of concept mapping, group activity, and feedback in altering pharmacy students' quiz performance and metacognitive skills.

The use of a metacognition and contemplation intervention in improving student abilities to recognize order entry errors

BACKGROUND AND PURPOSE

To describe a pilot study testing a metacognition and contemplative pedagogy intervention designed to improve student abilities to identify errors on orders entered into a computerized provider order entry (CPOE) system. Educational activity and setting: Student teams worked up five patient cases and entered new orders into a CPOE system. All orders entered by teams were analyzed for errors and faculty members identified fifteen orders representing the top errors and selected these for the pre-exercise, in-class contemplation activity, and post-exercise. Course instructors instructed students to identify all errors on these orders. Students completed a pre/post-survey rating their ability to enter new orders. Two weeks later, students worked up a new patient case and entered orders into the CPOE. These orders were graded and scores were compared to the 2017 cohort to determine any differences in error rates.

FINDINGS

Study participants included 205 students The average score of correctly identified errors increased from 22% on the pre-exercise to 62.8% on the post-exercise. On the survey of their skills, students rated their ability to "accurately enter medication orders into the [electronic health record] EHR" more highly after the intervention. The 2017 cohort made 73 errors on 170 orders compared to 45 errors on 165 orders by the 2018 cohort.

SUMMARY

Based on improvements in student ability to identify order entry errors and overall positive student feedback, this strategy may be adopted by other educators looking to improve student identification of errors on orders entered into a CPOE system.

Testing of a Program to Automatically Analyze Students' Concept Maps

Concept maps are graphical representations of how various concepts relate to one another. Assessment of concept maps developed by students in the pharmacy curriculum helps to evaluate student understanding of course material. However, providing feedback on concept maps can be time-consuming and often requires the grader to be a content expert. The purpose of this study was to develop and validate a software program to provide students with feedback on their concept map performance. Student maps for four different disease states were compared against expert concept maps. The analysis of the program compared favorably to a manual assessment of student maps for the maps' complexity and content but did not correlate for their organization. The value of using a software program to quickly and efficiently analyze concept maps is discussed.

Analysis of a novel enrichment strategy for an integrated medicinal chemistry and pharmacology course

INTRODUCTION

Study and application of integrated medicinal chemistry and pharmacology content affords opportunities for students to discuss and develop life-long learning skills.

METHODS

Five thematic enrichment activities were developed (problem solving, metacognition, reading comprehension, case-based problem solving, and structure-based therapeutic evaluation), each containing a self-study and live session featuring unit-specific content. Voluntary, longitudinal sessions were administered to 139 s professional year pharmacy students at the end of each unit of the first course of an integrated pharmacology and medicinal chemistry sequence (academic quarter system). Students provided five-point Likert-item feedback at the beginning of the course, after the first activity, and at course conclusion. Survey questions were linked to self-assessment domains of metacognition, content relevance, confidence, and affective response to content.

RESULTS

Survey responses indicated significant improvement in initial confidence (3.7 [1.1] to 4.2 [1.1]) and metacognition (3.2 [1] to 3.8 [1.1]) domains at course conclusion and significant, sustained improvement in affective domain following the first session (3.5 [1.1] to 4.1 [1.2] to 4.2 [1.2]). Perceived relevance of content did not change significantly (4.3 [1] to 4 [1.1] to 4.1 [1.2]).

CONCLUSIONS

Survey results were consistent with the notion that targeted learning interventions have a significant impact on content perception, which may be especially important for disciplines perceived by students as challenging. Introduction of learning topics with concurrent application may positively influence affective response to learning, which may beneficially impact latent student confidence and self-awareness.

Longitudinal evaluation of pharmacy students' metacognition in interpretation of evidence-based medicine

BACKGROUND AND PURPOSE

Metacognition is one's ability to "think about thinking" and to understand what knowledge and skills one has. Pharmacy professionals are required to have high metacognitive skills to ensure they are aware of what information they know and when to utilize resources. This longitudinal evaluation of pharmacy students' metacognitive ability was conducted via analysis of students' knowledge and confidence in evidence-based medicine (EBM) skills.

EDUCATIONAL ACTIVITY AND SETTING

A survey was administered to the graduating classes of 2017 and 2018 four times, from the beginning of the second-year drug literature evaluation course (baseline) through graduation. The surveys included seven self-confidence questions for which a corresponding knowledge question was asked. Students' metacognition was evaluated on the basis of whether the level of confidence was a "match" or a "mismatch" to their corresponding level of knowledge of each question. The proportion of matches over time was determined.

FINDINGS

Fifty-eight students (22%) completed all four surveys. The mean proportion of matches at baseline was 40% and increased over time until graduation when the proportion of matches was 74%. The proportion of matches at baseline for all seven questions were ranged from 33% to 57%. An improvement in meta-cognition was seen in 5 out 7 questions.

SUMMARY

This evaluation provides evidence that as students learned material and revisited concepts, their confidence, knowledge, and potentially metacognition regarding EBM concepts increased. However, as 26% of items measured were not a match at graduation, interventions need to be designed and tested to improve student metacognition.

A Student-Led Elective Provides Quality Improvement Feedback for a Required Compounding Course

Objective. To implement an advanced elective compounding course where pharmacy students conduct investigations to improve compounding-related issues that were subsequently evaluated in a required compounding course. Methods. The elective compounding course required students to engage in self-directed learning, critical thinking, creation and evaluation of laboratory data, and self- and group reflection. Students researched and developed "solutions" to compounded preparation problems, and their solutions were tested in the next iteration of a required compounding course. For example, students in the elective course identified sources of potency variability in a ketoprofen Pluronic organogel (PLO) emulsion preparation. The students identified six variables and executed an investigative action plan. They considered all data collected and proposed a method to reduce potency variation. The recommended solution was implemented in the next offering of a required compounding course and the potency variability results were compared to the previous required course's results. Results. The mean ketoprofen PLO emulsion potency achieved in the required course prior to implementing the elective course recommendation was 129% (SD 21%), n=158. After the recommended change from elective course was implemented, the mean potency was 118% (SD 21%), n=131. Conclusion. The teaching methods and activities conducted in the elective course provided students with a deeper level of learning and understanding of compounding science, while providing practical experience in scientific research methodology. The course also provided a cyclic quality improvement feedback mechanism for the required course.

The impact to demographic and academic factors on metacognition and academic self-efficacy: A study on Iranian students in health sciences

BACKGROUND

Metacognition and academic self-efficacy are two emerging resources in the process of learning. Basic levels of metacognition and academic self-efficacy may differ and be influenced by demographic and academic factors. This study investigated impacts of demographic and academic factors on metacognition, metacognitive skills, and academic self-efficacy in health sciences students.

MATERIALS AND METHODS

A cross-sectional study was conducted among 404 participating students attending various fields of study in health sciences at Shahrekord University of Medical Sciences. Demographic and academic data were collected. Global metacognition, metacognitive skills, and academic self-efficacy scores were assessed using published or online questionnaires. Means were compared using Student's t-test, whereas intra- and intergroups' scores were compared using one-way ANOVA test.

RESULTS

Global metacognition and academic self-efficacy were not impacted by demographic students' status. The gender and age impacted both knowledge and control of process- planning (favoring male gender), as well knowledge and control-of-self (disfavoring 20-30 age class) of metacognitive skills, (P < 0.05). Academic status did not influence academic self-efficacy. The school type influenced the metacognitive skill component to knowledge and control of process-regulation (disfavoring nursing school) (P < 0.05). The academic discipline impacted the global metacognition (P < 0.05) and its knowledge and control of self-component (P < 0.01).

CONCLUSIONS

Demographic and academic status does impact metacognitive skills and global metacognition scores. Given the heterogeneous level to innate metacognitive skills, this study sheds lights on usefulness to screen learners' subgroups that require supplementary educational instructions to uniformly optimize metacognitive skills.

Changes in Pharmacy Students' Metacognition Through Self-Evaluation During Advanced Pharmacy Practice Experiences

Objective. To explore whether metacognition can be improved in Doctor of Pharmacy (PharmD) students through routine self-assessment over a year-long advanced pharmacy practice experience (APPE) sequence. Methods. Differences between self-assessment scores and preceptors' scores for three cohorts of pharmacy students between 2015 and 2018 were compared between the first, second, and third trimester to determine whether students more accurately evaluated their performance over time. The primary endpoint was change in the absolute difference between student and preceptor evaluation (rubric and composite scores) between trimesters. Results. Of 2577 student and preceptor evaluations eligible for inclusion, 1713 were completed, matched, and analyzed. Using the same rubric as preceptors, students overestimated their performance by an average of 16 points during the first trimester, followed by 14 and 12 points during the second and third trimester, respectively. This reflected a significant improvement over time. No significance difference was found between student and preceptor composite scores. Faculty preceptorship, students' pre-APPE grade point average, and type of APPE were not associated with any difference in rubric or composite scores. Conclusion. This analysis revealed that the difference between student self-evaluation grades and preceptor evaluation grades was greatest during the first trimester and significantly decreased in the second and third trimesters. This could reflect students' development of metacognitive processes over time. Metacognition is a vital skill for pharmacy students to learn, and opportunities to develop this skill should be incorporated throughout the pharmacy curricula.

The use of exam wrappers to promote metacognition

INTRODUCTION

Self-directed learning, self-awareness, and metacognition are becoming more heavily emphasized in healthcare education. Metacognitive activities may be used to promote self-awareness and development of effective self-directed learning strategies. Exam wrappers (wrappers) are a metacognitive tool utilized in other areas of higher education that offer students the ability to learn from exam performance by reviewing missed items in a reflective manner. Wrappers encourage structured reflection and planning for future learning.

METHODS

Pharmacy students enrolled in a pathophysiology course were provided structured exam reviews with an opportunity to utilize a wrapper. A baseline questionnaire was given two weeks before the first exam to assess baseline characteristics. Wrappers were provided before Exams 2, 3, 4, and 5, and the effects on exam performance were measured. Mixed effects modeling was utilized for final analysis of effect on student performance.

RESULTS

Exam wrapper vs. non-wrapper groups were self-selected, with 53 students participating in at least one wrapper and 35 never utilizing a wrapper. Mixed effects modeling analysis found a non-significant increase in average exam performance (p = 0.142).

CONCLUSION

Wrappers have potential as an additional metacognitive tool for exam review. Further studies are needed to evaluate the effectiveness and impact more broadly in pharmacy education.