Formative assessment in postgraduate medical education - Perceptions of students and teachers

Simulation as an effective means of preparing trainees for active participation in MDT meetings

Introduction

Cancer multi-disciplinary team (MDT) meetings are an important component of consultant workload, however previous literature has suggested trainees are not satisfied with their current curriculum in preparing for MDT working.

Methods

This educational pilot assessed whether multi-speciality simulated scenarios with pre-defined learning objectives, could prepare specialist registrars for interacting within an MDT. Participants completed pre- and post-questionnaires assessing a number of areas including: current experience of training, confidence presenting patients and whether the course would alter future practice.

Results

Trainee confidence increased significantly from a mean of 5 to 7 (mean to nearest whole number, p < 0.01). Trainees rated the session highly for utility and altering their future practice (mean scores of 9 for both respectively, out of 10).

Conclusion

Simulation has shown success in other multidisciplinary teaching, however to our knowledge there are no cancer specific training programmes. Our results highlight a potential gap in UK specialist training, and suggest simulation may be beneficial in preparing trainees to present in MDT meetings.

Introduction of a formative assessment tool in a post-graduate training program in India: a mixed methods evaluation

BACKGROUND

Our institution has longstanding post-graduate education and training partnership programs in Emergency Medicine (EM) across India. A programmatic challenge has been the integration and uptake of evidence-based medicine and lifelong learning concepts. Formative assessment (FA) is intended to enable learners to monitor learning, identify strengths and weaknesses, and target areas of growth. As part of a program improvement initiative, we introduced an online FA tool to existing summative assessments. This study investigates how the FA tool was used and perceived by trainees.

METHODS

246 trainees across 19 sites were given access to the FA tool. Usage metrics were monitored over 12 months. Semi-structured interviews were conducted in person with trainees using a purposive sampling methodology. A hybrid thematic analysis approach was used to determine themes. Interviews were coded independently by two blinded researchers using NVivo software. The study was deemed exempt by our institutional review board.

RESULTS

There was high variability in trainees' utilization of the FA tool. Trainees who used the FA tool more performed better on summative exams (r = 0.35, p < 0.001). Qualitative analysis revealed that trainees were motivated to learn for improved clinical knowledge and to be a good physician, not only passing exams. Benefits of the tool included the relationship to clinical practice and thorough explanation of answers, while disadvantages included topics unrelated to India.

CONCLUSION

The integration of a FA tool has provided positive outcomes for trainees in EM education programs in India. Lessons learned may apply globally to other contexts and programs.

Assessment and Evaluation of Cardiology Residency Training in Türkiye: A National Survey

BACKGROUND

Workplace-based assessment methods are essential in the assessment and evaluation of competency-based cardiology residency training. This study aims to determine the assessment and evaluation methods used in cardiology residency training in Türkiye and to reach the opinions of the institutions on the applicability of the workplace-based assessments.

METHODS

In this descriptive study, a Google Survey was sent to the heads/trainers of residency educational centers and their opinions about the currently used assessment and evaluation methods, applicability of cardiology competency exams, and the workplace-based assessments were asked.

RESULTS

Responses were received from 65 (76.5%) of 85 training centers. Of the centers, 89.2% reported using resident report cards, 78.5% case-based discussion, 78.5% direct observation of procedural skills, 69.2% multiple-choice questions, 60% traditional oral exams, and less commonly other exam types. About 74% of responders gave a positive opinion on the requirement of being successful in the Turkish Cardiology Competency knowledge exam before specialty. Case-based discussion was the most common work-place-based assessments that the centers think could be applied as suggested by the current literature. A common idea was the adaptation of workplace-based assessments based on international standards and our national norms. The trainers supported a nationwide examination for all training centers to ensure standardization.

CONCLUSION

In Türkiye, it was promising to see that the trainers are positive about the applicability of workplace-based assessments, but they commonly thought that the proposed workplace-based assessments should be adapted before nationwide applicability. Medical educators and field experts need to work together on this issue.

Assessment Practices of Learning Outcomes of Postgraduate Students in Biomedical and Pharmaceutical Sciences at College of Health Sciences at Addis Ababa University: Student and Faculty Perspectives

Introduction

Higher education institutions are under increasing pressure to respond to societal needs which has in turn led to changes in the type of knowledge, competencies, and skills required from learners. Assessment of student learning outcomes is the most powerful educational tool for guiding effective learning. In Ethiopia, studies are scarce on assessment practices of learning outcomes of postgraduate students in biomedical and pharmaceutical sciences.

Objective

This study investigated the assessment practices of learning outcomes of postgraduate students pursuing studies in biomedical and pharmaceutical sciences at the College of Health Sciences of Addis Ababa University.

Methods

A quantitative cross-sectional study was conducted using structured questionnaires administered to postgraduate students and teaching faculty members in 13 MSc programs in biomedical and pharmaceutical sciences at the College of Health Sciences of Addis Ababa University. About 300 postgraduate and teaching faculty members were recruited with purposive sampling. The data collected included assessment methods, types of test items, and student preferences on assessment formats. Data were analyzed using quantitative approaches, descriptive statistics, and parametric tests.

Results

The study indicated that several assessment strategies and test items were practiced without a significant difference across fields of study. Regular attendance, oral questioning, quiz, group and individual assignments, seminar presentations, mid-term tests, and final written examination were commonly practiced assessment formats, while short question and long question essays were the most commonly used test items. However, students were not commonly assessed for skills and attitude. The students indicated they mostly preferred short essay questions, followed by practical-based examinations, long essay questions, and oral examination. The study identified several challenges to continuous assessment.

Conclusion

Practice of assessing students' learning outcomes involves multiple methods focusing on assessing mainly knowledge; however, the assessment of skills appears inadequate, and several challenges appear to be hindering implementation of continuous assessment.

Effective Feedback Strategy for Formative Assessment in an Integrated Medical Neuroscience Course

Purpose

Despite the different benefits of formative assessments in an integrated medical curriculum, the effective strategies to provide feedback to medical students to benefit from the different merits of formative assessment are not fully understood. This study aims to determine the effect of different strategies of formative feedback on students' outcomes in a medical neuroscience course.

Method

We compared medical students' performance in summative examinations in the academic year that formative feedback was provided using in-person discussion and compared such performances with the academic year when the feedback was provided by written rationales or a combination of written rationales and in-person discussion. We also surveyed medical students' preferences for whether written or in-person formative feedback is a better strategy to provide feedback at the end of each course.

Results

ANOVA found a significant difference in summative performance scores for those scoring ≥ 70% when formative feedback was provided by providing a rationale, in-person, and a combination of both ([F (2,80) = 247.60, P < 0.001]. Post hoc analysis revealed a significant and highest performance when feedback was provided using the written rationale approach (***P < 0.05), followed by in-person (**P < 0.05). In contrast, the least performance was recorded when formative feedback was provided using a combination of providing a written rationale for the answers to the questions and in-person discussion of the questions (*P < 0.05). Students' preferred approach for receiving formative feedback for their formative assessment was highest for written rationale (***P < 0.05), followed by in-person or a combination of in-person and written rationale (**P < 0.05).

Conclusion

Our results found that medical students preferred a written formative feedback approach, which was associated with better student performance on the summative examination. This study reveals the importance of developing effective strategies to provide formative feedback to medical students for medical students to fully benefit from the merits of formative assessment in an integrated medical school curriculum.

Fostering formative assessment: teachers' perception, practice and challenges of implementation in four Sudanese medical schools, a mixed-method study

Formative assessment (assessment for learning) enhances learning (especially deep learning) by using feedback as a central tool. However, implementing it properly faces many challenges. We aimed to describe the perception of medical teachers towards FA, their practice, challenges of implementing FA and present applicable solutions. A mixed-method, explanatory approach study was applied by administering a validated questionnaire to 190 medical teachers in four medical schools in Sudan. The obtained results were further studied using the Delphi method. Quantitative analysis revealed that medical teachers perceived their grasping of the concept of FAs and their ability to differentiate formative from summative assessments as very well (83.7%) and (77.4%), respectively. However, in contradiction to the former results, it was noteworthy that (41%) of them mistakenly perceived FA as an approach conducted for purposes of grading and certification. The qualitative study defined the challenges into two main themes: lack of understanding of formative assessment and lack of resources. Medical teachers' development and resource allocation were the main recommendations. We conclude that there is misunderstanding and malpractice in implementing formative assessment attributed to the lack of understanding of FA as well as the lack of resources. We as well present suggested solutions derived from the perception of the medical teachers in the study and evolved around three approaches: faculty development, managing the curriculum by allocating time and resources for FA, and advocacy among stakeholders.

Experiences of the International In-Training Examination (I-ITE) by Rwandan pediatric residents - a survey-study describing candidate feedback

Background: The University of Rwanda is the only African residency to have implemented the pediatric International In-Training Examination (I-ITE) as a tool to monitor resident knowledge acquisition. The objective of this study was to better understand the acceptance and relevance of this exam to residents from this setting, and their perceptions regarding this assessment tool. Methods: A retrospective, survey-study was undertaken to describe candidate feedback. Immediately on completing the I-ITE residents provided feedback by filling in an electronic questionnaire comprised of four closed Likert questions and an open text box for free-text feedback. Participants were pediatric residents from the University of Rwanda, the only university in Rwanda with a pediatric residency program. Quantitative analysis of the Likert questions was undertaken descriptively using SPSS. Free-text feedback was coded and analysed employing a phenomenological approach, with coding and analysis undertaken by two researchers. Results: Eighty-four residents completed a total of 213 I-ITE sittings during the five exam cycles undertaken during the study period. The survey was completed after 206 of the 213 exam sittings, giving a response rate of 97%. Five themes emerged from the qualitative analysis; 1) undertaking the I-ITE was a positive experience; 2) exam content; 3) formative nature of the assessment; 4) challenges to completing the exam; 5) practicalities to undertaking the exam. Conclusion: Qualitative feedback demonstrates that the I-ITE, a standardized, and independent exam, produced by the American Board of Pediatrics, was valued and well accepted by Rwanda pediatric residents. Its formative nature and the breadth and quality of the questions were reported to positively contribute to the residents' formative development.

Evaluation Evolution: Designing Optimal Evaluations to Enhance Learning in Nephrology Fellowship

Evaluations serve as the backbone of any educational program and can be broadly divided into formative and summative evaluations. Formative evaluations are "just in time" evaluations focused on informing the learning process, whereas summative evaluations compare fellows to a preset standard to determine their readiness for unsupervised practice. In the nephrology fellowship programs, evaluations assess competence in the framework of ACGME Milestones 2.0. A variety of learning venues, evaluators, and tools should be incorporated into the measurement process. It is important to determine which milestones can be best assessed in each education venue to decrease the burden of assessment fatigue. Additionally, programs can diversify the evaluators to include nurses, medical students, peers, and program coordinators in addition to faculty to provide a well-rounded assessment of the fellows and share the assessment burden. Lastly, the evaluation data should be presented to fellows in a format where it can inform goal setting. The evaluation system needs to evolve along with the changes being made in curriculum design. This will help to make fellowship learning effective and efficient.

Assessing medical students' perception and educational experience during COVID-19 pandemic

INTRODUCTION

The COVID-19 pandemic has significantly impacted the traditional delivery of medical education. Medical education programmes have had to cope with limitations on face-to-face learning, and accelerate the adoption of digital learning. In addition, the pandemic has potential serious implications on the psychological well-being of medical students. We aim to assess the changes in perceptions and experiences of medical students as a consequence of this pandemic.

METHODS

Cross-sectional survey of medical students at Trinity College Dublin (TCD) between March and April 2022 was performed. The survey explored student satisfaction with the current education program, teaching delivery and the impact of COVID-19 on education and student well-being.

RESULTS

175 medical students participated in the survey. Overall, the majority of students were happy/neutral with their medical education. 93 (53.1%) felt tutorials and problem-based learning (PBL) to be the most effective method of teaching, followed by laboratory and clinical placements in 78 participants (44.6%) and hybrid-learning in 85 participants (48.6%). There was a mixed reaction to the changes in the delivery of education brought about by the pandemic. 67 participants (40.6%) felt happy with the changes, another 64 participants (38.8%) felt neutral, whilst only 34 participants (20.6%) were unhappy. However, most participants felt the pandemic negatively impacted their mental health, with 96 participants (55.8%) reporting negative responses. 58% of participants (n = 102/175) reported utilising the student support services at university campus and 49% (n = 50) were satisfied with their services.

CONCLUSION

Digital content and delivery confer the benefit of greater flexibility in learning, the ability to learn at one's own pace and in a preferred environment, however lacks the advantage of bedside teaching and hands-on training. Our findings reinforce the potential advantages of online learning.

Factors affecting perceived credibility of assessment in medical education: A scoping review

Assessment is more educationally effective when learners engage with assessment processes and perceive the feedback received as credible. With the goal of optimizing the educational value of assessment in medical education, we mapped the primary literature to identify factors that may affect a learner's perceptions of the credibility of assessment and assessment-generated feedback (i.e., scores or narrative comments). For this scoping review, search strategies were developed and executed in five databases. Eligible articles were primary research studies with medical learners (i.e., medical students to post-graduate fellows) as the focal population, discussed assessment of individual learners, and reported on perceived credibility in the context of assessment or assessment-generated feedback. We identified 4705 articles published between 2000 and November 16, 2020. Abstracts were screened by two reviewers; disagreements were adjudicated by a third reviewer. Full-text review resulted in 80 articles included in this synthesis. We identified three sets of intertwined factors that affect learners' perceived credibility of assessment and assessment-generated feedback: (i) elements of an assessment process, (ii) learners' level of training, and (iii) context of medical education. Medical learners make judgments regarding the credibility of assessments and assessment-generated feedback, which are influenced by a variety of individual, process, and contextual factors. Judgments of credibility appear to influence what information will or will not be used to improve later performance. For assessment to be educationally valuable, design and use of assessment-generated feedback should consider how learners interpret, use, or discount assessment-generated feedback.

"Written Formative Assessments with Peer-Assisted Learning" an Innovative Teaching Program for Postgraduate Students in Community Medicine

Introduction:

There is a paucity of research on conducting written formative assessment with constructive feedback for theory paper writing for postgraduates of Community Medicine in India. The concept of “Written Formative assessments with Peer-Assisted Learning Program” was implemented to improve the first 2 levels of Miller's Pyramid and assess its impact on the summative assessment.

Materials and Methods:

The program was conducted for 2 batches of postgraduate students in the Community Medicine enrolled for the academic session of 2016–2019 and 2017–2020. The written formative assessment was conducted every Saturday for 1 h from August to March month in 2018 and 2019. After each test, answer papers were evaluated by the peer and faculty from the department. Written and oral feedback was given by the peer. After IEC approval, we planned to assess the program's effect on level 1 and level 2 Kirkpatrick's framework. The data were analyzed using SPSS statistical package version 24 (SPSS Inc., Chicago, IL, USA). P < 0.05 was considered statistically significant.

Results:

Total 23 formative written assessments were conducted per year. The proportions of knowledge, comprehension, and analytical type of questions asked were 47%, 32%, and 21%, respectively. The mean attendance rate was 76.28% ±16.4%. There was no statistically significant difference in the average percentage of marks in formative (weekly test) and summative assessment (university final examination). There was a statistically significant positive co-relation of projected mean marks and summative assessment marks with the co-efficient of the determination being 22.6%. There was overall positive feedback of the formative and peer-assisted learning (PAL) from post graduate students.

Conclusions:

Written Formative Assessment with PAL program is one of the effective programs for postgraduate students to gain confidence in writing and presentation skills and to score higher in theory examination.

Strengths, Weaknesses, and Suggestions for Improvement in Postgraduate Assessment in Community Medicine in India: A Delphi Study

Objectives:

It was to understand the strengths and weaknesses of the current postgraduate assessment system in community medicine in India, to identify recommendations for change, and to build a consensus around them.

Materials and Methods:

A conventional Delphi technique was preferred for consensus building among experts. We completed three Delphi rounds over a period of 4 weeks, and 16 experts participated in the study. Content analysis was done for open-ended responses, and consensus analysis was done for Likert-type scale questionnaire. In round three, we obtained their top five preferences for change in assessment.

Results:

The experts agreed to have an assessment system based on ongoing formative and one end-of-year summative assessment. Apart from this, they agreed on the various occasions for carrying out the formative assessment. Furthermore, they clearly agreed on measures such as blueprinting, improving test formats, and adequate briefing of test-taking students.

Conclusion and Recommendations:

Most of the consensus items were found to be in alignment with the modern assessment theory. Regulating body and policymakers should revise the current postgraduate assessment system in community medicine to enhance its validity and reliability.

Implementation and Use of Workplace-Based Assessment in Clinical Learning Environments: A Scoping Review

PURPOSE

Workplace-based assessment (WBA) serves a critical role in supporting competency-based medical education (CBME) by providing assessment data to inform competency decisions and support learning. Many WBA systems have been developed, but little is known about how to effectively implement WBA. Filling this gap is important for creating suitable and beneficial assessment processes that support large-scale use of CBME. As a step toward filling this gap, the authors describe what is known about WBA implementation and use to identify knowledge gaps and future directions.

METHOD

The authors used Arksey and O'Malley's 6-stage scoping review framework to conduct the review, including: (1) identifying the research question; (2) identifying relevant studies; (3) study selection; (4) charting the data; (5) collating, summarizing, and reporting the results; and (6) consulting with relevant stakeholders.

RESULTS

In 2019-2020, the authors searched and screened 726 papers for eligibility using defined inclusion and exclusion criteria. One hundred sixty-three met inclusion criteria. The authors identified 5 themes in their analysis: (1) Many WBA tools and programs have been implemented, and barriers are common across fields and specialties; (2) Theoretical perspectives emphasize the need for data-driven implementation strategies; (3) User perceptions of WBA vary and are often dependent on implementation factors; (4) Technology solutions could provide useful tools to support WBA; and (5) Many areas of future research and innovation remain.

CONCLUSIONS

Knowledge of WBA as an implemented practice to support CBME remains constrained. To remove these constraints, future research should aim to generate generalizable knowledge on WBA implementation and use, address implementation factors, and investigate remaining knowledge gaps.

Virtual exams: has COVID-19 provided the impetus to change assessment methods in medicine?

AIMS

The ongoing COVID-19 pandemic has disrupted and delayed medical and surgical examinations where attendance is required in person. Our article aims to outline the validity of online assessment, the range of benefits to both candidate and assessor, and the challenges to its implementation. In addition, we propose pragmatic suggestions for its introduction into medical assessment.

METHODS

We reviewed the literature concerning the present status of online medical and surgical assessment to establish the perceived benefits, limitations, and potential problems with this method of assessment.

RESULTS

Global experience with online, remote virtual examination has been largely successful with many benefits conferred to the trainee, and both an economic and logistical advantage conferred to the assessor or organization. Advances in online examination software and remote proctoring are overcoming practical caveats including candidate authentication, cheating prevention, cybersecurity, and IT failure.

CONCLUSION

Virtual assessment provides benefits to both trainee and assessor in medical and surgical examinations and may also result in cost savings. Virtual assessment is likely to be increasingly used in the post-COVID world and we present recommendations for the continued adoption of virtual examination. It is, however, currently unable to completely replace clinical assessment of trainees. Cite this article: Bone Jt Open 2021;2(2):111-118.

Process-Oriented Instrument and Taxonomy for Teaching Surgical Procedures in Medical Training: The Ultrasound-Guided Insertion of Central Venous Catheter

Procedural training is relevant for physicians who perform surgical procedures. In the medical education field, instructors who teach surgical procedures need to understand how their students are learning to give them feedback and assess them objectively. The sequence of steps of surgical procedures is an aspect rarely considered in medical education, and state-of-the-art tools for giving feedback and assessing students do not focus on this perspective. Process Mining can help to include this perspective in this field since it has recently been used successfully in some applications. However, these previous developments are more centred on students than on instructors. This paper presents the use of Process Mining to fill this gap, generating a taxonomy of activities and a process-oriented instrument. We evaluated both tools with instructors who teach central venous catheter insertion. The results show that the instructors found both tools useful to provide objective feedback and objective assessment. We concluded that the instructors understood the information provided by the instrument since it provides helpful information to understand students’ performance regarding the sequence of steps followed.

How are formative assessment methods used in the clinical setting? A qualitative study

OBJECTIVES

To explore how formative assessment methods are used and perceived by second-year junior doctors in different clinical settings.

METHODS

A focused ethnography study was carried out. Ten second-year junior doctors from different specialties were selected using purposive sampling. The junior doctors were observed during a day in their clinical workplace where formative assessment was in focus. They were subsequently phone interviewed using a semi-structured interview guide regarding their experiences and attitudes towards formative assessment. Field notes from observations and interview transcriptions were analyzed using an inductive content analysis approach, and the concept of "everyday resistance" was used as a theoretical lens.

RESULTS

Three themes were identified: First, there were several barriers to the use of formative assessment methods in the clinical context, including subtle tactics of everyday resistance such as avoidance, deprioritizing, and contesting formative assessment methods.  Secondly, junior doctors made careful selections when arranging a formative assessment. Finally, junior doctors had ambiguous attitudes towards the use of mandatory formative assessment methods and mixed experiences with their educational impact.

CONCLUSIONS

This study emphasizes that the use of formative assessment methods in the clinical setting is not a neutral and context-independent exercise, but rather is affected by a myriad of factors such as collegial relations, educational traditions, emotional issues, and subtle forms of resistance. An important implication for the health care sector will be to address these issues for formative assessment methods to be properly implemented in the clinic.

Internal medicine residents' achievement goals and efficacy, emotions, and assessments

BACKGROUND

Achievement goal theory is consistently associated with specific cognitions, emotions, and behaviours that support learning in many domains, but has not been examined in postgraduate medical education. The purpose of this research was to examine internal medicine residents' achievement goals, and how these relate to their sense of self-efficacy, epistemic emotions, and valuing of formative compared to summative assessments. These outcomes will be important as programs transition more to competency based education that is characterized by ongoing formative assessments.

METHODS

Using a correlational design, we distributed a self-report questionnaire containing 49 items measuring achievement goals, self-efficacy, emotions, and response to assessments to internal medicine residents. We used Pearson correlations to examine associations between all variables.

RESULTS

Mastery-approach goals were positively associated with self-efficacy and curiosity and negatively correlated with frustration and anxiety. Mastery-approach goals were associated with a greater value for feedback derived from annual ACP exams, end-of-rotation written exams, and annual OSCEs. Performance-approach goals were only associated with valuing ACP exams.

CONCLUSION

Mastery-approach goals were associated with self-efficacy and epistemic emotions among residents, two constructs that facilitate autonomous learning. Residents with mastery-approach goals also appeared to value a wider range of types of assessment data. This profile will likely be beneficial for learners in a competency-based environment that involves high levels of formative feedback.