Methods to assess students' acquisition, application and integration of basic science knowledge in an innovative competency-based curriculum

A narrative review of adaptive testing and its application to medical education

Adaptive testing has a long but largely unrecognized history. The advent of computer-based testing has created new opportunities to incorporate adaptive testing into conventional programmes of study. Relatively recently software has been developed that can automate the delivery of summative assessments that adapt by difficulty or content. Both types of adaptive testing require a large item bank that has been suitably quality assured. Adaptive testing by difficulty enables more reliable evaluation of individual candidate performance, although at the expense of transparency in decision making, and requiring unidirectional navigation. Adaptive testing by content enables reduction in compensation and targeted individual support to enable assurance of performance in all the required outcomes, although at the expense of discovery learning. With both types of adaptive testing, candidates are presented a different set of items to each other, and there is the potential for that to be perceived as unfair. However, when candidates of different abilities receive the same items, they may receive too many they can answer with ease, or too many that are too difficult to answer. Both situations may be considered unfair as neither provides the opportunity to demonstrate what they know. Adapting by difficulty addresses this. Similarly, when everyone is presented with the same items, but answer different items incorrectly, not providing individualized support and opportunity to demonstrate performance in all the required outcomes by revisiting content previously answered incorrectly could also be considered unfair; a point addressed when adapting by content. We review the educational rationale behind the evolution of adaptive testing and consider its inherent strengths and limitations. We explore the continuous pursuit of improvement of examination methodology and how software can facilitate personalized assessment. We highlight how this can serve as a catalyst for learning and refinement of curricula; fostering engagement of learner and educator alike.

Assessing medical students' perception and educational experience during COVID-19 pandemic

INTRODUCTION

The COVID-19 pandemic has significantly impacted the traditional delivery of medical education. Medical education programmes have had to cope with limitations on face-to-face learning, and accelerate the adoption of digital learning. In addition, the pandemic has potential serious implications on the psychological well-being of medical students. We aim to assess the changes in perceptions and experiences of medical students as a consequence of this pandemic.

METHODS

Cross-sectional survey of medical students at Trinity College Dublin (TCD) between March and April 2022 was performed. The survey explored student satisfaction with the current education program, teaching delivery and the impact of COVID-19 on education and student well-being.

RESULTS

175 medical students participated in the survey. Overall, the majority of students were happy/neutral with their medical education. 93 (53.1%) felt tutorials and problem-based learning (PBL) to be the most effective method of teaching, followed by laboratory and clinical placements in 78 participants (44.6%) and hybrid-learning in 85 participants (48.6%). There was a mixed reaction to the changes in the delivery of education brought about by the pandemic. 67 participants (40.6%) felt happy with the changes, another 64 participants (38.8%) felt neutral, whilst only 34 participants (20.6%) were unhappy. However, most participants felt the pandemic negatively impacted their mental health, with 96 participants (55.8%) reporting negative responses. 58% of participants (n = 102/175) reported utilising the student support services at university campus and 49% (n = 50) were satisfied with their services.

CONCLUSION

Digital content and delivery confer the benefit of greater flexibility in learning, the ability to learn at one's own pace and in a preferred environment, however lacks the advantage of bedside teaching and hands-on training. Our findings reinforce the potential advantages of online learning.

The Importance of Professional Development in a Programmatic Assessment System: One Medical School’s Experience

The Cleveland Clinic Lerner College of Medicine of Case Western Reserve University (CCLCM) was created in 2004 as a 5-year undergraduate medical education program with a mission to produce future physician-investigators. CCLCM’s assessment system aligns with the principles of programmatic assessment. The curriculum is organized around nine competencies, where each competency has milestones that students use to self-assess their progress and performance. Throughout the program, students receive low-stakes feedback from a myriad of assessors across courses and contexts. With support of advisors, students construct portfolios to document their progress and performance. A separate promotion committee makes high-stakes promotion decisions after reviewing students’ portfolios. This case study describes a systematic approach to provide both student and faculty professional development essential for programmatic assessment. Facilitators, barriers, lessons learned, and future directions are discussed.

Spiralling Pre-clerkship Concepts into the Clinical Phase: Augmenting Knowledge Transfer Using Innovative Technology-Enhanced Curriculum Activities

BACKGROUND

Clerkship is a challenging transition for medical students where they learn to apply functional knowledge and diagnostic reasoning skills learned in the pre-clinical phase into the clinical environment. Rather than a smooth continuum to facilitate application of knowledge, clerkship blocks are discrete, fragmented structures with little integration. Developments in cognitive psychology and increasing attention to the student learning environment are driving more purposeful integration in medical education. We sought to enhance knowledge transfer in the Family Medicine clerkship by developing an e-learning pathway with both asynchronous and synchronous components to integrate pre-clerkship problem-based learning (PBL) cases into more complex clinical scenarios.

METHODS

A parallel-convergent mixed methods evaluation was conducted, which included comparison of learning outcomes (exit exams) relative to the prior class, knowledge check quiz performance, and qualitative analysis of student and faculty perceptions.

RESULTS

Analyses revealed no significant difference between exit exam scores of the intervention and pre-intervention clerks (p = 0.30). There were statistically significant differences in mean quiz scores over the rotation (p = 0.0001). Moreover, learners and faculty each perceived the integration components as facilitating the transfer of pre-clinical learning into clerkship activities.

CONCLUSION

The novel e-learning pathway firmly anchored FM clerkship learning and will continue to ensure learners are ideally primed to optimize their direct clinical learning opportunities.

Prevalence of stress, stressors, and coping strategies among medical undergraduate students in a medical college of Mumbai

BACKGROUND

Stress is a subjective experience which creates an obstacle in a person's path of achieving his or her goals. It creates negative outcomes among the population. Medical curriculum is very vast and stressful. As doctors are the very important part of society, medical students must be allowed to learn their trade in a stress-free environment. The present study aimed to assess the perceived stress, various sources of stress among medical undergraduate students, and the coping strategies adopted.

MATERIALS AND METHODS

This was a cross-sectional study conducted among medical under-graduate students studying at a tertiary care hospital situated in Mumbai city of Maharashtra between January and June 2018. Using purposive sampling technique, a total of 450 medical students from 1^st year to final year were invited to participate in the study. A self-administered questionnaire consisting of sociodemographic details and perceived stress scale questions was used. Logistic regression analysis was carried out to assess the determinants of stress. Odds ratio, 95% confidence interval was calculated. P < 0.05 was considered statistically significant.

RESULTS

The overall response rate was 79.11%, with 356 out of 450 students returning the questionnaire. Among 356 participants, 324 participants (91%) were suffering from high levels of stress. Factors such as curriculum vastness, frequency of examination, competition with peer, performance in examinations, worry about future, loneliness, relation with opposite sex, and quality of food played a major role in creating additional stress.

CONCLUSIONS

The perceived stress was higher among female medical students. Academic factors are greater perceived cause of stress in medical students in this study. A substantial proportion of medical undergraduate students were found to be stressed which necessitates appropriate and timely interventions. Reframing the academic curriculum along with examination and evaluation patterns, incorporating extracurricular activities, and establishment of counselling cells in the institution is the need of the hour.

Two-dimensional integration approach to teaching cardiovascular physiology: effectiveness and students' perspectives

BACKGROUND

Pre-clerkship medical curriculums consist of a series of organ system-based courses and lectures but often lack an integration between organ systems. Such integration could be beneficial for clerkship years and students' future career. Hence, we aimed to share our process of organising an integrative teaching approach in a large class of pre-clerkship medical students and to reflect the students' perspective toward the teaching process in this observational study. In addition, we tested effectiveness of this integrative approach compared with the traditional teaching (lecture).

METHODS

We organised a two-dimensional (2D)-integrative teaching for 309 students in selected topics of cardiovascular physiology of the medical curriculum of the Faculty of Medicine, Chulalongkorn University, Thailand. The first dimension of integration is the incorporation of physiology of other organ systems into the cardiovascular physiology class. The second is the integration of multiple teaching methods and strategies, including small group discussion, student presentation, wrap-up, quiz, and question-and-answer sessions. Unless opting out, students evaluated this integrative teaching by filling in a questionnaire. The summative scores were also used to determine their comprehensive understandings of the content.

RESULTS

The course evaluation showed that most students (81.9-91.2%) had positive attitudes toward all organised sessions, i.e. this teaching method helps promote their basic and applied physiology knowledge, critical thinking, information searching, presentation, and teamwork skills. In general, students at all performance levels attained higher scores in the summative exam for the 2D-integrative-class-relevant questions (74.4±16.1%) than for the lecture-pertinent questions (65.2±13.6%).

CONCLUSIONS

In a large class size of pre-clerkship students, 2D-integrative teaching strategies with careful planning and preparation can be successfully implemented, based on positive attitudes and relatively high summative scores of students in this study. Hence, this comprehensive teaching could be incorporated in current medical curriculums, particularly for the complex learning topics.

Patients don't come with multiple choice options: essay-based assessment in UME

Curricular revision efforts have resulted in learner-centered programs that value content integration and active learning. Yet, less attention has been placed on assessment methods that are learner-centered and promote assessment for learning. The use of context rich short answer question (CR-SAQ) exams in the preclinical years of medical school was evaluated to determine if this format aligns with the criteria for assessment for learning. Medical students and preclinical faculty members were sent a survey comprised of closed and open-ended questions about their experience using CR-SAQ exams. Data were analyzed using a mixed-method design. Open-ended responses were evaluated using thematic analysis within the framework of criteria for assessment for learning. A total of 274 students (94%) and 24 faculty (75%) completed the survey. Fifty four percent of students reported preferring a CR-SAQ exam format over multiple choice questions (MCQ) format. Quantitative data and qualitative comments by students supported that CR-SAQ exams aligned with criteria for assessment for learning, including acceptability, authenticity, educational effect, and the cueing effect. Student concerns included preparation for USMLE Step 1 exam, as well as the validity and reproducibility of CR-SAQ assessments. Faculty largely agreed with the benefits of the CR-SAQ, but were concerned about feasibility, acceptability and reproducibility. The CR-SAQ exam format assessment strategy supports assessment for learning in an undergraduate medical education setting. Both benefits and drawbacks of this method are presented, however students and faculty describe a broader impact that this assessment method has on their development as a physician.

Ten maxims of formative assessment

In an era that is seemingly saturated with standardized tests of all hues and stripes, it is easy to forget that assessments not only measure the performance of students, but also consolidate and enhance their learning. Assessment for learning is best elucidated as a process by which the assessment information can be used by teachers to modify their teaching strategies while students adjust and alter their learning approaches. Effectively implemented, formative assessments can convert classroom culture to one that resonates with the triumph of learning. In this paper, we present 10 maxims that show ways that formative assessments can be better understood, appreciated, and implemented.

Assessment, feedback and the alchemy of learning

CONTEXT

Models of sound assessment practices increasingly emphasise assessment's formative role. As a result, assessment must not only support sound judgements about learner competence, but also generate meaningful feedback to guide learning. Reconciling the tension between assessment's focus on judgement and decision making and feedback's focus on growth and development represents a critical challenge for researchers and educators.

METHODS

We synthesise the literature related to this tension, framed around four trends in education research: (i) shifting perspectives on assessment; (ii) shifting perspectives on feedback; (iii) increasing attention on learners' perceptions of assessment and feedback, and (iv) increasing attention on the influence of culture on assessment and feedback. We describe factors that produce and sustain this tension.

RESULTS

The lines between assessment and feedback frequently blur in medical education. Models of programmatic assessment deliberately use the same data for both purposes: low-stakes individual data points are used formatively, but then are added together to support summative judgements. However, the translation of theory to practice is not straightforward. Efforts to embed meaningful feedback in programmes of learning face a multitude of threats. Learners may perceive assessment with formative intent as summative, restricting their engagement with it as feedback, and thus diminishing its learning value. A learning culture focused on assessment may limit learners' sense of safety to explore, to experiment, and sometimes to fail.

CONCLUSIONS

Successfully blending assessment and feedback demands clarity of purpose, support for learners, and a system and organisational commitment to a culture of improvement rather than a culture of performance.

The integrated curriculum in medical education: AMEE Guide No. 96

The popularity of the term "integrated curriculum" has grown immensely in medical education over the last two decades, but what does this term mean and how do we go about its design, implementation, and evaluation? Definitions and application of the term vary greatly in the literature, spanning from the integration of content within a single lecture to the integration of a medical school's comprehensive curriculum. Taking into account the integrated curriculum's historic and evolving base of knowledge and theory, its support from many national medical education organizations, and the ever-increasing body of published examples, we deem it necessary to present a guide to review and promote further development of the integrated curriculum movement in medical education with an international perspective. We introduce the history and theory behind integration and provide theoretical models alongside published examples of common variations of an integrated curriculum. In addition, we identify three areas of particular need when developing an ideal integrated curriculum, leading us to propose the use of a new, clarified definition of "integrated curriculum", and offer a review of strategies to evaluate the impact of an integrated curriculum on the learner. This Guide is presented to assist educators in the design, implementation, and evaluation of a thoroughly integrated medical school curriculum.

Cognition before curriculum: rethinking the integration of basic science and clinical learning

PURPOSE

Integrating basic science and clinical concepts in the undergraduate medical curriculum is an important challenge for medical education. The health professions education literature includes a variety of educational strategies for integrating basic science and clinical concepts at multiple levels of the curriculum. To date, assessment of this literature has been limited.

METHOD

In this critical narrative review, the authors analyzed literature published in the last 30 years (1982-2012) using a previously published integration framework. They included studies that documented approaches to integration at the level of programs, courses, or teaching sessions and that aimed to improve learning outcomes. The authors evaluated these studies for evidence of successful integration and to identify factors that contribute to integration.

RESULTS

Several strategies at the program and course level are well described but poorly evaluated. Multiple factors contribute to successful learning, so identifying how interventions at these levels result in successful integration is difficult. Evidence from session-level interventions and experimental studies suggests that integration can be achieved if learning interventions attempt to link basic and clinical science in a causal relationship. These interventions attend to how learners connect different domains of knowledge and suggest that successful integration requires learners to build cognitive associations between basic and clinical science.

CONCLUSIONS

One way of understanding the integration of basic and clinical science is as a cognitive activity occurring within learners. This perspective suggests that learner-centered, content-focused, and session-level-oriented strategies can achieve cognitive integration.

Motivating student learning using a formative assessment journey

Providing formative assessment opportunities has been recognised as a significant benefit to student learning. The outcome of any formative assessment should be one that ultimately helps improve student learning through familiarising students with the levels of learning required, informing them about gaps in their learning and providing feedback to guide the direction of learning. This article provides an example of how formative assessments can be developed into a formative assessment journey where a number of different assessments can be offered to students during the course of a module of teaching, thus utilising a spaced-education approach. As well as incorporating the specific drivers of formative assessment, we demonstrate how approaches deemed to be stimulating, interactive and entertaining with the aim of maximising enthusiasm and engagement can be incorporated. We provide an example of a mixed approach to evaluating elements of the assessment journey that focuses student reaction, appraisal of qualitative and quantitative feedback from student questionnaires, focus group analysis and teacher observations. Whilst it is not possible to determine a quantifiable effect of the assessment journey on student learning, usage data and student feedback shows that formative assessment can achieve high engagement and positive response to different assessments. Those assessments incorporating an active learning element and a quiz-based approach appear to be particularly popular. A spaced-education format encourages a building block approach to learning that is continuous in nature rather than focussed on an intense period of study prior to summative examinations.

Beyond assessment of learning toward assessment for learning: educating tomorrow's physicians

Beyond its importance in informing high-stakes decisions, the assessment process can also be designed to foster learning. To be effective, this requires developing a program in which curricular experiences, assessment practices and support activities are aligned to provide an educational culture that encourages self-regulated learning. We describe a program (based at Cleveland Clinic Lerner College of Medicine) in which explicit performance standards align these components and provide a roadmap for students to manage their learning. Information-rich assessment data, structured opportunities for reflection, and facilitated self-assessment using a portfolio approach are designed to support development of habits of reflective practice. Promotion depends on the achievement of competencies rather than grades. Preliminary evidence suggests that the program directs students towards learning, rather than on achieving a grade for grade's sake.

Integration of basic sciences in health's courses

Concepts from disciplines such as Biochemistry, Genetics, Cellular and Molecular Biology are essential to the understanding and treatment of an elevated number of illnesses, but often they are studied separately, with no integration between them. This article proposes a model for basic sciences integration based on problem-based learning (PBL) and compares failure rate, global final grade, approved student final grade, grade distribution and students' satisfaction with teacher conduction between integrated curriculum and traditional learning in health courses from Anhembi Morumbi University-a private institution from Brazil. Comparison between integrated and traditional curriculum was based on students' records obtained from first-year health sciences students. A total of 1,697 records from 2005 to 2007 (nonintegrated curriculum) and 785 records from 2008 (integrated curriculum) were selected for this study and they were necessary to get information about students' grades. Moreover, a questionnaire was applied in order to cover student's satisfaction with teacher conduction. The data presented in this study indicated that the integrated curriculum based on PBL was related to an improvement in student's grades and satisfaction compared with traditional teaching. We believe that the effectiveness in health education will be a combination of "classical" presentation of contents associated to actively involved students in the educational process and methodology based on problems in order to create the stimulus for the undergraduates continue to integrate basic and clinical investigation.

Involving users in the refinement of the competency-based achievement system: an innovative approach to competency-based assessment

BACKGROUND

Competency-based assessment innovations are being implemented to address concerns about the effectiveness of traditional approaches to medical training and the assessment of competence.

AIM

Integrating intended users' perspectives during the piloting and refinement process of an innovation is necessary to ensure the innovation meets users' needs. Failure to do so results in no opportunity for users to influence the innovation, nor for developers to assess why an innovation works or does not work in different contexts.

METHODS

A qualitative participatory action research approach was used. Sixteen first-year residents participated in three focus groups and two interviews during piloting. Verbatim transcripts were analyzed individually and then across all transcripts using a constant comparison approach.

RESULTS

The analysis revealed three key characteristics related to the impact on the residents' acceptance of the innovation as being a worthwhile investment of time and effort: access to frequent, timely, and specific feedback from preceptors. Findings were used to refine the innovation further.

CONCLUSION

This study highlights the necessary conditions for assessing the success of implementation of educational innovations. Reciprocal communication between users and developers is vital. This reflects the approaches recommended in the Ottawa Consensus Statement on research in assessment published in Medical Teacher in March 2011.

A comparison of student performance in multiple-choice and long essay questions in the MBBS stage I physiology examination at the University of the West Indies (Mona Campus)

This retrospective study compared the performance of preclinical medical students in the multiple-choice question (MCQ) and long essay question components of a comprehensive physiology final examination. During the 3 yr analyzed, 307 students had an average score of 47% (SD 9.9) in the long essay questions and 64% (SD 9.9) in the MCQs. Regression analysis showed a significant correlation (r = 0.62, P < 0.01) between MCQs and long essay questions. When student performance was grouped by final course grade, a statistically significant correlation between MCQ and long essay scores existed only for the 210 students who received a passing grade (r = 0.20, P < 0.01). The MCQ and long essay question scores were not correlated for the 57 students who failed (r = 0.25, P = 0.06) or for the 40 students who achieved honors and distinctions (r = -0.27, P = 0.11). MCQ scores were significantly higher (P < 0.01) than essay scores for each of the groups when analyzed by two-way ANOVA. The results of this study suggest that for most students, the strong correlation between scores on MCQs and essay questions indicates that student performance was independent of testing format. For students at either end on the performance spectrum, the lack of correlation suggests that the performance in one of the testing formats had a strong influence on the final course grade. In addition, those students who failed the course were likely to be weak in both testing modalities, whereas students in all grade groups were more likely to perform better in the MCQs than in the long essay questions.

Evaluation of essay questions used to assess medical students' application and integration of basic and clinical science knowledge

BACKGROUND

Educators need approaches to assess medical students' abilities to apply and integrate concepts essential to medical practice.

DESCRIPTION

We used a multimethod approach to examine the quality of essay questions intended to elicit medical students' ability to apply and integrate their understanding of medical concepts.

EVALUATION

Three educators assigned essay questions (n = 120) to one of four levels of cognition. Kappa was computed before and after discussion. Faculty (n = 46) critiqued essay quality using a checklist (97% response), and students completed a questionnaire about the learning environment (91% response).

CONCLUSIONS

We identified effective approaches to evaluate the quality of essay questions and to train faculty to write essay questions of sufficient complexity. This systematic review of essay questions also encouraged review of the curriculum to determine if core concepts were being taught. It is feasible to have faculty write and critique essay questions targeted at higher levels of cognition.