Evaluating the impact of moving from discipline-based to integrated assessment

Assessment approaches in undergraduate health professions education: towards the development of feasible assessment approaches for low-resource settings

BACKGROUND

Feasible and effective assessment approaches to measuring competency in health sciences are vital in competency-based education. Educational programmes for health professions in low- and middle-income countries are increasingly adopting competency-based education as a strategy for training health professionals. Importantly, the organisation of assessments and assessment approaches must align with the available resources and still result in the fidelity of implementation. A review of existing assessment approaches, frameworks, models, and methods is essential for the development of feasible and effective assessment approaches in low-resource settings.

METHODS

Published literature was sourced from 13 electronic databases. The inclusion criteria were literature published in English between 2000 and 2022 about assessment approaches to measuring competency in health science professions. Specific data relating to the aims of each study, its location, population, research design, assessment approaches (including the outcome of implementing such approaches), frameworks, models, and methods were extracted from the included literature. The data were analysed through a multi-step process that integrated quantitative and qualitative approaches.

RESULTS

Many articles were from the United States and Australia and reported on the development of assessment models. Most of the articles included undergraduate medical or nursing students. A variety of models, theories, and frameworks were reported and included the Ideal model, Predictive Learning Assessment model, Amalgamated Student Assessment in Practice (ASAP) model, Leadership Outcome Assessment (LOA) model, Reporter-Interpreter-Manager-Educator (RIME) framework, the Quarter model, and the model which incorporates four assessment methods which are Triple Jump Test, Essay incorporating critical thinking questions, Multistation Integrated Practical Examination, and Multiple Choice Questions (TEMM) model. Additional models and frameworks that were used include the Entrustable Professional Activities framework, the System of Assessment framework, the Reporter-Interpreter-Manager-Educator (RIME) framework, the Clinical Reasoning framework (which is embedded in the Amalgamated Student Assessment in Practice (ASAP) model), Earl's Model of Learning, an assessment framework based on the Bayer-Fetzer Kalamazoo Consensus Statement, Bloom's taxonomy, the Canadian Medical Education Directions for Specialists (CanMEDS) Framework, the Accreditation Council for Graduate Medical Education (ACGME) framework, the Dreyfus Developmental Framework, and Miller's Pyramid.

CONCLUSION

An analysis of the assessment approaches, frameworks, models, and methods applied in health professions education lays the foundation for the development of feasible and effective assessment approaches in low-resource settings that integrate competency-based education.

TRIAL REGISTRATION

This study did not involve any clinical intervention. Therefore, trial registration was not required.

The Effect of Histology Examination Format on Medical Student Preparation and Performance: Stand-Alone Versus Integrated Examinations

Most medical schools have transitioned from discipline-based to integrated curricula. Although the adoption of integrated examinations usually accompanies this change, stand-alone practical examinations are often retained for disciplines such as gross anatomy and histology. Due to a variety of internal and external factors, faculty at the University of Cincinnati College of Medicine recently began to phase out stand-alone histology practical examinations in favor of an integrated approach to testing. The purpose of this study was to evaluate this change by (1) comparing examination performance on histology questions administered as part of stand-alone versus integrated examinations and (2) ascertaining whether students alter their approach to learning histology content based on the examination format. Data from two courses over a period ranging from 2018 to 2022 were used to evaluate these questions. Results indicated histology question performance initially dropped after being included on integrated examinations. Stratification of students by class rank revealed this change had a greater impact on lower-performing students. Longitudinal data showed that performance 2 years after the change yielded scores similar to previous standards. Despite the initial performance drop, survey results indicate students overwhelmingly prefer when histology is included on integrated examinations. Additionally, students described alterations in study approaches that align with what is known to promote better long-term retention. The results presented in this study have important implications for those at other institutions who are considering making similar changes in assessment strategies.

Declining Student Performance and Satisfaction following Back-to-Back Scheduling of Foundational Science Exams: Experience at a Large US Osteopathic Medical School

Examinations are a significant source of academic stress, particularly in the demanding environment of medical education. In order to reduce the burden of anxiety, burnout and depression among students, medical schools aim at reducing academic stress by exploring alternatives to frequent, high-stakes assessment schedules. The bundling of examinations into integrated block assessments has emerged as a successful strategy to reach this goal, as the resulting reduction in examination days can provide uninterrupted periods of study time that allows for a deeper understanding of material as well as an opportunity for students to take wellness breaks between exams. The present study examines the outcomes of a natural experiment on back-to-back exam scheduling in two related medical school courses post hoc. The outcomes of the scheduling change on the academic performance and satisfaction of examinees were analyzed with a mixed-methods approach. The data show that the transition from a spaced-out to a back-to-back exam schedule was accompanied by a drop in academic performance and student satisfaction with the curricular schedule. The data presented suggest that without proper curricular integration, the block scheduling of exams has negative effects on learning outcomes and student satisfaction.

Exploring an integrated curriculum in pharmacy: Educators' perspectives

BACKGROUND AND PURPOSE

Lack of consensus regarding the benefits of an integrated curriculum, and ambiguity concerning what is being integrated within a pharmacy curriculum exists, but how an integrated curriculum is viewed, epistemologically, and subsequently incorporated into teaching practice has not been investigated. This study explores how educators conceptualize, experience and enact curricula integration both pedagogically and organizationally.

EDUCATIONAL ACTIVITY AND SETTING

In-depth qualitative interviews with faculty members purposively sampled for maximum variation in disciplinary background and teaching experience were undertaken at a single site. Interviews addressed two research questions: how pharmacy educators understand an integrated curriculum and educators' experiences of it.

FINDINGS AND DISCUSSION

Analysis of the interview data suggests four essential meanings of integration: integration as a method for organizing teaching and learning; integration as enacted by self and others; integration as tension between conflicting knowledge domains; and integration as an impossible concept to apply to teaching practice. Analysis suggests that integration is an abstract rather than enacted concept and although integration is viewed as learner-centered, integration is complex and associated with a loss of in-depth learning.

SUMMARY

Differences in how faculty members conceptualize the purpose and effects of integration mean that the level and type of integration reported varied. A clearer understanding of the rationale for change, and methods for better applying theory of integration to teaching practice, may be needed to achieve curriculum standards required by bodies accrediting undergraduate pharmacy programs.

Using generalizability analysis to estimate parameters for anatomy assessments: A multi-institutional study

With integrated curricula and multidisciplinary assessments becoming more prevalent in medical education, there is a continued need for educational research to explore the advantages, consequences, and challenges of integration practices. This retrospective analysis investigated the number of items needed to reliably assess anatomical knowledge in the context of gross anatomy and histology. A generalizability analysis was conducted on gross anatomy and histology written and practical examination items that were administered in a discipline-based format at Indiana University School of Medicine and in an integrated fashion at the University of Alabama School of Medicine and Rush University Medical College. Examination items were analyzed using a partially nested design s×(i:o) in which items were nested within occasions (i:o) and crossed with students (s). A reliability standard of 0.80 was used to determine the minimum number of items needed across examinations (occasions) to make reliable and informed decisions about students' competence in anatomical knowledge. Decision study plots are presented to demonstrate how the number of items per examination influences the reliability of each administered assessment. Using the example of a curriculum that assesses gross anatomy knowledge over five summative written and practical examinations, the results of the decision study estimated that 30 and 25 items would be needed on each written and practical examination to reach a reliability of 0.80, respectively. This study is particularly relevant to educators who may question whether the amount of anatomy content assessed in multidisciplinary evaluations is sufficient for making judgments about the anatomical aptitude of students. Anat Sci Educ 10: 109-119. © 2016 American Association of Anatomists.

A comparison of student performance on discipline-specific versus integrated exams in a medical school course

Curricular reform is a widespread trend among medical schools. Assessing the impact that pedagogical changes have on students is a vital step in review process. This study examined how a shift from discipline-focused instruction and assessment to integrated instruction and assessment affected student performance in a second-year medical school pathology course. We investigated this by comparing pathology exam scores between students exposed to traditional discipline-specific instruction and exams (DSE) versus integrated instruction and exams (IE). Exam content was controlled, and individual questions were evaluated using a modified version of Bloom's taxonomy. Additionally, we compared United States Medical Licensing Examination (USMLE) step 1 scores between DSE and IE groups. Our findings indicate that DSE students performed better than IE students on complete pathology exams. However, when exam content was controlled, exam scores were equivalent between groups. We also discovered that the integrated exams were composed of a significantly greater proportion of questions classified on the higher levels of Bloom's taxonomy and that IE students performed better on these questions overall. USMLE step 1 exam scores were similar between groups. The finding of a significant difference in content complexity between discipline-specific and integrated exams adds to recent literature indicating that there are a number of potential biases related to curricular comparison studies that must be considered. Future investigation involving larger sample sizes and multiple disciplines should be performed to explore this matter further.

Competency-based integrated practical examinations: Bringing relevance to basic science laboratory examinations

BACKGROUND

The practical examinations in subject-based curriculum have been criticized for lack of relevance and clinical application. We developed competency-based integrated practical examinations (IPEs) for first two years incorporating basic science principles with clinical relevance in our integrated curriculum.

AIM

To bring relevance to basic science laboratory practical examinations by conducting competency-based IPEs.

METHODS

IPEs were developed according to competency-based blueprinting for each integrated module. Clinical scenarios were used as triggers followed by tasks pertaining to laboratory tests, relevant physical diagnosis and ethics/professional aspects utilizing standardized patients. Checklists were developed for standardized marking. A feedback questionnaire and two focus group discussions were administered to a random group of students from both first and second year students. Faculty members' feedback was also recorded on a questionnaire.

RESULTS

Almost all the students agreed that IPE was a useful experience. Eighty-nine percent agreed that it was a fair examination and elicited a lesser degree of psychological stress. Eighty-two percent agreed that IPE encouraged critical thinking and application of knowledge. However, students suggested better organization and longer duration of stations. Faculty members also liked the experience.

CONCLUSION

In conclusion, IPEs were well-received and valued both by students and faculty members.

Competency-based assessment and cultural compression in medical education: lessons from educational anthropology

OBJECTIVE

This paper explores the thesis that medical education is the cultural transmission to learners of specific values, which are increasingly expressed as graduation competencies. As testing is a powerful way to transmit cultural values to learners in a brief period of time, competency-based assessments can be an instrument of cultural compression in medical education.

METHODS

The author reviewed medical literature to illustrate the concepts from educational anthropology, led the process one medical school used to develop its list of graduation competencies, and conducted a citation search about competency domains.

RESULTS

There is support in the literature for viewing medical education as an example of cultural transmission and compression and for the assertion that testing influences student behaviour. The graduation competency statements developed by the school reflect traditional and emergent values. The citation search data confirmed that some competency domains reflected traditional values, while others reflected more emergent values.

CONCLUSION

Concepts from educational anthropology are relevant to medical education and provide perspectives for understanding contemporary issues such as competency-based assessments.

Student perspectives of assessment by TEMM model in physiology

Assessment is the process by which the teacher and the student gain knowledge about student progress. Assessment systems should aim at evaluating the desired learning outcomes. In Melaka Manipal Medical College, (Manipal Campus), Manipal, India, the TEMM model (consisting of 4 assessment methods: Triple Jump Test, essay incorporating critical thinking questions, Multistation Integrated Practical Examination, and multiple choice questions) was introduced to 30 refresher students in the fourth block of the academic year. At the end of the block, a questionnaire was distributed to ask the students to rank the different assessments in the order of their preference with respect to seven items. Analysis of the results showed that not a single type of assessment was ranked highest for all the seven items, proving the earlier observation that a single assessment does not fulfill all aspects of assessment and that there is a need for an evaluating system with multiple ways of assessment.