A comparison of two learning approach inventories and their utility in predicting examination performance and study habits

The revised two-factor Study Process Questionnaire and the Approaches and Study Skills Inventory for Students are two instruments commonly used to measure student learning approach. Although they are designed to measure similar constructs, it is unclear whether the metrics they provide differ in terms of their real-world classification of learning approach. The purpose of this study is to compare outcomes of these two inventories in a study population from an undergraduate (baccalaureate) human anatomy course. The three central goals of this study are to compare the inventories in terms of 1) how students are classified, 2) the relationship between examination performance, time spent studying, and learning approach, and 3) instrument reliability. Results demonstrate that student classifications of corresponding scales of each inventory are significantly correlated, suggesting they measure similar constructs. Although the inventories had similar reliability, neither was consistently strong in predicting examination performance or study habits. Overall, these results suggest that the two inventories are comparable in terms of how they measure learning approach, but the lack of correspondence between learning approach scores and measurement outcomes questions their validity as tools that can be used universally in classrooms.NEW & NOTEWORTHY Although learning approach inventories have been used extensively in education research, there has been no direct comparison of how student classification differs between instruments or how classification influences the interpretation of how learning approach impacts student performance. This is especially relevant in light of recent research questioning the validity of the Study Process Questionnaire (LoGiudice AB, Norman GR, Manzoor S, Monteiro S. Adv Health Sci Educ Theory Pract 28: 47-63, 2023; Johnson SN, Gallagher ED, Vagnozzi AM. PLoS One 16: e0250600, 2021).

Comparing the performance of artificial intelligence learning models to medical students in solving histology and embryology multiple choice questions

INTRODUCTION

The appearance of artificial intelligence language models (AI LMs) in the form of chatbots has gained a lot of popularity worldwide, potentially interfering with different aspects of education, including medical education as well. The present study aims to assess the accuracy and consistency of different AI LMs regarding the histology and embryology knowledge obtained during the 1st year of medical studies.

METHODS

Five different chatbots (ChatGPT, Bing AI, Bard AI, Perplexity AI, and ChatSonic) were given two sets of multiple-choice questions (MCQs). AI LMs test results were compared to the same test results obtained from 1st year medical students. Chatbots were instructed to use revised Bloom's taxonomy when classifying questions depending on hierarchical cognitive domains. Simultaneously, two histology teachers independently rated the questions applying the same criteria, followed by the comparison between chatbots' and teachers' question classification. The consistency of chatbots' answers was explored by giving the chatbots the same tests two months apart.

RESULTS

AI LMs successfully and correctly solved MCQs regarding histology and embryology material. All five chatbots showed better results than the 1st year medical students on both histology and embryology tests. Chatbots showed poor results when asked to classify the questions according to revised Bloom's cognitive taxonomy compared to teachers. There was an inverse correlation between the difficulty of questions and their correct classification by the chatbots. Retesting the chatbots after two months showed a lack of consistency concerning both MCQs answers and question classification according to revised Bloom's taxonomy learning stage.

CONCLUSION

Despite the ability of certain chatbots to provide correct answers to the majority of diverse and heterogeneous questions, a lack of consistency in answers over time warrants their careful use as a medical education tool.

Evaluating the strengths and weaknesses of large language models in answering neurophysiology questions

Large language models (LLMs), like ChatGPT, Google's Bard, and Anthropic's Claude, showcase remarkable natural language processing capabilities. Evaluating their proficiency in specialized domains such as neurophysiology is crucial in understanding their utility in research, education, and clinical applications. This study aims to assess and compare the effectiveness of Large Language Models (LLMs) in answering neurophysiology questions in both English and Persian (Farsi) covering a range of topics and cognitive levels. Twenty questions covering four topics (general, sensory system, motor system, and integrative) and two cognitive levels (lower-order and higher-order) were posed to the LLMs. Physiologists scored the essay-style answers on a scale of 0-5 points. Statistical analysis compared the scores across different levels such as model, language, topic, and cognitive levels. Performing qualitative analysis identified reasoning gaps. In general, the models demonstrated good performance (mean score = 3.87/5), with no significant difference between language or cognitive levels. The performance was the strongest in the motor system (mean = 4.41) while the weakest was observed in integrative topics (mean = 3.35). Detailed qualitative analysis uncovered deficiencies in reasoning, discerning priorities, and knowledge integrating. This study offers valuable insights into LLMs' capabilities and limitations in the field of neurophysiology. The models demonstrate proficiency in general questions but face challenges in advanced reasoning and knowledge integration. Targeted training could address gaps in knowledge and causal reasoning. As LLMs evolve, rigorous domain-specific assessments will be crucial for evaluating advancements in their performance.

Authors' Reply: "Evaluating GPT-4's Cognitive Functions Through the Bloom Taxonomy: Insights and Clarifications"

Collapse

Blended learning in nursing pharmacology: elevating cognitive skills, engagement and academic outcomes

Pharmacological education is crucial for healthcare professionals to safely manage medications and reduce errors. Traditional lecture-based learning (LBL) often struggles to address this complexity, whereas newer methods, such as flipped classrooms and problem-based learning, yield mixed results, particularly in pre-clinical contexts, owing to students' limited experience. Our nursing pharmacology course under LBL recorded a high failure rate of 37.8% and marginal passing scores across five cohorts (n = 849 students). An analysis using Bloom's taxonomy revealed significant gaps in higher-order cognitive skills. As a remedy, the course was transformed into a novel blended learning format that integrated question-based learning (QBL) to enhance critical thinking across all cognitive levels. This model blends asynchronous and synchronous learning, is tailored to individual needs in large classes, and fosters continuous, student-centric learning. The redesign markedly decreased the failure rate by approximately 2.8-fold and increased the average grade by 11.8 points among 426 students. It notably improved the pass rates in advanced cognitive categories, such as "Evaluate" and "Create" by 19.0% and 24.2%, respectively. Additionally, the blended course showed increased student engagement, reflecting a dynamic and effective learning environment that significantly elevated participation and academic outcomes at all cognitive levels. This study demonstrated the profound impact of blended learning in pharmacology. By integrating QBL with various teaching methods, it surpasses traditional lecture-based limitations, enhancing engagement and understanding of complex topics by nursing students. Notable improvements in foundational and advanced learning suggest its broader application in health professionals' education, effectively equipping students for clinical pharmacology challenges.

Assessing ChatGPT's Mastery of Bloom's Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study

BACKGROUND

Large language models such as GPT-4 (Generative Pre-trained Transformer 4) are being increasingly used in medicine and medical education. However, these models are prone to "hallucinations" (ie, outputs that seem convincing while being factually incorrect). It is currently unknown how these errors by large language models relate to the different cognitive levels defined in Bloom's taxonomy.

OBJECTIVE

This study aims to explore how GPT-4 performs in terms of Bloom's taxonomy using psychosomatic medicine exam questions.

METHODS

We used a large data set of psychosomatic medicine multiple-choice questions (N=307) with real-world results derived from medical school exams. GPT-4 answered the multiple-choice questions using 2 distinct prompt versions: detailed and short. The answers were analyzed using a quantitative approach and a qualitative approach. Focusing on incorrectly answered questions, we categorized reasoning errors according to the hierarchical framework of Bloom's taxonomy.

RESULTS

GPT-4's performance in answering exam questions yielded a high success rate: 93% (284/307) for the detailed prompt and 91% (278/307) for the short prompt. Questions answered correctly by GPT-4 had a statistically significant higher difficulty than questions answered incorrectly (P=.002 for the detailed prompt and P<.001 for the short prompt). Independent of the prompt, GPT-4's lowest exam performance was 78.9% (15/19), thereby always surpassing the "pass" threshold. Our qualitative analysis of incorrect answers, based on Bloom's taxonomy, showed that errors were primarily in the "remember" (29/68) and "understand" (23/68) cognitive levels; specific issues arose in recalling details, understanding conceptual relationships, and adhering to standardized guidelines.

CONCLUSIONS

GPT-4 demonstrated a remarkable success rate when confronted with psychosomatic medicine multiple-choice exam questions, aligning with previous findings. When evaluated through Bloom's taxonomy, our data revealed that GPT-4 occasionally ignored specific facts (remember), provided illogical reasoning (understand), or failed to apply concepts to a new situation (apply). These errors, which were confidently presented, could be attributed to inherent model biases and the tendency to generate outputs that maximize likelihood.

Chatbot responses suggest that hypothetical biology questions are harder than realistic ones

The biology education literature includes compelling assertions that unfamiliar problems are especially useful for revealing students' true understanding of biology. However, there is only limited evidence that such novel problems have different cognitive requirements than more familiar problems. Here, we sought additional evidence by using chatbots based on large language models as models of biology students. For human physiology and cell biology, we developed sets of realistic and hypothetical problems matched to the same lesson learning objectives (LLOs). Problems were considered hypothetical if (i) known biological entities (molecules and organs) were given atypical or counterfactual properties (redefinition) or (ii) fictitious biological entities were introduced (invention). Several chatbots scored significantly worse on hypothetical problems than on realistic problems, with scores declining by an average of 13%. Among hypothetical questions, redefinition questions appeared especially difficult, with many chatbots scoring as if guessing randomly. These results suggest that, for a given LLO, hypothetical problems may have different cognitive demands than realistic problems and may more accurately reveal students' ability to apply biology core concepts to diverse contexts. The Test Question Templates (TQT) framework, which explicitly connects LLOs with examples of assessment questions, can help educators generate problems that are challenging (due to their novelty), yet fair (due to their alignment with pre-specified LLOs). Finally, ChatGPT's rapid improvement toward expert-level answers suggests that future educators cannot reasonably expect to ignore or outwit chatbots but must do what we can to make assessments fair and equitable.

Exergaming and education: a relational model for games selection and evaluation

Exergaming, or technology-driven physical exercise, has gained popularity in recent years. Its applications include physical education, health promotion, and rehabilitation. Although studies have obtained promising results regarding the positive effects of exergaming, the outcomes of exergaming for different populations remain undetermined. Inconsistencies in the literature on this topic have multiple potential explanations, including the content and demand of the exergames and the capability of the exergamer. A model with a sound theoretical framework is required to facilitate matching between games and gamers. This article proposes a relational model based on a matrix of Bloom's taxonomy of learning domains and the performance components of exergames. Appropriate matching of the physical demands of an exergame and the ability of the exergamer would enhance the effective usage of exergaming for individuals with various needs. This theory-based exergame model is developed to promote the general development, physical status, and psychosocial well-being of students, older adults, and individuals with rehabilitation needs. This model may provide a resource for future research on the application, effectiveness, and design of exergaming.

Students' foundational understanding of chemical reaction in the forensic science bachelor's degree program at the National Autonomous University of Mexico

OBJECTIVE

To compare the understanding of the concept of chemical reaction-as operationalized by Bloom's taxonomy of cognitive levels-of students in forensic science bachelor's degree with that achieved by students majoring in chemistry, as a prerequisite for future professional collaboration and communication.

MATERIALS AND METHODS

Using previously validated and published tests developed to assess students' knowledge, comprehension, and application of the concept of chemical reaction, we explored how conceptual understanding developed in students enrolled in (a) a forensic science degree program in a Mexican public university and in (b) chemistry undergraduate programs offered by the same university, and whether both groups achieved comparable attainment levels.

FINDINGS AND IMPLICATIONS

Despite receiving considerably less chemical instruction, forensic science students achieved comparable levels of conceptual understanding of chemical reaction to those exhibited by chemistry students. This finding is encouraging because it might mean that future forensic scientists could graduate with a solid foundation of chemical knowledge. More research, particularly on the learning of other key concepts, will be needed to verify these initial findings.

There Is More than Multiple Choice: Crowd-Sourced Assessment Tips for Online, Hybrid, and Face-to-Face Environments

Developing effective assessments of student learning is a challenging task for faculty and even more difficult for those in emerging disciplines that lack readily available resources and standards. With the power of technology-enhanced education and accessible digital learning platforms, instructors are also looking for assessments that work in an online format. This article will be useful for all teachers, but especially for entry-level instructors, in addition to more mature instructors who are looking to become more well versed in assessment, who seek a succinct summary of assessment types to springboard the integration of new forms of assessment of student learning into their courses. In this paper, ten assessment types, all appropriate for face-to-face, blended, and online modalities, are discussed. The assessments are mapped to a set of bioinformatics core competencies with examples of how they have been used to assess student learning. Although bioinformatics is used as the focus of the assessment types, the question types are relevant to many disciplines.

Application of Bloom's taxonomy to formative assessment in real-time online classes in Korea

PURPOSE

This study aims to design that using formative assessment as an instructional strategy in real-time online classes, and to explore the application of Bloom's taxonomy in the development of formative assessment items.

METHODS

We designed the instruction using formative assessment in real-time online classes, developed the items of formative assessment, analyzed the items statistically, and investigated students' perceptions of formative assessment through a survey.

RESULTS

It is designed to consist of 2-3 learning outcomes per hour of class and to conduct the formative assessment with 1-2 items after the lecture for each learning outcome. Formative assessment was 31 times in the physiology classes (total 48 hours) of three basic medicine integrated. There were nine "knowledge" items, 40 "comprehension" items, and 55 "application" items. There were 33 items (31.7%) with a correct rate of 80% or higher, which the instructor thought was appropriate. As a result of the survey on students' perceptions of formative assessment, they answered that it was able to concentrate on the class and that it was helpful in achieving learning outcomes.

CONCLUSION

The students focused during class because they had to take formative assessment immediately after the learning outcome lecture. "Integration of lesson and assessments" was maximized by solving the assessment items as well as through the instructor's immediate explanation of answers. Through formative assessment, the students were able to utilize metacognition by learning what content they understood or did not understand. Items that consider Bloom's taxonomy allow students to remember, understand, and apply to clinical contexts.

Flexible e-learning video approach to improve fundus examination skills for medical students: a mixed-methods study

BACKGROUND

Training for the fundus examination using traditional teaching is challenging, resulting in low generalist physicians' confidence in performing the funduscopic examination. There is growing evidence suggesting a flexible e-learning video approach's value in teaching physical examination procedures. However, whether the flexible e-learning video approach is superior to the traditional, face-to-face (F2F) lecture-based teaching for the funduscopic exam and the cognitive processes supporting its effectiveness has not yet been determined.

METHODS

We conducted a sequential explanatory mixed-method study to compare the flexible e-learning video approach's effectiveness versus the F2F lecture-based approach for teaching the funduscopic exam to medical students at Chiba University in Japan. Medical students were randomly assigned to either a flexible e-learning video approach group or a F2F lecture approach group. We then quantitatively measured the diagnostic accuracy of funduscopic findings before and after attending the specific classrooms. Next, we conducted student focus groups to explore the students' thinking processes in the flexible e-learning video approach vs. the F2F lecture-based teaching of fundus examination. The qualitative data were analyzed using the qualitative content analysis method.

RESULTS

The mean diagnostic accuracy scores in the post-test significantly increased from pre-test in the intervention group (36.6 to 63.4%, p < 0.001). Post-post comparisons across the two groups revealed a significant difference (intervention group 63.4% vs. control group 34.6%, p < 0.001). Six semi-structured focused group interviews were conducted (n = 36). In the flexible e-learning video approach group, we identified ten categories corresponding to four levels of the revised Bloom's taxonomy: remember, understand, apply, analyze. Five categories were identified in the traditional F2F lecture approach group corresponding to three revised Bloom's taxonomy levels: understand, apply, analyze. Interrater reliability was substantial (Cohen's kappa = 0.81).

CONCLUSIONS

Teaching medical students funduscopic examination using the flexible e-learning video approach leads to improved diagnostic accuracy of funduscopic examinations. The flexible e-learning video teaching method enabled higher cognitive activity levels than the traditional, lecture-based classroom, as assessed using the revised Bloom's taxonomy.

TRIAL REGISTRATION

This study was registered with the University Hospital Medical Information Network Clinical Trials Registry on 08/02/2020 (Unique trial number: UMIN 000039434 ).

A Health eLearning Ontology and Procedural Reasoning Approach for Developing Personalized Courses to Teach Patients about Their Medical Condition and Treatment

We propose a methodological framework to support the development of personalized courses that improve patients' understanding of their condition and prescribed treatment. Inspired by Intelligent Tutoring Systems (ITSs), the framework uses an eLearning ontology to express domain and learner models and to create a course. We combine the ontology with a procedural reasoning approach and precompiled plans to operationalize a design across disease conditions. The resulting courses generated by the framework are personalized across four patient axes-condition and treatment, comprehension level, learning style based on the VARK (Visual, Aural, Read/write, Kinesthetic) presentation model, and the level of understanding of specific course content according to Bloom's taxonomy. Customizing educational materials along these learning axes stimulates and sustains patients' attention when learning about their conditions or treatment options. Our proposed framework creates a personalized course that prepares patients for their meetings with specialists and educates them about their prescribed treatment. We posit that the improvement in patients' understanding of prescribed care will result in better outcomes and we validate that the constructs of our framework are appropriate for representing content and deriving personalized courses for two use cases: anticoagulation treatment of an atrial fibrillation patient and lower back pain management to treat a lumbar degenerative disc condition. We conduct a mostly qualitative study supported by a quantitative questionnaire to investigate the acceptability of the framework among the target patient population and medical practitioners.

Identification-Based Multiple-Choice Assessments in Anatomy can be as Reliable and Challenging as Their Free-Response Equivalents

Multiple-choice (MC) anatomy "spot-tests" (identification-based assessments on tagged cadaveric specimens) offer a practical alternative to traditional free-response (FR) spot-tests. Conversion of the two spot-tests in an upper limb musculoskeletal anatomy unit of study from FR to a novel MC format, where one of five tagged structures on a specimen was the answer to each question, provided a unique opportunity to assess the comparative validity and reliability of FR- and MC-formatted spot-tests and the impact on student performance following the change of test format to MC. Three successive year cohorts of health science students (n = 1,442) were each assessed by spot-tests formatted as FR (first cohort) or MC (following two cohorts). Comparative question difficulty was assessed independently by three examiners. There were more higher-order cognitive skill questions and more of the course objectives tested in the MC-formatted tests. Spot-test reliability was maintained with Cronbach's alpha reliability coefficients ≥ 0.80 and 80% of the MC items of high quality (having point-biserial correlation coefficients > 0.25). These results also demonstrated guessing was not an issue. The mean final score for the MC-formatted cohorts increased by 4.9%, but did not change for the final theory examination that was common to all three cohorts. Subgroup analysis revealed that the greatest change in spot-test marks was for the lower-performing students. In conclusion, our results indicate spot-tests formatted as MC are suitable alternatives to FR tests. The increase in mean scores for the MC-formatted spot-tests was attributed to the lower demand of the MC format.

Does case-based blended-learning expedite the transfer of declarative knowledge to procedural knowledge in practice?

BACKGROUND

Case-Based Learning (CBL) has seen widespread implementation in undergraduate education since the early 1920s. Ample data has shown CBL to be an enjoyable and motivational didactic tool, and effective in assisting the expansion of declarative and procedural knowledge in academia. Although a plethora of studies apply multiple choice questions (MCQs) in their investigation, few studies measure CBL or case-based blended learning (CBBL)-mediated changes in students' procedural knowledge in practice or employ comparison or control groups in isolating causal relationships.

METHODS

Utilizing the flexibilities of an e-learning platform, a CBBL framework consisting of a) anonymized patient cases, b) case-related textbook material and online e-CBL modules, and c) simulated patient (SP) contact seminars, was developed and implemented in multiple medical fields for undergraduate medical education. Additionally, other fields saw a solo implementation of e-CBL in the same format. E- cases were constructed according to the criteria of Bloom's taxonomy. In this study, Objective Structured Clinical Examination (OSCE) results from 1886 medical students were analyzed in total, stratified into the following groups: medical students in 2013 (n = 619) before CBBL implementation, and after CBBL implementation in 2015 (n = 624) and 2016 (n = 643).

RESULTS

A significant improvement (adjusted p = .002) of the mean OSCE score by 1.02 points was seen between 2013 and 2015 (min = 0, max = 25).

CONCLUSION

E-Case-Based Learning is an effective tool in improving performance outcomes and may provide a sustainable learning platform for many fields of medicine in future.

Comparison of level of cognitive process between case-based items and non-case-based items of the interuniversity progress test of medicine in the Netherlands

PURPOSE

It is assumed that case-based questions require higher order cognitive processing, whereas questions that are not case-based require lower order cognitive processing. In this study, we investigated to what extent case-based questions and questions that are not case-based, relate to Bloom's taxonomy.

METHODS

In this article, 4800 questions of the Progress Test were classified whether it was a case-based question and the level of Bloom's taxonomy. Lower-order questions require students to remember or/and basically understand the knowledge. Higher-order questions require students to apply, analyze, or/and evaluate. A phi-coefficient was calculated to investigate the relations between the presence of case-based questions and the required level of cognitive processing.

RESULTS

Our results demonstrated that case-based questions were measuring higher levels of cognitive processing in 98.1% of the questions. Of the non-case-based questions, 33.7% required a higher level of cognitive processing. The phi-coefficient demonstrated a significant moderate correlation between the presence of a patient case in a question and its required level of cognitive processing (phi-coefficient = 0.55, p<0.001).

CONCLUSION

Medical teachers should be aware of the association between item formats (case-based versus non-case-based) and the cognitive processes they elicit in order to meet a certain balance in a test, taking the learning objectives as well as the test difficulty into account.

An effective and novel method for teaching applied facial anatomy and related procedural skills to esthetic physicians

BACKGROUND

An understanding of facial anatomy is crucial for the safe practice of nonsurgical facial esthetic procedures. Contextual learning, aided with instructional design, enhances the trainees' overall learning experience and retention, and makes a positive impact on the performance of procedural skills. The present study aimed to develop a teaching approach based on Bloom's taxonomy involving cognitive, affective, and psychomotor learning domains.

MATERIALS AND METHODS

The practicability of Assess & Aware, Demonstrate, Decode, Act & Accomplish, Perform, Teach & Test (ADDAPT), a new approach to teaching applied facial anatomy and procedural skills to esthetic physicians in a large group setting, was evaluated in this study. Study participants were from two cohorts (n=124) who underwent 2 days of applied anatomy training in Singapore. Pre- and post-course multiple choice questions and objective structured practical examination were conducted to measure the effectiveness and applicability of the teaching model. Expert raters, table demonstrators, and participants rated the steps involved in the ADDAPT model on an 11-point Likert scale.

RESULTS

Pre- and post-course evaluation mean scores for multiple choice questions were 17.32 (SD ±3.36) and 22.61 (SD ±1.77) and for objective structured practical examination were 24.53 (SD ±4.63) and 43.57 (SD ±5.35), respectively (P<0.001). Inter-rater agreement, expressed as the intraclass correlation coefficient, was 0.91 (95% CI: 0.62-0.98) for expert raters and 0.90 (95% CI: 0.78-0.97) for table demonstrators, which reflects the real strength of sound educational practice. The trainees well accepted the model and found the sessions intellectually stimulating. Trainees' feedback stated that the learning experience was enhanced by the repeated observation and constructive feedback provided by the tutors.

CONCLUSION

The ADDAPT model is practical to instruct a large group of trainees in clinical anatomy and procedural skill training. This approach to instructional design may be feasible and transferable to other areas of psychomotor skill training in medical education.

Engagement and higher order skill proficiency of students completing a medical physiology course in three diverse learning environments

The primary aim of this study was to determine whether levels of student engagement, higher order skill proficiency, and knowledge acquisition demonstrated by medical students would differ when completing the same course in three diverse learning environments. Following Institutional Review Board approval, 56 first-year medical students, registered at the same medical school but attending class at three different campus centers, were enrolled in the study. All participants were completing a medical physiology course that utilized the same learning objectives but relied on different faculty incorporating diverse methodologies (percentage of class devoted to active learning strategies), course format (6-wk block vs. 17-wk semester), and student attendance. Students completed a validated survey of student engagement (SSE), a proctored online problem-based assessment of higher order skill proficiency [Collegiate Learning Assessment (CLA+); http://cae.org ], and the National Board of Medical Examiners (NBME) Physiology subject exam. In this limited sample, results indicate no significant differences between campus sites for any of the variables assessed. Levels of engagement were lower than expected compared with published values for graduate students. Higher order skill proficiency assessed by CLA+ was significantly higher than values reported for college seniors nationally. Surprisingly, SSE offered no prediction of performance on CLA+ or NBME, as there were no significant correlations between variables. These data indicate that, although first-year medical students may not perceive themselves as highly engaged, they are adept in using higher order skills and excel in meeting course learning objectives, regardless of learning environment.

Distractor Efficiency in an Item Pool for a Statistics Classroom Exam: Assessing Its Relation With Item Cognitive Level Classified According to Bloom's Taxonomy

Multiple-choice items are one of the most commonly used tools for evaluating students' knowledge and skills. A key aspect of this type of assessment is the presence of functioning distractors, i.e., incorrect alternatives intended to be plausible for students with lower achievement. To our knowledge, no work has investigated the relationship between distractor performance and the complexity of the cognitive task required to give the correct answer. The aim of this study was to investigate this relation, employing the first three levels of Bloom's taxonomy (Knowledge, Comprehension, and Application). Specifically, it was hypothesized that items classified into a higher level of Bloom's classification would show a greater number of functioning distractors. The study involved 174 items administered to a sample of 848 undergraduate psychology students during their statistics exam. Each student received 30 items randomly selected from the 174-item pool. The bivariate results mainly supported the authors' hypothesis: the highest percentage of functioning distractors was observed among the items classified into the Application category (η² = 0.024 and Phi = 0.25 for the dichotomized measure). When the analysis controlled for other item features, it lost statistical significance, partly because of the confounding effect of item difficulty.

Medical students create multiple-choice questions for learning in pathology education: a pilot study

BACKGROUND

Medical students facing high-stakes exams want study resources that have a direct relationship with their assessments. At the same time, they need to develop the skills to think analytically about complex clinical problems. Multiple-choice questions (MCQs) are widely used in medical education and can promote surface learning strategies, but creating MCQs requires both in-depth content knowledge and sophisticated analytical thinking. Therefore, we piloted an MCQ-writing task in which students developed MCQs for their peers to answer.

METHODS

Students in a fourth-year anatomic pathology course (N = 106) were required to write MCQs using the PeerWise platform. Students created two MCQs for each of four topic areas and the MCQs were answered, rated and commented on by their classmates. Questions were rated for cognitive complexity and a paper-based survey was administered to investigate whether this activity was acceptable, feasible, and whether it promoted desirable learning behaviours in students.

RESULTS

Students were able to create cognitively challenging MCQs: 313/421 (74%) of the MCQs which we rated required the respondent to apply or analyse pathology knowledge. However, students who responded to the end-of-course questionnaire (N = 62) saw the task as having little educational value. Students found PeerWise easy to use, and indicated that they read widely to prepare questions and monitored the quality of their questions. They did not, however, engage in extensive peer feedback via PeerWise.

CONCLUSIONS

Our study showed that the MCQ writing task was feasible and engaged students in self-evaluation and synthesising information from a range of sources, but it was not well accepted and did not strongly engage students in peer-learning. Although students were able to create complex MCQs, they found some aspects of the writing process burdensome and tended not to trust the quality of each other's MCQs. Because of the evidence this task did promote deep learning, it is worth continuing this mode of teaching if the task can be made more acceptable to students.

The Effect of Movement on Cognitive Performance

The study examines the relationship between walking, cognitive, and academic skills. Students from elementary, middle, high school, and college were required to walk for 10 min prior to completing feature detection, Simon-type memory, and mathematical problem-solving tasks. Participants were counterbalanced to remove a time bias. Ten minutes of walking had a significant positive effect on Simon-type memory and critical feature-detection tasks among all age groups. Separately, with mathematical problem-solving ability, higher performing high-school students demonstrated significant positive effects on mathematical reasoning tasks based on the Bloom Taxonomy. However, poorly achieving high-school students performed significantly better than those with higher grades in mathematics on tests of mathematical problem-solving ability based on the Bloom's Taxonomy. The study indicates that there is justification to employ relatively simple means to effect lifestyle, academic, and cognitive performance.

Development of an interprofessional lean facilitator assessment scale

High reliability is important for optimising quality and safety in healthcare organisations. Reliability efforts include interprofessional collaborative practice (IPCP) and Lean quality/process improvement strategies, which require skilful facilitation. Currently, no validated Lean facilitator assessment tool for interprofessional collaboration exists. This article describes the development and pilot evaluation of such a tool; the Interprofessional Lean Facilitator Assessment Scale (ILFAS), which measures both technical and 'soft' skills, which have not been measured in other instruments. The ILFAS was developed using methodologies and principles from Lean/Shingo, IPCP, metacognition research and Bloom's Taxonomy of Learning Domains. A panel of experts confirmed the initial face validity of the instrument. Researchers independently assessed five facilitators, during six Lean sessions. Analysis included quantitative evaluation of rater agreement. Overall inter-rater agreement of the assessment of facilitator performance was high (92%), and discrepancies in the agreement statistics were analysed. Face and content validity were further established, and usability was evaluated, through primary stakeholder post-pilot feedback, uncovering minor concerns, leading to tool revision. The ILFAS appears comprehensive in the assessment of facilitator knowledge, skills, abilities, and may be useful in the discrimination between facilitators of different skill levels. Further study is needed to explore instrument performance and validity.

Redesigning a course to help students achieve higher-order cognitive thinking skills: from goals and mechanics to student outcomes

Here we describe a 4-yr course reform and its outcomes. The upper-division neurophysiology course gradually transformed from a traditional lecture in 2004 to a more student-centered course in 2008, through the addition of evidence-based active learning practices, such as deliberate problem-solving practice on homework and peer learning structures, both inside and outside of class. Due to the incremental nature of the reforms and absence of pre-reform learning assessments, we needed a way to retrospectively assess the effectiveness of our efforts. To do this, we first looked at performance on 12 conserved exam questions. Students performed significantly higher post-reform on questions requiring lower-level cognitive skills and those requiring higher-level cognitive skills. Furthermore, student performance on conserved questions was higher post-reform in both the top and bottom quartiles of students, although lower-quartile student performance did not improve until after the first exam. To examine student learning more broadly, we also used Bloom's taxonomy to quantify a significant increase in the Bloom's level of exams, with students performing equally well post-reform on exams that had over twice as many questions at higher cognitive skill levels. Finally, we believe that four factors provided critical contributions to the success of the course reform, including: transformation efforts across multiple course components, alignment between formative and evaluative course materials, student buy-in to course instruction, and instructional support. This reform demonstrates both the effectiveness of incorporating student-centered, active learning into our course, and the utility of using Bloom's level as a metric to assess course reform.

Making authentic science accessible-the benefits and challenges of integrating bioinformatics into a high-school science curriculum

Despite the central place held by bioinformatics in modern life sciences and related areas, it has only recently been integrated to a limited extent into high-school teaching and learning programs. Here we describe the assessment of a learning environment entitled 'Bioinformatics in the Service of Biotechnology'. Students' learning outcomes and attitudes toward the bioinformatics learning environment were measured by analyzing their answers to questions embedded within the activities, questionnaires, interviews and observations. Students' difficulties and knowledge acquisition were characterized based on four categories: the required domain-specific knowledge (declarative, procedural, strategic or situational), the scientific field that each question stems from (biology, bioinformatics or their combination), the associated cognitive-process dimension (remember, understand, apply, analyze, evaluate, create) and the type of question (open-ended or multiple choice). Analysis of students' cognitive outcomes revealed learning gains in bioinformatics and related scientific fields, as well as appropriation of the bioinformatics approach as part of the students' scientific 'toolbox'. For students, questions stemming from the 'old world' biology field and requiring declarative or strategic knowledge were harder to deal with. This stands in contrast to their teachers' prediction. Analysis of students' affective outcomes revealed positive attitudes toward bioinformatics and the learning environment, as well as their perception of the teacher's role. Insights from this analysis yielded implications and recommendations for curriculum design, classroom enactment, teacher education and research. For example, we recommend teaching bioinformatics in an integrative and comprehensive manner, through an inquiry process, and linking it to the wider science curriculum.

Incorporation of Bloom's taxonomy into multiple-choice examination questions for a pharmacotherapeutics course

OBJECTIVE

To incorporate Bloom's taxonomy into multiple-choice examination questions in a pharmacotherapeutics course and assess its effectiveness in detecting areas of improvement in learning.

DESIGN

Bloom's taxonomy was incorporated into examination questions through a multi-step process: Sample questions representing each learning domain within Bloom's taxonomy (knowledge, comprehension, application, analysis, synthesis, and evaluation) were introduced to students during lecture presentations and discussions. Quiz and examination containing questions categorized according to Bloom's taxonomy were administered to students. During review sessions following each quiz or examination, the categorization of each question was provided to students and feedback from students was gathered.

ASSESSMENT

The effect of the 5 types of test questions on the correct response fraction and discrimination index was determined after combining synthesis and evaluation. Correct response fractions for knowledge, comprehension, and application questions were significantly higher than those for analysis and synthesis/evaluation questions (p<0.05). However, discrimination index for application and synthesis/evaluation questions were significantly higher than those for knowledge and comprehension questions (p<0.05). In interviews with students who had requested learning assistance, the majority realized the importance of critical-thinking skills in the learning process.

CONCLUSION

Well-designed multiple-choice questions incorporating different learning domains of Bloom's taxonomy may be a potential method of assessing critical-thinking skills in large classes of students.