Multiple choice questions can be designed or revised to challenge learners' critical thinking

A Massive Open Online Course Delivered Just-in-Time: A Cohort Study

BACKGROUND

Just-in-time education enables knowledge improvement during health crises. This study was conducted to determine whether nurses' knowledge improved after participating in an innovative online education intervention designed to upskill subacute nurses to prepare for a health crisis.

METHOD

The intervention was developed by university nursing academics and supported and promoted through a partnership with a government health department and clinical nurses. Quantitative data obtained with a pretest/post-test knowledge survey were analyzed with independent samples and paired t tests.

RESULTS

Statistically significant improvements were seen in nurses' knowledge according to qualification level. The mean difference for knowledge improvement for registered nurses was 1.90, SD = 2.54, and the mean difference for knowledge improvement for enrolled nurses was 2.93, SD = 2.43 (mean difference = 1.03; 95% CI [0.04, 2.03].

CONCLUSION

The intervention was highly effective during the emergent and ongoing phases of the coronavirus disease 2019 pandemic. This study showed that delivering just-in-time education can improve nurses' knowledge. [J Contin Educ Nurs. 2023;54(10):472-479.].

The Assessment Evaluation Rubric: Promoting Learning and Learner-Centered Teaching through Assessment in Face-to-Face or Distanced Higher Education

It is common to create courses for the higher education context that accomplish content-driven teaching goals and then develop assessments (quizzes and exams) based on the target content. However, content-driven assessment can tend to support teaching- or teacher-centered instruction. Adult learning and educational psychology theories suggest that instead, assessment should be aligned with learning, not teaching, objectives. To support the alignment of assessments with instruction in higher education, the Assessment Evaluation Rubric (AER) was developed. The AER can be utilized to guide the development and evaluation/revision of assessments that are already used. The AER describes, or permits the evaluation of, four features of an assessment: its general alignment with learning goal(s), whether the assessment is intended to/effective as formative or summative, whether some systematic approach to cognitive complexity is reflected, and whether the assessment (instructions as well as results) itself is clearly interpretable. Each dimension (alignment, utility, complexity, and clarity) has four questions that can be rated as present/absent. Other rating methods can also be conceptualized for the AER’s 16 questions, depending on the user’s intent. Any instructor can use the AER to evaluate their own assessments and ensure that they—or new assessments in development—will promote learning and learner-centered teaching. As instructors shift from face-to-face toward virtual or hybrid teaching models, or as they shift online instruction (back) to face-to-face teaching, it creates an ideal opportunity to ensure that assessment is optimizing learning and is valid for instructional decision-making.

Examining Bloom's Taxonomy in Multiple Choice Questions: Students' Approach to Questions

BACKGROUND

Analytic thinking skills are important to the development of physicians. Therefore, educators and licensing boards utilize multiple-choice questions (MCQs) to assess these knowledge and skills. MCQs are written under two assumptions: that they can be written as higher or lower order according to Bloom's taxonomy, and students will perceive questions to be the same taxonomical level as intended. This study seeks to understand the students' approach to questions by analyzing differences in students' perception of the Bloom's level of MCQs in relation to their knowledge and confidence.

METHODS

A total of 137 students responded to practice endocrine MCQs. Participants indicated the answer to the question, their interpretation of it as higher or lower order, and the degree of confidence in their response to the question.

RESULTS

Although there was no significant association between students' average performance on the content and their question classification (higher or lower), individual students who were less confident in their answer were more than five times as likely (OR = 5.49) to identify a question as higher order than their more confident peers. Students who responded incorrectly to the MCQ were 4 times as likely to identify a question as higher order than their peers who responded correctly.

CONCLUSIONS

The results suggest that higher performing, more confident students rely on identifying patterns (even if the question was intended to be higher order). In contrast, less confident students engage in higher-order, analytic thinking even if the question is intended to be lower order. Better understanding of the processes through which students interpret MCQs will help us to better understand the development of clinical reasoning skills.

Validation of data using RASCH analysis in a tool measuring changes in critical thinking in nursing students

BACKGROUND

In Australia there is no national examination to assess student competency with respect to critical thinking skills on graduation from undergraduate registered nurse program.

OBJECTIVES

To describe the use of Rasch analysis for the validation of a tool that measures critical thinking in nursing students in Australia.

DESIGN

A cross sectional survey was used to assess students' critical thinking.

SETTINGS

One Australian university.

PARTICIPANTS

Seventy-four first year and third year nursing students participated in the project.

METHODS

Nursing students at one university were requested to undertake a multiple choice question survey to determine their level of critical thinking. The survey tool was validated using Rasch analysis software.

RESULTS

Rasch analysis enabled identification of the difficulty of each question, enabling the assessment to identify difference between students and for each student cohort. The use of Rasch analysis demonstrated marked differences in the students' critical thinking abilities, showing higher levels of critical thinking for third year students with greater ability to answer the more difficult questions.

CONCLUSION

Rasch analysis enables the matching of test questions to the cognitive skill level of students, and is a useful tool for validating multiple choice questionnaires. Use of this method of analysis can provide a robust assessment in which questions are matched to the level of the learner and enable differentiation between high and low achieving students.

Development and validation of a written credentialing examination for overseas-educated dietitians

AIM

Health professionals seeking employment in foreign countries are commonly required to undertake competency assessment in order to practice. The present study aims to outline the development and validation of a written examination for Dietetic Skills Recognition (DSR), to assess the knowledge, skills, capabilities and professional judgement of overseas-educated dietitians against the competency standards applied to dietetic graduates in Australia.

METHODS

The present study reviews the design, rationale, validation and outcomes of a multiple choice question (MCQ) written examination for overseas-educated dietitians based on 5 years of administration. The validity of the exam is evaluated using Messick's validity framework, which focuses on five potential sources of validity evidence-content, internal structure, relationships with other variables, response process and consequences. The reference point for the exam pass mark or "cutscore" is the minimum standard required for safe practice.

RESULTS

In total, 114 candidates have completed the MCQ examination at least once, with an overall pass rate of 52% on the first attempt. Pass rates are higher from countries where dietetic education more closely reflects the Australian model. While the pass rate for each exam tends to vary with each cohort, the cutscore has remained relatively stable over eight administrations.

CONCLUSIONS

The findings provide important data supporting the validity of the MCQ exam. A more complete evaluation of the validity of the exam must be sought within the context of the whole DSR program of assessment. The DSR written component may serve as a model for use of the MCQ format for dietetic and other professional credentialing organisations.