Differences in Multiple-Choice Questions of Opposite Stem Orientations Based on a Novel Item Quality Measure

Objective. To determine whether there are differences in the performance and quality of multiple-choice items with opposite stem orientations (positive or negative), based on a novel item quality measure and conventional psychometric parameters.Methods. A retrospective study was conducted on multiple-choice assessment items used in years two and three of pharmacy school for pharmacotherapy and related courses administered between August 2018 and December 2019. Conventional psychometric parameters (difficulty and discrimination indices), average response time, nonfunctional distractor percentage, and a novel measure of item quality of negatively worded items were compared with those of control items, namely positively worded items (n=103 each). This novel measure uses difficulty and discrimination in tandem for the decision to reject, review, or retain items in an assessment. Statistical analyses were performed on continuous and categorical variables, on the relationship between difficulty and discrimination, and on differences in correlation coefficients between positively and negatively worded items.Results. Stem orientation was not significantly associated with the novel measure of item quality. Also, there were no significant differences between positively and negatively worded items in any of the psychometric parameters. There were significant, negative correlations between difficulty and discrimination indices in both groups, and the correlation coefficients were significantly stronger in positively versus negatively worded items.Conclusion. Items with opposite stem orientations show no differences in the novel item quality measure nor in conventional measures of performance and quality, except in difficulty-discrimination relationships. This suggests that negatively worded items should be used when necessary, but cautiously.

Evaluating knowledge and attitudes scales for the care of older adults among nursing students in Ghana

BACKGROUND

Understanding nursing students' knowledge about and attitudes toward older adults' using context-specific survey instruments can help to identify and design effective learning and teaching materials to improve the care for persons 60 years and above. However, there are no validated instruments to examine nursing students' knowledge and attitudes toward the care for older adults in the African context. The study aimed to evaluate the items on the Knowledge about Older Patients Quiz and Kogan's Attitudes towards Old People Scale suitable for the African context.

METHODS

A cross-sectional study was conducted using second-and third-year nursing students from two public Nursing Training Institutions in Ghana. Using Sahin's rule of sample size estimate of at least 150 participants for unidimensional dichotomous scales, 170 nursing students were recruited to participate after an information session in their classrooms. Data were collected from December 2019-March 2020 using the Knowledge about Older Patients Quiz and Kogan's Attitudes Towards Old People Scale. Item response theory was employed to evaluate the Knowledge about Older Patients Quiz difficulty level and discrimination indices. Corrected item-to-total correlation analysis was conducted for Kogan's Attitudes towards Old People Scale. The internal consistency for both scales was examined.

RESULTS

Of the 170 participants, 169 returned completed surveys. The mean age of participants was 21 years (SD = 3.7), and (54%) were female. Of the 30-items of the Knowledge about Older Patients Quiz, seven items were very difficult for most students to choose the correct response, and one was easy, as most of the students chose the correct response. Although 22 items demonstrated appropriate difficulty level, discrimination indices were used to select the final 15- items that discriminated moderately between upper and lower 25% performing students. The Kuder-Richardson-20 reliability was. 0.30, which was low. Considering Kogan's Attitudes towards Old People scale, 10-items were removed following negative and low corrected item-to-total correlation and a high Alpha coefficient if items were deleted. The final 22-items had a Cronbach alpha coefficient of 0.65, which was moderately satisfactory.

CONCLUSION

Evaluation of the scales demonstrated essential content validity and moderate internal consistency for the context of our study. Further research should focus on ongoing context-specific refinement of the survey instruments to measure nursing students' knowledge about and attitudes toward caring for older adults in the African context.

An Adaptive Blended Learning Model for the Implementation of an Integrated Medical Neuroscience Course During the Covid-19 Pandemic

The implementation of an integrated medical neuroscience course by technologically pivoting an in-person neuroscience course to online using an adaptive blended method may provide a unique approach for teaching a medical neuroscience course during the Covid-19 pandemic. An adaptive blended learning method was developed in response to the requirements necessitated by the Covid-19 pandemic. This model combined pedagogical needs with digital technology using online learning activities to implement student learning in a medical neuroscience course for year one medical students. This approach provided medical students with an individually customized learning opportunity in medical neuroscience. The students had the complete choice to engage the learning system synchronously or asynchronously and learn neuroscience materials at different locations and times in response to the demands required to deal with the pandemic. Students' performance in summative and formative examinations of the adaptive blended learning activities were compared with the previous performance obtained the previous year when the contents of the medical neuroscience course were implemented using the conventional "face-to-face" learning approach. While the cohort of our students in 2019 and 2020 changed, the contents, sessions, volume of material, and assessment were constant. This enabled us to compare the results of the 2019 and 2020 classes. Overall, students' performance was not significantly different between the adaptive blended learning and the in-person approach. More students scored between 70% and 79% during the adaptive blended learning compared with in-class teaching, while more students scored between 80% and 89% during the in-person learning than during the adaptive blended learning. Finally, the percentage of students that scored >90% was not significantly different for both Years 2019 and 2020. The adaptive blended learning approach was effective in enhancing academic performance for high-performing medical students. It also permitted the early identification of underachieving students, thereby serving as an early warning sign to permit timely intervention.

Working Towards Competence: A Novel Application of Borderline Regression to a Task-Specific Checklist for Technical Skills in Novices

OBJECTIVE

Demonstrated competence through frequent assessment is an expected goal for progressive development in competency-based medical education curricula. The Objective Structured Assessment of Technical Skill (OSATS) is considered a valid method of formative assessment, but in few instances have standards been set for determining competence. The present study used borderline regression methods to examine standard setting of performance on a complex technical task with novices assessed using an OSATS checklist.

METHODS

This was a single institution prospective single arm experimental design study. Participants were 58 non-medical undergraduate students with no previous surgical experience, who observed a computer-based training module on end-to-side vascular anastomosis. Subsequently, participants were provided two 20-minute training sessions, two weeks apart where they received expert feedback whilst performing the task on a low-fidelity model. After each training session, participants completed the task unaided. Sessions were recorded and assessed using an OSATS checklist retrospectively by experts.

RESULTS

Paired t-test analyses indicate that for both the checklist total score (t(52) = 8.05, p < 0.001) and the global rating score (t(53) = 8.15, p < 0.001), individuals performed significantly better in Phase 2. Borderline regression analyses indicated that in Phase 1 (R² = .60) and Phase 2 (R² = .75), the OSATS checklist could adequately capture variation in performance in novices. Further, the checklist could reliably classify novices at three of the five global rating performance levels. Pass rates determined by regression equations improved from Phase 1 to Phase 2 on all global rating levels.

CONCLUSIONS

With the increasing focus on competency-based medical education, it is imperative that training programs have the capacity to accurately assess outcomes and set minimum performance standards. Borderline regression methods can accurately differentiate novice learners of varying performance levels before and after training on a complex technical skill task using an OSATS checklist.

Validation of the General Medicine in-Training Examination Using the Professional and Linguistic Assessments Board Examination Among Postgraduate Residents in Japan

Purpose

In Japan, the General Medicine In-training Examination (GM-ITE) was developed by a non-profit organization in 2012. The GM-ITE aimed to assess the general clinical knowledge among residents and to improve the training programs; however, it has not been sufficiently validated and is not used for high-stake decision-making. This study examined the association between GM-ITE and another test measure, the Professional and Linguistic Assessments Board (PLAB) 1 examination.

Methods

Ninety-seven residents who completed the GM-ITE in fiscal year 2019 were recruited and took the PLAB 1 examination in Japanese. The association between two tests was assessed using the Pearson product-moment statistics. The discrimination indexes were also assessed for each question.

Results

A total of 91 residents at 17 teaching hospitals were finally included in the analysis, of whom 69 (75.8%) were women and 59 (64.8%) were postgraduate second year residents. All the participants were affiliated with community hospitals. Positive correlations were demonstrated between the GM-ITE and the PLAB scores (r = 0.58, p < 0.001). The correlations between the PLAB score and the scores in GM-ITE categories were as follows: symptomatology/clinical reasoning (r = 0.54, p < 0.001), physical examination/procedure (r = 0.38, p < 0.001), medical interview/professionalism (r = 0.25, p < 0.001), and disease knowledge (r = 0.36, p < 0.001). The mean discrimination index of each question of the GM-ITE (mean ± SD; 0.23 ± 0.15) was higher than that of the PLAB (0.16 ± 0.16; p = 0.004).

Conclusion

This study demonstrates incremental validity evidence of the GM-ITE to assess the clinical knowledge acquisition. The results indicate that GM-ITE can be widely used to improve resident education in Japan.

Development and validation of a written credentialing examination for overseas-educated dietitians

AIM

Health professionals seeking employment in foreign countries are commonly required to undertake competency assessment in order to practice. The present study aims to outline the development and validation of a written examination for Dietetic Skills Recognition (DSR), to assess the knowledge, skills, capabilities and professional judgement of overseas-educated dietitians against the competency standards applied to dietetic graduates in Australia.

METHODS

The present study reviews the design, rationale, validation and outcomes of a multiple choice question (MCQ) written examination for overseas-educated dietitians based on 5 years of administration. The validity of the exam is evaluated using Messick's validity framework, which focuses on five potential sources of validity evidence-content, internal structure, relationships with other variables, response process and consequences. The reference point for the exam pass mark or "cutscore" is the minimum standard required for safe practice.

RESULTS

In total, 114 candidates have completed the MCQ examination at least once, with an overall pass rate of 52% on the first attempt. Pass rates are higher from countries where dietetic education more closely reflects the Australian model. While the pass rate for each exam tends to vary with each cohort, the cutscore has remained relatively stable over eight administrations.

CONCLUSIONS

The findings provide important data supporting the validity of the MCQ exam. A more complete evaluation of the validity of the exam must be sought within the context of the whole DSR program of assessment. The DSR written component may serve as a model for use of the MCQ format for dietetic and other professional credentialing organisations.