Validation and perception of a key feature problem examination in neurology

Innovative assesment strategies: image based key feature questions for radiology postgraduate trainees

BACKGROUND

Innovative assessment strategies are essential for determining clinical understanding in the evolving field of health profession education. Key feature questions (KFQs) have been developed as assessment tools to assess the clinical understanding of students. The purpose of this study is to determine the effectiveness of image-based key feature questions (IBKFQs) compared with traditional multiple-choice questions (MCQs) in radiology examinations. Additionally, this study aims to determine the correlation between test scores obtained in both test formats.

METHOD

This quasi experimental, correlation study was conducted from September to December 2021 at a public medical university in Karachi, Pakistan. Thirty radiology residents from various training years participated in the study. Each resident completed a comprehensive written assessment comprising 50 MCQs and 50 IB-KFQs as part of their internal evaluation at the end of a module.

RESULTS

Out of thirty residents, 28 (93.3%) were females. The reliability score and Cronbach's alpha were 0.944 for the MCQs and 0.881 for the IB-KFQs. Spearman's correlation coefficient revealed a positive correlation between the MCQ and IB-KFQ scores (rho = 0.823, p < 0.001). The mean scores were similar for the IB-KFQs (29.24 ± 6.31) and MCQs (28.93 ± 11.41).

CONCLUSION

The findings of this study indicate that incorporating IB-KFQs alongside MCQs in written assessments of radiology residents is feasible. IB-KFQs offer a focused evaluation of critical skills such as film analysis, interpretation, and report writing. By complementing traditional MCQs, IB-KFQs enhance the assessment process.

Neurology Undergraduate Medical Education: A Scoping Review

PURPOSE

To map the current literature on undergraduate neurology medical education and research. Recommendations for future undergraduate neurology education and research are described.

METHOD

PRISMA-Scoping Review guidelines and Arksey and O'Malley's methodological framework are followed. Four databases and gray literature was searched with Oxford Evidence-Based Medicine level of evidence applied. A thematic framework was used to identify the main study outcomes. A narrative description and quantitative frequency analysis were used for results.

RESULTS

Nine-hundred and twenty-two articles were retrieved, 102 studies met the inclusion criteria. We identified four main study outcomes using a thematic framework. Our review found that (1) the main undergraduate neurology teaching styles are didactic and experiential teaching methods. (2) Research design of undergraduate neurology teaching is heterogenous. (3) The outcome measures most frequently used in undergraduate neurology research are student perception and knowledge.

CONCLUSION

Undergraduate neurology education research is challenging due to the heterogeneity in research design and teaching methodology. Evidence-based guidelines are limited. This gap in the literature represents an opportunity to develop tailored guidelines for undergraduate neurology education and research.

Development of Immediate Self-Feedback Very Short Answer Questions: Implementing Testlet Response Theory in Formative Examinations Across Multiple Occasions

Background

The novel Immediate Self-feedback Very Short Answer Question (ISF-VSAQ) format exhibited testlet effect due to its instant feedback feature. This study utilized the testlet-as-a-polytomous-item-modeling (TPIM), an extension of item response theory, to tackle the testlet-related challenges.

Methods

Ninety-four pre-clinical students participated in two ten-item ISF-VSAQ exams on cardiovascular drugs based on a parallel blueprint. Each question involves clinical vignettes followed by expected answers, self-marking, and understanding levels. Messick's validity framework guided the collection of validity evidence.

Results

Validity evidence included five sources: (1) Content: Experts reviews and alignment with the standard examination blueprint. (2) Response process: Students received an example and guide of ISF-VSAQ. Self-assessment steps were also explained, and all students were provided adequate time. (3) Internal structure: Higher marginal reliability was demonstrated in the TPIM model. While unidimensionality was violated in the second ISF-VSAQ, the TPIM maintained unidimensionality. Item discrimination was generally acceptable, except for one item on the second occasion with minor vignette changes. Items with new knowledge showed better discrimination and item information functions on the second occasion. (4) Relations to other variables: Path analysis shows that the first occasion's understanding and scoring abilities strongly influenced the second occasion, highlighting the testlet effect. (5) Consequences: Path analysis revealed that the understanding ability in the second attempt directly affected the summative cardiovascular (β = 0.26, p = 0.066) and comprehensive examination ability (β = 0.34, p = 0.010).

Conclusion

The TPIM model effectively tackles the testlet effect of the present ISF-VSAQ. Thus, when employing multiple instances of self-feedback format, analyzing it as a testlet is recommended.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-024-02167-w.

Psychometric evaluation of a national exam for clinical undergraduates

Background

As a high-stake national-level examination administrated before students' clerkship in China, the Standardized Competence Test for Clinical Medicine Undergraduates (SCTCMU) has received much attention from the relevant educational departments and society at large. Investigating SCTCMU's validity and reliability is critical to the national healthcare profession education.

Materials and methods

Raw responses from SCTCMU, answered by 44,332 examines of 4th-year undergraduate medical students on 300 multiple-choice items, were used to evaluate the quality of the exam via psychometric methods based on item response theory (IRT). The core assumptions and model-data fit of IRT models were evaluated, as well as the item properties and information functions.

Results

The IRT models were fitted to the observed assessment data, where all the required assumptions were met. The IRT analysis showed that most items had acceptable psychometric properties, and the passing score was located close to the lowest measurement error computed from the model outcomes.

Conclusion

The proposed modern psychometric method provides a practical and informative approach to calibrating and analyzing medical education assessments. This work showcases a realistic depiction of the IRT analysis process and therefore facilitates the work of applied researchers wanting to conduct, interpret, and report IRT analyses on medical assessments.

Using item response theory to appraise key feature examinations for clinical reasoning

BACKGROUND

Validation of examinations is usually based on classical test theory. In this study, we analysed a key feature examination according to item response theory and compared the results with those of a classical test theory approach.

METHODS

Over the course of five years, 805 fourth-year undergraduate students took a key feature examination on general medicine consisting of 30 items. Analyses were run according to a classical test theory approach as well as using item response theory. Classical test theory analyses are reported as item difficulty, discriminatory power, and Cronbach's alpha while item response theory analyses are presented as item characteristics curves, item information curves and a test information function.

RESULTS

According to classical test theory findings, the examination was labelled as easy. Analyses according to item response theory more specifically indicated that the examination was most suited to identify struggling students. Furthermore, the analysis allowed for adapting the examination to specific ability ranges by removing items, as well as comparing multiple samples with varying ability ranges.

CONCLUSIONS

Item response theory analyses revealed results not yielded by classical test theory. Thus, both approaches should be routinely combined to increase the information yield of examination data.

Key feature-cases as virtual patients in education of veterinary neurology

To provide students of veterinary medicine with the necessary day 1 competences, e-learning offerings are increasingly used in addition to classical teaching formats such as lectures. For example, virtual patients offer the possibility of case-based, computer-assisted learning. A concept to teach and test clinical decision-making is the key feature (KF) approach. KF questions consist of three to five critical points that are crucial for the case resolution. In the current study usage, learning success, usability and acceptance of KF cases as neurological virtual patients should be determined in comparison to the long cases format. Elective courses were offered in winter term 2019/20 and summer term 2020 and a total of 38 virtual patients with neurological diseases were presented in the KF format. Eight cases were provided with a new clinical decision-making application (Clinical Reasoning Tool) and contrasted with eight other cases without the tool. In addition to the evaluation of the learning analytics (e.g., processing times, success rates), an evaluation took place after course completion. After 229 course participations (168 individual students and additional 61 with repeated participation), 199 evaluation sheets were completed. The average processing time of a long case was 53 min, while that of a KF case 17 min. 78% of the long cases and 73% of KF cases were successfully completed. The average processing time of cases with Clinical Reasoning Tool was 19 min. The success rate was 58.3 vs. 60.3% for cases without the tool. In the survey, the long cases received a ranking (1 = very good, 6 = poor) of 2.4, while KF cases received a grade of 1.6, 134 of the respondents confirmed that the casework made them feel better prepared to secure a diagnosis in a real patient. Flexibility in learning (n = 93) and practical relevance (n = 65) were the most frequently listed positive aspects. Since KF cases are short and highlight only the most important features of a patient, 30% (n = 70) of respondents expressed the desire for more specialist information. KF cases are suitable for presenting a wide range of diseases and for training students' clinical decision-making skills. The Clinical Reasoning Tool can be used for better structuring and visualizing the reasoning process.

The Relationship between Experiences Level and Clinical Decision-Making Skill in Midwifery Students: A Cross-Sectional Study

Background: Development and assessment of clinical decision-making skills are essential in midwifery education because of their role in mothers' and infants' safety. Therefore, the present study's primary objective was to evaluate the relationship between experience levels and clinical decision-making skills using the key features (KFs) examination.

Methods: One hundred and two midwifery students in five different education levels participated in this cross-sectional study through convenient sampling. Twenty KFs questions were designed based on the principles of the KFs examination. The participants' information, including grade point average (GPA), theoretical and practical scores of the obstetrics course, were collected. KFs scores were compared according to students' training semester by one-way analysis of variance (ANOVA). Pearson correlation was conducted to explore the correlation between KFs scores and GPA as well as theoretical and practical scores. All statistical analyses were performed at a significance level of 0.05 (p≤0.05). We used five kinds of effect size calculators, which include mean difference (MD), standardized mean difference (cohend), partial Eta-squared, Cohenf, and partial omega-squared.

Results: There was no correlation between KFs scores and the grade point average, theoretical exam scores, and practical exam scores. KFs scores linearly rose as the learners' level increased with a mean± SD score of 7.61±1.09 during the third semester compared to 11.55 ± 1.89 during the eighth semester (p=0.001). The effect size of this result was large (partial omega square=0.35, partial eta square=0.38 & cohen’s f=0.73). The largest SMD was related to the comparison of KFs scores between the eighth and third semester (MD=3.58, SMD=2.554 [CI 95%: 1.719-3.389], p-value═ 0.001), and the lowest was related to the comparison between the third and fourth semesters (MD=0.354, SMD= 0.2 [CI 95%: -0.421-0.821], p=0.987).

Conclusion: Establishing proficiency in clinical decision-making skills is a linear process greatly enhanced by experience, clearly shown by the present study results. Using KFs examination and obtaining extensive evidence to its benefit can allow us to renegotiate proficiency evaluation methods for students in clinical fields. the education curriculum should focus more on identifying clinical KFs skills than merely teaching knowledge about disease processes.