Validity of a new assessment rubric for a short-answer test of clinical reasoning

Improving assessment of procedural skills in health sciences education: a validation study of a rubrics system in neurophysiotherapy

BACKGROUND

The development of procedural skills is essential in health sciences education. Rubrics can be useful for learning and assessing these skills. To this end, a set of rubrics were developed in case of neurophysiotherapy maneuvers for undergraduates. Although students found the rubrics to be valid and useful in previous courses, the analysis of the practical exam results showed the need to change them in order to improve their validity and reliability, especially when used for summative purposes. After reviewing the rubrics, this paper analyzes their validity and reliability for promoting the learning of neurophysiotherapy maneuvers and assessing the acquisition of the procedural skills they involve.

METHODS

In this cross-sectional and psychometric study, six experts and 142 undergraduate students of a neurophysiotherapy subject from a Spanish university participated. The rubrics' validity (content and structural) and reliability (inter-rater and internal consistency) were analyzed. The students' scores in the subject practical exam derived from the application of the rubrics, as well as the rubrics' criteria difficulty and discrimination indices were also determined.

RESULTS

The rubrics´ content validity was found to be adequate (Content Validity Index > 0.90). These showed a unidimensional structure, and an acceptable internal consistency (α = 0.71) and inter-rater reliability (Fleiss' ƙ=0.44, ICC = 0.94). The scores of the subject practical exam practically covered the entire range of possible theoretical scores, showing all the criterion medium-low to medium difficulty indices - except for the one related to the physical therapist position-. All the criterion exhibited adequate discrimination indices (rpbis > 0.39), as did the rubric as a whole (Ferguson's δ = 0.86). Students highlighted the rubrics´ usefulness for learning the maneuvers, as well as their validity and reliability for formative and summative assessment.

CONCLUSIONS

The changed rubrics constitute a valid and reliable instrument for evaluating the execution quality of neurophysiotherapy maneuvers from a summative evaluation viewpoint. This study facilitates the development of rubrics aimed at promoting different practical skills in health-science education.

Application of test-enhanced learning (TEL) in obstetrics and gynecology: a prospective study

OBJECTIVE

Clinical reasoning is an essential skill, the foundations of which should be acquired during medical school. Within the format of test-based learning, such examinations can also be used to support the long-term retention of procedural knowledge necessary for clinical reasoning. The aim was to investigate whether repeated exposure to clinical cases in obstetrics and gynecology (OBGYN) with built-in questions leads to higher learning outcome than pure reading cases and what influence the delay between the intervention and the final test has on the retention of the respective content.

METHODS

In this non-randomised crossover study, 5th-year medical students (duration of the study is 6 years) taking a 1-week clinical attachment in OBGYN participated in computer-based case seminars in winter term 2020/2021, in which different case histories on gynecological-obstetric diseases were presented. Case content was identical for all groups, but the presentation format (cases with key feature questions vs read-only cases) of individual case vignettes changed weekly. The also intervention was repeated after 2 weeks for each group. Knowledge was assessed in an entry and an exit exam consisting of 40 short-answer questions.

RESULTS

A total of 94 out of 118 eligible students participated in the study (response rate: 79.7%). Learning outcome was significantly higher for items presented in the key feature format compared to items presented as read-only cases (74.2 ± 8.6% vs. 71.0 ± 9.2%; p = 0.017). Furthermore, the analysis showed that the temporal distance of the intervention package from the final examination had no influence on retention.

CONCLUSION

This is the first study to demonstrate an effect of test-enhanced learning on clinical reasoning in the subject of OGBYN. In this cross-over study, repeated testing was more effective than repeated case-based learning alone. Curricular implementation of longitudinal key feature testing can thus improve learning outcomes for OBGYN.

Rubric for the evaluation of competencies in traumatology in the Degree of Physiotherapy: Delphi approach

BACKGROUND

In health professions, the curriculum that must be met in order to obtain the academic certificate is based on the development of the so-called competencies. The broad content of the Practicum of the Degree of Physiotherapy has led to the creation of multiple types of evaluation, which makes it difficult for faculty members to reach a consensus on competencies. The aim of this study was to develop and validate content of a rubric for the evaluation of acquired competencies related to physiotherapeutic performance and intervention in traumatology within the Practicum of the Degree of Physiotherapy.

METHODS

Following the Delphi methodology, a group of experts from all over the Spanish territory participated in the study. Through on-line questionnaires, several sequential rounds were established, alternated by controlled feedback until obtaining a consensus in the opinion of the experts, which allowed elaborating the final rubric.

RESULTS

Initially, 16 experts were contacted, of whom 10 worked and completed the final content of the rubric. For the 3 rounds that were conducted, the initial 142 interventions of the initial proposition, which correspond to specific competencies, were reduced to the final 29 items that compose the specific evaluation rubric presented in this study.

CONCLUSIONS

This rubric is an evaluation instrument with valid content for the assessment of specific competencies of Traumatology in the Practicum of the Degree of Physiotherapy.

Students' Perceptions of Instructional Rubrics in Neurological Physical Therapy and Their Effects on Students' Engagement and Course Satisfaction

One of the main challenges faced by physical therapy (PT) students is to learn the practical skills involved in neurological physical therapy (PT). To help them to acquire these skills, a set of rubrics were designed for formative purposes. This paper presents the process followed in the creation of these rubrics and their application in the classroom, noting that students perceived them as valid, reliable, and highly useful for learning. The perception of the validity and usefulness of the rubrics has different closely related dimensions, showing homogeneous values across the students´ sociodemographic and educational variables, with the exception of dedication to studying, which showed a significant relationship with schoolwork engagement and course satisfaction. The adequacy of the hypothesized structural model of the relationships among the variables was confirmed. Direct effects of the perception of the rubrics' validity and engagement on course satisfaction were found, as well as direct effects of the assessment of the usefulness of the rubrics on schoolwork engagement and indirect effects on course satisfaction through this latter variable. The results are discussed taking into account the conclusions of previous research and different instructional implications.

How to perform an excellent radiology board examination: a web-based checklist

BACKGROUND

Board exams are now considered as means of quality procedures that aim to keep the professional knowledge and skills of the physicians at the highest level. In addition, for an assessment to be scientifically valid, it has to be done within defined standards. Although there are different sources in this field, there is a need for a resource that details the steps required for the examinations to be performed perfectly, brings descriptions of the reasons for the procedure and associates the steps with assessment standards. Experts with national and international experience both in radiology and medical education contributed to the preparation of this checklist.

RESULTS

The guide includes 174 elements to consider before, after the exam order and examination. From the perspective of assessment standards, it has been observed that the steps to be considered before the exam have a greater impact on the validity and reliability of the exam. The standard in which the questions are most associated was validity with 117 (67.24%) questions.

CONCLUSIONS

We think that our guide, which will be accessible in the web environment, will be useful to the teams with a development goal or just start the exam, the candidates who will take the exam and the examiners.

Criterion scores, construct validity and reliability of a web-based instrument to assess physiotherapists' clinical reasoning focused on behaviour change: 'Reasoning 4 Change'

Background and aim: 'Reasoning 4 Change' (R4C) is a newly developed instrument, including four domains (D1-D4), to assess clinical practitioners' and students' clinical reasoning with a focus on clients' behaviour change in a physiotherapy context. To establish its use in education and research, its psychometric properties needed to be evaluated. The aim of the study was to generate criterion scores and evaluate the reliability and construct validity of a web-based version of the R4C instrument. Methods: Fourteen physiotherapy experts and 39 final-year physiotherapy students completed the R4C instrument and the Pain Attitudes and Beliefs Scale for Physiotherapists (PABS-PT). Twelve experts and 17 students completed the R4C instrument on a second occasion. The R4C instrument was evaluated with regard to: internal consistency (five subscales of D1); test-retest reliability (D1-D4); inter-rater reliability (D2-D4); and construct validity in terms of convergent validity (D1.4, D2, D4). Criterion scores were generated based on the experts' responses to identify the scores of qualified practitioners' clinical reasoning abilities. Results: For the expert and student samples, the analyses demonstrated satisfactory internal consistency (α range: 0.67-0.91), satisfactory test-retest reliability (ICC range: 0.46-0.94) except for D3 for the experts and D4 for the students. The inter-rater reliability demonstrated excellent agreement within the expert group (ICC range: 0.94-1.0). The correlations between the R4C instrument and PABS-PT (r range: 0.06-0.76) supported acceptable construct validity. Conclusions: The web-based R4C instrument shows satisfactory psychometric properties and could be useful in education and research. The use of the instrument may contribute to a deeper understanding of physiotherapists' and students' clinical reasoning, valuable for curriculum development and improvements of competencies in clinical reasoning related to clients' behavioural change.