From theory to practice in learning about healthcare associated infections: reliable assessment of final year medical students' ability to reflect

Improving Self-Reflection Assessment Practices: Comparative Judgment as an Alternative to Rubrics

CONSTRUCT

The authors aimed to investigate the utility of the comparative judgment method for assessing students' written self-reflections.

BACKGROUND

Medical practitioners' reflective skills are increasingly considered important and therefore included in the medical education curriculum. However, assessing students' reflective skills using rubrics does not appear to guarantee adequate inter-rater reliabilities. Recently, comparative judgment was introduced as a new method to evaluate performance assessments. This study investigates the merits and limitations of the comparative judgment method for assessing students' written self-reflections. More specifically, it examines the reliability in relation to the time spent assessing, the correlation between the scores obtained using the two methods (rubrics and comparative judgment), and, raters' perceptions of the comparative judgment method.

APPROACH

Twenty-two self-reflections, that had previously been scored using a rubric, were assessed by a group of eight raters using comparative judgment. Two hundred comparisons were completed and a rank order was calculated. Raters' impressions were investigated using a focus group.

FINDINGS

Using comparative judgment, each self-reflection needed to be compared seven times with another self-reflection to reach a scale separation reliability of .55. The inter-rater reliability of rating (ICC, (1, k)) using rubrics was .56. The time investment required for these reliability levels in both methods was around 24 minutes. The Kendall's tau rank correlation indicated a strong correlation between the scores obtained via both methods. Raters reported that making comparisons made them evaluate the quality of self-reflections in a more nuanced way. Time investment was, however, considered heavy, especially for the first comparisons. Although raters appreciated that they did not have to assign a grade to each self-reflection, the fact that the method does not automatically lead to a grade or feedback was considered a downside.

CONCLUSIONS

First evidence was provided for the comparative judgment method as an alternative to using rubrics for assessing students' written self-reflections. Before comparative judgment can be implemented for summative assessment, more research is needed on the time investment required to ensure no contradictory feedback is given back to students. Moreover, as the comparative judgment method requires an additional standard setting exercise to obtain grades, more research is warranted on the merits and limitations of this method when a pass/fail approach is used.

Interassessor agreement of portfolio-based competency assessment for orthotists/prosthetists in Australia: a mixed method study

BACKGROUND

Internationally qualified orthotists/prosthetists who want to practice in Australia must pass a portfolio-based competency assessment. Testing the agreement between independent assessors is important to engender confidence in the assessment, and continually improve the processes.

OBJECTIVES

To quantify interassessor agreement for all 68 performance indicators in the Australian Orthotic Prosthetic Association's Entry Level Competency Standards and where there was significant disagreement between assessors, to explore the reasons why.

STUDY DESIGN

Mixed methods: explanatory sequential.

METHOD

Fifteen portfolios were assigned to independent assessors. Assessors determined whether the evidence presented met the requirements of each performance indicator. Interassessor agreement was calculated using Gwet's Agreement Coefficient 1 (AC1), and these data informed semistructured interviews to explore the reasons for disagreement.

RESULTS

Most performance indicators (87%) had moderate to substantial agreement (AC1 > 0.71), which could be attributed to a variety of factors including the use of a simple assessment rubric with supporting guidelines and assessor training to establish shared expectations. The remaining performance indicators (13%) had fair to slight agreement (AC1 ≤ 0.7). Interviews with assessors suggested that disagreement could be attributed to the complexity of some performance indicators, unconscious bias, and the appropriateness of the evidence presented.

CONCLUSIONS

Although most performance indicators in Australian Orthotic Prosthetic Association's Entry Level Competency Standard were associated with moderate to substantial interassessor agreement, there are opportunities to improve agreement by simplifying the wording of some performance indicators and revising guidelines to help applicants curate the most appropriate evidence for each performance indicator.

Educational effects using a robot patient simulation system for development of clinical attitude

INTRODUCTION

The aim of this study was to assess the effectiveness of improving the attitude of dental students towards the use of a full-body patient simulation system (SIMROID) compared to the traditional mannequin (CLINSIM) for dental clinical education.

MATERIALS AND METHODS

The participants were 10 male undergraduate dental students who had finished clinical training in the university hospital 1 year before this study started. They performed a crown preparation on an upper pre-molar tooth using SIMROID and CLINSIM as the practical clinical trials. The elapsed time for preparation was recorded. The taper of the abutment teeth was measured using a 3-dimensional shape-measuring device after this trial. In addition, a self-reported questionnaire was collected that included physical pain, treatment safety and maintaining a clean area for each simulator. Qualitative data analysis of a free format report about SIMROID was performed using text mining analysis. This trial was performed twice at 1-month intervals.

RESULTS

The students considered physical pain, treatment safety and a clean area for SIMROID significantly better than that for CLINSIM (P < .01). The elapsed time of preparation in the second practical clinical trial was significantly lower than in the first for SIMROID and CLINSIM (P < .01). However, there were no significant differences between the abutment tapers for both systems. For the text mining analysis, most of the students wrote that SIMROID was similar to real patients.

CONCLUSION

The use of SIMROID was proven to be effective in improving the attitude of students towards patients, thereby giving importance to considerations for actual patients during dental treatment.

Exploring the approaches used to teach concepts of hand hygiene to Australian medical students

BACKGROUND

Recent audit data has revealed that the hand hygiene (HH) rates of Australian medical students is suboptimal. It has been suggested that new approaches are needed to teach students about infection control. As a first step, we undertook a study to determine the current educational approaches used to teach Australian medical students about HH. Secondly, this study aimed to explore the perceived barriers and to explore what other teaching approaches could be used to improve the levels of knowledge and compliance.

METHODS

A self-administered questionnaire was sent to the Dean of Medical Education at each of the medical schools in late 2012.

RESULTS

Of the 19 medical schools in Australia, 17 agreed to participate. The most commonly reported approaches currently used to teach students about HH are skills stations (17/17) and case scenarios/lectures (15/17). Clinical practical exams (15/17) and competency checks (11/17) are mostly used to assess the HH practices of medical students. Participants nominated the following as barriers to improving HH compliance: negative role modelling by senior doctors, and negative attitudes of students. Practical exercises (15/17), online teaching (12/17) and reflection (12/17) were suggested as other useful approaches that could be used to teach these concepts.

CONCLUSION

Practical laboratory-based approaches were suggested as the most useful teaching and learning approach. Given the trend towards blended learning, universities may want to consider new HH teaching approaches that combine campus-based learning with online components and reflection. Early exposure and the continued reinforcement of HH concepts through the student's medical degree are essential.

Facilitators and barriers around teaching concepts of hand hygiene to undergraduate medical students

BACKGROUND

Currently, there is limited literature examining the impact and appropriateness of the educational approaches used to teach medical students concepts around hand hygiene (HH).

AIM

To explore: (i) the perspectives of key academics and medical students towards HH and factors influencing compliance; (ii) the current teaching practices around HH in the medical school. We also aimed to examine options for new teaching and learning approaches to help improve student knowledge and attitudes towards HH.

METHODS

Individual, in-depth interviews were conducted with key academics and medical students. Transcripts were analysed thematically.

FINDINGS

Participants felt that students do not value the teaching around HH, nor do they find it interesting, especially in comparison to other subjects taught in the medical programme. The use of professional modelling, assessment tasks and feedback from patients and colleagues were the major educational recommendations; these could be implemented to help improve the attitudes of medical students towards HH and hopefully their compliance. Regular, small, group-scenario-based and/or practical hands-on sessions were also proposed. The need for a culture change was the overarching theme to foster sustainable HH practices among medical students.

CONCLUSION

Assessment and scenario-based learning and teaching approaches should be considered to help improve HH behaviours among medical students. Sustainability of these practices is likely to require role models and culture change around infection control.

Does self reflection and insight correlate with academic performance in medical students?

BACKGROUND

Medical students in academic difficulty are often described as lacking insight. The Self Reflection and Insight Scale (SRIS) is a tool for measuring insight which has been validated in medical students. We investigated whether self reflection and insight scores correlate with academic performance in Year 4 medical students from a six year undergraduate medical degree, and whether self reflection and insight changes after one year of clinical training.

METHODS

Self reflection and insight scores were measured in 162 students at the start of Year 4 at the University of Western Australia. Performance in end of year written and clinical exams was monitored and correlated with SRIS. Seventy of the students were surveyed again at the start of Year 5 to see if scores changed or were stable after one year of full time clinical training.

RESULTS

We found no correlation between self reflection or insight and academic performance in written and clinical exams. There was a significant increase in recognition of the need for self reflection in Year 5 compared with Year 4.

CONCLUSIONS

While no correlation was found between this measure of self reflection and insight with academic performance, there was an increase in students' recognition of the need for reflection after one year of clinical studies. This study is a valuable first step towards a potentially exciting research domain and warrants further longitudinal evaluation with larger cohorts of students using additional measures of achievement.

Hand hygiene knowledge and attitudes: comparisons between student nurses

It is recognised that early intervention in healthcare workers’ education is important in terms of embedding effective infection prevention and control knowledge into practice. One of the most important aspects of this education is hand hygiene compliance and technique expectations. This small study was undertaken in collaboration with the University of Dundee, School of Nursing and Midwifery and NHS Tayside. It explored the differences in knowledge, attitudes and practice towards hand hygiene among second and third year student nurses with the aim of reviewing and informing the development of future educational material at undergraduate level in order to continually enhance knowledge and skills and bridge the theory–practice gap. This study concluded that third year student nurses did have a slightly better knowledge base than the second years and that although knowledge, attitudes and practice were reported to be of a good standard overall, there were still some important issues that must be addressed.

Innovative infection prevention and control teaching for nursing students: a personal reflection

A crucial part of our role as Infection Prevention and Control Practitioners is to develop, facilitate and evaluate the teaching and learning of students and healthcare workers. However, creating an environment for participation, interaction and socialisation whilst effectively stimulating reflective and critical thinking in order to promote good practice can be challenging. This paper is a critical reflection, based on Gibb’s model of reflection to demonstrate the development, facilitation, support and evaluation of learning using an innovative, interactive method.

'Hawks' and 'doves': effect of feedback on grades awarded by supervisors of student selected components

BACKGROUND

Supervisors of some student selected components (SSCs) may appear to give higher grades than others. It is not known if feedback can influence the behaviour of supervisors in the grades they award. We have introduced feedback letters in our institution.

AIMS

(1) To assess the feasibility of objectively identifying SSCs where grades awarded are consistently higher or lower than the average; (2) To assess the effect of feedback on the grades awarded by supervisors of SSCs.

METHODS

The breakdown of SSC grades was examined over four consecutive years, before and after feedback letters were introduced in 2005. The grades awarded globally, and in five individual SSCs, were compared using the chi(2) goodness-of-fit test.

RESULTS

(1) Individual SSCs were identified which awarded grades that were consistently different from the average. (2) Overall grades awarded in 2003/04 and 2004/05 (before feedback) were similar (chi2=0.37, df=2, p=0.83). Likewise, overall grades awarded in 2005/06 and 2006/07 (after feedback) were similar (chi2=1.72, df=2, p=0.42). Comparison of 2003/04 with 2005/06 (chi2=16.0, df=2, p<0.001), and 2006/07 (chi2=26.6, df=2, p<0.001), and of 2004/05 with 2005/06 (chi2=13.5, df=2, p=0.001), and 2006/07 (chi2=23.7, df=2, p<0.001), revealed highly significant differences. The grades awarded after feedback were higher than the grades awarded before feedback.

CONCLUSIONS

The chi2 goodness-of-fit test may be used to identify individual SSCs where the grades awarded are different from the average, although the interpretation of the results thus obtained is fraught with difficulty. Our data also suggest that it is possible to influence assessors in the grades they award.

Training of physicians for the twenty-first century: role of the basic sciences

BACKGROUND

Rapid changes in the healthcare environment and public dissatisfaction with the cost and quality of medical care have prompted a critical analysis of how physicians are trained in the United States. Accrediting agencies have catalyzed a transformation from a process based to a competency-based curriculum, both at the undergraduate and the graduate levels.

AIM

The objective of this overview is to determine how these changes are likely to alter the role of basic science in medical education.

METHODS

Policy statements related to basic science education from the National Board of Medical Examiners (NBME), the Accreditation Council for Graduate Medical Education (ACGME), American Board of Medical Specialties (ABMS), and the Federation of State Medical Boards (FSMB) were reviewed and assessed for common themes.

RESULTS

Three primary roles for the basic sciences in medical education are proposed: (1) basic science to support the development of clinical reasoning skills; (2) basic science to support a critical analysis of medical and surgical interventions ("evidence-based medicine"); and (3) basic and translational science to support analysis of processes to improve healthcare ("science of healthcare delivery").

CONCLUSION

With these roles in mind, several methods to incorporate basic sciences into the curriculum are suggested.

Survey of teaching/learning of healthcare-associated infections in UK and Irish medical schools

All medical doctors have an important role to play in the diagnosis, management and prevention of healthcare-associated infection (HCAI). Strengthening the contribution of medical doctors and medical students to HCAI prevention programmes should include measures that enhance knowledge, improve practice and develop appropriate attitudes to the safety and quality of patient care. The Hospital Infection Society (HIS) funded a review of medical education on HCAI throughout medical schools in the UK and the Republic of Ireland. A questionnaire was drafted and circulated to all medical schools and 31 of 38 (82%) responded. The prevalence and transmission of HCAI were taught by 97% and 100% of medical schools, respectively, but the importance of HCAI as a quality and safety issue was covered in only 60% of medical schools. Multiple choice questions (MCQs) and objective structure clinical examinations (OSCEs) were the most popular methods of assessment. Lectures, discussion of cases and practical demonstrations were considered useful by >90% of respondents and online material and log books by 67% and 60%, respectively. More than 80% were willing to share a common pool of educational resources. An agreed curriculum should be developed for educating medical students in HCAI prevention and control, to outline optimum methods for assessment and develop a shared pool of educational resources.