The role of assessment in expanding professional horizons

Assessment of anaesthesia trainees using performance metrics derived from electronic health records: a longitudinal cohort analysis

BACKGROUND

The development of competencies in anaesthesia residents is assessed by direct observation of their performance and written and oral examinations. Little is known about how residents' competencies are reflected by objective data in anaesthetic records. We investigated whether performance metrics derived from electronic anaesthesia records are associated with anaesthesia residents' progress of training and European written examination timing and results.

METHODS

We recruited 46 anaesthesia providers who trained at the Vienna University Hospital between September 2013 and June 2021, and we reviewed the anaesthesia records of all cases they managed during the study period. We derived six performance measures based on perioperative data routinely collected: intraoperative hypotension and hypothermia, glycaemic control, postoperative nausea and vomiting prevention, lung-protective ventilation, and postoperative kidney injury. We evaluated the association between these quality metrics with training level and written exam completion status in anaesthesia residents after adjusting for patient and surgical factors.

RESULTS

We found a statistically significant association between the level of training and most performance measures. The probability of preventing intraoperative hypotension increased (OR 1.16, 95% CI 1.12 - 1.20) with the level of training, as did the probability of preventing hypothermia (OR 1.08, 95% CI 1.05 - 1.11) and administering appropriate postoperative nausea and vomiting prophylaxis (OR 1.21, 95% CI 1.15 - 1.27). However, the odds of preventing acute kidney injury decreased (OR 0.91, 95% CI 0.83 - 0.99), as did the use of lung-protective ventilation (OR 0.94, 95% 0.92 - 0.97). All participating residents who took the written exam passed it, precluding a direct pass versus fail comparison, but the exam completion status was associated with increased odds of lung-protective ventilation (OR 1.42, 95% CI 1.03 - 1.95) and decreased odds of preventing intraoperative hypotension (OR 0.7, 95% CI 0.49 - 0.99). Glycaemic control was not associated with either of the training milestones.

CONCLUSIONS

The training level of anaesthesia residents had a significant association with several performance metrics. Passing the written exam only had a modest effect. Performance measures based on patient outcomes and intraoperative care may serve as assessment methods for anaesthesia residents' progress tracking.

The pivot to online teaching: an opportunity to create effective problem-based learning environments for dietetic education

BACKGROUND

Dietetics, as an autonomous profession guided by evidence-based practice, requires a dynamic and adaptable workforce to meet evolving challenges in healthcare, public health nutrition and epidemiological research. This study aimed to explore the opportunities presented by the transition to online education of preparing dietetic students for professional demands, particularly, through the lens of problem-based learning (PBL) within the Community of Inquiry (CoI) framework.

METHODS

This paper examines the shift from traditional classroom and practice placement education to online and blended learning accelerated by the COVID-19 pandemic. It focuses on the application of PBL, rooted in constructivist theories and the CoI framework, which includes social, cognitive and teaching presences.

RESULTS

PBL encourages active and collaborative learning, aligning with the multifaceted roles of dietetic professionals. The CoI framework guides the development of effective online learning environments, emphasising the importance of social, cognitive and teaching presence.

CONCLUSION

The transition to online education presents significant opportunities for dietetic education. Careful instructional design and effective teaching strategies are crucial for creating successful online learning environments, as demonstrated through an illustrative case study.

Portfolio use in postgraduate medical education: a systematic scoping review

BACKGROUND

Portfolios are increasingly commonplace in postgraduate medical education. However, poor understanding of and variations in their content, quality, and structure have hindered their use across different settings, thus dampening their efficacy.

METHODS

This systematic scoping review on portfolios in postgraduate medical education utilized Krishna's Systematic Evidence Based Approach (SEBA). Braun and Clarke's thematic analysis and Hsieh and Shannon's directed content analysis were independently used to evaluate the data.

RESULTS

In total, 12 313 abstracts were obtained, and 76 full-text articles included. Six key themes/categories were identified: (i) portfolio definitions and functions, (ii) platforms, (iii) design, (iv) implementation, (v) use in assessments, and (vi) evaluations of their usage.

CONCLUSIONS

Portfolios allow for better appreciation and assessments of knowledge, skills, and attitudes in time-, learner-, and context-specific competencies through the establishment of smaller micro-competencies and micro-credentialling. Organized into three broad stages-development, implementation, and improvement-a six-step framework for optimizing and instituting portfolios in postgraduate medical education is offered.

The impact of coronavirus disease 2019 pandemic on dental school assessments - Current status and future perspectives

PURPOSE

To evaluate course directors' feedback on the assessment methods used during the coronavirus disease 2019 (COVID-19) pandemic and identify effective approaches for future assessments in dental education.

METHODS

Course directors at the US dental schools were surveyed for changes in assessments implemented during the early stages of the pandemic (March-July 2020) using the Qualtrics platform. The survey questions addressed assessment methods utilized in didactic, preclinical, and clinical arenas pre-COVID-19 (before March 2020) and during the early phase of the pandemic (between March and July 2020) and identified any sustained changes in assessments post-COVID-19. Of the 295 responses for the type of courses directed, 48%, 22%, and 30% responses were for didactic, pre-clinical, and clinical assessments, respectively. Chi-square tests and 95% confidence intervals were used to assess quantitative differences.

RESULTS

Computer-based un-proctored and remote- proctored assessments increased whereas paper-based in-person proctored assessments decreased during an early pandemic. For pre-clinical and clinical courses, objective-structured clinical exams and case-based assessments increased whereas, for didactic courses, the number of presentations, short-answer, and multiple-choice questions-based assessments increased. Specimen-based assessments and patient-based encounters decreased significantly in didactic and clinical courses, respectively. Manikin-based exams increased in clinical but not in pre-clinical courses. Survey respondents disagreed that alternative assessments helped students learn better, resulted in better course evaluations, or were an equivalent replacement for pre-COVID-19 assessments. Interestingly, 49% of respondents indicated a likelihood of continuing alternative assessments whereas 36% were unlikely and 15% were neutral.

CONCLUSIONS

A combination of effective pre-pandemic and innovative alternative assessments developed during the pandemic may be the new normal in the dental education curriculum.

Development and Integration of DOPS as Formative Tests in Head and Neck Ultrasound Education: Proof of Concept Study for Exploration of Perceptions

In Germany, progress assessments in head and neck ultrasonography training have been carried out mainly theoretically and lack standardisation. Thus, quality assurance and comparisons between certified courses from various course providers are difficult. This study aimed to develop and integrate a direct observation of procedural skills (DOPS) in head and neck ultrasound education and explore the perceptions of both participants and examiners. Five DOPS tests oriented towards assessing basic skills were developed for certified head and neck ultrasound courses on national standards. DOPS tests were completed by 76 participants from basic and advanced ultrasound courses (n = 168 documented DOPS tests) and evaluated using a 7-point Likert scale. Ten examiners performed and evaluated the DOPS after detailed training. The variables of "general aspects" (6.0 Scale Points (SP) vs. 5.9 SP; p = 0.71), "test atmosphere" (6.3 SP vs. 6.4 SP; p = 0.92), and "test task setting" (6.2 SP vs. 5.9 SP; p = 0.12) were positively evaluated by all participants and examiners. There were no significant differences between a basic and advanced course in relation to the overall results of DOPS tests (p = 0.81). Regardless of the courses, there were significant differences in the total number of points achieved between individual DOPS tests. DOPS tests are accepted by participants and examiners as an assessment tool in head and neck ultrasound education. In view of the trend toward "competence-based" teaching, this type of test format should be applied and validated in the future.

Challenges in the Teaching-Learning Process of the Newly Implemented Module on Bioethics in the Undergraduate Medical Curriculum in India

The National Medical Commission of India introduced the Competency Based Curriculum in Medical Education for undergraduate medical students in 2019 with a new module named Attitude, Ethics and Communication (AETCOM) across the country. There was a consensus for teaching medical ethics in an integrated way, suggesting dedicated hours in each phase of undergraduate training. The AETCOM module was prepared and circulated as a guide to acquire necessary competency in attitudinal, ethical and communication domains. This study was aimed to explore the perceptions of students and medical teachers and identify the challenges in teaching and learning process of the newly implemented AETCOM module. It was a mixed method designed study with structured questionnaires for students and teachers at various medical schools in India. Based on the quantitative data, in-depth interviews with medical teachers were undertaken. Challenges were perceived by both students and teachers. The students had a mixed perception, facing difficulties in passive learning with scarce resource materials. Challenges identified by teachers were a lack of knowledge and skills required for teaching bioethics, the logistics of managing large numbers of students in the stipulated time frame, interdisciplinary integration—both horizontal and vertical, and assessment program in terms competency-based education. The study draws the attention of all stakeholders for a revision and efforts for further improvement in the teaching and assessment process, and setting a standard model in medical education in India.

Simulation-based mastery learning in gastrointestinal endoscopy training

Simulation-based mastery learning (SBML) is an emerging form of competency-based training that has been proposed as the next standard method for procedural task training, including that in gastrointestinal endoscopy. Current basic gastrointestinal endoscopy training relies on the number of procedures performed, and it has been criticized for its lack of objective standards that result in variable skills among trainees and its association with patient safety risk. Thus, incorporating simulators into a competency-based curriculum seems ideal for gastrointestinal endoscopy training. The curriculum for SBML in gastrointestinal endoscopy is currently being developed and has promising potential to translate into the clinical performance. Unlike the present apprenticeship model of "see one, do one, teach one," SBML integrates a competency-based curriculum with specific learning objectives alongside simulation-based training. This allows trainees to practice essential skills repeatedly, receive feedback from experts, and gradually develop their abilities to achieve mastery. Moreover, trainees and trainers need to understand the learning targets of the program so that trainees can focus their learning on the necessary skills and trainers can provide structured feedback based on the expected outcomes. In addition to learning targets, an assessment plan is essential to provide trainees with future directions for their improvement and ensure patient safety by issuing a passing standard. Finally, the SBML program should be planned and managed by a specific team and conducted within a developed and tested curriculum. This review discusses the current state of gastrointestinal endoscopy training and the role of SBML in that field.

The prognostic validity of the formative for the summative MEQ (Modified Essay Questions)

Objective: The purpose of formative examinations is that students and lecturers receive an early feedback regarding the success of learning behavior and teaching methods. These also serve as practice for later summative exams. The aim of this paper is to investigate to what extent the result of the formative MEQ* at the end of the first semester at Witten/Herdecke University (UW/H) in the study program human medicine can be used as a predictor for the summative MEQ-1 at the end of the second semester which is part of the equivalence examination replacing the state examination. Methodology: The predictive value of the score achieved in the MEQ* on the MEQ-1 score, as well as the potential influence of the variables gender, age, high school graduation grade (German Abiturnote), professional background, and self-efficacy expectancy, was determined for students of human medicine. Results: Data from two cohorts of UW/H with a total of 88 students were included. Scores on the formative MEQ* correlate with those on the summative MEQ-1 in both cohorts. In regression analyses, only the score on the MEQ* proves to be a significant predictor of performance on the MEQ-1 (40.5% variance explanation). Particularly significant predictors are the scores in the subjects anatomy and clinical reasoning. Vocational training or pre-study only appear to contribute to higher scores in the MEQ* after the first semester, but have no further significance in predicting scores in the MEQ-1. Conclusion: The MEQ* was confirmed to be a good predictor of the MEQ-1. Thus, it serves as a formative exam to inform students about their current state of knowledge with regard to the summative exam MEQ-1, so that they can adequately adapt their learning strategies in the course of the second semester.

Professional Assessment Tool for Team Improvement: An assessment tool for paediatric intensive care unit nurses' technical and nontechnical skills

BACKGROUND

Cardiorespiratory arrests are rare in paediatric intensive care units, yet intensive care nurses must be able to initiate resuscitation before medical assistance is available. For resuscitation to be successful, instant decision-making, team communication, and the coordinating role of the first responsible nurse are crucial. In-house resuscitation training for nurses includes technical and nontechnical skills.

OBJECTIVES

The aim of this study was to develop a valid, reliable, and feasible assessment instrument, called the Professional Assessment Tool for Team Improvement, for the first responsible nurse's technical and nontechnical skills.

METHODS

Instrument development followed the COnsensus-based Standards for the selection of health Measurement Instruments guidelines and professionals' expertise. To establish content validity, experts reached consensus via group discussions about the content and the operationalisation of this team role. The instrument was tested using two resuscitation assessment scenarios. Inter-rater reliability was established by assessing 71 nurses in live scenario sessions and videotaped sessions, using intraclass correlation coefficients and Cohen's kappa. Internal consistency for the total instrument was established using Cronbach's alpha. Construct validity was assessed by examining the associations between raters' assessments and nurses' self-assessment scores.

RESULTS

The final instrument included 12 items, divided into four categories: Team role, Teamwork and communication, Technical skills, and Reporting. Intraclass correlation coefficients were good in both live and videotaped sessions (0.78-0.87). Cronbach's alpha was stable around 0.84. Feasibility was approved (assessment time reduced by >30%).

CONCLUSIONS

The Professional Assessment Tool for Team Improvement appears to be a promising valid and reliable instrument to assess both technical and nontechnical skills of the first responsible paediatric intensive care unit nurse. The ability of the instrument to detect change over time (i.e., improvement of skills after training) needs to be established.

Mental health education in undergraduate medical curricula across Nepalese universities

BACKGROUND

Mental and substance use disorders account for 30 % of the non-fatal disease burden and 10 % of the overall disease burden but the treatment gap is daunting. With just one psychiatrist per 200,000 populations in Nepal, the only convincing way to improve access to the services quickly is by mobilizing non-specialized medical practitioner. A robust mental health component within the training curriculum of general medical doctors could produce medical graduates with adequate knowledge and skills to deliver basic mental health service. We reviewed the mental health curricula for medical students of all the medical universities in Nepal.

METHODS

Information on existing mental health curricula was collected from the faculty of the respective universities with respect to content coverage, teaching methods and evaluation patterns. The mental health curricula were described in relation to teaching duration, duration of clinical rotation, duration of internship, and the relative weight of mental health in examination marks. Teaching methods were classified broadly as passive and active. Assessment methods were documented. Content coverage of the curricula was evaluated with respect to history taking and general physical examination, the priority mental health conditions, topics on behavioural sciences, and child mental health or other topics.

RESULTS

The duration of teaching on mental health in general medical doctor training in Nepalese medical universities ranges from 25 to 92 h. All medical universities have a relative focus on the priority mental neurological and substance use disorders. The clinical rotation on mental health is mostly two weeks, except in one university where it can be extended up to 4 weeks with an elective clinical rotation. The relative weight of summative assessment ranges from 0.21 to 2.5 % total marks of the entire training.

CONCLUSIONS

Considerable disparities exist in course content, teaching/learning modalities and assessments for mental health across Nepalese medical universities. The relative proportion of mental health in medical curricula as well as teaching/learning and assessments are far below ideal in these universities. These findings suggest a need for increasing time allocation, adopting newer teaching learning methods, and also having a mandatory clinical rotation during training and during internship.

Pharmacy students' perceptions of assessment and its impact on learning

INTRODUCTION

Outcomes-based education requires active student learning with assessment strategies that foster deep approaches to learning, which are often influenced by students' perceptions of assessment. We aimed to investigate the perceptions of pharmacy students at an Australian university about their experiences of assessment and its impact on their learning.

METHODS

A mixed-methods study was conducted involving a self-administered questionnaire and semi-structured focus groups. Descriptive statistics were used to rate the perceived value of different assessment tools and confirmed by themes arising from the focus groups. Examination results over a five-year period were also collated to assess congruence between perceptions and academic performance.

RESULTS

From the 123 questionnaire and nine focus group participants, short-answer questions were the most positively-received form of assessment due to students being able to demonstrate and receive marks for partial knowledge. Multiple-choice questions received mixed response as they were cited as being useful in assessing student knowledge but potentially difficult to interpret/answer correctly. Reflective pieces received the lowest ratings and were considered the least beneficial. Key identified themes were ensuring quality assurance of assessment processes, use of authentic assessment, timely feedback, and appropriate match between workload and assessment weightings. Overall, there was congruence between students' exam scores and their perceptions of the different assessment types.

CONCLUSIONS

Strategic planning and delivery of correctly-weighted authentic assessments with the provision of constructive feedback are key elements for active engagement of students and achievement of life-long learning outcomes.

Developing a novel framework for non-technical skills learning strategies for undergraduates: A systematic review

OBJECTIVES

There is substantial lack of guidance when it comes to the implementation of non-technical skills (NTS) in undergraduate medical education. This review aimed to identify and critically evaluate published literature on learning strategies for NTS in undergraduate medical education and to derive a training framework targeted towards standardizing future training interventions.

METHODS

A systematic review of the MEDLINE database was performed using a prospective protocol following PRISMA guidelines. Studies evaluating undergraduate medical students exposed to NTS interventions, which measured subjective or objective outcomes in selected attributes, were included.

RESULTS

Initial systematic search yielded a total of 5079 articles, out of which 68 fulfilled the inclusion criteria. A total of 24 NTS were identified, with communication skills being the most commonly reported skill evaluated (n = 37). A variety of educational tools were used (n = 32), noteworthy being the use of simulated patients. Great heterogeneity was also observed in measured outcomes and methods of assessment. A 'triad of outcomes' in NTS training was devised (knowledge, skill performance and attitude towards skills) and used for classification of all reported outcomes. Extracted data were used to design a non-technical skill training framework.

CONCLUSIONS

The existing literature describes a plethora of NTS interventions in undergraduate medical education, with varied outcomes and assessments. We hereby propose the 'NTS Training Framework', in an attempt to coordinate future research and catalyze the identification of an ideal NTS course structure to form tomorrow's physicians.

[Examinations while studying medicine - more than simply grades]

Assessment drives learning. Examinations need to be aligned primarily with learning objectives, as well as teaching and assessment methods of the courses on offer. In doing so, various examination instruments are required to measure on levels of competency that build on one another. An appropriate mix is essential to reflect the variety of learning outcomes of a chosen curriculum. Furthermore, examinations also possess the characteristics of evaluation: They reflect the knowledge and abilities of students and assess the teaching at a defined location. Digital examinations in the form of multiple-choice-question (MCQ) testing enable a higher degree of automation and accelerate the processes of creation, implementation, and evaluation of the examination results. Thus, they enjoy increasing popularity, provided that the technical requirements for large semester cohorts are met. Shifting examination processes to computers or tablets entails not only a wealth of new challenges but also opportunities.

Step Up-Not On-The Step 2 Clinical Skills Exam: Directors of Clinical Skills Courses (DOCS) Oppose Ending Step 2 CS

Recently, a student-initiated movement to end the United States Medical Licensing Examination Step 2 Clinical Skills and the Comprehensive Osteopathic Medical Licensing Examination Level 2-Performance Evaluation has gained momentum. These are the only national licensing examinations designed to assess clinical skills competence in the stepwise process through which physicians gain licensure and certification. Therefore, the movement to end these examinations and the ensuing debate merit careful consideration. The authors, elected representatives of the Directors of Clinical Skills Courses, an organization comprising clinical skills educators in the United States and beyond, believe abolishing the national clinical skills examinations would have a major negative impact on the clinical skills training of medical students, and that forfeiting a national clinical skills competency standard has the potential to diminish the quality of care provided to patients. In this Perspective, the authors offer important additional background information, outline key concerns regarding the consequences of ending these national clinical skills examinations, and provide recommendations for moving forward: reducing the costs for students, exploring alternatives, increasing the value and transparency of the current examinations, recognizing and enhancing the strengths of the current examinations, and engaging in a national dialogue about the issue.

Towards the implementation of OSCE in undergraduate nursing curriculum: A qualitative study

BACKGROUND

Despite the fact that Objective Structured Clinical Examination is considered to be an efficient assessment method, their implementation in the undergraduate nursing curriculum encounters difficulties. However, the initiative of the European Higher Education Area to promote competency-based assessment may represent an opportunity to introduce this instrument in undergraduate nursing curriculum.

OBJECTIVE

To explore the perception of nursing faculty members regarding the implementation of the OSCE as an assessment tool in Catalan Nursing Schools.

DESIGN/PARTICIPANTS/SETTING

In this qualitative study, fifteen teachers participated in semi-structured interviews in eight Catalan Nursing Schools.

METHODS

Semi-structured interviews were conducted. A thematic content analysis was used to identify major themes in the interview data and collaborative analysis was undertaken to ensure rigorous results.

RESULTS

The relevant aspects that are emphasized by teachers included the consideration of the dual purpose of the OSCE via its formative and evaluative facets by enhancing the feedback received by students about their performance on the OSCE. The OSCE should be administered towards the end of the degree program and should complement other methods of assessment. Despite its high cost, the OSCE was deemed to be efficient as it enables student competencies to be assessed with objective criteria, which is a difficult task with other instruments. OSCE implementation is feasible with the institutional support of and collaborative work between schools.

CONCLUSIONS

The implementation of the OSCE in the Catalan undergraduate nursing degree programs is feasible if the project receives the support of all involved parties and if creative strategies are determined to reduce economic costs and optimize resources. With adequate feedback, the OSCE is an assessment tool that can provide high-impact training to students.

Supervisor perspectives on the summative in-training assessment

BACKGROUND

Assessment is a fundamental component of medical education and exists in many formats. In-training assessments are one such example and they serve to provide feedback to learners about their performance during a period of clinical attachment. However, in addition to trainee knowledge and performance, many factors influence the assessment given to a trainee.

METHOD

This study used an anonymous survey to investigate the perceptions of supervisors of the influences on their assessments of Australian dermatology trainees, focusing on the summative in-training assessment (SITA) format.

RESULTS

A response rate of 41% was achieved. The importance of reporting underperformance and providing feedback to trainees was agreed on, but current limitations in the ability of the tool to do this were noted. Implications for practice are discussed including the education and support of supervisors, consideration of logistical issues, the process of SITA completion and supervisor appointment. Further research into the impact of supervisor concerns about potential challenges to a judgement and hesitations about making negative comments about a trainee are required. Examination of the trainee perspective is also required.

CONCLUSION

Quality feedback is essential for learners to guide and improve their performance. Supervisors face many potential influences on their assessments and if these are too great, they may jeopardise the quality of the assessment given. Attention to highlighted areas may serve to improve the process, so allowing trainees to develop into the best clinicians they can be.

Encouraging formative assessments of leadership for foundation doctors

BACKGROUND

Clinical leadership is considered essential for maintaining and improving patient care and safety in the UK, and is incorporated in the curriculum for all trainee doctors.

CONTEXT

Despite the growing focus on the importance of leadership, and the introduction of the Medical Leadership Competency Framework (MLCF) in the UK, leadership education for doctors in training is still in its infancy. Assessment is focused on clinical skills, and trainee doctors receive very little formal feedback on their leadership competencies.

INNOVATION

In this article we describe the approach taken by Health Education Kent, Sussex and Surrey (HEKSS) to raise the profile of leadership amongst doctors in training in the South Thames Foundation School (STFS). An annual structured formative assessment in leadership for each trainee has been introduced, supported by leadership education for both trainees and their supervisors in HEKSS trusts. We analysed over 500 of these assessments from the academic year 2012/13 for foundation doctors in HEKSS trusts, in order to assess the quality of the feedback. From the analysis, potential indicators of more effective formative assessments were identified. These may be helpful in improving the leadership education programme for future years.

IMPLICATIONS

There is a wealth of evidence to highlight the importance and value of formative assessments; however, particularly for foundation doctors, these have typically been focused on assessing clinical capabilities. This HEKSS initiative encourages doctors to recognise leadership opportunities at the beginning of their careers, seeks to help them understand the importance of acquiring leadership skills and provides structured feedback to help them improve. Leadership education for doctors in training is still in its infancy.

Clinical vignettes improve performance in anatomy practical assessment

Although medical curricula now adopt an integrated teaching approach, this is not adequately reflected in assessment of anatomy knowledge and skills. In this study, we aimed to explore the impact of the addition of clinical vignette to item stems on students' performance in anatomy practical examinations. In this study, 129 undergraduate medical students of Durham University took part in a 30-item anatomy practical test consisting of those with and without clinical stem, in a crossover design. Classical test theory was used to analyze item difficulty, discrimination index, point biserial, and reliability. Student performance on items with clinical stem and the percentage of students who correctly answered each item was significantly improved by the addition of a clinical stem in the Year 2 cohort. Also, items with a clinical stem showed much better discrimination index than non-clinical items in the Year 2 cohort. In contrast, there was no significant difference in item performance, student performance and discrimination index between items with a clinical and non-clinical stem in Year 1 cohort. Over 65% of test items in both year groups were of good quality with point biserial exceeding 0.2. However, Year 1 test reliability for non-clinical items was better than clinical items. The results raise question as to what level to apply this method of assessment in undergraduate education. Although interpreted on the basis of a relatively small item sample, the findings support the need for improving anatomy practical examinations in line with overriding curricula changes.

Gastrointestinal Endoscopy Competency Assessment Tool: reliability and validity evidence

BACKGROUND

Rigorously developed and validated direct observational assessment tools are required to support competency-based colonoscopy training to facilitate skill acquisition, optimize learning, and ensure readiness for unsupervised practice.

OBJECTIVE

To examine reliability and validity evidence of the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT) for colonoscopy for use within the clinical setting.

DESIGN

Prospective, observational, multicenter validation study. Sixty-one endoscopists performing 116 colonoscopies were assessed using the GiECAT, which consists of a 7-item global rating scale (GRS) and 19-item checklist (CL). A second rater assessed procedures to determine interrater reliability by using intraclass correlation coefficients (ICCs). Endoscopists' first and second procedure scores were compared to determine test-retest reliability by using ICCs. Discriminative validity was examined by comparing novice, intermediate, and experienced endoscopists' scores. Concurrent validity was measured by correlating scores with colonoscopy experience, cecal and terminal ileal intubation rates, and physician global assessment.

SETTING

A total of 116 colonoscopies performed by 33 novice (<50 previous procedures), 18 intermediate (50-500 previous procedures), and 10 experienced (>1000 previous procedures) endoscopists from 6 Canadian hospitals.

MAIN OUTCOME MEASUREMENTS

Interrater and test-retest reliability, discriminative, and concurrent validity.

RESULTS

Interrater reliability was high (total: ICC=0.85; GRS: ICC=0.85; CL: ICC=0.81). Test-retest reliability was excellent (total: ICC=0.91; GRS: ICC=0.93; CL: ICC=0.80). Significant differences in GiECAT scores among novice, intermediate, and experienced endoscopists were noted (P<.001). There was a significant positive correlation (P<.001) between scores and number of previous colonoscopies (total: ρ=0.78, GRS: ρ=0.80, CL: Spearman's ρ=0.71); cecal intubation rate (total: ρ=0.81, GRS: Spearman's ρ=0.82, CL: Spearman's ρ=0.75); ileal intubation rate (total: Spearman's ρ=0.82, GRS: Spearman's ρ=0.82, CL: Spearman's ρ=0.77); and physician global assessment (total: Spearman's ρ=0.90, GRS: Spearman's ρ=0.94, CL: Spearman's ρ=0.77).

LIMITATIONS

Nonblinded assessments.

CONCLUSION

This study provides evidence supporting the reliability and validity of the GiECAT for use in assessing the performance of live colonoscopies in the clinical setting.

Right person, right skills, right job: the contribution of objective structured clinical examinations in advancing staff nurse experts

Recruitment processes need to discriminate among candidates to ensure that the right person with the right skills is selected for advancement opportunities. An innovative recruitment process using an objective structured clinical examination grounded in best practice guidelines resulted in improved recruitment practices for senior nursing clinical expert roles. Candidates' skills, knowledge, and attitudes in the areas of patient focus, clinical expertise, teamwork, and leadership were assessed using a clinical simulation. Candidates achieving advancement were assessed at 6 months to validate the efficacy of the process.

From Flexner to competencies: reflections on a decade and the journey ahead

This article is a sequel to one published in 2002 only a few years after the initiation of the shift to competency-based medical education (CBME). The authors reflect on the major forces that have influenced the movement and tipped the balance toward widespread adoption of CBME in the United States, primarily in graduate medical education. These forces include regulatory bodies, international counterparts, and the general public. The authors highlight the most important lessons learned over the decade. These include (1) the need for standardization of language to develop a shared vision of the path ahead, (2) the power of direct observation in assessment, (3) the challenge of developing meaningful measures of performance, (4) desired outcomes as the starting point for curriculum development, (5) dependence on reflection in the development of expertise, (6) the need for exploiting the role of learners in their learning, and (7) competent clinical systems as the required learning environment for producing competent physicians.The authors speculate on why this most recent attempt to shift to CBME differs from previous aborted attempts. They conclude by explaining how the recent lessons learned inform the vision of what successful implementation of CBME would look like, and discussing the importance of milestones, entrustable professional activities, and an integrated, rather than a reductionist, approach to assessment of competence. The fundamental question at each step along the way in implementing CBME should be "How do we improve medical education to provide better care for patients?"

Relationship of pass/fail grading and curriculum structure with well-being among preclinical medical students: a multi-institutional study

PURPOSE

Psychological distress is common among medical students. Curriculum structure and grading scales are modifiable learning environment factors that may influence student well-being. The authors sought to examine relationships among curriculum structures, grading scales, and student well-being.

METHOD

The authors surveyed 2,056 first- and second-year medical students at seven U.S. medical schools in 2007. They used the Perceived Stress Scale, Maslach Burnout Inventory, and Medical Outcomes Study Short Form (SF-8) to measure stress, burnout, and quality of life, respectively. They measured curriculum structure using hours spent in didactic, clinical, and testing experiences. Grading scales were categorized as two categories (pass/fail) versus three or more categories (e.g., honors/pass/fail).

RESULTS

Of the 2,056 students, 1,192 (58%) responded. In multivariate analyses, students in schools using grading scales with three or more categories had higher levels of stress (beta 2.65; 95% CI 1.54-3.76, P<.0001), emotional exhaustion (beta 5.35; 95% CI 3.34-7.37, P<.0001), and depersonalization (beta 1.36; 95% CI 0.53-2.19, P=.001) and were more likely to have burnout (OR 2.17; 95% CI 1.41-3.35, P=.0005) and to have seriously considered dropping out of school (OR 2.24; 95% CI 1.54-3.27, P<.0001) compared with students in schools using pass/fail grading. There were no relationships between time spent in didactic and clinical experiences and well-being.

CONCLUSIONS

How students are evaluated has a greater impact than other aspects of curriculum structure on their well-being. Curricular reform intended to enhance student well-being should incorporate pass/fail grading.

Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference

In this article, we outline criteria for good assessment that include: (1) validity or coherence, (2) reproducibility or consistency, (3) equivalence, (4) feasibility, (5) educational effect, (6) catalytic effect, and (7) acceptability. Many of the criteria have been described before and we continue to support their importance here. However, we place particular emphasis on the catalytic effect of the assessment, which is whether the assessment provides results and feedback in a fashion that creates, enhances, and supports education. These criteria do not apply equally well to all situations. Consequently, we discuss how the purpose of the test (summative versus formative) and the perspectives of stakeholders (examinees, patients, teachers-educational institutions, healthcare system, and regulators) influence the importance of the criteria. Finally, we offer a series of practice points as well as next steps that should be taken with the criteria. Specifically, we recommend that the criteria be expanded or modified to take account of: (1) the perspectives of patients and the public, (2) the intimate relationship between assessment, feedback, and continued learning, (3) systems of assessment, and (4) accreditation systems.

How to measure success: the impact of scholarly concentrations on students--a literature review

PURPOSE

Scholarly concentrations (SCs) are elective or required curricular experiences that give students opportunities to study subjects in-depth beyond the conventional medical curriculum and require them to complete an independent scholarly project. This literature review explores the question, "What impact do SC programs have on medical students?"

METHOD

In 2008, the authors retrieved published articles using Medline, ERIC, and PsycINFO electronic databases and scanned reference lists to locate additional citations. They extracted data from selected articles using a structured form and used Kirkpatrick's evaluation model to organize learner outcomes into four categories: reactions, learning, behavior, and results.

RESULTS

Of 1,640 citations, 82 full-text papers were considered, and 39 studies met inclusion criteria. Most articles described SC programs that offered students research opportunities. Fourteen articles provided evidence that SC experiences influenced students' choice of clinical specialty or fostered their interest in research. Eight articles reported that SCs improved students' understanding of research principles and methods. Nineteen articles reported publications and presentations to document students' ability to apply acquired knowledge and skills. Twelve studies confirmed the entry of SC graduates into academic medicine with continued engagement in research or success in obtaining grant funding. Students' criticisms focused on requiring research during clinical training and the effort needed to complete scholarly projects.

CONCLUSIONS

The diversity of articles and variable results prevent definitive conclusions about the value of SCs. Findings suggest several implications for future SC program evaluations and educational research. The authors advocate increased rigor in evaluation designs to demonstrate SCs' true impact.

Evaluation of essay questions used to assess medical students' application and integration of basic and clinical science knowledge

BACKGROUND

Educators need approaches to assess medical students' abilities to apply and integrate concepts essential to medical practice.

DESCRIPTION

We used a multimethod approach to examine the quality of essay questions intended to elicit medical students' ability to apply and integrate their understanding of medical concepts.

EVALUATION

Three educators assigned essay questions (n = 120) to one of four levels of cognition. Kappa was computed before and after discussion. Faculty (n = 46) critiqued essay quality using a checklist (97% response), and students completed a questionnaire about the learning environment (91% response).

CONCLUSIONS

We identified effective approaches to evaluate the quality of essay questions and to train faculty to write essay questions of sufficient complexity. This systematic review of essay questions also encouraged review of the curriculum to determine if core concepts were being taught. It is feasible to have faculty write and critique essay questions targeted at higher levels of cognition.

The objective structured clinical examination (OSCE): optimising its value in the undergraduate nursing curriculum

This article explores the use of the objective structured clinical examination (OSCE) in undergraduate nursing education. The advantages and limitations of this assessment approach are discussed and various applications of the OSCE are described. Attention is given to the complexities of evaluating some psychosocial competency components. The issues are considered in an endeavour to delineate the competency components, or skill sets, that best lend themselves to assessment by the OSCE. We conclude that OSCEs can be used most effectively in nurse undergraduate curricula to assess safe practice in terms of performance of psychomotor skills, as well as the declarative and schematic knowledge associated with their application. OSCEs should be integrated within a curriculum in conjunction with other relevant student evaluation methods.

Reliability and validity of reflection exercises for obstetrics and gynecology residents

OBJECTIVE

Self-evaluation is an essential skill throughout a physician's career, and reflection is thought to be a necessary mechanism for effective self-evaluation. The aim of our study was to establish the reliability and validity of structured assessments of critical self-reflection.

STUDY DESIGN

Thirty-two residents completed 6 exercises that were scored from 0 (no description of event) to 6 (deep reflection). We calculated interrater and internal consistency reliability for the exercises and compared scores by postgraduate year and with other competency assessments.

RESULTS

Residents completed 183 reflections. Interrater reliability was 0.89. Surgical skill reflections scored highest (score, 3.2 +/- 0.91 [SD]). Five exercises had adequate internal consistency reliability (0.62). Senior residents received higher reflection scores than junior residents; the magnitude of difference was similar for other competency measures and not statistically significant. Reflection scores were correlated with professionalism and communication skill assessments (score, 0.36-0.37; P < .01) but not with medical knowledge.

CONCLUSION

Self-reflection can be assessed reliably with scored exercises that demonstrate concurrent validity with other assessments. We encourage further research that should include multiple training programs to further evaluate our approach for the assessment of reflection in postgraduate education.

Student perspectives of assessment by TEMM model in physiology

Assessment is the process by which the teacher and the student gain knowledge about student progress. Assessment systems should aim at evaluating the desired learning outcomes. In Melaka Manipal Medical College, (Manipal Campus), Manipal, India, the TEMM model (consisting of 4 assessment methods: Triple Jump Test, essay incorporating critical thinking questions, Multistation Integrated Practical Examination, and multiple choice questions) was introduced to 30 refresher students in the fourth block of the academic year. At the end of the block, a questionnaire was distributed to ask the students to rank the different assessments in the order of their preference with respect to seven items. Analysis of the results showed that not a single type of assessment was ranked highest for all the seven items, proving the earlier observation that a single assessment does not fulfill all aspects of assessment and that there is a need for an evaluating system with multiple ways of assessment.

The effect of in-training assessment on clinical confidence in postgraduate education

INTRODUCTION

The literature on how in-training assessment (ITA) works in practice and what educational outcomes can actually be achieved is limited. One of the aims of introducing ITA is to increase trainees' clinical confidence; this relies on the assumption that assessment drives learning through its content, format and programming. The aim of this study was to investigate the effect of introducing a structured ITA programme on junior doctors' clinical confidence. The programme was aimed at first year trainees in anaesthesiology.

METHODS

The study involved a nationwide survey of junior doctors' self-confidence in clinical performance before (in 2001) and 2 years after (in 2003) the introduction of an ITA programme. Respondents indicated confidence on a 155-item questionnaire related to performance of clinical skills and tasks reflecting broad aspects of competence. A total of 23 of these items related to the ITA programme.

RESULTS

The response rate was 377/531 (71%) in 2001 and 344/521 (66%) in 2003. There were no statistically significant differences in mean levels of confidence before and 2 years after the introduction of the ITA programme - neither in aspects that were related to the programme nor in those that were unrelated to the programme.

DISCUSSION

This study demonstrates that the introduction of a structured ITA programme did not have any significant effect on trainees' mean level of confidence on a broad range of aspects of clinical competence. The importance of timeliness and rigorousness in the application of ITA is discussed.

Implementation of a formal in-training assessment programme in anaesthesiology and preliminary results of acceptability

BACKGROUND

A new reform on postgraduate education in Denmark requires a formal in-training assessment in all specialties. The aim of this study was to survey the implementation and acceptability of the first example of a nation-wide in-training assessment programme for first-year trainees in anaesthesiology developed by a working group under the Danish Society of Anaesthesiology and Intensive Care Medicine.

METHODS

A questionnaire about the implementation of the programme in practice and the characteristics of the trainees was sent to the educational responsible consultant (ERC) in each of the 26 anaesthetic departments in the country with first-year trainees in anaesthesiology. Standard evaluations of the assessment programme were regularly collected from trainees.

RESULTS

Twenty-five (96%) departments returned the questionnaire. In total the departments reported on 100 trainees and 83 of these had been enrolled in the programme. Thirteen departments reported in total on 27 trainees who had completed their first year of training and these departments had applied a median 21 (range 17-21) of the 21 tests included in the entire programme. Time constraints and resistance among senior clinicians were the most frequently cited barriers to implementation. Evaluations from trainees showed a generally positive attitude towards most of the programme. They especially praised the programme's effect on structuring training and having a positive effect on learning.

CONCLUSION

The in-training assessment programme has been widely implemented across the country. The majority of the programme was acceptable to trainees and had a positive effect on structuring training and on fostering learning.

Reflective critique and collaborative practice in evaluation: promoting change in medical education

The School of Medicine at the University of Tasmania has recently begun to implement a process of curriculum evaluation,which aims to reflect contemporary best practice in evaluation in tertiary pedagogy and medical education. Best practice must accommodate a broadening interest in cooperative and collaborative evaluation strategies among stakeholders, advances in applied qualitative educational research and recognition that critical reflection on practice is the cornerstone of professional education. This paper reports a recent evaluation strategy in a specific year-long unit in the second year of a six-year undergraduate medical degree. The paper begins by presenting the context; it then discusses and rationalises the evaluation strategy and presents findings. The paper concludes by arguing that curriculum evaluation as best practice must be reflective, informed by the scholarship of medical education, and internalized as a dynamic process that can promote sustainable change and improvement in medical curricula. Such an approach will contribute to an undergraduate medical curriculum that prepares students for the demands and complexities of medical practice.

Consultants' opinion on a new practice-based assessment programme for first-year residents in anaesthesiology

BACKGROUND

Assessment in postgraduate education is moving towards using a broad spectrum of practice-based assessment methods. This approach was recently introduced in first-year residency in anaesthesiology in Denmark. The new assessment programme covers: clinical skills, communication skills, organizational skills and collaborative skills, scholarly proficiencies and professionalism. Eighteen out of a total of 21 assessment instruments were used for pass/fail decisions. The aim of this study was to survey consultants' opinions of the programme in terms of the representativeness of competencies tested, the suitability of the programme as a basis for pass/fail decisions and the relevance and sufficiency of the content of the different assessment instruments.

METHODS

A description of the assessment programme and a questionnaire were sent to all consultants of anaesthesiology in Denmark. The questionnaire consisted of items, to be answered on a five-point scale, asking the consultants' opinions about representativeness, suitability and content of the programme.

RESULTS

The response rate was 251/382 (66%). More than 75% of the respondents agreed that the assessment programme offered adequate coverage of the competencies of a first-year resident and was appropriate for making pass/fail decisions. There was strong agreement that the content of the 18 tests used for pass/fail decisions was relevant and sufficient for pass/fail decisions.

CONCLUSION

Judging from the consultants' opinions, the assessment programme for first-year residency in anaesthesiology appears to be appropriate regarding the range of competencies assessed, the appropriateness as a basis for pass/fail decisions, and regarding the content of the tests used for pass/fail decisions. Further studies are needed to assess the feasibility and acceptability of the programme in practice.