Workplace-based assessment as an educational tool: Guide supplement 31.3--viewpoint

Educators' perceptions and challenges of student assessment process at Prince Sattam Bin Abdulaziz University dentistry program: a qualitative study

BACKGROUND

As part of Saudi Vision 2030, there is increasing demand for dentistry colleges to provide training in the student assessment process. Assessment is the process of accurately determining a learner's skills across multiple educational domains. The objectives of this study were to investigate teachers' perspectives, assessment challenges, and make recommendations for improving the assessment process of undergraduate dental students at Prince Sattam bin Abdulaziz University in Al-Kharj, Saudi Arabia.

METHODS

A qualitative study employed the grounded theory approach following purposive sampling. Four focus group interviews were conducted with course directors from the College of Dentistry at Prince Sattam bin Abdulaziz University (PSAU) using open-ended questions for data collection. Otter software was used for the transcription and NVivo 14 for the data analysis.

RESULTS

Four themes emerged: perspectives on the assessment process, summative and formative assessments, challenges of assessments, and proposed solutions to the assessment challenges. Most educators perceived assessments as assessments of learning, with the planning and execution of assessments requiring regulation. Different feedback models were occasionally used by examiners to improve student performance. Examiner standardization training, communication, and calibration were lacking, according to the educators in this study.

CONCLUSION

The challenges of the assessment process in the College of Dentistry at PSAU are multifactorial including the examiners themselves, students, and the college. These challenges indicated the need for a tailor-made, appropriately designed faculty development training program related to different methods of student assessment.

To evaluate the effect of small-group demonstration-based training module in improving clinical skills in undergraduate medical students

BACKGROUND

In today's modern era, the two most important aspects of medical education are evaluation- and outcome-based learning. Directly Observed Procedural Skills (DOPSs ) is a well-known method of evaluation which constitutes a direct observation of the practical skills being performed by the student and simultaneous written feedback by the teacher.

METHOD

A total of 40 undergraduate students were taught by a module based on DOPS. A pre-test and post-test was conducted on DOPS examination pattern and was compared by the Cochran's Q test.

RESULTS

In this study, a total of 40 medical undergraduate students and 10 teachers participated. Each student was given one pre-test and five post-test with ultrasound proven mild to moderate splenomegaly. Each student was individually assessed on a module based on DOPS and was given direct feedback by the teachers. A total of 34 (85%) students strongly agreed that they felt comfortable and confident with this methodology.

CONCLUSION

The results of this study revealed that DOPS tests can be used as an effective evaluation method to assess medical students because of its appropriate validity and reliability, positive impact on learning, and high satisfaction level amongst students. However, special attention needs to be given to the quality of these tests.

Implementation and Use of Workplace-Based Assessment in Clinical Learning Environments: A Scoping Review

PURPOSE

Workplace-based assessment (WBA) serves a critical role in supporting competency-based medical education (CBME) by providing assessment data to inform competency decisions and support learning. Many WBA systems have been developed, but little is known about how to effectively implement WBA. Filling this gap is important for creating suitable and beneficial assessment processes that support large-scale use of CBME. As a step toward filling this gap, the authors describe what is known about WBA implementation and use to identify knowledge gaps and future directions.

METHOD

The authors used Arksey and O'Malley's 6-stage scoping review framework to conduct the review, including: (1) identifying the research question; (2) identifying relevant studies; (3) study selection; (4) charting the data; (5) collating, summarizing, and reporting the results; and (6) consulting with relevant stakeholders.

RESULTS

In 2019-2020, the authors searched and screened 726 papers for eligibility using defined inclusion and exclusion criteria. One hundred sixty-three met inclusion criteria. The authors identified 5 themes in their analysis: (1) Many WBA tools and programs have been implemented, and barriers are common across fields and specialties; (2) Theoretical perspectives emphasize the need for data-driven implementation strategies; (3) User perceptions of WBA vary and are often dependent on implementation factors; (4) Technology solutions could provide useful tools to support WBA; and (5) Many areas of future research and innovation remain.

CONCLUSIONS

Knowledge of WBA as an implemented practice to support CBME remains constrained. To remove these constraints, future research should aim to generate generalizable knowledge on WBA implementation and use, address implementation factors, and investigate remaining knowledge gaps.

Evaluating the effectiveness of a preclinical practice of tooth preparation using digital training system: A randomised controlled trial

PURPOSE

This study aimed to evaluate the effectiveness of preclinical training on ceramic crown preparation using digital training system compared with traditional training method.

MATERIALS AND METHODS

A sample of 66 participating fourth-year undergraduate dental students were randomly assigned to one of two groups for ceramic crown preparation: experimental group as trained using digital method with Online Peer-Review System (OPRS) and Real-time Dental Training and Evaluation System (RDTES); control group as trained using traditional method with instructor demonstration and evaluation. At the completion of training periods, both groups performed the preparation of ceramic crown of upper left central incisor, which were blindly scored by the experienced instructors under the pre-defined assessment criteria. The results of both were compared using Student's t-test or Wilcoxon signed rank test. The level of significance was P < .05. The questionnaires regarding the benefits or drawbacks of digital training system were answered by the students of experimental group.

RESULTS

Five of 15 items in the assessment outcome of the digital group were significantly better than the traditional group. The questionnaire results from the students of the experimental group indicated 96.97% of the students agreed or strongly agreed that using digital training system could better improve the practical ability than traditional method. The total scores of practical results were significantly positively correlated with the points of the questionnaires.

CONCLUSION

The digital training system with OPRS and RDTES might be a good alternative to the traditional training method in the preclinical course of dental practice.

Value of supervised learning events in predicting doctors in difficulty

CONTEXT

In the UK, supervised learning events (SLE) replaced traditional workplace-based assessments for foundation-year trainees in 2012. A key element of SLEs was to incorporate trainee reflection and assessor feedback in order to drive learning and identify training issues early. Few studies, however, have investigated the value of SLEs in predicting doctors in difficulty. This study aimed to identify principles that would inform understanding about how and why SLEs work or not in identifying doctors in difficulty (DiD).

METHODS

A retrospective case-control study of North West Foundation School trainees' electronic portfolios was conducted. Cases comprised all known DiD. Controls were randomly selected from the same cohort. Free-text supervisor comments from each SLE were assessed for the four domains defined in the General Medical Council's Good Medical Practice Guidelines and each scored blindly for level of concern using a three-point ordinal scale. Cumulative scores for each SLE were then analysed quantitatively for their predictive value of actual DiD. A qualitative thematic analysis was also conducted.

RESULTS

The prevalence of DiD in this sample was 6.5%. Receiver operator characteristic curve analysis showed that Team Assessment of Behaviour (TAB) was the only SLE strongly predictive of actual DiD status. The Educational Supervisor Report (ESR) was also strongly predictive of DiD status. Fisher's test showed significant associations of TAB and ESR for both predicted and actual DiD status and also the health and performance subtypes. None of the other SLEs showed significant associations. Qualitative data analysis revealed inadequate completion and lack of constructive, particularly negative, feedback. This indicated that SLEs were not used to their full potential.

CONCLUSIONS

TAB and the ESR are strongly predictive of DiD. However, SLEs are not being used to their full potential, and the quality of completion of reports on SLEs and feedback needs to be improved in order to better identify and manage DiD.

Workplace-based assessment: a review of user perceptions and strategies to address the identified shortcomings

Workplace based assessments (WBAs) are now commonplace in postgraduate medical training. User acceptability and engagement is essential to the success of any medical education innovation. To this end, possessing an insight into trainee and trainer perceptions towards WBAs will help identify the major problems, permitting strategies to be introduced to improve WBA implementation. A review of literature was performed to identify studies examining trainee and trainer perceptions towards WBAs. Studies were excluded if non-English or sampling a non-medical/dental population. The identified literature was synthesised for the purpose of this critical narrative review. It is clear that there is widespread negativity towards WBAs in the workplace. This has negatively impacted on the effectiveness of WBA tools as learning aids. This negativity exists in trainees but also to an extent in their trainers. Insight gained from the literature reveals three dominant problems with WBA implementation: poor understanding as to the purpose of WBAs; insufficient time available for undertaking these assessments; and inadequate training of trainers. Approaches to addressing these three problems with WBA implementation are discussed. It is likely that a variety of solutions will be required. The prevalence of negativity towards WBAs is substantial in both trainees and trainers, eroding the effectiveness of learning that is consequent upon them. The educational community must now listen to the concerns being raised by the users and consider the range of strategies being proposed to improve the experiences of trainees, and their trainers.

The influence of students' prior clinical skills and context characteristics on mini-CEX scores in clerkships--a multilevel analysis

BACKGROUND

In contrast to objective structured clinical examinations (OSCEs), mini-clinical evaluation exercises (mini-CEXs) take place at the clinical workplace. As both mini-CEXs and OSCEs assess clinical skills, but within different contexts, this study aims at analyzing to which degree students' mini-CEX scores can be predicted by their recent OSCE scores and/or context characteristics.

METHODS

Medical students participated in an end of Year 3 OSCE and in 11 mini-CEXs during 5 different clerkships of Year 4. The students' mean scores of 9 clinical skills OSCE stations and mean 'overall' and 'domain' mini-CEX scores, averaged over all mini-CEXs of each student were computed. Linear regression analyses including random effects were used to predict mini-CEX scores by OSCE performance and characteristics of clinics, trainers, students and assessments.

RESULTS

A total of 512 trainers in 45 clinics provided 1783 mini-CEX ratings for 165 students; OSCE results were available for 144 students (87%). Most influential for the prediction of 'overall' mini-CEX scores was the trainers' clinical position with a regression coefficient of 0.55 (95%-CI: 0.26-0.84; p < .001) for residents compared to heads of department. Highly complex tasks and assessments taking place in large clinics significantly enhanced 'overall' mini-CEX scores, too. In contrast, high OSCE performance did not significantly increase 'overall' mini-CEX scores.

CONCLUSION

In our study, Mini-CEX scores depended rather on context characteristics than on students' clinical skills as demonstrated in an OSCE. Ways are discussed which focus on either to enhance the scores' validity or to use narrative comments only.

The use of mini-CEX in UK foundation training six years following its introduction: lessons still to be learned and the benefit of formal teaching regarding its utility

BACKGROUND

The mini-clinical evaluation exercise (mini-CEX) is a widely used tool with a strong theoretical basis. It was introduced to UK foundation training in 2005.

AIMS

To assess current experiences, opinions and attitudes towards mini-CEX amongst foundation doctors, and explore what factors underpin these.

METHODS

Data were collected from foundation trainees via an on-line questionnaire.

RESULTS

Ninety-eight per cent of respondents had used mini-CEX during FY1, however, only 32% had ever received formal teaching regarding its use. In terms of understanding of the purpose of mini-CEX, only 30% of trainees commented on there being a formative aspect or requirement for feedback. The majority of trainees did not feel that mini-CEX was a useful part of their training. The main themes were the poor attitude and understanding of assessors and difficulties finding sufficient time. However, those who had received formal teaching as students regarding the use of mini-CEX were significantly more likely as postgraduates to find it beneficial (p = 0.031).

CONCLUSIONS

A more concerted effort to educate trainees and assessors regarding the correct use of mini-CEX will enhance its educational value. Increased education during undergraduate training regarding use of formative assessment may lead to more effective utilisation in the postgraduate setting.

The role of feedback in improving the effectiveness of workplace based assessments: a systematic review

BACKGROUND

With recent emphasis placed on workplace based assessment (WBA) as a method of formative performance assessment, there is limited evidence in the current literature regarding the role of feedback in improving the effectiveness of WBA. The aim of this systematic review was to elucidate the impact of feedback on the effectiveness of WBA in postgraduate medical training.

METHODS

Searches were conducted using the following bibliographic databases to identify original published studies related to WBA and the role of feedback: Medline (1950-December 2010), Embase (1980-December 2010) and Journals@Ovid (English language only, 1996-December 2010). Studies which attempted to evaluate the role of feedback in WBA involving postgraduate doctors were included.

RESULTS

15 identified studies met the inclusion criteria and minimum quality threshold. They were heterogeneous in methodological design. 7 studies focused on multi source feedback, 3 studies were based on mini-clinical evaluation exercise, 2 looked at procedural based assessment, one study looked at workplace based assessments in general and 2 studies looked at a combination of 3 to 6 workplace based assessments. 7 studies originated from the United Kingdom. Others were from Canada, the United States and New Zealand. Study populations were doctors in various grades of training from a wide range of specialties including general practice, general medicine, general surgery, dermatology, paediatrics and anaesthetics. All studies were prospective in design, and non-comparative descriptive or observational studies using a variety of methods including questionnaires, one to one interviews and focus groups.

CONCLUSIONS

The evidence base contains few high quality conclusive studies and more studies are required to provide further evidence for the effect of feedback from workplace based assessment on subsequent performance. There is, however, good evidence that if well implemented, feedback from workplace based assessments, particularly multisource feedback, leads to a perceived positive effect on practice.

Opportunities for learning in the surgical workplace and how they can be exploited: a practical guide

Training times and the working week have been shortened but the amount needed to be learned remains the same. This article demonstrates how trainees in surgery can incorporate learning opportunities into their normal working day.