American Dental Education Association Compendium Entrustable Professional Activities Workgroup report

PURPOSE

Entrustable professional activities (EPAs) are discrete clinical tasks that can be evaluated to help define readiness for independent practice in the health professions and are intended to increase trust in the dental graduate. EPAs provide a framework that bridges competencies to clinical practice. This report describes the work of the American Dental Education Association (ADEA) Compendium EPA Workgroup to develop a list of EPAs for dental education and supportive resources, including specifications and a glossary.

METHODS

Preliminary work including literature and resource review, mapping of existing competencies, and review of other health professions' EPAs informed the development of our EPAs list. Workgroup members achieved consensus using a modified Delphi process. A Qualtrics survey using a validated rubric for the assessment of EPAs as described in peer-reviewed literature was used. Dental educators, including academic deans, were surveyed for feedback on the content and format of the EPAs.

RESULTS

Based on findings in the literature analysis of existing EPAs and competencies in health professions, a list of EPAs was developed along with a description of specifications. The EPA workgroup (nine members from multiple institutions) used the Delphi process in receiving feedback from various experts. A list of 11 core EPAs was vetted by dental educators including academic deans (n = ∼23), and the process of development was reviewed by EPAs experts outside dental education. A glossary was developed to align language.

CONCLUSION

These EPAs define the scope of dental practice. This report represents Phase 1 of the EPA framework development and vetting process. Future directions will include a broader vetting of the EPA list, faculty development, and national standardized technology that support this work to optimize implementation.

Experiences of the use of portfolio in dental education - A qualitative meta-ethnographic literature review

INTRODUCTION

Portfolio with a collection of evidence has become popular in higher education, including dental education. It is valuable to study the experiences of the use and implementation processes. Meta-ethnography can be a suitable method to analyse, synthesize and construct interpretations of qualitative research. Our aim was to explore experiences from the use of a portfolio/e-portfolio in dental education, from the students' and teachers' perspectives.

MATERIALS AND METHODS

A systematic search in the databases PubMed, Scopus and ERC was performed, and the established seven steps of a meta-ethnographic review were used. 278 papers were initially identified, and seven were included in the final analysis.

RESULTS

Two themes (Issues to Address and Consequences) and five subthemes (Purpose, Roles, Support and Structure, Challenges and Enablers, and Gains) were constructed.

DISCUSSION

Our synthesis reflects various challenges, yet the learning gains are recognized and expressed to be important once the students and teachers have overcome early thresholds. Beyond the conclusions drawn in each paper, our synthesis provides new perspectives on the complexity of an implementation process and the balance of not seeing the woods for the trees being overwhelmed by technical and other practical aspects, reducing the opportunity for learning.

CONCLUSION

The portfolio implementation in undergraduate dental education should address clarification to all stakeholders of the purpose and role, presenting a purposeful portfolio structure and timely support.

Rubric vs. numeric rating scale: agreement among evaluators on endodontic treatments performed by dental students

BACKGROUND

Students´ assessment should be carried out in an effective and objective manner, which reduces the possibility of different evaluators giving different scores, thus influencing the qualification obtained and the consistency of education. The aim of the present study was to determine the agreement among four evaluators and compare the overall scores awarded when assessing portfolios of endodontic preclinical treatments performed by dental students by using an analytic rubric and a numeric rating scale.

METHODS

A random sample of 42 portfolios performed by fourth-year dental students at preclinical endodontic practices were blindly assessed by four evaluators using two different evaluation methods: an analytic rubric specifically designed and a numeric rating scale. Six categories were analyzed: radiographic assessment, access preparation, shaping procedure, obturation, content of the portfolio, and presentation of the portfolio. The maximum global score was 10 points. The overall scores obtained with both methods from each evaluator were compared by Student's t, while agreement among evaluators was measured by Intraclass correlation coefficients (ICC). The influence of the difficulty of the endodontic treatment on the evaluators´ scores was analyzed by one-way ANOVA. Statistical tests were performed at a pre-set alpha of 0.05 using Stata 16.

RESULTS

Difficulty of canal treatment did not influence the scores of evaluators, irrespective of the evaluation method used. When the analytic rubric was used, inter-evaluator agreement was substantial for radiographic assessment, access preparation, shaping procedure, obturation, and overall scores. Inter-evaluator agreement ranged from moderate to fair with the numeric rating scale. Mean higher overall scores were achieved when numeric rating scale was used. Presentation and content of the portfolio showed slight and fair agreement, respectively, among evaluators, regardless the evaluation method applied.

CONCLUSIONS

Assessment guided by an analytic rubric allowed evaluators to reach higher levels of agreement than those obtained when using a numeric rating scale. However, the rubric negatively affected overall scores.

Rubric development and peer review for predoctoral dental education

PURPOSE

This article outlines a process and template for developing rubrics that can be shared across predoctoral dental education programs.

METHODS

The rubrics were developed using a peer-review process for incorporating clinical skills and professional behaviors required for predoctoral dental students in their development of competent independent practice.

RESULTS

This article shares the process, templates, and rubrics that were developed for a new predoctoral dental program. These rubrics can be implemented across various educational settings including didactic curriculum, preclinical, clinical, and extra-mural educational experiences. Successful rubric implementation requires the identification of a software and its key features.

CONCLUSION

These rubrics are presented with the intent to share among dental education institutions looking to identify novel ways for longitudinal student assessment. These rubrics also offer the opportunity for collaborative use among various dental schools.

The best mirror of the students' longitudinal performance: Portfolio or structured oral exam assessment at clerkship?

OBJECTIVE

This study aimed to compare the strength of association (i.e., explained variability) of the cumulative grade point average (GPA) with the grades obtained in the clerkship portfolio and the final structured oral exam by dental students.

METHODS

A prospective longitudinal study was designed to analyze quantitative data from three cohorts of dental school students. Univariate and multivariate linear regression models were built to evaluate the association between the students' cumulative GPA with the grades obtained in their clerkship portfolio and the final structured oral exam.

RESULTS

In total, 171 students in the last year of the undergraduate program were considered (76% women, age average 24.8 ± 1.6 years). The dental students' grades of both portfolio and structured oral exam were significantly associated with the GPA score but with different strengths of association. The clerkship portfolio was more strongly associated with cumulative GPA than the structured oral exam (R²  = 19.6% versus R²  = 7.6%). On the opposite, the association between the structured oral exam and GPA can be interpreted as a lower precision in its practical significance and thus reflecting different concurrent validity.

CONCLUSIONS

Considering the results of this study, it could probably incline the balance toward the portfolio because it may be closer to a programmatic assessment model, with timely feedback, development of metacognition, and the achievement of formative process measurement rather than evidence of a single instance of examination.

Reliability of a grading rubric designed to evaluate the reflective ability of predoctoral dental students

INTRODUCTION

Reflection and self-assessment are critical skills for healthcare providers. Identification of gaps in knowledge, skills and attitudes, along with the ability to critically think and problem solve to fill gaps, is the ultimate outcome for lifelong learning. The aims of this study were to (a) refine an instrument used for measuring reflective ability, and conduct comprehensive reliability testing, and (b) describe a process for rater calibration.

MATERIALS AND METHODS

Students develop e-portfolios over a four-year span with assignments that require reflection and self-assessment. The final piece of the portfolio includes a global reflection written the last semester of the programme. Three faculty raters independently evaluated 106 dental students' global reflections using the revised grading rubric. An intraclass correlation coefficient measured the level of agreement between the three raters.

RESULTS

Analysis of the 318 faculty ratings (106/rater) resulted in an intraclass correlation of .708. Based on a 5-point grading scale (0 = does not respond to the assignment to 5 = reconstructing), the ratings of the 106 global reflections ranged from 1.3 to 5.0 (M = 3.1, SD =0.66).

DISCUSSION

This study provides confidence in the reliability of a grading rubric designed to assess reflective ability, along with suggestions for calibration. An overall mean of 3.1 (Level 3 = relating-includes evidence of lessons learned) illustrates the complexity of teaching reflection and self-assessment.

CONCLUSION

Use of a reliable grading rubric for assessing reflective writing could assist schools interested in incorporating reflection and self-assessment into the curriculum, ultimately supporting lifelong and enhanced health care.

Twelve tips for embedding assessment for and as learning practices in a programmatic assessment system

Programmatic assessment supports the evolution from assessment of learning to fostering assessment for learning and as learning practices. A well-designed programmatic assessment system aligns educational objectives, learning opportunities, and assessments with the goals of supporting student learning, making decisions about student competence and promotion decisions, and supporting curriculum evaluation. We present evidence-based guidance for implementing assessment for and as learning practices in the pre-clinical knowledge assessment system to help students learn, synthesize, master and retain content for the long-term so that they can apply knowledge to patient care. Practical tips are in the domains of culture and motivation of assessment, including how an honour code and competency-based grading system can support an assessment system to develop student self-regulated learning and professional identity, curricular assessment structure, such as how and when to utilize low-stakes and cumulative assessment to drive learning, exam and question structure, including what authentic question and exam types can best facilitate learning, and assessment follow-up and review considerations, such exam retake processes to support learning, and academic success structures. A culture change is likely necessary for administrators, faculty members, and students to embrace assessment as most importantly a learning tool for students and programs.

ADEA-ADEE Shaping the Future of Dental Education III: Assessment in competency-based dental education: Ways forward

Assessment in competency-based dental education continues to be a recognized area for growth and development within dental programs around the world. At the joint American Dental Education Association (ADEA) and the Association for Dental Education in Europe (ADEE) 2019 conference, Shaping the Future of Dental Education III, the workshop on assessment was designed to continue the discussion started in 2017 at the ADEA-ADEE Shaping the Future of Dental Education II.¹ The focus of the 2019 conference involved examining the potential of entrustable professional activities (EPAs) and current thinking about workplace-based assessment (WBA) within competency-based education in the 21st century. Approximately 30 years ago, George Miller wrote about the assessment of competence in medical education and challenged faculty to reach for higher levels of assessment than knowledge or skill.² Acknowledging that no one assessment method can result in a valid assessment of competence, Miller proposed a four-level framework for assessment. The lowest level involves measuring what students know ("knows"), followed by assessment of the skill with which knowledge is applied in relevant tasks or problems ("knows how"). Next is an assessment of task performance in standardized settings ("shows how"), and finally, the highest level assesses the student's performance in the unstandardized clinical workplace ("does"). The 2019 assessment workshop focused on advances in the assessment of learners in the unstandardized workplace-the highest level of Miller's assessment pyramid ("does"). Research has shown that dental education has struggled to implement assessment strategies that meet this level.³ The workshop brought together individuals from around the world, with an interest in assessment in dental education, to consider how assessment in the "does" level, specifically EPAs and WBA, factors into competence assessment in dentistry/dental education.