Reflective critique and collaborative practice in evaluation: promoting change in medical education

Developing an evaluation approach for the in-depth review of a new undergraduate medical programme as a complex system

BACKGROUND

A new undergraduate medical programme was instituted at the University of Botswana in 2009. In 2016, the Faculty of Medicine decided to conduct a comprehensive review of the programme. Participants at a planning workshop decided the review had to lead to an in-depth understanding of the programme. The challenge was to develop an evaluation process to achieve this aim.

APPROACH

Standards and theories used in other programme evaluations were investigated. A steering committee developed a 'six-step process' for the review of the programme as a complex system. The process used three evaluation models and various sources of evaluation standards to derive 90 evaluation questions. Quantitative and qualitative data were collected using 96 different instruments, with data triangulation prominently featured. Data analysis used an interpretivist approach. The review process was validated against the Cilliers's ten features of a complex system.

VALIDATION

The review process was validated against Cilliers's ten features of a complex system. We found that the 'six-step process' illuminated each of these features in the MBBS programme in turn, and was therefore a valid way of evaluating this programme as a complex system.

DISCUSSION

In the process of designing and using the 'six-step process' to evaluate a complex medical programme important lessons were learnt: starting the process with complexity theory at the forefront; being as inclusive as possible in data collection; also applying complexity theory to the evaluation of smaller programme components as 'mini-complex systems'; and managing one's inevitable insider bias.

CONCLUSION AND UTILITY

The 'six-step process' as it stands or in adaptation is likely to be useful in similar situations, where evaluators perceive the object of their evaluation to be a complex system, or a component of such a system.

Advancing Research Training in Medical Education: Global Perspectives and Paradigms for Future Development

BACKGROUND

This study delves into the dynamic field of medical education research, emphasizing the integration of research training within medical curricula. It seeks to understand the impact of such integration on the competencies of future medical professionals.

OBJECTIVE

The primary aim is to systematically categorize and analyze the current trends and future directions in research training in medical education. This involves assessing the influence of research training on medical students' skills and the methodologies used in such research.

METHODS

The research employs an extensive bibliographic literature review across multiple databases. It classifies studies like experiential or case studies, editorials, and original research articles. This classification is based on criteria such as geographical location, research objectives, theoretical frameworks, and methodologies.

RESULTS

Findings reveal a diverse landscape in medical education research, with a significant emphasis on research training. The research showcases varying methodologies and approaches used globally, highlighting the thematic focus and geographical distribution of these studies.

CONCLUSION

Research training in medical education is a globally expansive and evolving field. It underscores the importance of continuous investigation, particularly focusing on integrating research elements at curricular levels and exploring innovative educational strategies. The study also points out potential research gaps, especially in underrepresented regions, indicating directions for future research efforts.

Creation of the Association of Pediatric Program Directors Faculty Development for Educators "Nuts and Bolts" Teaching Aids

Faculty development (FD) continues to be a great need and challenge for faculty engaged in graduate medical education (GME) and such educator development should ultimately benefit learners in GME programs. As a Task Force within the Association of Pediatric Program Directors (APPD) FD Learning Community, our Educator subcommittee utilized multiple needs assessments to develop a novel educator development program called the APPD FD for Educators Nuts and Bolts (Nuts & Bolts) consisting of 1 to 2 page FD teaching aids. This paper outlines the development of these teaching aids and can be used as a reference for developing future Nuts & Bolts teaching aids. The development was based on 1) a defining framework (Glassick's criteria), 2) a clear model for evaluation of the program (Logic model), and 3) development of an assessment tool to demonstrate effectiveness of the teaching aids in FD at the local level. These 3 components help establish the scholarly nature of the FD Nuts & Bolts program. Multiple lessons learned from development of this program are presented to inform others engaged in educator program development. Detailed proposals for use of FD Nuts & Bolts for faculty improvement and plans for generating additional teaching aids are presented.