The effectiveness of portfolios for post-graduate assessment and education: BEME Guide No 12

How to use polarity thinkingTM to manage tensions between accountability and learner agency when using a multipurpose portfolio?

Portfolios are often implemented to target multiple purposes, e.g. assessment, accountability and/or self-regulated learning. However, in educational practice, it appears to be difficult to combine different purposes in one portfolio, as interdependencies between the purposes can cause tensions. This paper explored directions to manage tensions that are inextricably linked to multipurpose portfolio use. We used a systems thinking methodology, that was based on the polarity thinkingTM framework. This framework provides a step-by-step approach to chart a polarity map® that can help to balance the tensions present in specific settings. We followed the steps of the framework to chart a polarity map for multipurpose portfolio use. Based on literature and our prior research, we selected one overarching polarity: accountability and learner agency. This polarity seems responsible for multiple tensions related to multipurpose portfolio use. We formulated values (potential benefits) and fears (tensions that can arise) of the two poles of this polarity. Then, we organised a session with stakeholders who work with the portfolio of the Dutch General Practice speciality programme. Together we formulated action steps and early warnings that can help to balance accountability and learner agency during multipurpose portfolio use. In addition to previous recommendations concerning portfolio use, we advocate that it is important to create a shared frame of reference between all involved with the multipurpose portfolio. During this process, the acknowledgement and discussion of tensions related to multipurpose portfolio use are vital.

How does portfolio use support self-regulated learning during general practitioner specialty training? A qualitative focus group study

OBJECTIVES

Portfolios are used to support self-regulated learning (SRL), but the research literature is still inconclusive on their effectiveness. This study explored experiences with portfolio use among different stakeholders, to answer the research question: How does portfolio use support SRL during general practitioner (GP) specialty training?

DESIGN

We used a qualitative research design, based on phenomenology.

SETTING

Three of the eight training institutes of Dutch GP specialty training participated in this study.

PARTICIPANTS

The three stakeholder groups that use the portfolio were included in nine homogenous focus groups: trainees (n=16), supervisors (n=16) and faculty (n=17). All participants had at least 6-month experience with portfolio use.

RESULTS

Three themes were identified: SRL with(out) the portfolio, stakeholder dynamics and ambiguities. Respondents were doubtful about the learning benefits of portfolio use, as most trainees used their portfolio to 'check off' what was considered required. Stakeholder dynamics contributed to checking off behaviour in two ways. First, trainees experienced documenting learning activities to be superfluous, since the close relationship with their supervisor already supported SRL sufficiently. Second, faculty often (unintentionally) took portfolio ownership away from trainees, as they instructed trainees to deliver portfolio content that was valuable for assessment. Without ownership, trainees struggled to use the portfolio for SRL. Besides, ambiguities related to portfolio use amplified checking off behaviour.

CONCLUSIONS

Portfolio use did not support SRL in our setting. The multipurpose use of the portfolio (for the support of SRL and assessment) was identified as the primary obstacle. Underlying is a conflict that is often present in current medical curricula: agency versus accountability. If the support of SRL is considered a valuable and attainable purpose of portfolio use, it is important to realise that deliberate attention for this purpose is required during the design, guidance, assessment and evaluation of the portfolio.

Portfolios in practice: Developing advancing practice within a musculoskeletal competency-based model

INTRODUCTION

The development of professional portfolios and the relevance of this within professional practice, competency and capability is gaining significant credibility in line with professional requirements. Nursing and medicine in terms of historical perspectives have long held the need for clinicians to maintain a portfolio for professional validation, whilst in other professional groups it is a requirement of registration. The allied health professionals, physiotherapy and ultimately musculoskeletal practice within this context are rapidly developing advancing and consultant practice. This professional development further requires appropriate verification and validation of practice, and achieving this can be through formal and non-formal routes.

PURPOSE

This paper looks to explore this and give direction to professionals developing portfolios whilst placing the requirements in context to contemporary practice in the U.K. Universities, professional bodies and special interest groups are now aligning in the need to support practice in a multi-format way, that moves away from traditional methods of evaluation into more diverse models of competency-based assessment.

IMPLICATIONS

With improvement in technology, the development of national frameworks and standards, portfolios in practice although commonly considered as standard practice will be a requirement not only of registration but as a criteria of maintaining status, career development and expansion of roles.

BACKGROUND

Musculoskeletal (MSK) physiotherapy in the U.K. has moved forward significantly in the last 20 years. Sitting within a clinical reasoned framework, the introduction of additional skills such is image requesting, injection therapies, and non-medical prescribing has further underpinned the advanced practice agenda (Langridge et al., 2015). While these advancements in practice are driving the profession forward, challenges remain in providing the workforce with a clear process of career development. Alongside developing professional pathways methods of evidencing advanced knowledge and skills acquired outside formal routes are required to support practitioners' career pathway into advancing practice.

Implementation of a Web-Based Educational Intervention for Promoting Flipped Classroom Pedagogy: A Mixed-Methods Study

Introduction

Flipped classroom pedagogy (FCP) is recognised as an effective and efficient teaching-learning method. However, nursing students and teachers may be hesitant to adopt FCP due to technophobia and time constraints linked to academic and clinical responsibilities. This necessitates the provision of promotional training for adopting FCP. However, there is a dearth of research regarding how the practice of FCP could be promoted and in demonstrating its efficacy in developing countries. Thus, this study aimed to examine the educational impact of a web-based education intervention, branded as the Flipped Classroom Navigator (FCN), for promoting FCP in nursing education in Sri Lanka.

Methods

This mixed-methods study employed pre- and post-training knowledge tests, the Instructional Materials Motivation Survey (IMMS), the Perceived Transfer of Learning Questionnaire survey, and collected students' and teachers' open-ended responses to evaluate the impact of the FCN. Fifty-five undergraduate nursing students and 15 university teachers participated in the study from two state universities in Sri Lanka. Repeated-measures ANOVA, t-tests, Levene's test of homogeneity, Cohen's d, and an inductive thematic approach were employed in the data analysis.

Results

Post-training knowledge test scores were significantly higher than the pre-training knowledge test scores indicating improved understanding of FCP. Participants were also highly motivated to learn in the FCN instructional materials. Participants exhibited positive attitudes towards FCN training with transfer of learning to their teaching-learning practice. The inductive thematic analysis identified the following themes: user experiences, FCN learning content, behaviour changes, and suggested improvements.

Conclusion

Overall, the FCN enhanced both students' and teachers' knowledge and understanding of FCP in undergraduate nursing education.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-022-01706-7.

Perceptions of portfolio assessment in family medicine graduates: a qualitative interview study

BACKGROUND

The use of the portfolio methodology in medical education can serve as a tool for learning, assessment, and reflection on practice. This study concentrates on perceptions of the portfolio assessment methodology among participants in the Saudi Diploma of Family Medicine program.

METHODS

In this qualitative interview study, data were collected and analysed using a grounded theory approach.

RESULTS

Nine codes emerged: (1) Importance of understanding the definition, objectives, and process of portfolio assessment, (2) Impact of different understandings on the part of various trainers, (3) Role of the type of assessment, (4) Workload and stress of portfolio assessment, (5) Effectiveness of the portfolio contents, (6) Role of the mentor's feedback, (7) Role in the learning process, (8) Role in practice, (9) Suggestions for portfolio improvement. Open codes were then regrouped into three axial codes: context, strategy, and outcome of portfolio assessment.

CONCLUSION

This study explored a general explanation of portfolio assessment shaped by the postgraduate students. It identifies the importance of portfolio understanding in student acceptability of the portfolio assessment methodology. Thus, proper implementation is vital for the success of assessing the student by the portfolio methodology. The students perceived reflection as the most valuable part of the process, which facilitated their learning, confidence, and self-assessment. Mentor feedback is a good strategy for coping with portfolio challenges. Our findings provide some evidence of positive outcomes of portfolio assessment in practice and professional development.

How does portfolio use affect self-regulated learning in clinical workplace learning: What works, for whom, and in what contexts?

INTRODUCTION

Portfolio use to support self-regulated learning (SRL) during clinical workplace learning is widespread, but much is still unknown regarding its effectiveness. This review aimed to gain insight in the extent to which portfolio use supports SRL and under what circumstances.

METHODS

A realist review was conducted in two phases. First, stakeholder interviews and a scoping search were used to formulate a program theory that explains how portfolio use could support SRL. Second, an in-depth literature search was conducted. The included papers were coded to extract context-mechanism-outcome configurations (CMOs). These were synthesized to answer the research question.

RESULTS

Sixteen papers were included (four fulfilled all qualitative rigor criteria). Two primary portfolio mechanisms were established: documenting as a moment of contemplation (learners analyze experiences while writing portfolio reports) and documentation as a reminder of past events (previous portfolio reports aid recall). These mechanisms may explain the positive relationship between portfolio use and self-assessment, reflection, and feedback. However, other SRL outcomes were only supported to a limited extent: formulation of learning objectives and plans, and monitoring. The partial support of the program theory can be explained by interference of contextual factors (e.g., system of assessment) and portfolio-related mechanisms (e.g., mentoring).

DISCUSSION

Portfolio research is falling short both theoretically-in defining and conceptualizing SRL-and methodologically. Nevertheless, this review indicates that portfolio use has potential to support SRL. However, the working mechanisms of portfolio use are easily disrupted. These disruptions seem to relate to tensions between different portfolio purposes, which may undermine learners' motivation.

How to enhance students' learning in a patient-centered longitudinal integrated clerkship: factors associated with students' learning experiences

PURPOSE

Longitudinal integrated clerkships (LICs) have been introduced in medical schools, as learning relationships with clinical faculty or peers are important components of medical education. The purpose of this study was to investigate the characteristics of student-faculty and student-student interactions in the LIC and to identify other factors related to whether students understood and acquired the program's main outcomes.

METHODS

The study was conducted among the 149 third-year students who participated in the LIC in 2019. We divided the students into groups of eight. These groups were organized into corresponding discussion classes, during which students had discussions with clinical faculty members and peers and received feedback. Clinical faculty members and students were matched through an e-portfolio, where records were approved and feedback was given. A course evaluation questionnaire was completed and analysed.

RESULTS

A total of 144 valid questionnaires were returned. Logistic regression analysis showed that relevant feedback in discussion classes (adjusted odds ratio [AOR], 5.071; p<0.001), frequency of e-portfolio feedback (AOR, 1.813; p=0.012), and motivation by e-portfolio feedback (AOR, 1.790; p=0.026) predicted a greater likelihood of understanding the continuity of the patient's medical experience. Relevant feedback from faculty members in discussion classes (AOR, 3.455; p<0.001) and frequency of e-portfolio feedback (AOR, 2.232; p<0.001) also predicted a greater likelihood of understanding the concept of patient-centered care.

CONCLUSION

Student-faculty interactions, including relevant feedback in discusstion classes, frequency of e-portfolio feedback, and motivation by e-portfolio feedback were found to be important factors in the LIC program.

Subjective achievement from psychiatry rotation in the Japanese postgraduate residency system: a longitudinal questionnaire study

BACKGROUND

Psychiatry rotation has been mandatory in the Japanese postgraduate residency system since 2020. Some psychiatry-related competency items are stipulated as mandatory for residents. The current study aimed to clarify whether psychiatry rotation affected residents' subjective achievement of these competency items.

METHODS

This longitudinal study was conducted among postgraduate residents who completed a rotation in the psychiatry department at Nagasaki University Hospital across two academic years (2020-2021). The survey was administered at the start and at the end of the psychiatry rotation. Residents evaluated their subjective understanding and confidence regarding initiating treatment for these competency items using a six-point Likert scale. The average scores for each item were compared between pre-rotation and post-rotation.

RESULTS

In total, 99 residents (91.7%) responded to this survey. Residents had significantly higher scores at post-rotation compared with pre-rotation in all psychiatry-related competency items in both subjective understanding and confidence in initiating treatment. Additionally, strong effect sizes were found for many items.

CONCLUSION

Residents improved learning about psychiatry-related competency items through psychiatry rotation. This finding suggests that it is reasonable for psychiatry rotation to be mandatory in the current Japanese postgraduate residency system. The importance of psychiatry is likely to increase in both undergraduate and postgraduate medical education in the future. It is necessary to continuously update educational strategies to meet changing social needs over time. As this study was conducted at a single institution, a multi-center study is needed to expand the current findings.

Training to Support ePortfolio Users During Clinical Placements: a Scoping Review

ePortfolios are frequently used to support students' competency development, and teachers' and clinical mentors' supervision during clinical placements. User training is considered a critical success factor for the implementation of these ePortfolios. However, there is ambiguity about the design and outcomes of ePortfolio user training. A scoping review was conducted to consolidate evidence from studies describing the design of ePortfolio user training initiatives and their outcomes. The search yielded 1180 articles of which 16 were included in this review. Based on the results, an individual, ongoing training approach which grounds in a fitting theoretical framework is recommended.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s40670-022-01583-0.

Everybody is able to reflect, or aren't they? Evaluating the development of medical professionalism via a longitudinal portfolio mentoring program from a student perspective

Introduction: Reflective competence is fundamental for responsible medical practice and must be systematically incorporated in medical training. To promote this, a longitudinal portfolio-based mentoring program was made mandatory at the Medical Faculty of the University of Tübingen in 2013. This study examines medical students' attitudes toward professional reflection and toward the program in general to draw conclusions about conditions as well as the needs-based design of the program. Method: In winter semester 2017/18, a retrospective questionnaire survey with free text fields was conducted (total sample: N=1.405; students S 1-9; response 37%; S 1-4 "Pre-clinic": n=231; S 5-9 "Clinic": n=241). Opinion trends of semester groups were identified through seven semi-structured interviews with semester speaker and peer tutors. Results: Differences in understanding and attitudes resulted in three positions: 1=approval, 2=ambivalence, 3=rejection. All three groups included individuals from pre-clinical and clinical settings with varying levels of experience. Prior experience and hidden curriculum influenced the position. Opinion trends confirmed the feedback. Conclusion: Although reflection appears in the National Competence-based Learning Objectives Catalogue for Medicine (NKLM), reflective competence is not regarded as equivalent to other study content. Motivation, commitment on the part of the mentors, and a trusting mentor-mentee relationship are effective. The flexibility of the portfolio in terms of content and methodology, as well as the curricular integration of the program are also beneficial.

Evaluation in health professions education-Is measuring outcomes enough?

INTRODUCTION

In an effort to increase the rigour of evaluation in health professions education (HPE), a range of evaluation approaches are used. These largely focus on outcome evaluation as opposed to programme evaluation. We aim to review and critique the use of outcome evaluation models, using the Kirkpatrick Model as an example given its wide acceptance and use, and advocate for the use of programme evaluation models that help us understand how and why outcomes are occurring.

METHODS

We systematically searched OVID medline, Scopus, CINAHL and Pubmed, and hand searched six leading HPE journals to provide an overview of the use of the Kirkpatrick Model as well as a range of programme evaluation models in HPE. In addition to this, we synthesised the existing critiques of the Kirkpatrick Model as an example of outcome evaluation, to highlight the limitations of such models.

RESULTS

The use of the Kirkpatrick Model in HPE is widespread and increasing; however, studies focus on categorising outcomes, rather than explaining how and why they occur. The main criticisms of the model are as follows: it is outcomes focused and fails to consider factors that can impact training outcomes; it assumes positive casual linkages between the levels; there is an assumption that the higher-level outcomes are more important; and unintended impacts are not considered. The use of the Kirkpatrick Model by the MERSQI, BEME and WHO contribute to the myth that the Kirkpatrick Model is the gold standard for programme evaluation.

DISCUSSION

Moving forward, evaluations of HPE interventions must shift from focusing largely on measuring outcomes of interventions with little consideration for how and why these outcomes are occurring to programme evaluation that investigates what contributes to these outcomes. Other models that facilitate the evaluation of the complex processes that occur in HPE should be used instead of Kirkpatrick's.

Improving Medical Education in Hematology and Transfusion Medicine in Canada: Standards and Limitations

The paradigm of medical education is evolving with the introduction of competency-based medical education (CBME) and it is crucial that residency programs adapt. In this paper, we provide an overview of the current status of medical education in Hematology in Canada including models of training, assessment methods, anticipated challenges, and the effects of the COVID-19 pandemic. We will also discuss additional training that can be pursued after a Hematology residency, with a particular focus On Transfusion Medicine as it was one of the first programs to implement a competency-based curriculum. Finally, we explore the future directions of medical education in Hematology and Transfusion Medicine.

Ottawa 2020 consensus statement for programmatic assessment - 1. Agreement on the principles

INTRODUCTION

In the Ottawa 2018 Consensus framework for good assessment, a set of criteria was presented for systems of assessment. Currently, programmatic assessment is being established in an increasing number of programmes. In this Ottawa 2020 consensus statement for programmatic assessment insights from practice and research are used to define the principles of programmatic assessment.

METHODS

For fifteen programmes in health professions education affiliated with members of an expert group (n = 20), an inventory was completed for the perceived components, rationale, and importance of a programmatic assessment design. Input from attendees of a programmatic assessment workshop and symposium at the 2020 Ottawa conference was included. The outcome is discussed in concurrence with current theory and research.

RESULTS AND DISCUSSION

Twelve principles are presented that are considered as important and recognisable facets of programmatic assessment. Overall these principles were used in the curriculum and assessment design, albeit with a range of approaches and rigor, suggesting that programmatic assessment is an achievable education and assessment model, embedded both in practice and research. Knowledge on and sharing how programmatic assessment is being operationalized may help support educators charting their own implementation journey of programmatic assessment in their respective programmes.

Acceptance and validity of the methods used to implement a competency based medical education programme in an Intensive Care Department of a teaching referral center

OBJECTIVES

1. To determine the satisfaction of tutors and residents with a specific methodology used to implement CoBaTrICE. 2. To determine the reliability and validity of the global rating scales designed ad hoc to assess the performance of the residents for training purposes.

DESIGN

Prospective cohort study.

PARTICIPANTS

All the residents and tutors of the ICU Department of the Hospital Universitario y Politécnico la Fe de Valencia.

INTERVENTION

CoBaTrICE implementation started in March 2016, it was based on: (1) Training the tutors in feedback techniques; (2) Performing multiple objective and structured work based assessments to achieve the competences of the program; and (3) The use of an electronic portfolio to promote learning reflection and to collect the evidence that learning was taking place.

METHODS

The acceptance of CoBaTrICE was explored through a satisfaction survey conducted after 9 months of implementation of the training program. The 15 residents and 5 tutors of the ICU Department were asked about the methodology of the formative assessments, the quality of the feedback, self-learning regulation and the electronic portfolio usefulness. The validity of the global rating scales was assessed through the tests alfa de Cronbach, reliability and generalizability indexes, and intraclass correlation coefficient.

RESULTS

The implementation of CoBaTrICE was satisfactory in all the dimensions studied. The global rating scales used for formative purposes showed reliability and validity.

CONCLUSIONS

The methodology used to implement CoBaTrICE was highly valued by tutors and residents. The global rating scales used for formative purposes showed reliability and validity.

Exploring the strategies for successfully building e-portfolios in medical schools

PURPOSE

The purpose of this study is to examine the characteristics of resistance among medical students toward e-portfolios and find the strategies for them to successfully prepare e-portfolios.

METHODS

Participants were a group of 258 medical students. The questionnaire comprised 13 items developed based on the innovation resistance theory. The data were analyzed using descriptive analysis and Spearman's correlation analysis using PASW SPSS version 18.0 (SPSS Inc., Chicago, USA).

RESULTS

Students perceived that e-portfolios have a high degree of relative advantage, trialability, and complexity as their innovation resistance characteristics. Regarding perceived risk, they did not want others to see their information, but they had a high degree of demand for communication with their professors.

CONCLUSION

The successful use of portfolios can serve as a tool for student management and assessment that can reflect their introspection, personal development, and academic performance. This study proposed the strategies to promote the use of e-portfolios by strengthening education on the need for them, specific feedback of instructors, and students' autonomy.

Fairness in human judgement in assessment: a hermeneutic literature review and conceptual framework

Human judgement is widely used in workplace-based assessment despite criticism that it does not meet standards of objectivity. There is an ongoing push within the literature to better embrace subjective human judgement in assessment not as a 'problem' to be corrected psychometrically but as legitimate perceptions of performance. Taking a step back and changing perspectives to focus on the fundamental underlying value of fairness in assessment may help re-set the traditional objective approach and provide a more relevant way to determine the appropriateness of subjective human judgements. Changing focus to look at what is 'fair' human judgement in assessment, rather than what is 'objective' human judgement in assessment allows for the embracing of many different perspectives, and the legitimising of human judgement in assessment. However, this requires addressing the question: what makes human judgements fair in health professions assessment? This is not a straightforward question with a single unambiguously 'correct' answer. In this hermeneutic literature review we aimed to produce a scholarly knowledge synthesis and understanding of the factors, definitions and key questions associated with fairness in human judgement in assessment and a resulting conceptual framework, with a view to informing ongoing further research. The complex construct of fair human judgement could be conceptualised through values (credibility, fitness for purpose, transparency and defensibility) which are upheld at an individual level by characteristics of fair human judgement (narrative, boundaries, expertise, agility and evidence) and at a systems level by procedures (procedural fairness, documentation, multiple opportunities, multiple assessors, validity evidence) which help translate fairness in human judgement from concepts into practical components.

A Systematic Scoping Review on Portfolios of Medical Educators

BACKGROUND

Heralded as a teaching, assessment and reflective tool, and increasingly as a longitudinal and holistic perspective of the educator's development, medical educator's portfolios (MEP)s are increasingly employed to evaluate progress, assess for promotions and career switches, used as a reflective tool and as a means of curating educational activities. However, despite its blossoming role, there is significant dissonance in the content and structure of MEPs. As such, a systematic scoping review (SSR) is proposed to identify what is known of MEPs and its contents.

METHODS

Krishna's Systematic Evidenced Based Approach (SEBA) was adopted to structure this SSR in SEBA of MEPs. SEBA's constructivist approach and relativist lens allow data from a variety of sources to be considered to paint a holistic picture of available information on MEPs.

RESULTS

From the 12 360 abstracts reviewed, 768 full text articles were evaluated, and 79 articles were included. Concurrent thematic and content analysis revealed similar themes and categories including: (1) Definition and Functions of MEPs, (2) Implementing and Assessing MEPs, (3) Strengths and limitations of MEPs and (4) electronic MEPs.

DISCUSSION

This SSR in SEBA proffers a novel 5-staged evidence-based approach to constructing MEPs which allows for consistent application and assessment of MEPs. This 5-stage approach pivots on assessing and verifying the achievement of developmental milestones or 'micro-competencies' that facilitate micro-credentialling and effective evaluation of a medical educator's development and entrust-ability. This allows MEPs to be used as a reflective and collaborative tool and a basis for career planning.

The Pharmacy Game-GIMMICS® a Simulation Game for Competency-Based Education

The profile of the profession of pharmacists has profoundly changed over the last decades. Pharmacy education has moved towards competency-based education. The pharmacy game, called GIMMICS^®, developed at the University of Groningen, is unique in combining simulation with serious gaming to teach a wide range of competencies. In this article, we describe the learning goals, the assessment methods, the teaching tools, and the students’ view of the pharmacy game. The learning goals are to train the competencies of collaboration, leadership, communication, and pharmaceutical expertise. The core of the game is the simulation of community pharmacy practice activities, such as patient counseling, processing of prescriptions, and collaboration with other health professionals. Students are assessed individually and as a pharmacy team. The pharmacy team, with the largest number of patients wins the game. Student evaluations show that they value the course. Currently, seven universities from around the globe have adopted the pharmacy game in their curriculum, adjusting the course to their country’s pharmacy practice and educational system.

E-portfolios and Entrustable Professional Activities to support competency-based education in dietetics

The concept of Entrustable Professional Activities, recently pioneered in medical education, has emerged to support the implementation of competency-based education. Although competency-based frameworks are widely used in healthcare professional education to develop outcomes-based curricula, assessment of student competency in professional placement settings remains challenging. The novel concept of Entrustable Professional Activities together with established methods of competency assessment, namely e-portfolios and self-assessment, was implemented in the "[La Trobe University Dietetic program in 2015-2016. This study aimed to appraise the e-portfolio and evaluate the use of Entrustable Professional Activities to assess competence. A mixed-methods evaluation, using qualitative and quantitative surveys with follow-up structured consultations, was conducted with final year dietetics students and their supervisors. Dietetics students were comfortable with Entrustable Professional Activities and competency-based assessment, whereas supervisors preferred Entrustable Professional Activity based assessment. All stakeholders valued student self-assessment and the ongoing use of structured e-portfolios to develop and document competency. The use of structured e-portfolios, student self-assessment, and the emerging concept of Entrustable Professional Activities are useful tools to support dietetics student education in professional placement settings.

A BEME systematic review of teaching interventions to equip medical students and residents in early recognition and prompt escalation of acute clinical deteriorations: BEME Guide No. 62

Background: Current educational interventions and teaching for acute deteriorations seem to address acute care learning in discreet segments. Technology enhanced and team training methodologies are in vogue though well studied in the nursing profession, teaching avenues for junior 'doctors in training' seem to be a lacuna.Aims: The BEME systematic review was designed to (1) appraise the existing published evidence on educational interventions that are intended for 'doctors in training' to teach early recognition and prompt escalation in acute clinical deteriorations (2) to synthesise evidence & to evaluate educational effectiveness.Methodology: The method applied was a descriptive, justification & clarification review. Databases searched included PubMed, PsycINFO, Science Direct and Scopus for original research and grey literature with no restrictions to year or language. Abstract review, full text decisions and data extraction were completed by two primary coders with final consensus by a third reviewer.Results: 5592 titles and abstracts were chosen after removal of 905 duplications. After exclusion of 5555 studies, 37 full text articles were chosen for coding. 22 studies met final criteria of educational effectiveness, relevance to acute care. Educational platforms varied from didactics to blended learning approaches, small group teaching sessions, simulations, live & cadaveric tissue training, virtual environments and insitu team-based training. Translational outcomes with reduction in long term (up to 3-6 years) morbidity & mortality with financial savings were reported by 18% (4/22) studies. Interprofessional training were reported in 41% (9/22) of studies. Recent evidence demonstrated effectiveness of virtual environment and mobile game-based learning.Conclusions: There were significant improvements in teaching initiatives with focus on observable behaviours and translational real patient outcomes. Serious game-based learning and virtual multi-user collaborative environments might enhance individual learners' cognitive deliberate practice. Acute care learning continuum with programmatic acute care portfolios could be a promise of the future.

How is self-regulated learning documented in e-portfolios of trainees? A content analysis

BACKGROUND

It is assumed that portfolios contribute to self-regulated learning (SRL). Presence of these SRL processes within the documentation kept in portfolios is presupposed in common educational practices, such as the assessment of reflective entries. However, questions can be asked considering the presence of SRL within portfolios. The aim of this study was to gain insight into the documentation of SRL processes within the electronic (e)-portfolio content of medical trainees. SRL consists of numerous processes, for this study the focus was on self-assessment via reflection and feedback, goal-setting and planning, and monitoring, as these are the processes that health professions education research mentions to be supported by portfolios.

METHODS

A database containing 1022 anonymous e-portfolios from General Practitioner trainees was used to provide descriptive statistics of the various available e-portfolio forms. This was followed by a quantitative content analysis of 90 e-portfolios, for which, a codebook was constructed to rate the documentation of the included SRL processes.

RESULTS

The numbers of forms in the e-portfolios varied to a great extent. Content analysis showed a limited documentation of reflective entries, and available entries mainly described events and experiences without explanations and context. Feedback was generally limited to comments on what went well and lacked specificity, context and suggestions for future action. Learning goals and plans were short of specificity, but did contain challenging topics and different goals were compatible with each other. 75% of the e-portfolios showed (limited) signs of monitoring.

CONCLUSIONS

The e-portfolio content showed limited documentation of SRL processes. As documentation of SRL requires time and asks for a high level of introspection and writing skills, one cannot expect documentation of SRL processes to appear in e-portfolio content without efforts.

Pre-service and in-service education and training for maternal and newborn care providers in low- and middle-income countries: An evidence review and gap analysis

BACKGROUND

Good quality midwifery care has the potential to reduce both maternal and newborn mortality and morbidity in high, low, and lower-middle income countries (LMIC) and needs to be underpinned by effective education. There is considerable variation in the quality of midwifery education provided globally.

OBJECTIVE

To determine what are the most efficient and effective ways for LMICs to conduct pre-service and in-service education and training in order to adequately equip care providers to provide quality maternal and newborn care.

DESIGN

Rapid Systematic Evidence Review METHODS: A systematic search of the following databases was conducted: Medline, CINAHL, LILACs, PsycInfo, ERIC, and MIDIRs. Studies that evaluated the effects of pre-service and in-service education that were specifically designed to train, educate or upskill care providers in order to provide quality maternal and newborn care were included. Data was extracted and presented narratively.

FINDINGS

Nineteen studies were included in the review. Of these seven were evaluations of pre-service education programmes and 12 were evaluations of in-service education programmes. Whilst studies demonstrated positive effects on knowledge and skills, there was a lack of information on whether this translated into behaviour change and positive effects for women and babies. Moreover, the level of the evidence was low and studies often lacked an educational framework and theoretical basis. Mapping the skills taught in each of the programmes to the Quality Maternal and Newborn Care framework (Renfrew et al., 2014) identified that interventions focused on very specific or individual clinical skills and not on the broader scope of midwifery.

KEY CONCLUSIONS

There is a very limited quantity and quality of peer reviewed published studies of the effectiveness of pre service and in service midwifery education in LMICs; this is at odds with the importance of the topic to survival, health and well-being. There is a preponderance of studies which focus on training for specific emergencies during labour and birth. None of the in-service programmes considered the education of midwives to international standards with the full scope of competencies needed. There is an urgent need for the development of theoretically informed pre-service and in-service midwifery education programmes, and well-conducted evaluations of such programmes. Upscaling quality midwifery care for all women and newborn infants is of critical importance to accelerate progress towards Sustainable Development Goal 3. Quality midwifery education is an essential pre-requisite for quality care. To deliver SDG 3, the startling underinvestment in midwifery education identified in this review must be reversed.

A New Educational Framework to Improve Lifelong Learning for Cardiologists

Lifelong learning is essential for the practicing cardiologist. Present lifelong learning mechanisms are stagnant and at risk for not meeting the needs of currently practicing cardiologists. With the increasing burden of cardiovascular disease, growing complexity of patient care, and ongoing pressures of nonclinical responsibilities, educational programming must evolve to meet the demands of the contemporary cardiovascular professional. A paradigm shift, replete with modern and practical educational tools, is needed in the lifelong learning armamentarium. Emerging evidence of novel educational strategies in graduate medical education supports the promise of broader application of these tools to different stages of professional life. In this commentary from the Fellows-in-Training Section Leadership Council, the authors propose 3 novel educational tools-personalized learning, adaptive learning, and the flipped classroom-to improve lifelong learning to meet the educational needs of fellows-in-training to practicing cardiologists alike.

Implementing and evaluating an e-portfolio for postgraduate family medicine training in the Western Cape, South Africa

BACKGROUND

In South Africa it is compulsory to submit a satisfactory portfolio of learning to gain entrance to the national exit examination of the College of Family Physicians and to qualify as a family physician. A paper-based portfolio has been implemented thus far and the need for an electronic portfolio (e-portfolio) was identified. The aim of the study was to describe and evaluate the implementation of an e-portfolio for the training of family medicine registrars in the Western Cape province of South Africa.

METHODS

Mixed methods were used. A quasi-experimental study evaluated paper- and e-portfolios from the same 28 registrars in 2015 compared to 2016. Semi-structured interviews were conducted with 11 registrars or supervisors to explore their experiences of using the e-portfolio. Quantitative data was analysed in the Statistical Package for Social Sciences and qualitative data in Atlas.ti.

RESULTS

Most respondents found the e-portfolio easier to use and more accessible. It made progress easier to monitor and provided sufficient evidence of learning. Feedback was made easier and more explicit. There were concerns regarding face-to-face feedback being negatively affected. It was suggested to have a feedback template to further improve feedback. Several aspects were significantly better in the e-portfolio such as feedback on the registrar's general behaviour, alignment with learning outcomes, less feedback based on hearsay and acknowledgement of the feedback by the registrar. Although not statistically significant, there was an increase in the usage of the e-portfolio, compared to the paper portfolio.

CONCLUSION

In general, the e-portfolio is an improvement on the paper-based portfolio. It is easier to access, more user-friendly and less cumbersome. It makes feedback and monitoring of progress and development of registrars easier and more visible and provides sufficient evidence of learning. Its implementation throughout South Africa is recommended.

Narrative medicine as a medical education tool: A systematic review

Aim: Narrative medicine has been promoted as an innovative and effective means of stimulating medical students' professional development by teaching them to approach their patients' experiences of illness with more understanding and compassion. This systematic literature review aims to answer the following question: what evidence of effect is available in the literature about models for teaching narrative medicine? Methods: We conducted a narrative review of 36 articles and used the Best Evidence in Medical Education (BEME) Global Scale and Kirkpatrick Scale for strength and importance of evidence to categorize reported assessment strategies and to evaluate the effectiveness of their narrative medicine programs. Results: We found evidence that narrative medicine is an effective pedagogic tool with a clear and replicable structure and methodology. We also determined that a positive impact could be measured when pertaining to participation and modification of attitudes, knowledge, and skills. However, unequivocal evidence of the effect of narrative medicine on students' behavior or ongoing interaction with colleagues and patients is still lacking. Conclusion: While many recent publications describe the goals and virtues of a narrative-based approach, more research is needed to determine whether or not there is an ideological consensus undergirding this approach. In addition, it is still unclear whether the long-term impact of narrative medicine classroom interventions are felt by patients, or whether such interventions positively impact patient care.

Current Trends and Opportunities for Competency Assessment in Pharmacy Education-A Literature Review

An increasing emphasis on health professional competency in recent times has been matched by an increased prevalence of competency-based education models. Assessments can generate information on competence, and authentic, practice-based assessment methods are critical. Assessment reform has emerged as an academic response to the demands of the pharmacy profession and the need to equip graduates with the necessary knowledge, skills and attributes to face the challenges of the modern workforce. The objective of this review was to identify and appraise the range of assessment methods used in entry-level pharmacy education and examine current trends in health professional assessment. The initial search located 2854 articles. After screening, 36 sources were included in the review, 13 primary research studies, 12 non-experimental pharmacy research papers, and 11 standards and guidelines from the grey literature. Primary research studies were critically appraised using the Medical Education Research Study Quality Instrument (MERSQI). This review identified three areas in pharmacy practice assessment which provide opportunities for expansion and improvement of assessment approaches: (1) integrated approaches to performance assessment; (2) simulation-based assessment approaches, and; (3) collection of validity evidence to support assessment decisions. Competency-based assessment shows great potential for expanded use in pharmacy, but there is a need for further research and development to ensure its appropriate and effective use.

Understanding how to enhance efficacy and effectiveness of feedback via e-portfolio: a realist synthesis protocol

INTRODUCTION

The validity of feedback as one of the defining components for electronic portfolios (e-portfolios) to be effective and efficacious has yet to be demonstrated. While the literature has shown individual beneficial features of e-portfolios and feedback per se, evidence of feedback as mediated through technology directly resulting in improved educational practice is scarce. The explanation of how feedback via e-portfolio improves educational practice is particularly vague.

METHODS AND ANALYSIS

The aim of this research is to unpack how and why feedback via e-portfolio is likely to flourish or wither in its path. Given the complexity of intervention, we will apply a theory-driven approach for evidence synthesis called realist synthesis. Informed by realist philosophy of science, it seems the most appropriate method because it explores observed outcomes (O) in terms of causal relationship between relevant contexts (C) and generating mechanisms (M). Initial programme theory will be developed through literature scoping. Later on it will be tested against purposively gathered evidence (through database and journal search), which simultaneously will be evaluated for rigour and relevance (whether method used are trustworthy and whether data contributes to theory building). We strive to (1) uncover 'context sensitive' mechanisms that generate feedback via e-portfolio to be (in) effective and (2) define in what circumstances is this mostly likely to occur.

ETHICS AND DISSEMINATION

The synthesis report will be written according to the RAMESES guidelines and its findings will be published in peer reviewed articles and presented at relevant conferences. The aim is to inform: (1) policy and decision makers for future-course design; (2) medical educators/clinical supervisors and learners for improved educational use. No formal ethical approval is required.

PROSPERO REGISTRATION NUMBER

120863.

'I did not check if the teacher gave feedback': a qualitative analysis of Taiwanese postgraduate year 1 trainees' talk around e-portfolio feedback-seeking behaviours

OBJECTIVES

Despite feedback being an extensively researched and essential component of teaching and learning, there is a paucity of research examining feedback within a medical education e-portfolio setting including feedback-seeking behaviours (FSBs). FSBs can be understood within a cost-value perspective. The objective of this research is to explore the factors that influence postgraduate year 1 (PGY1) trainee doctors' FSBs via e-portfolios.

SETTING

Postgraduate education provision in the largest teaching hospital in Taiwan.

PARTICIPANTS

Seventy-one PGY1s (66% male).

METHODS

A qualitative semistructured one-to-one interview method was adopted. Interviews were audio recorded, transcribed verbatim, anonymised and checked for completeness. Data were analysed inductively via thematic framework analysis and deductively informed using FSB theory. The process comprised data familiarisation, identification of the themes, charting and data interpretation.

RESULTS

Two main themes of FSB related and e-portfolio related were identified. We present the theme focussing on FSB here to which n=32 (22 males, 10 females) of the n=71 participants contributed meaningfully. Subthemes include factors variously affecting PGY1s' positive and negative FSBs via e-portfolios at the individual, process and technological levels. These factors include learner-related (internal values vs social influence, forced reflection); teacher-related (committed educators vs superficial feedback); technology-related (face-saving vs lagging systems; inadequate user-interface) and process-related (delayed feedback, too frequent feedback) factors.

CONCLUSIONS

Our findings reveal the complexity of PGY1s' FSBs in an e-portfolio context and the interaction of numerous facilitating and inhibiting factors. Further research is required to understand the range of facilitating and inhibiting factors involved in healthcare learners' FSBs across different learning, social, institutional and national cultural settings.

Examining the readiness of best evidence in medical education guides for integration into educational practice: A meta-synthesis

BACKGROUND

To support evidence-informed education, health professions education (HPE) stakeholders encourage the creation and use of knowledge syntheses or reviews. However, it is unclear if these knowledge syntheses are ready for translation into educational practice. Without understanding the readiness, defined by three criteria-quality, accessibility and relevance-we risk translating weak evidence into practice and/or providing information that is not useful to educators.

METHODS

A librarian searched Web of Science for knowledge syntheses, specifically Best Evidence in Medical Education (BEME) Guides. This meta-synthesis focuses on BEME Guides because of their explicit goal to inform educational practice and policy. Two authors extracted data from all Guides, guided by the 25-item STructured apprOach to the Reporting In healthcare education of Evidence Synthesis (STORIES).

RESULTS

Forty-two Guides published in Medical Teacher between 1999 and 2017 were analyzed. No Guide met all STORIES criteria, but all included structured summaries and most described their literature search (n = 39) and study inclusion/exclusion (n = 40) procedures. Eleven Guides reported the presence of theory and/or educational principles, and eight consulted with external subject matter experts. Accessibility to each Guide's full-text and supplemental materials was variable.

DISCUSSION

For a subset of HPE knowledge syntheses, BEME Guides, this meta-synthesis identifies factors that support readiness and indicates potential areas of improvement, such as consistent access to Guides and inclusion of external subject matter experts on the review team. This analysis is useful for understanding the current readiness of HPE knowledge syntheses and informing future reviews to evolve so they can catalyze translation of evidence into educational practice.

Self-regulated learning in the clinical context: a systematic review

OBJECTIVES

Research has suggested beneficial effects of self-regulated learning (SRL) for medical students' and residents' workplace-based learning. Ideally, learners go through a cyclic process of setting learning goals, choosing learning strategies and assessing progress towards goals. A clear overview of medical students' and residents' successful key strategies, influential factors and effective interventions to stimulate SRL in the workplace is missing. This systematic review aims to provide an overview of and a theoretical base for effective SRL strategies of medical students and residents for their learning in the clinical context.

METHODS

This systematic review was conducted according to the guidelines of the Association for Medical Education in Europe. We systematically searched PubMed, EMBASE, Web of Science, PsycINFO, ERIC and the Cochrane Library from January 1992 to July 2016. Qualitative and quantitative studies were included. Two reviewers independently performed the review process and assessed the methodological quality of included studies. A total of 3341 publications were initially identified and 18 were included in the review.

RESULTS

We found diversity in the use of SRL strategies by medical students and residents, which is linked to individual (goal setting), contextual (time pressure, patient care and supervision) and social (supervisors and peers) factors. Three types of intervention were identified (coaching, learning plans and supportive tools). However, all interventions focused on goal setting and monitoring and none on supporting self-evaluation.

CONCLUSIONS

Self-regulated learning in the clinical environment is a complex process that results from an interaction between person and context. Future research should focus on unravelling the process of SRL in the clinical context and specifically on how medical students and residents assess their progress towards goals.

Proposed Performance-Based Metrics for the Future Funding of Graduate Medical Education: Starting the Conversation

Graduate medical education (GME) in the United States is financed by contributions from both federal and state entities that total over $15 billion annually. Within institutions, these funds are distributed with limited transparency to achieve ill-defined outcomes. To address this, the Institute of Medicine convened a committee on the governance and financing of GME to recommend finance reform that would promote a physician training system that meets society's current and future needs. The resulting report provided several recommendations regarding the oversight and mechanisms of GME funding, including implementation of performance-based GME payments, but did not provide specific details about the content and development of metrics for these payments. To initiate a national conversation about performance-based GME funding, the authors asked: What should GME be held accountable for in exchange for public funding? In answer to this question, the authors propose 17 potential performance-based metrics for GME funding that could inform future funding decisions. Eight of the metrics are described as exemplars to add context and to help readers obtain a deeper understanding of the inherent complexities of performance-based GME funding. The authors also describe considerations and precautions for metric implementation.

Emergency nursing students' and nurse educators' perception of care delivery: A comparison study of self-assessment ratings

BACKGROUND

Postgraduate emergency nursing studies strengthen the emergency nursing workforce and contribute to specialist patient care. Formative and summative assessments in postgraduate study are important for student learning as they enable self-reflection and feedback.

AIM

To compare formative and summative appraisal assessments between postgraduate emergency nursing students and nurse educators.

METHOD

Retrospective review of assessments completed by postgraduate student and educator in the final multi-system complex subject of the postgraduate emergency nursing course during 2015. Data were extracted from university records. Data for the two appraisal periods were compared using chisquare test of independence.

RESULTS

Data from 52 emergency nursing students and 16 educators were extracted. Significant differences were detected between most student and educator ratings for both assessment appraisals. The lowest self-ratings of independence for students and educators at the summative assessment were for the provision and coordination of patient care and critical analysis of research evidence. Educators did agree that students were performing independently on domains relating to legal, professional and ethical frameworks.

CONCLUSION

Postgraduate emergency nursing study is important for the emergency nursing workforce. Examining student learning and educator assessment in this context enables a broader understanding of student learning and transition to emergency nursing speciality practice.

Multiuser virtual worlds in healthcare education: A systematic review

BACKGROUND

The use of multiuser virtual worlds (MUVWs) for collaborative learning has generated interest among healthcare educators. Published evidence to support its use is growing, but none has synthesized the evidence to guide future work.

OBJECTIVE

This study sought to provide a comprehensive and systematic evaluation of MUVWs in healthcare education.

DESIGN

A systematic review METHODS: A systematic search of five databases including CINAHL, Cochrane library, EMBASE, PubMed, and Scopus, was conducted from inception up to January 2017. Two independent researchers selected studies that met the inclusion criteria and assessed for methodological quality using the Medical Education Research Study Quality Instrument (MERSQI). A total of 18 studies were reviewed and their data were synthesized narratively using a 3-P model (presage-process-product).

RESULTS

Average scores in the MERSQI for methodological quality are 10/18, which is modest. A rally by the government or professional bodies towards more collaborative working among healthcare professionals is a key driver behind implementing MUVWs. Funding is important for its development and evaluation. Team training in acute care and communication training were the most frequent learning objectives, and predominant learning activities include practice on simulation scenario and debriefing. Two-thirds of the studies did not explain their theoretical framework that underpinned their design and implementation of MUVWs. While MUVWs in healthcare education is generally well-received, learning outcomes remain inconclusive.

CONCLUSION

Despite a growth of studies on the use of MUVW in healthcare education, there is a need for more understanding of the application of theories to inform the learning activities. Therefore, we suggest educators to incorporate a theoretical model to explain the learning processes behind MUVWs. To improve the quality of evidence, we call for researchers to employ a more rigorous and broader approach to evaluation that explicates longer-term outcomes, including cost benefit analyses.

What do we know about coaching in medical education? A literature review

CONTEXT

Coaching has been employed successfully in the competitive sports, professional music, and business and corporate worlds. It is now emerging as a training modality in medical education.

OBJECTIVES

This paper reviews the current evidence on coaching strategies for doctors and medical students.

METHODS

An applied literature search was conducted in PubMed, MEDLINE and Web of Science. Predetermined definitions of coaching interventions and their evaluations were used to narrow 993 papers down to 21, which were included in the final review. The 21 papers were critiqued with reference to validated scoring metrics.

RESULTS

There are many papers discussing the merits of coaching in the world of medicine, but few evaluations of coaching interventions. Existing coaching methodologies can be broadly summarised into three categories: coaching for doctor/student well-being and resilience; coaching for improved non-technical skills, and coaching for technical skills. Identification of suitable papers for inclusion is complicated by theoretical uncertainty regarding coaching: many papers use the term as a synonym for teaching or mentoring. The strongest evidence for coaching lies in the teaching of technical skills.

CONCLUSIONS

There is weak- to medium-strength evidence to support coaching as a method of improving doctor well-being and enhancing non-technical skills, although the evidence base is limited as a whole. This review identifies strong evidence to support coaching as a method to improve technical skills. There is great scope for further studies investigating the power of coaching in medical students and doctors.

Factors Influencing Acceptability and Perceived Impacts of a Mandatory ePortfolio Implemented by an Occupational Therapy Regulatory Organization

INTRODUCTION

The use of ePortfolios has been implemented in several regulatory organizations to encourage clinicians' engagement in continuing professional development (CPD). However, their use has achieved mixed success, and multiple personal and contextual factors can influence their impacts on practice change. The aim of this study was to identify which factors influence the acceptability and perceived impacts of an ePortfolio implemented by an occupational therapy regulatory organization in one Canadian province.

METHOD

A cross-sectional online survey design was used. The survey was sent to registered occupational therapists in Quebec. Multiple regression analyses were conducted to identify factors influencing acceptability and outcomes: ease of use, satisfaction, impact on implementation of the CPD plan, and competence improvement.

RESULTS

The survey was fully completed by 546 participants. Factors significantly influencing the ePortfolio acceptability and perceived impacts were attitude toward and familiarity with the portfolio, confidence in reflective skills, engagement in the CPD plan, and desire for feedback. Time spent completing the ePortfolio and the fact of completing it in teams were negatively associated with the outcomes.

DISCUSSION

Shaping more favorable user attitudes, helping users recognize and experience the tool's benefits for their practice, and fostering confidence in their reflective skills are important factors that can be addressed to improve ePortfolio acceptability and outcomes. Contextual factors, such as time spent completing the ePortfolio and completing it in teams, seem to reflect greater difficulty with using the tool. Study findings can contribute to improving ePortfolio implementation in the CPD context.

Stakeholder perspectives on workplace-based performance assessment: towards a better understanding of assessor behaviour

Workplace-Based Assessment (WBA) plays a pivotal role in present-day competency-based medical curricula. Validity in WBA mainly depends on how stakeholders (e.g. clinical supervisors and learners) use the assessments-rather than on the intrinsic qualities of instruments and methods. Current research on assessment in clinical contexts seems to imply that variable behaviours during performance assessment of both assessors and learners may well reflect their respective beliefs and perspectives towards WBA. We therefore performed a Q methodological study to explore perspectives underlying stakeholders' behaviours in WBA in a postgraduate medical training program. Five different perspectives on performance assessment were extracted: Agency, Mutuality, Objectivity, Adaptivity and Accountability. These perspectives reflect both differences and similarities in stakeholder perceptions and preferences regarding the utility of WBA. In comparing and contrasting the various perspectives, we identified two key areas of disagreement, specifically 'the locus of regulation of learning' (i.e., self-regulated versus externally regulated learning) and 'the extent to which assessment should be standardised' (i.e., tailored versus standardised assessment). Differing perspectives may variously affect stakeholders' acceptance, use-and, consequently, the effectiveness-of assessment programmes. Continuous interaction between all stakeholders is essential to monitor, adapt and improve assessment practices and to stimulate the development of a shared mental model. Better understanding of underlying stakeholder perspectives could be an important step in bridging the gap between psychometric and socio-constructivist approaches in WBA.

Electronic portfolio use in pediatric residency and perceived efficacy as a tool for teaching lifelong learning

BACKGROUND

Residency programs use electronic portfolios (efolios) to organize data, track resident performance, and sometimes teach and assess lifelong learning (LLL) skills. Published studies on efolios in graduate medical education are mostly descriptions of implementation at individual institutions.

METHODS

An anonymous online survey was sent to 199 pediatric residency program directors across the United States. Efolio usage patterns were described and compared between program directors that perceived efolios effective at fostering LLL and those that did not.

RESULTS

Surveys were completed by 82 of 199 program directors (41%), and 55% used efolios. The 20% (9 of 45) of program directors that believed efolios were effective at teaching LLL more often used self-assessment (88% vs. 50%, p = 0.05) and goal-setting (75% vs. 40%, p = 0.03) functionalities. Common efolio challenges included limited usability and difficulty integrating data. Most non-users (65%) would like to invest in efolios.

CONCLUSIONS

Respondents reported technical and convenience-related challenges to efolio use, which need to be addressed for efolios to meet their potential as valuable learning tools. The use of self-assessments and goal-setting features was associated with program directors' perceptions that efolios were effective at fostering LLL.

Clinical Skills Passport: A Method to Increase Participation in Clinical Skills by Medical Students During a Surgery Clerkship

OBJECTIVE

To prospectively evaluate the effect of introducing a clinical skills "passport" on medical students' reporting of their experience with basic clinical skills.

DESIGN

A prospective longitudinal intervention study was conducted. Medical students were administered a questionnaire at the conclusion of their 12-week surgery clerkship regarding their experience with 15 clinical skills, inquiring whether they had "learned on surgery clerkship", "learned before surgery clerkship", or "not learned". Preintervention baseline data were obtained for 2 consecutive academic years (n = 213 students). In the third year, students (n = 124) were given a clinical skills passport to document performance of 8 of the 15 skills under the supervision of a surgical resident or faculty member. After excluding from analysis those students who learned a skill before their surgery clerkship, the fraction of students who reported learning clinical skills during their surgery clerkship before and after the introduction of the clinical skills passport was compared using Fisher exact test and chi-squared test, where appropriate (p ≤ 0.003 was considered significant; Bonferroni correction for multiple comparisons).

SETTING

Washington University School of Medicine.

PARTICIPANTS

A total of 337 medical students completing the junior surgery clerkship over a 3-year period were included in the study.

RESULTS

All 337 students completed a survey. Considering each skill individually, survey response rate was 5045/5055 (99.8%). Combining all responses for all skills, the fraction of students reporting that skills were learned on the surgery clerkship increased after the introduction of the clinical skills passport (1498/1938 [77%] preintervention vs. 974/1109 [88%] postintervention, p < 0.0001, chi square). After the introduction of the clinical skills passport, the fraction of students reporting that a skill was "learned on surgery clerkship" significantly increased for the 8 skills listed on the passport (1026/1699 [83%] preintervention vs. 685/714 [96%] postintervention, p < 0.0001, chi square), but did not increase for the 7 skills not listed on the passport (472/695 [68%] preintervention vs. 289/395 [73%] postintervention, p = 0.08, chi square). Considering each skill individually, after the introduction of the clinical skills passport, the fraction of students reporting that a skill was "learned on surgery clerkship" significantly increased for 4/15 skills (all listed on the clinical passport) and decreased for 0/15 skills (p < 0.003, Fisher exact test, 2 tailed).

CONCLUSIONS

Institution of a clinical skills passport system during a surgery clerkship increased medical student reporting of their performance of basic clinical skills.

Evaluation of the training capacity of the Spanish Resident Book of Otolaryngology (FORMIR) as an electronic portfolio

BACKGROUND

and objectives We have evaluated the training capacity of the Spanish resident training book as an electronic portfolio to achieve the learning objectives of otorhinolaryngology (ENT) residents.

METHODS

A multi-method qualitative investigation with transversal characteristics, temporal and retrospective guidance was performed on Spanish ENT residents using a structured questionnaire, a semi-structured interview, and a computer application on the FORMIR website.

RESULTS

A 56.5% of ENT-residents specialising in one of the 63 accredited Spanish hospitals between 2009-2012 participated in the study. The results obtained show that the ENT residents who used the e-portfolio were better able to implement self-guided study, were more aware of their learning objectives, fulfilled the training programme more efficiently, identified the causes of learning gaps more clearly, and considered FORMIR in e-portfolio format to be an ideal training tool to replace the resident training book in paper format.

CONCLUSIONS

The ENT residents greatly appreciated the training benefits of FORMIR as an e-portfolio, especially its simple and intuitive interface, the ease and comfort with which they could record their activities, the automatic and numeric feedback on the acquisition of their competencies (which facilitates self-guided learning), its storage capacity for evidence, and its ability to be used as UEMS logbook as well as a standard curriculum vitae. All these features make FORMIR a training and evaluation tool that outperforms similar instruments available to ENT residents. They do not hesitate to identify it as the ideal resident training book for facilitating their specialised training.

Utilisation of an electronic portfolio to engage rehabilitation professionals in continuing professional development: results of a provincial survey

INTRODUCTION

ePortfolios are frequently used to support continuing professional development (CPD) of rehabilitation professionals. Though this tool is now widely implemented in many professions by regulatory organisations, very few studies have investigated the use and impact among rehabilitation professionals. Implementation of comprehensive ePortfolios that are centred on the needs of rehabilitation professionals requires documenting their level of use and perceived outcomes.

OBJECTIVES

The objectives were to describe how occupational therapists use a mandatory ePortfolio that has been recently implemented by a regulatory organisation in Quebec (Canada) and the perceived outcomes of this requirement on continuing professional development and practice change.

METHODS

An online survey was sent to all registered occupational therapists in Quebec using the ePortfolio. The survey content was developed based on a literature review and expert consultation. Results were analysed using descriptive statistics.

RESULTS

A total of 546 respondents completed the survey. Results show relatively high levels of ease and satisfaction with the tool, but a limited perception of the tool's impacts on the improvement of professional competencies and change in practices.

CONCLUSIONS

Occupational therapists reported that use of the ePortfolio supports their engagement in CPD but has limited impact on practice. Promotion of work-based learning, team use and mentor support could increase its meaningfulness for professionals. Implications for Rehabilitation To improve attitudes and beliefs about benefits related to portfolio use, rehabilitation practitioners need a very clear understanding of the purpose and usefulness of a portfolio in clinical practice. Most of the respondents saw the ePortfolio as helping them develop and implement a continuing professional development plan and reflect on the changes needed in their practice. Portfolio use in teams and productive reflection should be promoted in order to target shared objectives for continuous practice improvement. Rehabilitation professionals trained in portfolio use during their entry-level studies have a slightly more positive attitude towards portfolio use and impact of this use compared with than clinicians who have not had this training.

When Assessment Data Are Words: Validity Evidence for Qualitative Educational Assessments

Quantitative scores fail to capture all important features of learner performance. This awareness has led to increased use of qualitative data when assessing health professionals. Yet the use of qualitative assessments is hampered by incomplete understanding of their role in forming judgments, and lack of consensus in how to appraise the rigor of judgments therein derived. The authors articulate the role of qualitative assessment as part of a comprehensive program of assessment, and translate the concept of validity to apply to judgments arising from qualitative assessments. They first identify standards for rigor in qualitative research, and then use two contemporary assessment validity frameworks to reorganize these standards for application to qualitative assessment.Standards for rigor in qualitative research include responsiveness, reflexivity, purposive sampling, thick description, triangulation, transparency, and transferability. These standards can be reframed using Messick's five sources of validity evidence (content, response process, internal structure, relationships with other variables, and consequences) and Kane's four inferences in validation (scoring, generalization, extrapolation, and implications). Evidence can be collected and evaluated for each evidence source or inference. The authors illustrate this approach using published research on learning portfolios.The authors advocate a "methods-neutral" approach to assessment, in which a clearly stated purpose determines the nature of and approach to data collection and analysis. Increased use of qualitative assessments will necessitate more rigorous judgments of the defensibility (validity) of inferences and decisions. Evidence should be strategically sought to inform a coherent validity argument.

Status of portfolios in undergraduate medical education in the LCME accredited US medical school

AIM

We sought to investigate the number of US medical schools utilizing portfolios, the format of portfolios, information technology (IT) innovations, purpose of portfolios and their ability to engage faculty and students.

METHODS

A 21-question survey regarding portfolios was sent to the 141 LCME-accredited, US medical schools. The response rate was 50% (71/141); 47% of respondents (33/71) reported that their medical school used portfolios in some form. Of those, 7% reported the use of paper-based portfolios and 76% use electronic portfolios. Forty-five percent reported portfolio use for formative evaluation only; 48% for both formative and summative evaluation, and 3% for summative evaluation alone.

RESULTS

Seventy-two percent developed a longitudinal, competency-based portfolio. The most common feature of portfolios was reflective writing (79%). Seventy-three percent allow access to the portfolio off-campus, 58% allow usage of tablets and mobile devices, and 9% involve social media within the portfolio. Eighty percent and 69% agreed that the portfolio engaged students and faculty, respectively. Ninety-seven percent reported that the portfolios used at their institution have room for improvement.

CONCLUSION

While there is significant variation in the purpose and structure of portfolios in the medical schools surveyed, most schools using portfolios reported a high level of engagement with students and faculty.

Content validity of workplace-based portfolios: A multi-centre study

BACKGROUND

Portfolios are used as tools to coach and assess students in the workplace. This study sought to evaluate the content validity of portfolios as reflected in their capacity to adequately assess achieved competences of medical students during clerkships.

METHODS

We reviewed 120 workplace portfolios at three medical universities (Belgium and the Netherlands). To validate their content, we developed a Validity Inventory for Portfolio Assessment (VIPA) based on the CanMEDS roles. Two raters evaluated each portfolio and indicated for each VIPA item whether the portfolio provided sufficient information to enable satisfactory assessment of the item. We ran a descriptive analysis on the validation data and computed Cohen's Kappa to investigate interrater agreement.

RESULTS

The portfolios adequately covered the items pertaining to the communicator (90%) and professional (87%) roles. Coverage of the medical expert, collaborator, scholar and manager roles ranged between 75% and 85%. The health advocate role, covering 59%, was clearly less well represented. This role also exhibited little interrater agreement (Kappa < 0.4).

CONCLUSIONS

This study lends further credence to the evidence that portfolios can indeed adequately assess the different CanMEDS roles during clerkships, the health advocate role, which was less well represented in the portfolio content, excepted.

A systematic review of assessment and intervention strategies for effective clinical communication in culturally and linguistically diverse students

OBJECTIVE

Culturally and linguistically diverse (CALD) students often experience difficulties with the clinical communication skills that are essential for successful interactions in the workplace. However, there is little evidence on the effectiveness of assessment and intervention strategies for this population. The two aims of this study were: to evaluate the effectiveness of assessment tools in identifying and describing the clinical communication difficulties of CALD health care students; and to determine whether communication programmes improved their clinical communication skills.

METHODS

Systematic review based on the Cochrane protocol. Articles were identified through a search of established databases using MeSH and key search terms. Studies published in English from 1990 to March 2015 were included if they described assessment strategies or a training programme for communication skills of CALD students. Studies were excluded if they did not describe implementation of a specific assessment or intervention programme. Data were extracted independently by the first author and verified by the second author. Quality was measured by the Best Evidence Medical Education guide and the Educational Interventions Critical Appraisal Tool. The Kirkpatrick hierarchy was used to measure impact. Meta-analysis was not conducted because of the heterogeneity of programme design and outcome measures.

RESULTS

One hundred and twenty-nine articles met the criteria for full text review. Eighty-six articles were excluded. Thirteen articles addressing assessment and 30 articles reporting on communication training programmes were included in this review. Assessment tools used rubrics and rating scales effectively. Intervention studies focused on speech and language skills (n = 20), interpersonal skills (n = 7) and faculty-level support (n = 5). Although 17 studies reported positive findings on student satisfaction, only eight reported improved skills post-training.

CONCLUSIONS

The development of effective assessment and intervention programmes should have an integrated design and include specific outcome measures to increase educational impact.