A Delphi approach to define learning outcomes and assessment

Evaluating user experience with immersive technology in simulation-based education: A modified Delphi study with qualitative analysis

BACKGROUND

Immersive technology is becoming more widespread in simulation-based medical education with applications that both supplement and replace traditional teaching methods. There is a lack of validated measures that capture user experience to inform of the technology utility. We aimed to establish a consensus of items and domains that different simulation experts would include in a measure for immersive technology use.

METHODS

A 3-stage modified Delphi using online software was conducted to support the conceptual framework for the proposed measure. The first round was informed by prior work on immersive technology in simulation. In the first round, participants were asked to describe what we could measure in simulation-based education and technology. Thematic analysis generated key themes that were presented to the participants in the second round. Ranking of importance in round 2 was determined by mean rank scores. The final round was an online meeting for final consensus discussion and most important domains by experts were considered.

RESULTS

A total of 16 simulation experts participated in the study. A consensus was reached on the ideal measure in immersive technology simulation that would be a user questionnaire and domains of interest would be: what was learnt, the degree of immersion experienced, fidelity provided, debrief, psychological safety and patient safety. No consensus was reached with the barriers that this technology introduces in education.

CONCLUSIONS

There is varied opinion on what we should prioritise in measuring the experience in simulation practice. Importantly, this study identified key areas that aids our understanding on how we can measure new technology in educational settings. Synthesising these results in to a multidomain instrument requires a systematic approach to testing in future research.

Educational Research Report Development of International Learning Outcomes for Shelter Medicine in Veterinary Education: A Delphi Approach

Shelter medicine is a veterinary discipline of growing importance. Formally accepted as a clinical specialty in the US in 2014, the practice of shelter medicine worldwide is expanding. As a topic in veterinary pre-registration (undergraduate) education, it is frequently used as an opportunity to teach primary care skills, but increasingly recognized as a subject worthy of teaching in its own right. The aim of this study was to use a Delphi consensus methodology to identify learning outcomes relevant to shelter medicine education. Shelter medicine educators worldwide in a variety of settings, including universities, non-governmental organizations and shelters were invited to participate. Participants were initially invited to share shelter medicine teaching materials. These were synthesized and formatted into Learning Outcomes (LOs) based on Bloom's taxonomy and organized into five subject-specific domains. Participants were then asked to develop and evaluate the identified LOs in two rounds of online surveys. Consensus was determined at > 80% of panelists selecting "agree" or "strongly agree" in response to the statement "please indicate whether you would advise that it should be included in a shelter medicine education program" for each LO. In the second survey, where re-wording of accepted LOs was suggested, preference was determined at > 50% agreement. Through this method, 102 agreed LOs have been identified and refined. These LOs, as well as those which did not reach consensus, are presented here. These are intended for use by shelter medicine educators worldwide, to enable and encourage the further development of this important veterinary discipline.

Defining interdisciplinary competencies for audiological rehabilitation: findings from a modified Delphi study

OBJECTIVES

The aim of this study is to derive a consensus on an interdisciplinary competency framework regarding a holistic approach for audiological rehabilitation (AR), which includes disciplines from medicine, engineering, social sciences and humanities.

DESIGN

We employed a modified Delphi method. In the first round survey, experts were asked to rate an initial list of 28 generic interdisciplinary competencies and to propose specific knowledge areas for AR. In the second round, experts were asked to reconsider their answers in light of the group answers of the first round.

STUDY SAMPLE

An international panel of 27 experts from different disciplines in AR completed the first round. Twenty-two of them completed the second round.

RESULTS

We developed a competency framework consisting of 21 generic interdisciplinary competencies grouped in five domains and nine specific competencies (knowledge areas) in three clusters. Suggestions for the implementation of the generic competencies in interdisciplinary programmes were identified.

CONCLUSIONS

This study reveals insights into the interdisciplinary competencies that are unique for AR. The framework will be useful for educators in developing interdisciplinary programmes as well as for professionals in considering their lifelong training needs in AR.

Designing Learning Outcomes for Handoff Teaching of Medical Students Using Group Concept Mapping: Findings From a Multicountry European Study

PURPOSE

To develop, by consultation with an expert group, agreed learning outcomes for the teaching of handoff to medical students using group concept mapping.

METHOD

In 2013, the authors used group concept mapping, a structured mixed-methods approach, applying both quantitative and qualitative measures to identify an expert group's common understanding about the learning outcomes for training medical students in handoff. Participants from four European countries generated and sorted ideas, then rated generated themes by importance and difficulty to achieve. The research team applied multidimensional scaling and hierarchical cluster analysis to analyze the themes.

RESULTS

Of 127 experts invited, 45 contributed to the brainstorming session. Twenty-two of the 45 (48%) completed pruning, sorting, and rating phases. They identified 10 themes with which to select learning outcomes and operationally define them to form a basis for a curriculum on handoff training. The themes "Being able to perform handoff accurately" and "Demonstrate proficiency in handoff in workplace" were rated as most important. "Demonstrate proficiency in handoff in simulation" and "Engage with colleagues, patients, and carers" were rated most difficult to achieve.

CONCLUSIONS

The study identified expert consensus for designing learning outcomes for handoff training for medical students. Those outcomes considered most important were among those considered most difficult to achieve. There is an urgent need to address the preparation of newly qualified doctors to be proficient in handoff at the point of graduation; otherwise, this is a latent error within health care systems. This is a first step in this process.

Use of a group concept mapping approach to define learning outcomes for an interdisciplinary module in medicine

Learning outcomes are typically developed using standard group-based consensus methods. Two main constraints with standard techniques such as the Delphi method or expert working group processes are: (1) the ability to generate a comprehensive set of outcomes and (2) the capacity to reach agreement on them. We describe the first application of Group Concept Mapping (GCM) to the development of learning outcomes for an interdisciplinary module in medicine and engineering. The biomedical design module facilitates undergraduate participation in clinician-mentored team-based projects that prepare students for a multidisciplinary work environment. GCM attempts to mitigate the weaknesses of other consensus methods by excluding pre-determined classification schemes and inter-coder discussion, and by requiring just one round of data structuring. Academic members from medicine and engineering schools at three EU higher education institutions participated in this study. Data analysis, which included multidimensional scaling and hierarchical cluster analysis, identified two main categories of outcomes: technical skills (new advancement in design process with special attention to users, commercialization and standardization) and transversal skills such as working effectively in teams and creative problem solving. The study emphasizes the need to address the highest order of learning taxonomy (analysis, synthesis, problem solving, creativity) when defining learning outcomes.