Emerging Issues in Assessment in Medical Education: A Collection

Challenges and future directions in the measurement of evidence-based practice: Qualitative analysis of umbrella review findings

RATIONALE, AIMS AND OBJECTIVES: An important aspect of scholarly discussions about evidence-based practice (EBP) is how EBP is measured. Given the conceptual and empirical developments in the study of EBP over the last 3 decades, there is a need to better understand how to best measure EBP in educational and clinical contexts. The aim of this study was to identify and describe the main challenges, recommendations for practice, and areas of future research in the measurement of EBP across the health professions as reported by systematic reviews (SRs).

METHODS

We conducted a secondary analysis of qualitative data obtained in the context of a previously published umbrella review that aimed to compare SRs on EBP measures. Two reviewers independently extracted excerpts from the results and discussion/conclusion sections of the 10 included SRs that aligned with the three research aims. An iterative six-phase reflexive thematic analysis according to Braun and Clarke was conducted.

RESULTS

Our thematic analysis produced five themes describing the main challenges associated with measuring EBP, four themes outlining main recommendations for practice, and four themes representing areas of future research. Challenges include limited psychometric testing and validity evidence for existing EBP measures; limitations with the self-report format; lack of construct clarity of EBP measures; inability to capture the complexity of the EBP process and outcomes; and the context-specific nature of EBP measures. Reported recommendations for practice include acknowledging the multidimensionality of EBP; adapting EBP measures to the context and re-examining the validity argument; and considering the feasibility and acceptability of measures. Areas of future research included the development of comprehensive, multidimensional EBP measures and the need for expert consensus on the operationalization of EBP.

CONCLUSIONS

This study suggests that existing measures may be insufficient in capturing the multidimensional, contextual and dynamic nature of EBP. There is a need for a clear operationalization of EBP and an improved understanding and application of validity theory.

Quality and constructed knowledge: Truth, paradigms, and the state of the science

CONTEXT AND TRUTH

Education is a social science. Social science knowledge is related to its context of origin. The concept of global 'truth' in education is therefore of limited use when truth is tempered by context. The wider applicability of our knowledge can only be judged if we look at the context in which that knowledge was produced and the assumptions that underpin it. This calls into question the idea that educational research is a quest for global 'truth', although in relation to programme evaluation, truth tied to context is an aim. An analysis is presented of the effects of social construction on research and evaluation processes, on the selection of paradigms, reporting and interpreting findings, and on the ethics of all this.

QUALITY AND IMPROVEMENT

Quality improvement is based on information selected, constructed and interpreted by those who gather, analyse or use it. The strength, and not the weakness, of our knowledge is that it is socially constructed, contextual and of its time. Increasingly looking for our own truth about educational quality, and not importing the truth of others, is crucial to the state of the science. In terms of quality development, using others' findings must be based on informed local judgement. In social science, those judgements are linked to social context and their associated ideologies.

IMPLICATIONS FOR FUTURE WORK

The hallmark of social science is not a narrowing of focus and the search for one truth, but is a broadening of concepts, theories, paradigms, reported experience and method, and an intention for each to tell their own truth well. This will lead to a wealth of diverse views and analysed experience. The science of medical education must seek many truths.