1
|
Chiu H, Wood TJ, Garber A, Halman S, Rekman J, Gofton W, Dudek N. The Ottawa resident observation form for nurses (O-RON): evaluation of an assessment tool's psychometric properties in different specialties. BMC Med Educ 2024; 24:487. [PMID: 38698352 PMCID: PMC11067073 DOI: 10.1186/s12909-024-05476-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/18/2023] [Accepted: 04/26/2024] [Indexed: 05/05/2024]
Abstract
BACKGROUND Workplace-based assessment (WBA) used in post-graduate medical education relies on physician supervisors' feedback. However, in a training environment where supervisors are unavailable to assess certain aspects of a resident's performance, nurses are well-positioned to do so. The Ottawa Resident Observation Form for Nurses (O-RON) was developed to capture nurses' assessment of trainee performance and results have demonstrated strong evidence for validity in Orthopedic Surgery. However, different clinical settings may impact a tool's performance. This project studied the use of the O-RON in three different specialties at the University of Ottawa. METHODS O-RON forms were distributed on Internal Medicine, General Surgery, and Obstetrical wards at the University of Ottawa over nine months. Validity evidence related to quantitative data was collected. Exit interviews with nurse managers were performed and content was thematically analyzed. RESULTS 179 O-RONs were completed on 30 residents. With four forms per resident, the ORON's reliability was 0.82. Global judgement response and frequency of concerns was correlated (r = 0.627, P < 0.001). CONCLUSIONS Consistent with the original study, the findings demonstrated strong evidence for validity. However, the number of forms collected was less than expected. Exit interviews identified factors impacting form completion, which included clinical workloads and interprofessional dynamics.
Collapse
Affiliation(s)
- Hedva Chiu
- Department of Medicine, Division of Physical Medicine & Rehabilitation, University of Ottawa, Ottawa, Canada.
| | - Timothy J Wood
- Department of Innovation in Medical Education, University of Ottawa, Ottawa, Canada
| | - Adam Garber
- Department of Obstetrics and Gynecology, University of Ottawa, Ottawa, Canada
| | - Samantha Halman
- Department of Medicine, Division of General Internal Medicine, University of Ottawa, Ottawa, Canada
| | - Janelle Rekman
- Department of Surgery, Division of General Surgery, University of Ottawa, Ottawa, Canada
| | - Wade Gofton
- Department of Surgery, Division of Orthopedic Surgery, University of Ottawa, Ottawa, Canada
| | - Nancy Dudek
- Department of Medicine, Division of Physical Medicine & Rehabilitation), The Ottawa Hospital, University of Ottawa, Ottawa, ON, Canada
| |
Collapse
|
2
|
Wood TJ, Daniels VJ, Pugh D, Touchie C, Halman S, Humphrey-Murto S. Implicit versus explicit first impressions in performance-based assessment: will raters overcome their first impressions when learner performance changes? Adv Health Sci Educ Theory Pract 2023:10.1007/s10459-023-10302-2. [PMID: 38010576 DOI: 10.1007/s10459-023-10302-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/25/2023] [Accepted: 11/12/2023] [Indexed: 11/29/2023]
Abstract
First impressions can influence rater-based judgments but their contribution to rater bias is unclear. Research suggests raters can overcome first impressions in experimental exam contexts with explicit first impressions, but these findings may not generalize to a workplace context with implicit first impressions. The study had two aims. First, to assess if first impressions affect raters' judgments when workplace performance changes. Second, whether explicitly stating these impressions affects subsequent ratings compared to implicitly-formed first impressions. Physician raters viewed six videos where learner performance either changed (Strong to Weak or Weak to Strong) or remained consistent. Raters were assigned two groups. Group one (n = 23, Explicit) made a first impression global rating (FIGR), then scored learners using the Mini-CEX. Group two (n = 22, Implicit) scored learners at the end of the video solely with the Mini-CEX. For the Explicit group, in the Strong to Weak condition, the FIGR (M = 5.94) was higher than the Mini-CEX Global rating (GR) (M = 3.02, p < .001). In the Weak to Strong condition, the FIGR (M = 2.44) was lower than the Mini-CEX GR (M = 3.96 p < .001). There was no difference between the FIGR and the Mini-CEX GR in the consistent condition (M = 6.61, M = 6.65 respectively, p = .84). There were no statistically significant differences in any of the conditions when comparing both groups' Mini-CEX GR. Therefore, raters adjusted their judgments based on the learners' performances. Furthermore, raters who made their first impressions explicit showed similar rater bias to raters who followed a more naturalistic process.
Collapse
Affiliation(s)
- Timothy J Wood
- Faculty of Medicine, University of Ottawa, 850 Peter Morand Crescent, Ottawa, ON, K1G-5Z3, Canada.
| | - Vijay J Daniels
- Department of Medicine, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Canada
| | - Debra Pugh
- Faculty of Medicine, University of Ottawa, 850 Peter Morand Crescent, Ottawa, ON, K1G-5Z3, Canada
- Department of Medicine, The Ottawa Hospital, Ottawa, Canada
- Medical Council of Canada, Ottawa, Canada
| | - Claire Touchie
- Faculty of Medicine, University of Ottawa, 850 Peter Morand Crescent, Ottawa, ON, K1G-5Z3, Canada
- Department of Medicine, The Ottawa Hospital, Ottawa, Canada
| | - Samantha Halman
- Faculty of Medicine, University of Ottawa, 850 Peter Morand Crescent, Ottawa, ON, K1G-5Z3, Canada
- Department of Medicine, The Ottawa Hospital, Ottawa, Canada
| | - Susan Humphrey-Murto
- Faculty of Medicine, University of Ottawa, 850 Peter Morand Crescent, Ottawa, ON, K1G-5Z3, Canada
- Department of Medicine, The Ottawa Hospital, Ottawa, Canada
| |
Collapse
|
3
|
Leclercq N, Marshall L, Weekers T, Basu P, Benda D, Bevk D, Bhattacharya R, Bogusch P, Bontšutšnaja A, Bortolotti L, Cabirol N, Calderón-Uraga E, Carvalho R, Castro S, Chatterjee S, De La Cruz Alquicira M, de Miranda JR, Dirilgen T, Dorchin A, Dorji K, Drepper B, Flaminio S, Gailis J, Galloni M, Gaspar H, Gikungu MW, Hatteland BA, Hinojosa-Diaz I, Hostinská L, Howlett BG, Hung KLJ, Hutchinson L, Jesus RO, Karklina N, Khan MS, Loureiro J, Men X, Molenberg JM, Mudri-Stojnić S, Nikolic P, Normandin E, Osterman J, Ouyang F, Oygarden AS, Ozolina-Pole L, Ozols N, Parra Saldivar A, Paxton RJ, Pitts-Singer T, Poveda K, Prendergast K, Quaranta M, Read SFJ, Reinhardt S, Rojas-Oropeza M, Ruiz C, Rundlöf M, Sade A, Sandberg C, Sgolastra F, Shah SF, Shebl MA, Soon V, Stanley DA, Straka J, Theodorou P, Tobajas E, Vaca-Uribe JL, Vera A, Villagra CA, Williams MK, Wolowski M, Wood TJ, Yan Z, Zhang Q, Vereecken NJ. Global taxonomic, functional, and phylogenetic diversity of bees in apple orchards. Sci Total Environ 2023; 901:165933. [PMID: 37536603 DOI: 10.1016/j.scitotenv.2023.165933] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/01/2023] [Revised: 07/27/2023] [Accepted: 07/29/2023] [Indexed: 08/05/2023]
Abstract
An essential prerequisite to safeguard pollinator species is characterisation of the multifaceted diversity of crop pollinators and identification of the drivers of pollinator community changes across biogeographical gradients. The extent to which intensive agriculture is associated with the homogenisation of biological communities at large spatial scales remains poorly understood. In this study, we investigated diversity drivers for 644 bee species/morphospecies in 177 commercial apple orchards across 33 countries and four global biogeographical biomes. Our findings reveal significant taxonomic dissimilarity among biogeographical zones. Interestingly, despite this dissimilarity, species from different zones share similar higher-level phylogenetic groups and similar ecological and behavioural traits (i.e. functional traits), likely due to habitat filtering caused by perennial monoculture systems managed intensively for crop production. Honey bee species dominated orchard communities, while other managed/manageable and wild species were collected in lower numbers. Moreover, the presence of herbaceous, uncultivated open areas and organic management practices were associated with increased wild bee diversity. Overall, our study sheds light on the importance of large-scale analyses contributing to the emerging fields of functional and phylogenetic diversity, which can be related to ecosystem function to promote biodiversity as a key asset in agroecosystems in the face of global change pressures.
Collapse
Affiliation(s)
- N Leclercq
- Agroecology Lab, Université Libre de Bruxelles (ULB), Boulevard du Triomphe CP 264/02, B-1050 Brussels, Belgium.
| | - L Marshall
- Agroecology Lab, Université Libre de Bruxelles (ULB), Boulevard du Triomphe CP 264/02, B-1050 Brussels, Belgium; Naturalis Biodiversity Center, Darwinweg 2, 2333 CR, Leiden, Netherlands
| | - T Weekers
- Agroecology Lab, Université Libre de Bruxelles (ULB), Boulevard du Triomphe CP 264/02, B-1050 Brussels, Belgium
| | - P Basu
- Centre for Pollination Studies, University of Calcutta, Kolkata, India
| | - D Benda
- Department of Zoology, Faculty of Science, Charles University, Prague, Czech Republic; Department of Entomology, National Museum, Prague, Czech Republic
| | - D Bevk
- Department of Organisms and Ecosystems Research, National Institute of Biology, Ljubljana, Slovenia
| | - R Bhattacharya
- Centre for Pollination Studies, University of Calcutta, Kolkata, India
| | - P Bogusch
- Department of Biology, Faculty of Science, University of Hradec Králové, Hradec Králové, Czech Republic
| | - A Bontšutšnaja
- Institute of Agricultural and Environmental Sciences, Estonian University of Life Sciences, Tartu, Estonia
| | - L Bortolotti
- CREA Research Centre for Agriculture and Environment, Bologna, Italy
| | - N Cabirol
- Department of Ecology and Natural Resources, Faculty of Science, UNAM, México City, Mexico
| | - E Calderón-Uraga
- Department of Ecology and Natural Resources, Faculty of Science, UNAM, México City, Mexico
| | - R Carvalho
- Centre for Functional Ecology, Associate Laboratory TERRA, Department of Life Sciences, University of Coimbra, Coimbra, Portugal
| | - S Castro
- Centre for Functional Ecology, Associate Laboratory TERRA, Department of Life Sciences, University of Coimbra, Coimbra, Portugal
| | - S Chatterjee
- Centre for Pollination Studies, University of Calcutta, Kolkata, India
| | - M De La Cruz Alquicira
- Department of Ecology and Natural Resources, Faculty of Science, UNAM, México City, Mexico
| | - J R de Miranda
- Department of Ecology, Swedish University of Agricultural Sciences, Uppsala, 750 05, Sweden
| | - T Dirilgen
- School of Agriculture and Food Science and Earth Institute, University College Dublin, Belfield, Dublin 4, Ireland
| | - A Dorchin
- Laboratory of Zoology, Université de Mons, Mons, Belgium; The Steinhardt Museum of Natural History, Tel Aviv University, 69978 Tel Aviv, Israel; Department of Entomology, Royal Museum for Central Africa, Tervuren, Belgium
| | - K Dorji
- College of Natural Resources, Royal University of Bhutan, Punakha, Bhutan
| | - B Drepper
- Division of Forest, Nature and Landscape, University of Leuven, Leuven, Belgium
| | - S Flaminio
- CREA Research Centre for Agriculture and Environment, Bologna, Italy; Laboratory of Zoology, Université de Mons, Mons, Belgium
| | - J Gailis
- Institute for Plant Protection Research Agrihorts, Latvia University of Life Sciences and Technologies, Jelgava, Latvia
| | - M Galloni
- Department of Biological, Geological, and Environmental Sciences, University of Bologna, Bologna, Italy
| | - H Gaspar
- Centre for Functional Ecology, Associate Laboratory TERRA, Department of Life Sciences, University of Coimbra, Coimbra, Portugal
| | - M W Gikungu
- Department of Zoology, National Museums of Kenya, Nairobi, Kenya
| | - B A Hatteland
- Division for Biotechnology and Plant Health, Norwegian Institute of Bioeconomy Research, Aas, Norway; Department of Biological Sciences, University of Bergen, Bergen, Norway
| | - I Hinojosa-Diaz
- Department of Zoology, Institute of Biology, UNAM, México City, Mexico
| | - L Hostinská
- Department of Biology, Faculty of Science, University of Hradec Králové, Hradec Králové, Czech Republic
| | - B G Howlett
- The New Zealand Institute for Plant & Food Research Limited, Lincoln, Canterbury, New Zealand
| | - K-L J Hung
- Department of Ecology and Evolutionary Biology, University of Toronto, Toronto, ON M5S 3B2, Canada; Oklahoma Biological Survey, University of Oklahoma, Norman, OK 73019, USA
| | - L Hutchinson
- School of Agriculture, Policy and Development, University of Reading, Reading, United Kingdom
| | - R O Jesus
- Graduate Program in Ecology, State University of Campinas, Campinas, São Paulo, Brazil
| | - N Karklina
- Institute for Plant Protection Research Agrihorts, Latvia University of Life Sciences and Technologies, Jelgava, Latvia
| | - M S Khan
- Department of Entomology, University of Agriculture, Peshawar, Pakistan
| | - J Loureiro
- Centre for Functional Ecology, Associate Laboratory TERRA, Department of Life Sciences, University of Coimbra, Coimbra, Portugal
| | - X Men
- Institute of Plant Protection, Shandong Academy of Agricultural Sciences/Shandong Provincial Key Laboratory of Plant Virology,Jinan 250100, China
| | - J-M Molenberg
- Agroecology Lab, Université Libre de Bruxelles (ULB), Boulevard du Triomphe CP 264/02, B-1050 Brussels, Belgium
| | - S Mudri-Stojnić
- Department of Biology and Ecology, Faculty of Sciences, University of Novi Sad, Trg Dositeja Obradovića 2, 21000 Novi Sad, Serbia
| | - P Nikolic
- Faculty of Agriculture, University of Banja Luka, Banja Luka, Bosnia and Herzegovina
| | - E Normandin
- Centre sur la biodiversité, Département des sciences biologiques, Université de Montréal, QC, Québec H1X 2B2, Canada
| | - J Osterman
- General Zoology, Institute for Biology, Martin Luther University Halle-Wittenberg, Hoher Weg 8, 06120 Halle (Saale), Germany; Nature Conservation and Landscape Ecology, University of Freiburg, Tennenbacherstrasse 4, 79106, Freiburg im Breisgau, Germany
| | - F Ouyang
- State Key Laboratory of Integrated Management of Pest Insects and Rodents, Institute of Zoology, Chinese Academy of Sciences, Beijing 100101, China
| | - A S Oygarden
- Department of Natural Sciences and Environmental Health, University of South-Eastern Norway, Bø, Norway
| | - L Ozolina-Pole
- Institute for Plant Protection Research Agrihorts, Latvia University of Life Sciences and Technologies, Jelgava, Latvia
| | - N Ozols
- Institute for Plant Protection Research Agrihorts, Latvia University of Life Sciences and Technologies, Jelgava, Latvia
| | - A Parra Saldivar
- Instituto de Entomología, Universidad Metropolitana de Ciencias de la Educación (UMCE), Santiago, Chile
| | - R J Paxton
- General Zoology, Institute for Biology, Martin Luther University Halle-Wittenberg, Hoher Weg 8, 06120 Halle (Saale), Germany
| | - T Pitts-Singer
- USDA Agricultural Research Service, Pollinating Insects Research Unit, Logan, UT 84322, USA
| | - K Poveda
- Department of Entomology, Cornell University, 4126 Comstock Hall, Ithaca, NY 14853, USA
| | - K Prendergast
- Molecular and Life Sciences, Curtin University, Bentley, WA 6102, Australia
| | - M Quaranta
- CREA Research Centre for Agriculture and Environment, Bologna, Italy
| | - S F J Read
- The New Zealand Institute for Plant & Food Research Limited, Lincoln, Canterbury, New Zealand
| | - S Reinhardt
- Department of Natural Sciences and Environmental Health, University of South-Eastern Norway, Bø, Norway
| | - M Rojas-Oropeza
- Department of Ecology and Natural Resources, Faculty of Science, UNAM, México City, Mexico
| | - C Ruiz
- Departamento Biología Animal, Edafología y Geología, Facultad de Ciencias, Universidad de La Laguna, La Laguna, 38206, Tenerife, Spain
| | - M Rundlöf
- Department of Biology, Lund University, Lund, Sweden
| | - A Sade
- Department of Evolutionary and Environmental Biology, University of Haifa, Mt. Carmel, 31905 Haifa, Israel
| | - C Sandberg
- Department of Biology, Lund University, Lund, Sweden; Calluna AB, Husargatan 3, Malmö, 211 28, Sweden
| | - F Sgolastra
- Department of Agricultural and Food Sciences, University of Bologna, Bologna, Italy
| | - S F Shah
- Department of Entomology, University of Agriculture, Peshawar, Pakistan
| | - M A Shebl
- Department of Plant Protection, Faculty of Agriculture, Suez Canal University, Ismailia 41522, Egypt
| | - V Soon
- Natural History Museum and Botanical Garden, University of Tartu, Vanemuise 46, 51003 Tartu, Estonia
| | - D A Stanley
- School of Agriculture and Food Science and Earth Institute, University College Dublin, Belfield, Dublin 4, Ireland
| | - J Straka
- Department of Zoology, Faculty of Science, Charles University, Prague, Czech Republic
| | - P Theodorou
- General Zoology, Institute for Biology, Martin Luther University Halle-Wittenberg, Hoher Weg 8, 06120 Halle (Saale), Germany
| | - E Tobajas
- Department of Biology, Lund University, Lund, Sweden; Department of Animal Biology, University of Salamanca, Campus Miguel de Unamuno, Salamanca, 37007, Spain
| | - J L Vaca-Uribe
- Laboratorio de Investigaciones en Abejas LABUN, Departamento de Biología, Facultad de Ciencias, Universidad Nacional de Colombia, Bogotá,111321, Colombia
| | - A Vera
- Departamento de Biología, Universidad Metropolitana de Ciencias de la Educación (UMCE), Santiago, Chile
| | - C A Villagra
- Instituto de Entomología, Universidad Metropolitana de Ciencias de la Educación (UMCE), Santiago, Chile
| | - M-K Williams
- Department of Biology, Utah State University, Logan, UT 84322, USA
| | - M Wolowski
- Institute of Natural Sciences, Federal University of Alfenas, Alfenas, Minas Gerais, Brazil
| | - T J Wood
- Laboratory of Zoology, Université de Mons, Mons, Belgium
| | - Z Yan
- State Key Laboratory of Integrated Management of Pest Insects and Rodents, Institute of Zoology, Chinese Academy of Sciences, Beijing 100101, China
| | - Q Zhang
- Beijing Biodiversity Conservation Research Center/Beijing Milu Ecological Research Center, Beijing 100076, China
| | - N J Vereecken
- Agroecology Lab, Université Libre de Bruxelles (ULB), Boulevard du Triomphe CP 264/02, B-1050 Brussels, Belgium
| |
Collapse
|
4
|
Wood TJ, Müller A, Praz C, Michez D. Elevated rates of dietary generalization in eusocial lineages of the secondarily herbivorous bees. BMC Ecol Evol 2023; 23:67. [PMID: 37986035 PMCID: PMC10662511 DOI: 10.1186/s12862-023-02175-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2023] [Accepted: 11/16/2023] [Indexed: 11/22/2023] Open
Abstract
BACKGROUND Within the Hymenoptera, bees are notable for their relationship with flowering plants, being almost entirely dependent on plant pollen and nectar. Though functionally herbivorous, as a result of their role as pollinators, bees have received comparatively little attention as models for insect herbivory. Bees often display dietary specialization, but quantitative comparison against other herbivorous insects has not previously been conducted. RESULTS In the most comprehensive analysis to date for 860 bee species, dietary specialization amounted to 50.1% of studied species collecting pollen from between 1 and 2 botanical families with a relatively long tail of dietary generalists, with 11.1% of species collecting from more than 10 botanical families. This distribution deviated from the truncated Pareto distribution of dietary breadth seen in other herbivorous insect lineages. However, this deviation was predominantly due to eusocial bee lineages, which show a range of dietary breadths that conformed to a normal distribution, while solitary bees show a typical truncated distribution not strongly different from other herbivorous insects. We hypothesize that the relatively low level of dietary specialization in bees as a whole reflects the relaxation of the constraints typically observed in herbivorous insects with a comparatively reduced importance of plant chemistry and comparatively increased importance of phenology and foraging efficiency. The long flight periods of eusocial bees that are necessary to allow overlapping generations both allows and necessitates the use of multiple flowering resources, whereas solitary bees with short flight periods have more limited access to varied resources within a constrained activity period. CONCLUSIONS Collectively, solitary bees show slightly lower specialization compared to other herbivorous insects, possibly due to their balanced relationship with plants, rather than direct antagonism such as seen in the direct consumption of plant tissues. An additional factor may be the mediocre diversity of bees at low latitudes combined with low levels of dietary specialization, whereas these areas typically display a high rate of specialization by herbivorous insects in general. Though the most important factors structuring dietary specialization in bees appear to differ from many other herbivorous insects, solitary bees show a surprisingly similar overall pattern of dietary specialization.
Collapse
Affiliation(s)
- T J Wood
- University of Mons, Research Institute for Biosciences, Laboratory of Zoology, Place du parc 20, 7000, Mons, Belgium.
| | - A Müller
- ETH Zurich, Institute of Agricultural Sciences, Biocommunication and Entomology, Schmelzbergstrasse 9/LFO, 8092, Zurich, Switzerland
| | - C Praz
- University of Neuchâtel, Institute of Biology, Rue Emile-Argand 11, 2000, Neuchâtel, Switzerland
- InfoFauna - Swiss Zoological Records Center, Avenue de Bellevaux 51, 2000, Neuchâtel, Switzerland
| | - D Michez
- University of Mons, Research Institute for Biosciences, Laboratory of Zoology, Place du parc 20, 7000, Mons, Belgium
| |
Collapse
|
5
|
Shaw T, LaDonna KA, Hauer KE, Khalife R, Sheu L, Wood TJ, Montgomery A, Rauscher S, Aggarwal S, Humphrey-Murto S. Having a Bad Day Is Not an Option: Learner Perspectives on Learner Handover. Acad Med 2023; 98:S58-S64. [PMID: 37983397 DOI: 10.1097/acm.0000000000005433] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/12/2023]
Abstract
PURPOSE Learner handover is the sharing of learner-related information between supervisors involved in their education. The practice allows learners to build upon previous assessments and can support the growth-oriented focus of competency-based medical education. However, learner handover also carries the risk of biasing future assessments and breaching learner confidentiality. Little is known about learner handover's educational impact, and what is known is largely informed by faculty and institutional perspectives. The purpose of this study was to explore learner handover from the learner perspective. METHOD Constructivist grounded theory was used to explore learners' perspectives and beliefs around learner handover. Twenty-nine semistructured interviews were completed with medical students and residents from the University of Ottawa and University of California, San Francisco. Interviews took place between April and December 2020. Using the constant comparative approach, themes were identified through an iterative process. RESULTS Learners were generally unaware of specific learner handover practices, although most recognized circumstances where both formal and informal handovers may occur. Learners appreciated the potential for learner handover to tailor education, guide entrustment and supervision decisions, and support patient safety, but worried about its potential to bias future assessments and breach confidentiality. Furthermore, learners were concerned that information-sharing may be more akin to gossip rather than focused on their educational needs and feared unfair scrutiny and irreversible long-term career consequences from one shared mediocre performance. Altogether, these concerns fueled an overwhelming pressure to perform. CONCLUSIONS While learners recognized the rationale for learner handover, they feared the possible inadvertent short- and long-term impact on their training and future careers. Designing policies that support transparency and build awareness around learner handover may mitigate unintended consequences that can threaten learning and the learner-supervisor relationship, ensuring learner handover benefits the learner as intended.
Collapse
Affiliation(s)
- Tammy Shaw
- T. Shaw is assistant professor, Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - Kori A LaDonna
- K.A. LaDonna is associate professor, Department of Innovation in Medical Education and Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - Karen E Hauer
- K.E. Hauer is associate dean for competency assessment and professional standards and professor, Department of Medicine, University of California, San Francisco, School of Medicine, San Francisco, California
| | - Roy Khalife
- R. Khalife is assistant professor, Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - Leslie Sheu
- L. Sheu is a physician, Private Medical, San Francisco, California
| | - Timothy J Wood
- T.J. Wood is professor, Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - Anne Montgomery
- A. Montgomery is associate program director, Washington Regional, University of Arkansas for Medical Sciences, Little Rock, Arkansas
| | - Scott Rauscher
- S. Rauscher is project coordinator, Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada
| | - Simran Aggarwal
- S. Aggarwal is a first-year resident in pediatrics, McMaster University, Hamilton, Ontario, Canada
| | - Susan Humphrey-Murto
- S. Humphrey-Murto is associate professor, Department of Medicine and Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada
| |
Collapse
|
6
|
Dewhirst S, Wood TJ, Cheung WJ, Frank JR. Assessing the utility of a novel entrustment-supervision assessment tool. Med Educ 2023; 57:949-957. [PMID: 37387266 DOI: 10.1111/medu.15156] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/30/2022] [Revised: 06/05/2023] [Accepted: 06/10/2023] [Indexed: 07/01/2023]
Abstract
BACKGROUND Work-based assessments (WBAs) are increasingly used to inform decisions about trainee progression. Unfortunately, WBAs often fail to discriminate between trainees of differing abilities and have poor reliability. Entrustment-supervision scales may improve WBA performance, but there is a paucity of literature directly comparing them to traditional WBA tools. METHODS The Ottawa Emergency Department Shift Observation Tool (O-EDShOT) is a previously published WBA tool employing an entrustment-supervision scale with strong validity evidence. This pre-/post-implementation study compares the performance of the O-EDShOT with that of a traditional WBA tool using norm-based anchors. All assessments completed in 12-month periods before and after implementing the O-EDShOT were collected, and generalisability analysis was conducted with year of training, trainees within year and forms within trainee as nested factors. Secondary analysis included assessor as a factor. RESULTS A total of 3908 and 3679 assessments were completed by 99 and 116 assessors, for 152 and 138 trainees in the pre- and post-implementation phases respectively. The O-EDShOT generated a wider range of awarded scores than the traditional WBA, and mean scores increased more with increasing level of training (0.32 vs. 0.14 points per year, p = 0.01). A significantly greater proportion of overall score variability was attributable to trainees using the O-EDShOT (59%) compared with the traditional tool (21%, p < 0.001). Assessors contributed less to overall score variability for the O-EDShOT than for the traditional WBA (16% vs. 37%). Moreover, the O-EDShOT required fewer completed assessments than the traditional tool (27 vs. 51) for a reliability of 0.8. CONCLUSION The O-EDShOT outperformed a traditional norm-referenced WBA in discriminating between trainees and required fewer assessments to generate a reliable estimate of trainee performance. More broadly, this study adds to the body of literature suggesting that entrustment-supervision scales generate more useful and reliable assessments in a variety of clinical settings.
Collapse
Affiliation(s)
| | - Timothy J Wood
- Department of Innovation in Medical Education, University of Ottawa, Ottawa, Canada
| | - Warren J Cheung
- Department of Emergency Medicine, University of Ottawa, Ottawa, Canada
| | - Jason R Frank
- Department of Emergency Medicine, University of Ottawa, Ottawa, Canada
| |
Collapse
|
7
|
Singh S, Cheung WJ, Dewhirst S, Wood TJ, Landreville JM. The influence of clinical coaching teams on quality of entrustable professional activity assessments. AEM Educ Train 2023; 7:e10879. [PMID: 37361186 PMCID: PMC10290210 DOI: 10.1002/aet2.10879] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/14/2022] [Revised: 03/18/2023] [Accepted: 03/28/2023] [Indexed: 06/28/2023]
Abstract
Background Coaching is an important component of workplace-based assessment in competency-based medical education. Longitudinal coaching relationships have been proposed to enhance the trainee-supervisor relationship and promote high-quality assessment. Objective The objective of this study was to determine the influence of longitudinal coaching relationships on the quality of entrustable professional activity (EPA) assessments. Methods EPAs (n = 174) completed by emergency medicine (EM) supervisors between July 2020 and June 2021 were extracted and divided into two groups; one group consisted of EPAs completed by supervisors when a longitudinal coaching relationship existed (n = 87) and the other group consisted of EPAs completed by the same supervisors when no coaching relationship existed (n = 87). Three physicians were recruited to rate the EPAs using the Quality of Assessment and Learning (QuAL) score, a previously published measure of EPA quality. An analysis of variance was performed to compare mean QuAL scores between the groups. Linear regression analysis was conducted to examine the relationship between trainee performance (EPA rating) and EPA assessment quality (QuAL score). Results All raters completed the survey. The mean ± SD QuAL score in the coaching relationship group (3.63 ± 0.91) was higher than the no coaching relationship group (3.51 ± 1.10) but the difference was not statistically significant (p = 0.40). Supervisor was a significant predictor of QuAL score (p = 0.012) and supervisor alone accounted for 26% of the variability in QuAL scores (R2 = 0.26). There was no significant relationship between trainee performance and EPA assessment quality. Conclusions The presence of a longitudinal coaching relationship did not influence the quality of EPA assessments.
Collapse
Affiliation(s)
| | - Warren J. Cheung
- Department of Emergency MedicineUniversity of OttawaOttawaOntarioCanada
- Royal College of Physicians and Surgeons of CanadaOttawaOntarioCanada
| | | | - Timothy J. Wood
- Department of Innovation in Medical EducationUniversity of OttawaOttawaOntarioCanada
| | | |
Collapse
|
8
|
Humphrey-Murto S, Ho Lee S, Gottlieb M, Horsley T, Shea B, Fournier K, Tran C, Chan T, Wood TJ, Cate OT. Protocol for an extended scoping review on the use of virtual nominal group technique in research. PLoS One 2023; 18:e0280764. [PMID: 36662907 PMCID: PMC9858029 DOI: 10.1371/journal.pone.0280764] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2022] [Accepted: 01/08/2023] [Indexed: 01/21/2023] Open
Abstract
INTRODUCTION Consensus group methods such as the Nominal Group Technique (NGT) and Delphi method are commonly used in research to elicit and synthesize expert opinions when evidence is lacking. Traditionally, the NGT involves a face-to-face interaction. However, due to the COVID-19 pandemic, many in-person meetings have moved to online settings. It is unclear to what extent the NGT has been undertaken in virtual settings. The overarching aim of this scoping review is to explore the use of the virtual NGT in research. Our specific objectives are to answer the following questions: To what extent has the NGT been used virtually? What modifications were made to accommodate this online format? What advantages and disadvantages were noted by authors in comparison with the face-to-face mode of the technique? MATERIALS AND METHODS This scoping review will follow the steps outlined by Arksey and O'Malley and the PRISMA-ScR guidelines. Several pilot searches were completed to refine inclusion and exclusion criteria. Media Synchronicity Theory will provide a conceptual framework to inform the research, including data extraction and summarizing results. As an additional extension to the literature review, online interviews with corresponding authors will be conducted to gather further information.
Collapse
Affiliation(s)
- Susan Humphrey-Murto
- Department of Medicine, University of Ottawa, Ottawa, ON, Canada
- Department of Innovation in Medical Education, University of Ottawa, Ottawa, ON, Canada
| | - Seung Ho Lee
- Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Michael Gottlieb
- Department of Emergency Medicine, Rush University Medical Center, Chicago, IL, United States of America
| | - Tanya Horsley
- Royal College of Physicians and Surgeons of Canada, Research, Ottawa, ON, Canada
- School of Epidemiology and Public Health, University of Ottawa, Ottawa, ON, Canada
| | - Bev Shea
- School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Karine Fournier
- Health Sciences Library, University of Ottawa, Ottawa, ON, Canada
| | - Christopher Tran
- Department of Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Teresa Chan
- Department of Medicine, Division of Emergency Medicine; Division of Education & Innovation; Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
| | - Timothy J. Wood
- Department of Innovation in Medical Education, University of Ottawa, Ottawa, ON, Canada
| | - Olle ten Cate
- Utrecht Center for Research and Development of Health Professions Education, Division of Education, University Medical Center Utrecht, Utrecht, Utrecht, The Netherlands
| |
Collapse
|
9
|
Archibald D, Grant R, Tuot DS, Liddy C, Sewell JL, Price DW, Grad R, Shipman SA, Campbell C, Guglani S, Wood TJ, Keely E. Development of eConsult reflective learning tools for healthcare providers: a pragmatic mixed methods approach. BMC Prim Care 2023; 24:15. [PMID: 36647016 PMCID: PMC9841624 DOI: 10.1186/s12875-022-01948-9] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/24/2022] [Accepted: 12/14/2022] [Indexed: 01/18/2023]
Abstract
BACKGROUND Electronic consultation (eConsult) programs are crucial components of modern healthcare that facilitate communication between primary care providers (PCPs) and specialists. eConsults between PCPs and specialists. They also provide a unique opportunity to use real-world patient scenarios for reflective learning as part of professional development. However, tools that guide and document learning from eConsults are limited. The purpose of this study was to develop and pilot two eConsult reflective learning tools (RLTs), one for PCPs and one for specialists, for those participating in eConsults. METHODS We performed a four-phase pragmatic mixed methods study recruiting PCPs and specialists from two public health systems located in two countries: eConsult BASE in Canada and San Francisco Health Network eConsult in the United States. In phase 1, subject matter experts developed preliminary RLTs for PCPs and specialists. During phase 2, a Delphi survey among 20 PCPs and 16 specialists led to consensus on items for each RLT. In phase 3, we conducted cognitive interviews with three PCPs and five specialists as they applied the RLTs on previously completed consults. In phase 4, we piloted the RLTs with eConsult users. RESULTS The RLTs were perceived to elicit critical reflection among participants regarding their knowledge and practice habits and could be used for quality improvement and continuing professional development. CONCLUSION PCPs and specialists alike perceived that eConsult systems provided opportunities for self-directed learning wherein they were motivated to investigate topics further through the course of eConsult exchanges. We recommend the RLTs be subject to further evaluation through implementation studies at other sites.
Collapse
Affiliation(s)
- Douglas Archibald
- grid.28046.380000 0001 2182 2255Department of Family Medicine, C.T. Lamont Primary Health Care Research Centre, University of Ottawa, Ottawa, Canada ,grid.418792.10000 0000 9064 3333Bruyère Research Institute, Ottawa, Canada ,grid.28046.380000 0001 2182 2255Faculty of Education, University of Ottawa, Ottawa, Canada
| | - Rachel Grant
- grid.28046.380000 0001 2182 2255Faculty of Education, University of Ottawa, Ottawa, Canada
| | - Delphine S. Tuot
- grid.266102.10000 0001 2297 6811Department of Medicine, Division of Nephrology, Zuckerberg San Francisco General Hospital, University of California San Francisco, San Francisco, CA USA
| | - Clare Liddy
- grid.28046.380000 0001 2182 2255Department of Family Medicine, C.T. Lamont Primary Health Care Research Centre, University of Ottawa, Ottawa, Canada ,grid.418792.10000 0000 9064 3333Bruyère Research Institute, Ottawa, Canada ,grid.412687.e0000 0000 9606 5108Ontario eConsult Centre of Excellence, The Ottawa Hospital, Ottawa, Canada
| | - Justin L. Sewell
- grid.266102.10000 0001 2297 6811Department of Medicine, Division of Gastroenterology, Zuckerberg San Francisco General Hospital, University of California San Francisco, San Francisco, CA USA
| | - David W. Price
- grid.430503.10000 0001 0703 675XDepartment of Family Medicine, University of Colorado Anschutz School of Medicine, Aurora, CO USA ,The American Board of Family Medicine, Lexington, KY USA
| | - Roland Grad
- grid.14709.3b0000 0004 1936 8649Department of Family Medicine, McGill University, Montreal, Canada
| | - Scott A. Shipman
- grid.414000.10000 0000 8652 9597Association of American Medical Colleges, Washington, DC USA ,grid.254748.80000 0004 1936 8876Creighton University, Omaha, NE USA
| | - Craig Campbell
- grid.28046.380000 0001 2182 2255Department of Medicine, University of Ottawa, Ottawa, Canada
| | - Sheena Guglani
- grid.418792.10000 0000 9064 3333Bruyère Research Institute, Ottawa, Canada ,grid.412687.e0000 0000 9606 5108Ontario eConsult Centre of Excellence, The Ottawa Hospital, Ottawa, Canada
| | - Timothy J. Wood
- grid.28046.380000 0001 2182 2255Department of Innovation in Medical Education, University of Ottawa, Ottawa, Canada
| | - Erin Keely
- grid.412687.e0000 0000 9606 5108Ontario eConsult Centre of Excellence, The Ottawa Hospital, Ottawa, Canada ,grid.28046.380000 0001 2182 2255Department of Medicine, University of Ottawa, Ottawa, Canada
| |
Collapse
|
10
|
Alibhai KM, Fowler A, Gawad N, Wood TJ, Raîche I. Assessment of laparoscopic skills: comparing the reliability of global rating and entrustability tools. Can Med Educ J 2022; 13:36-45. [PMID: 36440072 PMCID: PMC9684047 DOI: 10.36834/cmej.72369] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
BACKGROUND Competence by design (CBD) residency programs increasingly depend on tools that provide reliable assessments, require minimal rater training, and measure progression through the CBD milestones. To assess intraoperative skills, global rating scales and entrustability ratings are commonly used but may require extensive training. The Competency Continuum (CC) is a CBD framework that may be used as an assessment tool to assess laparoscopic skills. The study aimed to compare the CC to two other assessment tools: the Global Operative Assessment of Laparoscopic Skills (GOALS) and the Zwisch scale. METHODS Four expert surgeons rated thirty laparoscopic cholecystectomy videos. Two raters used the GOALS scale while the remaining two raters used both the Zwisch scale and CC. Each rater received scale-specific training. Descriptive statistics, inter-rater reliabilities (IRR), and Pearson's correlations were calculated for each scale. RESULTS Significant positive correlations between GOALS and Zwisch (r = 0.75, p < 0.001), CC and GOALS (r = 0.79, p < 0.001), and CC and Zwisch (r = 0.90, p < 0.001) were found. The CC had an inter-rater reliability of 0.74 whereas the GOALS and Zwisch scales had inter-rater reliabilities of 0.44 and 0.43, respectively. Compared to GOALS and Zwisch scales, the CC had the highest inter-rater reliability and required minimal rater training to achieve reliable scores. CONCLUSION The CC may be a reliable tool to assess intraoperative laparoscopic skills and provide trainees with formative feedback relevant to the CBD milestones. Further research should collect further validity evidence for the use of the CC as an independent assessment tool.
Collapse
Affiliation(s)
- Kameela Miriam Alibhai
- Division of General Surgery, Department of Surgery, Faculty of Medicine, University of Ottawa, Ontario, Canada
| | - Amanda Fowler
- Division of General Surgery, Department of Surgery, Faculty of Medicine, Memorial University, Newfoundland and Labrador, Canada
| | - Nada Gawad
- Division of General Surgery, Department of Surgery, Faculty of Medicine, University of Ottawa, Ontario, Canada
- Department of Innovation in Medical Education (DIME), University of Ottawa, Ontario, Canada
| | - Timothy J Wood
- Department of Innovation in Medical Education (DIME), University of Ottawa, Ontario, Canada
| | - Isabelle Raîche
- Division of General Surgery, Department of Surgery, Faculty of Medicine, University of Ottawa, Ontario, Canada
- Department of Innovation in Medical Education (DIME), University of Ottawa, Ontario, Canada
| |
Collapse
|
11
|
DeBiasio C, Su C, Campbell C, Wood TJ. A virtual rural medicine self-learning module for preclerkship students. Can Med Educ J 2022; 13:104-105. [PMID: 36310906 PMCID: PMC9588178 DOI: 10.36834/cmej.73580] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
Implication Statement All Ontario medical schools have a mandatory preclerkship rural placement. Despite these mandatory placements, there is a lack of Canadian rural medicine pre-departure training for students. We describe a virtual self-learning module (SLM) aimed to enhance medical students' abilities to provide compassionate care during their rural medicine placements. This SLM improves students' knowledge about rural health issues while also encouraging exploration of rural career options. Better preparation for a rural medicine placement may lead to an improved experience and eventually an increase in the number of students pursuing rural medicine careers.
Collapse
Affiliation(s)
| | - Charles Su
- Faculty of Medicine, University of Ottawa, Ontario, Canada
| | - Craig Campbell
- Faculty of Medicine, University of Ottawa, Ontario, Canada
| | - Timothy J Wood
- Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa
| |
Collapse
|
12
|
Landreville JM, Wood TJ, Frank JR, Cheung WJ. Does direct observation influence the quality of workplace-based assessment documentation? AEM Educ Train 2022; 6:e10781. [PMID: 35903424 PMCID: PMC9305723 DOI: 10.1002/aet2.10781] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/14/2022] [Revised: 06/07/2022] [Accepted: 06/08/2022] [Indexed: 05/30/2023]
Abstract
BACKGROUND A key component of competency-based medical education (CBME) is direct observation of trainees. Direct observation has been emphasized as integral to workplace-based assessment (WBA) yet previously identified challenges may limit its successful implementation. Given these challenges, it is imperative to fully understand the value of direct observation within a CBME program of assessment. Specifically, it is not known whether the quality of WBA documentation is influenced by observation type (direct or indirect). METHODS The objective of this study was to determine the influence of observation type (direct or indirect) on quality of entrustable professional activity (EPA) assessment documentation within a CBME program. EPA assessments were scored by four raters using the Quality of Assessment for Learning (QuAL) instrument, a previously published three-item quantitative measure of the quality of written comments associated with a single clinical performance score. An analysis of variance was performed to compare mean QuAL scores among the direct and indirect observation groups. The reliability of the QuAL instrument for EPA assessments was calculated using a generalizability analysis. RESULTS A total of 244 EPA assessments (122 direct observation, 122 indirect observation) were rated for quality using the QuAL instrument. No difference in mean QuAL score was identified between the direct and indirect observation groups (p = 0.17). The reliability of the QuAL instrument for EPA assessments was 0.84. CONCLUSIONS Observation type (direct or indirect) did not influence the quality of EPA assessment documentation. This finding raises the question of how direct and indirect observation truly differ and the implications for meta-raters such as competence committees responsible for making judgments related to trainee promotion.
Collapse
Affiliation(s)
| | - Timothy J. Wood
- Department of Innovation in Medical EducationUniversity of OttawaOttawaOntarioCanada
| | - Jason R. Frank
- Department of Emergency MedicineUniversity of OttawaOttawaOntarioCanada
| | - Warren J. Cheung
- Department of Emergency MedicineUniversity of OttawaOttawaOntarioCanada
| |
Collapse
|
13
|
Dudek N, Duffy MC, Wood TJ, Gofton W. The Ottawa Resident Observation Form for Nurses (O-RON): Assessment of Resident Performance through the Eyes of the Nurses. J Surg Educ 2021; 78:1666-1675. [PMID: 34092533 DOI: 10.1016/j.jsurg.2021.03.014] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/17/2020] [Revised: 02/06/2021] [Accepted: 03/21/2021] [Indexed: 06/12/2023]
Abstract
OBJECTIVE Most work-place based assessment relies on physician supervisors making observations of residents. Many areas of performance are not directly observed by physicians but rather by other healthcare professionals, most often nurses. Assessment of resident performance by nurses is captured with multi-source feedback tools. However, these tools combine the assessments of nurses with other healthcare professionals and so their perspective can be lost. A novel tool was developed and implemented to assess resident performance on a hospital ward from the perspective of the nurses. DESIGN Through a nominal group technique, nurses identified dimensions of performance that are reflective of high-quality physician performance on a hospital ward. These were included as items in the Ottawa Resident Observation Form for Nurses (O-RON). The O-RON was voluntarily completed during an 11-month period. Validity evidence related to quantitative and qualitative data was collected. SETTING The Orthopedic Surgery Residency Program at the University of Ottawa. PARTICIPANTS 49 nurses on the Orthopedic Surgery wards at The Ottawa Hospital (tertiary care). RESULTS The O-RON has 15 items rated on a 3-point frequency scale, one global judgment yes/no question regarding whether they would want the resident on their team and a space for comments. 1079 O-RONs were completed on 38 residents. There was an association between the response to the global judgment question and the frequency of concerns (p < 0.01). With 8 forms per resident, the reliability of the O-RON was 0.80. Open-ended responses referred to aspects of interpersonal skills, responsiveness, dependability, communication skills, and knowledge. CONCLUSIONS The O-RON demonstrates promise as a work-place based assessment tool to provide residents and training programs with feedback on aspects of their performance on a hospital ward through the eyes of the nurses. It appears to be easy to use, has solid evidence for validity and can provide reliable data with a small number of completed forms.
Collapse
Affiliation(s)
- Nancy Dudek
- Department of Medicine (Division of Physical Medicine & Rehabilitation) and The Ottawa Hospital, University of Ottawa, Ottawa, Ontario, Canada.
| | - Melissa C Duffy
- Department of Educational Studies, University of South Carolina, College of Education, University of South Carolina, Wardlaw College, Columbia, South Carolina
| | - Timothy J Wood
- Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada
| | - Wade Gofton
- Department of Surgery (Division of Orthopedic Surgery) and The Ottawa Hospital, University of Ottawa, Division of Orthopedic Surgery, Ottawa, Ontario, Canada
| |
Collapse
|
14
|
Humphrey-Murto S, Shaw T, Touchie C, Pugh D, Cowley L, Wood TJ. Are raters influenced by prior information about a learner? A review of assimilation and contrast effects in assessment. Adv Health Sci Educ Theory Pract 2021; 26:1133-1156. [PMID: 33566199 DOI: 10.1007/s10459-021-10032-3] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/17/2020] [Accepted: 01/25/2021] [Indexed: 06/12/2023]
Abstract
Understanding which factors can impact rater judgments in assessments is important to ensure quality ratings. One such factor is whether prior performance information (PPI) about learners influences subsequent decision making. The information can be acquired directly, when the rater sees the same learner, or different learners over multiple performances, or indirectly, when the rater is provided with external information about the same learner prior to rating a performance (i.e., learner handover). The purpose of this narrative review was to summarize and highlight key concepts from multiple disciplines regarding the influence of PPI on subsequent ratings, discuss implications for assessment and provide a common conceptualization to inform research. Key findings include (a) assimilation (rater judgments are biased towards the PPI) occurs with indirect PPI and contrast (rater judgments are biased away from the PPI) with direct PPI; (b) negative PPI appears to have a greater effect than positive PPI; (c) when viewing multiple performances, context effects of indirect PPI appear to diminish over time; and (d) context effects may occur with any level of target performance. Furthermore, some raters are not susceptible to context effects, but it is unclear what factors are predictive. Rater expertise and training do not consistently reduce effects. Making raters more accountable, providing specific standards and reducing rater cognitive load may reduce context effects. Theoretical explanations for these findings will be discussed.
Collapse
Affiliation(s)
- Susan Humphrey-Murto
- Department of Medicine, Faculty of Medicine, The Ottawa Hospital-Riverside Campus, University of Ottawa, 1967 Riverside Drive, Box 67, Ottawa, ON, Canada.
- Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada.
| | - Tammy Shaw
- Department of Medicine, Faculty of Medicine, The Ottawa Hospital-General Campus, Ottawa, ON, Canada
| | - Claire Touchie
- Department of Medicine, Faculty of Medicine, The Ottawa Hospital-Riverside Campus, University of Ottawa, 1967 Riverside Drive, Box 67, Ottawa, ON, Canada
- Medical Council of Canada, Ottawa, ON, Canada
| | - Debra Pugh
- Department of Medicine, Faculty of Medicine, The Ottawa Hospital-Riverside Campus, University of Ottawa, 1967 Riverside Drive, Box 67, Ottawa, ON, Canada
- Medical Council of Canada, Ottawa, ON, Canada
| | - Lindsay Cowley
- Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Timothy J Wood
- Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada
| |
Collapse
|
15
|
Humphrey-Murto S, Walker K, Aggarwal S, Dhillon NPK, Rauscher S, Wood TJ. The impact of local health professions education grants: is it worth the investment? Can Med Educ J 2021; 12:44-53. [PMID: 34249190 PMCID: PMC8263034 DOI: 10.36834/cmej.71357] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
BACKGROUND Local grants programs are important since funding for medical education research is limited. Understanding which factors predict successful outcomes is highly relevant to administrators. The purpose of this project was to identify factors that contribute to the publication of local medical education grants in a Canadian context. METHODS Surveys were distributed to previous Department of Innovation in Medical Education (DIME) and Department of Medicine (DOM) grant recipients (n = 115) to gather information pertaining to PI demographics and research outcomes. A backward logistic regression was used to determine the effects several variables on publication success. RESULTS The overall publication rate was 64/115 (56%). Due to missing data, 91 grants were included in the logistic regression. Variables associated with a higher rate of publication; cross departmental compared to single department OR = 2.82 (p = 0.04), being presented OR = 3.30 (p = 0.01), and multiple grant acquisition OR = 3.85 (p = 0.005). CONCLUSION Although preliminary, our data suggest that increasing research publications from local grants may be facilitated by pooling funds across departments, making research presentations mandatory, and allowing successful researchers to re-apply.
Collapse
Affiliation(s)
| | - Kyle Walker
- Department Medicine, University of Ottawa, Ontario, Canada
| | | | | | - Scott Rauscher
- Department of Innovation in Medical Education Research Support Unit, University of Ottawa, Ontario, Canada
| | - Timothy J Wood
- Department of Innovation in Medical Education (DIME), University of Ottawa, Ontario, Canada
| |
Collapse
|
16
|
Gomes MM, Driman D, Park YS, Wood TJ, Yudkowsky R, Dudek NL. Teaching and assessing intra-operative consultations in competency-based medical education: development of a workplace-based assessment instrument. Virchows Arch 2021; 479:803-813. [PMID: 33966099 PMCID: PMC8516791 DOI: 10.1007/s00428-021-03113-6] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/12/2021] [Revised: 04/22/2021] [Accepted: 04/27/2021] [Indexed: 02/02/2023]
Abstract
Competency-based medical education (CBME) is being implemented worldwide. In CMBE, residency training is designed around competencies required for unsupervised practice and use entrustable professional activities (EPAs) as workplace “units of assessment”. Well-designed workplace-based assessment (WBA) tools are required to document competence of trainees in authentic clinical environments. In this study, we developed a WBA instrument to assess residents’ performance of intra-operative pathology consultations and conducted a validity investigation. The entrustment-aligned pathology assessment instrument for intra-operative consultations (EPA-IC) was developed through a national iterative consultation and used clinical supervisors to assess residents’ performance at an anatomical pathology program. Psychometric analyses and focus groups were conducted to explore the sources of evidence using modern validity theory: content, response process, internal structure, relations to other variables, and consequences of assessment. The content was considered appropriate, the assessment was feasible and acceptable by residents and supervisors, and it had a positive educational impact by improving performance of intra-operative consultations and feedback to learners. The results had low reliability, which seemed to be related to assessment biases, and supervisors were reluctant to fully entrust trainees due to cultural issues. With CBME implementation, new workplace-based assessment tools are needed in pathology. In this study, we showcased the development of the first instrument for assessing resident’s performance of a prototypical entrustable professional activity in pathology using modern education principles and validity theory.
Collapse
Affiliation(s)
- Marcio M Gomes
- Department of Pathology and Laboratory Medicine, University of Ottawa, Ottawa, Canada.
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada.
- The Ottawa Hospital, Ottawa, Canada.
| | - David Driman
- Department of Pathology and Laboratory Medicine, Western University, London, Canada
| | - Yoon Soo Park
- Department of Medical Education, University of Illinois At Chicago, Chicago, IL, USA
| | - Timothy J Wood
- Department of Innovation in Medical Education, University of Ottawa, Ottawa, Canada
| | - Rachel Yudkowsky
- Department of Medical Education, University of Illinois At Chicago, Chicago, IL, USA
| | - Nancy L Dudek
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- The Ottawa Hospital, Ottawa, Canada
- Department of Medicine, University of Ottawa, Ottawa, Canada
| |
Collapse
|
17
|
Gawad N, Wood TJ, Malvea A, Cowley L, Raiche I. The Impact of Surgeon Experience on Script Concordance Test Scoring. J Surg Res 2021; 265:265-271. [PMID: 33964636 DOI: 10.1016/j.jss.2021.03.057] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2020] [Revised: 03/24/2021] [Accepted: 03/29/2021] [Indexed: 10/21/2022]
Abstract
OBJECTIVE The Script Concordance Test (SCT) is a test of clinical decision-making that relies on an expert panel to create its scoring key. Existing literature demonstrates the value of specialty-specific experts, but the effect of experience among the expert panel is unknown. The purpose of this study was to explore the role of surgeon experience in SCT scoring. DESIGN An SCT was administered to 29 general surgery residents and 14 staff surgeons. Staff surgeons were stratified as either junior or senior experts based on years since completing residency training (<15 versus >25 years). The SCT was scored using the full expert panel, the senior panel, the junior panel, and a subgroup junior panel in practice <5 years. A one-way ANOVA was used to compare the scores of first (R1) and fifth (R5) year residents using each scoring scheme. Cognitive interviews were analyzed for differences between junior and senior expert panelist responses. RESULTS There was no statistically significant difference between the mean score of six R1s and five R5s using the full expert panel (R1 69.08 versus R5 67.06, F1,9 = 0.10, P = 0.76), the junior panel (R1 66.73 versus R5 62.50, F1,9 = 0.35, P = 0.57), or the subgroup panel in practice <5 years (R1 61.07 versus R5 58.79, F1,9 = 0.18, P = 0.75). However, the average score of R1s was significantly lower than R5s when using the senior faculty panel (R1 52.04 versus R5 63.26, F1,9 = 26.90, P = 0.001). Cognitive interview data suggests that some responses of junior experts demonstrate less confidence than those of senior experts. CONCLUSIONS SCT scores are significantly affected by the responses of the expert panel. Expert differences between first and fifth year residents were only demonstrated when using an expert panel consisting of senior faculty members. Confidence may play a role in the response selections of junior experts. When constructing an SCT expert panel, consideration must be given to the experience of panel members.
Collapse
Affiliation(s)
- Nada Gawad
- Division of General Surgery, Department of Surgery, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada; Department of Innovation in Medical Education (DIME), University of Ottawa, Ottawa, Ontario, Canada.
| | - Timothy J Wood
- Department of Innovation in Medical Education (DIME), University of Ottawa, Ottawa, Ontario, Canada
| | - Anahita Malvea
- Division of General Surgery, Department of Surgery, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - Lindsay Cowley
- Department of Innovation in Medical Education (DIME), University of Ottawa, Ottawa, Ontario, Canada
| | - Isabelle Raiche
- Division of General Surgery, Department of Surgery, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada; Department of Innovation in Medical Education (DIME), University of Ottawa, Ottawa, Ontario, Canada
| |
Collapse
|
18
|
Tran C, Archibald D, Humphrey-Murto S, Wood TJ, Dudek N, Liddy C, Keely E. eConsult Specialist Quality of Response (eSQUARE): A novel tool to measure specialist correspondence via electronic consultation. J Telemed Telecare 2021; 28:280-290. [PMID: 33657913 PMCID: PMC9066665 DOI: 10.1177/1357633x21998216] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
High-quality correspondence between healthcare providers is critical for effective patient care. We developed an assessment tool to measure the quality of specialist correspondence to primary care providers (PCPs) via electronic consultation (eConsult), where specialists provide advice without specialist-patient interactions. We incorporated fourteen previously described features of high-quality eConsult correspondence into an assessment tool named the eConsult Specialist Quality of Response (eSQUARE). Six PCPs and two specialists applied the 10-item eSQUARE tool to 30 eConsults of varying quality as informed by PCP survey data. Content, response process, and internal structure validity evidence was gathered. Psychometric properties were calculated using descriptive statistics and generalizability analyses. Mean total score for low-quality eConsults (M = 24 ± 5.6) was significantly lower than moderate-quality eConsults (M = 38 ± 4.7; p<0.001) which was significantly lower than high-quality eConsults (M = 46 ± 3.0; p = 0.002). Reliability measures were high, including generalizability coefficient (0.96), inter-item (≥0.55) and item-total correlations (≥0.68). A decision study demonstrated that a single rater was adequate to achieve a reliability measure of ≥0.70. This study demonstrates initial validity evidence including multiple reliability measures for the eSQUARE. A single rater is adequate to achieve reliability measures for formative feedback. Future studies can apply the eSQUARE when planning educational initiatives aiming to improve specialist-to-PCP correspondence via eConsult.
Collapse
Affiliation(s)
- Christopher Tran
- Department of Medicine, University of Ottawa, Canada.,Ottawa Hospital Research Institute, The Ottawa Hospital, Canada
| | - Douglas Archibald
- Department of Family Medicine, University of Ottawa, Canada.,Bruyère Research Institute, Canada
| | - Susan Humphrey-Murto
- Department of Medicine, University of Ottawa, Canada.,Department of Innovation in Medical Education, University of Ottawa, Canada
| | - Timothy J Wood
- Department of Innovation in Medical Education, University of Ottawa, Canada
| | - Nancy Dudek
- Department of Medicine, University of Ottawa, Canada
| | - Clare Liddy
- Department of Family Medicine, University of Ottawa, Canada.,Bruyère Research Institute, Canada
| | - Erin Keely
- Department of Medicine, University of Ottawa, Canada.,Ottawa Hospital Research Institute, The Ottawa Hospital, Canada
| |
Collapse
|
19
|
Shaw T, Wood TJ, Touchie C, Pugh D, Humphrey-Murto SM. How biased are you? The effect of prior performance information on attending physician ratings and implications for learner handover. Adv Health Sci Educ Theory Pract 2021; 26:199-214. [PMID: 32577927 DOI: 10.1007/s10459-020-09979-6] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/14/2020] [Accepted: 06/15/2020] [Indexed: 06/11/2023]
Abstract
Learner handover (LH), the process of sharing of information about learners between faculty supervisors, allows for longitudinal assessment fundamental in the competency-based education model. However, the potential to bias future assessments has been raised as a concern. The purpose of this study is to determine whether prior performance information such as LH influences the assessment of learners in the clinical context. Between December 2017 and June 2018, forty-two faculty members and final-year residents from the Department of Medicine at the University of Ottawa were assigned to one of three study groups through quasi-randomisation, taking into account gender, speciality and rater experience. In a counter-balanced design, each group received either positive, negative or no LH prior to watching six simulated learner-patient encounter videos. Participants rated each video using the mini-CEX and completed a questionnaire on the raters' general impressions of LH. A significant difference in the mean mini-CEX competency scale scores between the negative (M = 5.29) and positive (M = 5.97) LH groups (P < .001, d = 0.81) was noted. Similar findings were found for the single overall clinical competence ratings. In the post-study questionnaire, 22/28 (78%) of participants had correctly deduced the purpose of the study and 14/28 (50%) felt LH did not influence their assessment. LH influenced mini-CEX scores despite raters' awareness of the potential for bias. These results suggest that LH could influence a rater's performance assessment and careful consideration of the potential implications of LH is required.
Collapse
Affiliation(s)
- Tammy Shaw
- Department of Medicine, University of Ottawa, Ottawa, ON, Canada.
- The Ottawa Hospital - General Campus, 501 Smyth Road Box 209, Ottawa, ON, K1H 8L6, Canada.
| | - Timothy J Wood
- Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Claire Touchie
- Department of Medicine, University of Ottawa, Ottawa, ON, Canada
- Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada
- Medical Council of Canada, Ottawa, Canada
| | - Debra Pugh
- Department of Medicine, University of Ottawa, Ottawa, ON, Canada
- Medical Council of Canada, Ottawa, Canada
| | - Susan M Humphrey-Murto
- Department of Medicine, University of Ottawa, Ottawa, ON, Canada
- Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada
| |
Collapse
|
20
|
Gawad N, Wood TJ, Cowley L, Raiche I. How do cognitive processes influence script concordance test responses? Med Educ 2021; 55:354-364. [PMID: 33185303 DOI: 10.1111/medu.14416] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/13/2020] [Revised: 10/15/2020] [Accepted: 11/09/2020] [Indexed: 06/11/2023]
Abstract
INTRODUCTION The script concordance test (SCT) is a test of clinical decision-making (CDM) that compares the thought process of learners to that of experts to determine to what extent their cognitive 'scripts' align. Without understanding test-takers' cognitive process, however, it is unclear what influences their responses. The objective of this study was to gather response process validity evidence by studying the cognitive process of test-takers to determine whether the SCT tests CDM and what cognitive processes may influence SCT responses. METHODS Cases from an SCT used in a national validation study were administered and semi-structured cognitive interviews were conducted with ten residents and five staff surgeons. A retrospective verbal probing technique was used. Data was independently analysed and coded by two analysts. Themes were identified as factors that influence SCT responses during the cognitive interview. RESULTS Cognitive interviews demonstrated variability in CDM among test-takers. Consistent with dual process theory, test-takers relied on scripts formed through past experiences, when available, to make decisions and used conscious deliberation in the absence of experience. However, test-takers' response process was also influenced by their comprehension of specific terms, desire for additional information, disagreement with the planned management, underlying knowledge gaps and desire to demonstrate confidence or humility. CONCLUSION The rationale behind SCT answers may be influenced by comprehension, underlying knowledge and social desirability in addition to formed scripts and/or conscious deliberation. Having test-takers verbalise their rationale for responses provides a depth of assessment that is otherwise lost in the SCT's current format. With the improved ability to standardise CDM assessment using the SCT, consideration of test-makers improving the SCT construction process and combining the SCT question format with verbal responses may improve the use of the SCT for CDM assessment.
Collapse
Affiliation(s)
- Nada Gawad
- Division of General Surgery, Department of Surgery, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada
- Department of Innovation in Medical Education (DIME), University of Ottawa, Ottawa, ON, Canada
| | - Timothy J Wood
- Department of Innovation in Medical Education (DIME), University of Ottawa, Ottawa, ON, Canada
| | - Lindsay Cowley
- Department of Innovation in Medical Education (DIME), University of Ottawa, Ottawa, ON, Canada
| | - Isabelle Raiche
- Division of General Surgery, Department of Surgery, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada
- Department of Innovation in Medical Education (DIME), University of Ottawa, Ottawa, ON, Canada
| |
Collapse
|
21
|
Guay JM, Wood TJ, Touchie C, Ta CA, Halman S. Will I publish this abstract? Determining the characteristics of medical education oral abstracts linked to publication. Can Med Educ J 2020; 11:e46-e53. [PMID: 33349753 PMCID: PMC7749698 DOI: 10.36834/cmej.69558] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
BACKGROUND Prior studies have shown that most conference submissions fail to be published. Understanding factors that facilitate publication may be of benefit to authors. Using data from the Canadian Conference on Medical Education (CCME), our goal was to identify characteristics of conference submissions that predict the likelihood of publication with a specific focus on the utility of peer-review ratings. METHODS Study characteristics (scholarship type, methodology, population, sites, institutions) from all oral abstracts from 2011-2015 and peer-review ratings for 2014-2015 were extracted by two raters. Publication data was obtained using online database searches. The impact of variables on publication success was analyzed using logistic regressions. RESULTS In total, 953 oral abstracts were reviewed from 2011 to 2015. Overall, the publication rate was 30.5% (291/953). Of 531 abstracts with peer-review ratings, between 2014 and 2015, 162 (31%) were published. Of the nine analyzed variables, those associated with a greater odds of publication were: multiple vs. single institutions (odds ratio (OR) = 1.72), post-graduate research vs. others (OR=1.81) and peer-review ratings (OR=1.60). Factors with decreased odds of publication were curriculum development (OR=0.17) and innovation vs. others (OR=0.22). CONCLUSION Similar to other studies, the publication rate of CCME presentations is low. However, peer ratings were predictive of publication success suggesting that ratings could be a useful form of feedback to authors.
Collapse
Affiliation(s)
| | - Timothy J. Wood
- Department of Innovation in Medical Education, University of Ottawa, Ontario, Canada
| | - Claire Touchie
- Department of Medicine, University of Ottawa, Ontario, Canada
- Department of Innovation in Medical Education, University of Ottawa, Ontario, Canada
- Medical Council of Canada, Ontario, Canada
| | - Chi Anh Ta
- Department of Innovation in Medical Education, University of Ottawa, Ontario, Canada
| | - Samantha Halman
- Department of Medicine, University of Ottawa, Ontario, Canada
| |
Collapse
|
22
|
Abstract
PURPOSE Accreditation aims to ensure all training programs meet agreed-upon standards of quality. The process is complex, resource intensive, and costly. Its benefits are difficult to assess because contextual confounds obscure comparisons between systems that do and do not include accreditation. This study explores accreditation's influence "within system" by investigating the relationship between accreditation cycle and performance on a national licensing examination. METHOD Scores on the computer-based portion of the Medical Council of Canada Qualifying Examination Part I, from 1993 to 2017, were examined for all 17 Canadian medical schools. Typically completed upon graduation from medical school, results within each year were transformed for comparability across administrations and linked to timing within each school's accreditation cycle. ANOVAs were used to assess the relationship between accreditation timing and examination scores. Secondary analyses isolated 4-year from 3-year training programs and separated data generated before versus after implementation of a national midcycle informal review program. RESULTS Performance on the licensing exam was highest during and shortly after an accreditation site visit, falling significantly until the midpoint in the accreditation cycle (d = 0.47) before rising again. This pattern disappeared after introduction of informal interim review, but too little data have accumulated post implementation to determine if interim review is sufficient to break the influence of accreditation cycle. CONCLUSIONS Formal, externally driven, accreditation cycles appear associated with educational processes in ways that translated into student outcomes on a national licensing examination. Whether informal, internal, interim reviews can mediate this effect remains to be seen.
Collapse
Affiliation(s)
- Marguerite Roy
- M. Roy is medical education researcher, Medical Council of Canada, Ottawa, Ontario, Canada; ORCID: http://orcid.org/0000-0001-8462-7721
| | - Timothy J Wood
- T.J. Wood is professor, Department of Innovation in Medical Education, and director of evaluation-undergraduate medical education, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada; ORCID: http://orcid.org/0000-0001-9177-704X
| | - Danielle Blouin
- D. Blouin is professor, Faculty of Health Sciences, Department of Emergency Medicine, and Faculty of Education, Queens University, Kingston, Ontario, Canada; ORCID: http://orcid.org/0000-0002-5448-8326
| | - Kevin W Eva
- K.W. Eva is professor and director of education research and scholarship, Department of Medicine, and associate director and senior scientist, Centre for Health Education Scholarship, University of British Columbia, Vancouver, British Columbia, Canada; ORCID: http://orcid.org/0000-0002-8672-2500
| |
Collapse
|
23
|
Cheung WJ, Wood TJ, Gofton W, Dewhirst S, Dudek N. The Ottawa Emergency Department Shift Observation Tool (O-EDShOT): A New Tool for Assessing Resident Competence in the Emergency Department. AEM Educ Train 2020; 4:359-368. [PMID: 33150278 PMCID: PMC7592826 DOI: 10.1002/aet2.10419] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/13/2019] [Revised: 11/01/2019] [Accepted: 11/13/2019] [Indexed: 05/23/2023]
Abstract
OBJECTIVES The outcome of emergency medicine (EM) training is to produce physicians who can competently run an emergency department (ED) shift. However, there are few tools with supporting validity evidence specifically designed to assess multiple key competencies across an entire shift. The investigators developed and gathered validity evidence for a novel entrustment-based tool to assess a resident's ability to safely run an ED shift. METHODS Through a nominal group technique, local and national stakeholders identified dimensions of performance that are reflective of a competent ED physician and are required to safely manage an ED shift. These were included as items in the Ottawa Emergency Department Shift Observation Tool (O-EDShOT), and each item was scored using an entrustment-based rating scale. The tool was implemented in 2018 at the University of Ottawa Department of Emergency Medicine, and quantitative data and qualitative feedback were collected over 6 months. RESULTS A total of 1,141 forms were completed by 78 physicians for 45 residents. An analysis of variance demonstrated an effect of training level with statistically significant increases in mean O-EDShOT scores with each subsequent postgraduate year (p < 0.001). Scores did not vary by ED treatment area. Residents rated as able to safely run the shift had significantly higher mean ± SD scores (4.8 ± 0.3) than those rated as not able (3.8 ± 0.6; p < 0.001). Faculty and residents reported that the tool was feasible to use and facilitated actionable feedback aimed at progression toward independent practice. CONCLUSIONS The O-EDShOT successfully discriminated between trainees of different levels regardless of ED treatment area. Multiple sources of validity evidence support the O-EDShOT as a tool to assess a resident's ability to safely run an ED shift. It can serve as a stimulus for daily observation and feedback making it practical to use within an EM residency program.
Collapse
Affiliation(s)
- Warren J. Cheung
- Department of Emergency MedicineUniversity of OttawaOttawaOntarioCanada
| | - Timothy J. Wood
- Department of Innovation in Medical EducationUniversity of OttawaOttawaOntarioCanada
| | - Wade Gofton
- Department of SurgeryDivision of Orthopaedic SurgeryUniversity of OttawaOttawaOntarioCanada
| | | | - Nancy Dudek
- Department of MedicineDivision of Physical Medicine and RehabilitationUniversity of OttawaOttawaOntarioCanada
| |
Collapse
|
24
|
Gawad N, Wood TJ, Cowley L, Raiche I. The cognitive process of test takers when using the script concordance test rating scale. Med Educ 2020; 54:337-347. [PMID: 31912562 DOI: 10.1111/medu.14056] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/13/2019] [Revised: 12/24/2019] [Accepted: 01/02/2020] [Indexed: 06/10/2023]
Abstract
CONTEXT Clinical decision making (CDM) skills are important to learn and assess in order to establish competence in trainees. A common tool for assessing CDM is the script concordance test (SCT), which asks test takers to indicate how a new clinical finding influences a proposed plan using a Likert-type scale. Most criticisms of the SCT relate to its rating scale but are largely theoretical. The cognitive process of test takers when selecting their responses using the SCT rating scale remains understudied, but is essential to gathering validity evidence for use of the SCT in CDM assessment. METHODS Cases from an SCT used in a national validation study were administered to 29 residents and 14 staff surgeons. Semi-structured cognitive interviews were then conducted with 10 residents and five staff surgeons based on the SCT results. Cognitive interview data were independently coded by two data analysts, who specifically sought to elucidate how participants mapped their internally generated responses to any of the rating scale options. RESULTS Five major issues were identified with the response matching cognitive process: (a) the meaning of the '0' response option; (b) which response corresponds to agreement with the planned management; (c) the rationale for picking '±1' versus '±2'; (d) which response indicates the desire to undertake the planned management plus an additional procedure, and (e) the influence of time on response selection. CONCLUSIONS Studying how test takers (experts and trainees) interpret the SCT rating scale has revealed several issues related to inconsistent and unintended use. Revising the scale to address the variety of interpretations could help to improve the response process validity of the SCT and therefore improve the SCT's ability to be used in CDM skills assessments.
Collapse
Affiliation(s)
- Nada Gawad
- Division of General Surgery, Department of Surgery, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
- Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada
| | - Timothy J Wood
- Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada
| | - Lindsay Cowley
- Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada
| | - Isabelle Raiche
- Division of General Surgery, Department of Surgery, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
- Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada
| |
Collapse
|
25
|
Wood TJ, Humphrey-Murto S, Moineau G, Forgie M, Puddester D, Leddy JJ. Does Emotional Intelligence at medical school admission predict future licensing examination performance? Can Med Educ J 2020; 11:e35-e45. [PMID: 32215141 PMCID: PMC7082480 DOI: 10.36834/cmej.67884] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
BACKGROUND Medical school admissions committees are seeking alternatives to traditional academic measures when selecting students; one potential measure being emotional intelligence (EI). If EI is to be used as an admissions criterion, it should predict future performance. The purpose of this study is to determine if EI scores at admissions predicts performance on a medical licensure examination. METHODS All medical school applicants to the University of Ottawa in 2006 and 2007 were invited to complete the Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT v2.0) after their interview. Students were tracked through medical school into licensure and EI scores were correlated to their scores on the Medical Council of Canada Qualifying Examination (MCCQE) attempted between 2010 and 2014. RESULTS The correlation between the MSCEIT and the MCCQE Part I was r (200) = .01 p =. 90 The covariates of age and gender accounted for a significant amount of variance in MCCQE Part I scores (R 2 = .10, p<.001, n=202) but the addition of the MSCEIT scores was not statistically significant (R 2 change = .002, p=.56). The correlation between the MSCEIT and the MCCQE Part II was r(197) = .06, p = .41. The covariates of age and gender accounted for some variance in MCCQE Part II scores (R 2 = .05, p = .007, n=199) but the addition of the MSCEIT did not (R 2 change = .002 p =.55). CONCLUSION The low correlations between EI and licensure scores replicates other studies that have found weak correlations between EI scores and tests administered at admissions and during medical school. These results suggest caution if one were to use EI as part of their admissions process.
Collapse
Affiliation(s)
- Timothy J. Wood
- Department of Innovation in Medical Education,University of Ottawa, Ontario, Canada
| | - Susan Humphrey-Murto
- Department of Innovation in Medical Education,University of Ottawa, Ontario, Canada
- Department of Medicine, University of Ottawa, Ontario, Canada
| | | | - Melissa Forgie
- Department of Medicine, University of Ottawa, Ontario, Canada
| | - Derek Puddester
- Department of Psychiatry, University of Ottawa, Ontario, Canada
| | - John J. Leddy
- Department of Cellular and Molecular Medicine, University of Ottawa, Ontario, Canada
| |
Collapse
|
26
|
Abstract
Background: It is a doctrine that OSCE checklists are not sensitive to increasing levels of expertise whereas rating scales are. This claim is based primarily on a study that used two psychiatry stations and it is not clear to what degree the finding generalizes to other clinical contexts. The purpose of our study was to reexamine the relationship between increasing training and scoring instruments within an OSCE.Approach: A 9-station OSCE progress test was administered to Internal Medicine residents in post-graduate years (PGY) 1-4. Residents were scored using checklists and rating scales. Standard scores from three administrations (27 stations) were analyzed.Findings: Only one station produced a result in which checklist scores did not increase as a function of training level, but the rating scales did. For 13 stations, scores increased as a function of PGY equally for both checklists and rating scales.Conclusion: Checklist scores were as sensitive to the level of training as rating scales for most stations, suggesting that checklists can capture increasing levels of expertise. The choice of which measure is used should be based on the purpose of the examination and not on a belief that one measure can better capture increases in expertise.
Collapse
Affiliation(s)
- Timothy J Wood
- Department of Innovation in Medical Education, University of Ottawa, Ottawa, Canada
| | - Debra Pugh
- Department of Medicine, University of Ottawa, Ottawa, Canada
| |
Collapse
|
27
|
Affiliation(s)
- Susan Humphrey-Murto
- Department of Medicine, Faculty of Medicine, University of Ottawa Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa clinician educator, and clinical hematologist, Department of Medicine, Faculty of Medicine, University of Ottawa Department of Psychiatry, and education fellow, Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa Department of Medicine, Uniformed Services University of the Health Sciences
| | | | | | | | | |
Collapse
|
28
|
Humphrey-Murto S, LeBlanc A, Touchie C, Pugh D, Wood TJ, Cowley L, Shaw T. The Influence of Prior Performance Information on Ratings of Current Performance and Implications for Learner Handover: A Scoping Review. Acad Med 2019; 94:1050-1057. [PMID: 30946129 DOI: 10.1097/acm.0000000000002731] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/12/2023]
Abstract
PURPOSE Learner handover (LH) is the sharing of information about trainees between faculty supervisors. This scoping review aimed to summarize key concepts across disciplines surrounding the influence of prior performance information (PPI) on current performance ratings and implications for LH in medical education. METHOD The authors used the Arksey and O'Malley framework to systematically select and summarize the literature. Cross-disciplinary searches were conducted in six databases in 2017-2018 for articles published after 1969. To represent PPI relevant to LH in medical education, eligible studies included within-subject indirect PPI for work-type performance and rating of an individual current performance. Quantitative and thematic analyses were conducted. RESULTS Of 24,442 records identified through database searches and 807 through other searches, 23 articles containing 24 studies were included. Twenty-two studies (92%) reported an assimilation effect (current ratings were biased toward the direction of the PPI). Factors modifying the effect of PPI were observed, with larger effects for highly polarized PPI, negative (vs positive) PPI, and early (vs subsequent) performances. Specific standards, rater motivation, and certain rater characteristics mitigated context effects, whereas increased rater processing demands heightened them. Mixed effects were seen with nature of the performance and with rater expertise and training. CONCLUSIONS PPI appears likely to influence ratings of current performance, and an assimilation effect is seen with indirect PPI. Whether these findings generalize to medical education is unknown, but they should be considered by educators wanting to implement LH. Future studies should explore PPI in medical education contexts and real-world settings.
Collapse
Affiliation(s)
- Susan Humphrey-Murto
- S. Humphrey-Murto is associate professor, Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada. A. LeBlanc is a fifth-year respirology resident, Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada. C. Touchie is associate professor, Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada. D. Pugh is associate professor, Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada. T.J. Wood is full professor, Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada. L. Cowley is a research assistant, Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada. T. Shaw is lecturer, Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | | | | | | | | | | | | |
Collapse
|
29
|
Wood TJ, Kaplan I, Zhang Y, Szendrei Z. Honeybee dietary neonicotinoid exposure is associated with pollen collection from agricultural weeds. Proc Biol Sci 2019; 286:20190989. [PMID: 31213190 DOI: 10.1098/rspb.2019.0989] [Citation(s) in RCA: 21] [Impact Index Per Article: 4.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022] Open
Abstract
Neonicotinoid insecticides have been linked to bee declines. However, tracking the primary exposure route for bees in the field has proven to be a major logistical challenge, impeding efforts to restore pollinator health in agricultural landscapes. We quantified neonicotinoid concentrations and botanical species composition in 357 pollen samples collected from 114 commercial honeybee colonies placed along a gradient of agricultural intensity between June and September. Neonicotinoid concentrations increased through the season, peaking at the end of August. As a result, concentrations in pollen were negatively associated with collection from woody and crop plants that flower early-mid season, and positively associated with collection from herbaceous plants that flower mid-late season. Higher clothianidin and thiamethoxam residues were correlated with samples containing a greater proportion of pollen collected from agricultural weeds. The percentage of agricultural land within 1500 m was positively correlated with thiamethoxam concentration; however, this spatial relationship was far weaker than the relationship with the proportion of pollen collected from herbaceous plants. These results indicate that both plant species identity and agricultural dominance are important in determining honeybee neonicotinoid exposure through the pollen diet, but that uncultivated plants associated with agriculture are the source of the greatest acute exposure.
Collapse
Affiliation(s)
- T J Wood
- 1 Department of Entomology, Michigan State University , East Lansing, MI , USA
| | - I Kaplan
- 2 Department of Entomology, Purdue University , West Lafayette, IN , USA
| | - Y Zhang
- 1 Department of Entomology, Michigan State University , East Lansing, MI , USA
| | - Z Szendrei
- 1 Department of Entomology, Michigan State University , East Lansing, MI , USA
| |
Collapse
|
30
|
Saliken D, Dudek N, Wood TJ, MacEwan M, Gofton WT. Comparison of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) to a Single-Item Performance Score. Teach Learn Med 2019; 31:146-153. [PMID: 30514128 DOI: 10.1080/10401334.2018.1503961] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/06/2018] [Revised: 06/22/2018] [Accepted: 07/12/2018] [Indexed: 06/09/2023]
Abstract
UNLABELLED Construct: We compared a single-item performance score with the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) for their ability in assessing surgical competency. BACKGROUND Surgical programs are adopting competency-based frameworks. The adoption of these frameworks for assessment requires tools that produce accurate and valid assessments of knowledge and technical performance. An assessment tool that is quick to complete could improve feasibility, reduce delays, and result in a higher volume of assessments of learners. Previous work demonstrated that the 9-item O-SCORE can produce valid results; the goal of this study was to determine if a single-item performance rating (Is candidate competent to independently complete procedure: yes or no) completed at a separate viewing would correlate to the O-SCORE, thus increasing feasibility of procedural competence assessment. APPROACH Nineteen residents and 2 staff orthopedic surgeons from the University of Ottawa volunteered for a 2-part OSCE-style station including a written questionnaire and videotaped simulated open reduction and internal fixation midshaft radius fracture. Each performance was rated independently by 3 orthopedic surgeons using a single-item performance score (Time 1). The performances were assessed again 6 weeks later by the 3 raters using the O-SCORE (Time 2). Correlation between the single-item performance score and the O-SCORE were evaluated. RESULTS Three orthopedic surgeons completed 21 ratings each resulting in 63 orthopedic ratings. There was a high level of correlation and agreement between the single-item performance score at Time 1 and Time 2 (κ correlation =0.72-1.00; p < .001; percentage agreement =90%-100%). The reliability of the O-SCORE at Time 2 with three raters was 0.83 and the internal consistency was 0.89. There was a tendency for each rater to assign more yes responses to the more senior trainees. CONCLUSIONS A single-item performance score correlated highly with the O-SCORE in an orthopedic setting. A single-item score could be used to supplement a multi-item score with similar results in orthopedics. There is still benefit in completing multi-item scores such as the O-SCORE evaluations to guide specific areas of improvement and direct feedback.
Collapse
Affiliation(s)
- David Saliken
- a Department of Surgery , RebalanceMD , Victoria , British Columbia , Canada
| | - Nancy Dudek
- b Department of Surgery , University of Ottawa , Ottawa , Ontario , Canada
| | - Timothy J Wood
- c Department of Innovation in Medical Education , University of Ottawa , Ottawa , Ontario , Canada
| | - Matthew MacEwan
- d Department of Orthopedic Surgery , University of Ottawa , Ottawa , Ontario , Canada
- e Departments of Surgery and Innovation in Medical Education , University of Ottawa , Ottawa , Ontario , Canada
| | - Wade T Gofton
- c Department of Innovation in Medical Education , University of Ottawa , Ottawa , Ontario , Canada
| |
Collapse
|
31
|
Terrell E, Agarwal A, Wood TJ, Writer H, Pound CM. An Evaluation of the Scholarly Activity Guidance and Evaluative (SAGE) Program. MedEdPublish (2016) 2019; 8:65. [PMID: 38089366 PMCID: PMC10712647 DOI: 10.15694/mep.2019.000065.1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/06/2024] Open
Abstract
This article was migrated. The article was marked as recommended. Objective: To evaluate the SAGE program five years following implementation. Methods: Our program evaluation was based on Guskey's five-level framework for evaluation of professional development. Residents and supervisors were invited to participate. Participants' reactions, learning, perceived organization support, use of new knowledge or skills and learning outcomes were examined through questionnaires and interviews. Results: 54% of residents and 65% supervisors were mostly or very satisfied with SAGE. 75% of residents felt there was moderate or great institutional support of resident research. Most residents and supervisors reported satisfaction with institutional research resources. Residents participating in the SAGE program reported a greater number of grant submissions and awards, but fewer conference presentations. Conclusions: SAGE has been well received by residents and supervisors. Findings suggest the program has fostered the development of research abilities and improved institutional support. It remains unclear if demonstrable learning outcomes have increased since program implementation. We also identified several barriers which will serve as targets for program improvement in future.
Collapse
Affiliation(s)
| | - Amisha Agarwal
- Children’s Hospital of Eastern Ontario Research Institute
| | | | | | | |
Collapse
|
32
|
Abstract
This article was migrated. The article was marked as recommended. Introduction: With the advent of competency-based medical education there is an emphasis on formative workplace based assessment. The quality of these assessments is a concern for medical educators and their trainees. Faculty development (FD) strategies to improve assessment quality have resulted in some success. However, few faculty participate, and those who do are likely more motivated to improve, making it difficult to demonstrate a conclusive benefit. To address these weaknesses, we designed a FD initiative to improve the quality of completed in-training evaluation reports (ITERs). All faculty within a division participated. We hypothesized that clinical supervisors would improve their ITER quality based on feedback, regardless of their own motivation to do so, with a simple, point-in-time intervention. Methods: In this three-phase study, two independent raters used the Completed Clinical Evaluation Report Rating (CCERR) to assess the quality of ITERs completed by all faculty in the Division of Orthopedic Surgery at the University of Ottawa. In phase one, ITERs from the previous nine months were evaluated. In phase two, the participants were aware that their ITERs were being evaluated, but they did not receive feedback. In phase three, participants received regular feedback on their performance in the form of their mean CCERR scores. Mean CCERR scores from the different phases of the study were compared. Results: CCERR scores were similar for all three phases (one: 17.56 ± 1.02, two: 17.65 ± 0.96, three: 17.54 ± 0.75, p=0.98). Discussion and Conclusions: There was no evidence in our study that participants' improved their ITER quality despite being aware that they were being evaluated and/or receiving feedback. Potentially, this was related to a lack of motivation. Alternatively, the intensity and/or frequency of the feedback may have been inadequate to create change. These results raise concerns that some faculty development may not necessarily be better than none.
Collapse
|
33
|
Wood TJ, Pugh D, Touchie C, Chan J, Humphrey-Murto S. Can physician examiners overcome their first impression when examinee performance changes? Adv Health Sci Educ Theory Pract 2018; 23:721-732. [PMID: 29556923 DOI: 10.1007/s10459-018-9823-4] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/05/2017] [Accepted: 03/15/2018] [Indexed: 06/08/2023]
Abstract
There is an increasing focus on factors that influence the variability of rater-based judgments. First impressions are one such factor. First impressions are judgments about people that are made quickly and are based on little information. Under some circumstances, these judgments can be predictive of subsequent decisions. A concern for both examinees and test administrators is whether the relationship remains stable when the performance of the examinee changes. That is, once a first impression is formed, to what degree will an examiner be willing to modify it? The purpose of this study is to determine the degree that first impressions influence final ratings when the performance of examinees changes within the context of an objective structured clinical examination (OSCE). Physician examiners (n = 29) viewed seven videos of examinees (i.e., actors) performing a physical exam on a single OSCE station. They rated the examinees' clinical abilities on a six-point global rating scale after 60 s (first impression or FIGR). They then observed the examinee for the remainder of the station and provided a final global rating (GRS). For three of the videos, the examinees' performance remained consistent throughout the videos. For two videos, examinee performance changed from initially strong to weak and for two videos, performance changed from initially weak to strong. The mean FIGR rating for the Consistent condition (M = 4.80) and the Strong to Weak condition (M = 4.87) were higher compared to their respective GRS ratings (M = 3.93, M = 2.73) with a greater decline for the Strong to Weak condition. The mean FIGR rating for the Weak to Strong condition was lower (3.60) than the corresponding mean GRS (4.81). This pattern of findings suggests that raters were willing to change their judgments based on examinee performance. Future work should explore the impact of making a first impression judgment explicit versus implicit and the role of context on the relationship between a first impression and a subsequent judgment.
Collapse
Affiliation(s)
- Timothy J Wood
- Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa, PMC 102H, 850 Peter Morand Crescent, Ottawa, ON, K1G-573, Canada.
| | - Debra Pugh
- Department of Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Canada
| | - Claire Touchie
- Department of Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Canada
| | - James Chan
- Department of Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Canada
| | - Susan Humphrey-Murto
- Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa, PMC 102H, 850 Peter Morand Crescent, Ottawa, ON, K1G-573, Canada
- Department of Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Canada
| |
Collapse
|
34
|
Wood TJ, Gibbs J, Rothwell N, Wilson JK, Gut L, Brokaw J, Isaacs R. Limited phenological and dietary overlap between bee communities in spring flowering crops and herbaceous enhancements. Ecol Appl 2018; 28:1924-1934. [PMID: 30184292 DOI: 10.1002/eap.1789] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/09/2018] [Revised: 06/29/2018] [Accepted: 07/15/2018] [Indexed: 06/08/2023]
Abstract
Wild bee populations have undergone declines in recent years across much of the Western world, and these declines have the potential to limit yield in pollination-dependent crops. Highbush blueberry, Vaccinium corymbosum, and tart cherry, Prunus cerasus, are spring-blooming crops that rely on the movement of pollen by bees and other insects for pollination. Wild bee populations can be increased on farmland by providing floral resources, but whether the addition of these plants translates into increased pollinator density on crop flowers has not been documented in most cropping systems. To determine the importance of providing additional floral resources for wild bee pollinator communities, we selected blueberry fields and tart cherry orchards with and without herbaceous floral enhancements in western Michigan, USA. The bee communities visiting crop flowers, enhancements and control grassy field margins were sampled over a 5-yr period. In addition, the pollen diets of the most abundant wild bee crop pollinators were quantified across Michigan to better understand their foraging niches and to identify potentially important alternative host plants. The presence of floral enhancements did not increase the abundance of wild bees on either blueberry or cherry flowers during bloom. The bee community visiting blueberry was evenly composed of short-season bees that fly only during the spring and long-season bees that fly in both spring and summer. In contrast, the bee community visiting cherry was dominated by short-season spring bees. The majority of pollen collected by the wild bee communities visiting blueberry and cherry was from spring-flowering woody plants, with limited use of the herbaceous enhancements. Enhancements attracted greater abundance and species richness of bees compared to control areas, including twice as many floral specialists. Conserving summer-flying, grassland-associated bees is an appropriate goal for pollinator conservation programs. However, herbaceous enhancements may not provide adequate resources for the wild bees that pollinate spring-flowering crops. This study demonstrates that an examination of the pollen collected by wild bees across their flight periods can identify plant species to help them persist in intensively managed landscapes.
Collapse
Affiliation(s)
- T J Wood
- Department of Entomology, Michigan State University, East Lansing, Michigan, 48824, USA
| | - J Gibbs
- Department of Entomology, Michigan State University, East Lansing, Michigan, 48824, USA
| | - N Rothwell
- Northwest Michigan Horticulture Research Center, Michigan State University, Traverse City, Michigan, 49684, USA
| | - J K Wilson
- Department of Entomology, Michigan State University, East Lansing, Michigan, 48824, USA
| | - L Gut
- Department of Entomology, Michigan State University, East Lansing, Michigan, 48824, USA
| | - J Brokaw
- Department of Entomology, Michigan State University, East Lansing, Michigan, 48824, USA
| | - R Isaacs
- Department of Entomology, Michigan State University, East Lansing, Michigan, 48824, USA
| |
Collapse
|
35
|
Abstract
UNLABELLED Construct: The purpose of this study was to provide validity evidence for the mini-clinical evaluation exercise (mini-CEX) as an assessment tool for clinical skills in the workplace. BACKGROUND Previous research has demonstrated validity evidence for the mini-CEX, but most studies were carried out in internal medicine or single disciplines, therefore limiting generalizability of the findings. If the mini-CEX is to be used in multidisciplinary contexts, then validity evidence should be gathered in similar settings. The purpose of this study was to gather further validity evidence for the mini-CEX but in a broader context. Specifically we sought to explore the effects of discipline and rater type on mini-CEX scores, internal structure, and the relationship between mini-CEXs and OSCEs in a multidisciplinary context. APPROACH During clerkship, medical students completed eight different rotations (family medicine, internal medicine, surgery, psychiatry, pediatrics, emergency, anesthesiology and obstetrics and gynecology). During each rotation, mini-CEX forms and a written examination were completed. Two multidisciplinary OSCEs (in Clerkship Year 3 and start of Year 4) assessed clinical skills. The reliability of the mini-CEX was assessed using Generalizability analyses. To assess the influence of discipline and rater type, mean scores were analyzed using a factorial analysis of variance. The total mini-CEX score was correlated to scores from the students' respective OSCEs and corresponding written exams. RESULTS Eighty-two students met inclusion criteria for a total of 781 ratings (average of 9.82 mini-CEX forms per student). There was a significant effect of discipline (p < .001, = .16), and faculty provided lower scores than nonfaculty raters (7.12 vs. 7.41; p = .002, = .02). The g-coefficient was .53 when discipline was included as a facet and .23 when rater type was a facet. There were low, but statistically significant correlations between the mini-CEX and scores for the 4th-year OSCE Total Score and the OSCE communication scores, r(80) = .40, p < .001 and r(80) = .29, p = .009. The mini-CEX was not correlated with the written examination scores for any of the disciplines. CONCLUSIONS Our results provide conflicting findings for validity evidence for the mini-CEX. Mini-CEX ratings were correlated to multidisciplinary OSCEs but not written examinations, supporting the validity argument. However, reliability of the mini-CEX was low to moderate, and error accounted for the greatest amount of variability in scores. There was variation in scores due to discipline and resident raters gave higher scores than faculty. These results should be considered when considering the use of the mini-CEX in different contexts.
Collapse
Affiliation(s)
| | - Mylène Côté
- a Department of Medicine , University of Ottawa , Ottawa , Ontario , Canada
| | - Debra Pugh
- a Department of Medicine , University of Ottawa , Ottawa , Ontario , Canada
| | - Timothy J Wood
- b Department of Innovation in Medical Education , University of Ottawa , Ottawa , Ontario , Canada
| |
Collapse
|
36
|
Abstract
A common research study in assessment involves measuring the amount of knowledge, skills, or attitudes that participants' possess. In the continuing professional development arena, a researcher might also want to assess this information as an outcome of an educational activity. At some point, the researcher may wish to publish the results from these assessment-based studies. The goal of this commentary is to highlight common problems that could negatively influence the likelihood of an assessment-based manuscript being published.
Collapse
Affiliation(s)
- Timothy J Wood
- Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario
| |
Collapse
|
37
|
Cheung WJ, Dudek NL, Wood TJ, Frank JR. Supervisor-trainee continuity and the quality of work-based assessments. Med Educ 2017; 51:1260-1268. [PMID: 28971502 DOI: 10.1111/medu.13415] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/28/2017] [Revised: 05/30/2017] [Accepted: 07/11/2017] [Indexed: 05/12/2023]
Abstract
CONTEXT Work-based assessments (WBAs) represent an increasingly important means of reporting expert judgements of trainee competence in clinical practice. However, the quality of WBAs completed by clinical supervisors is of concern. The episodic and fragmented interaction that often occurs between supervisors and trainees has been proposed as a barrier to the completion of high-quality WBAs. OBJECTIVES The primary purpose of this study was to determine the effect of supervisor-trainee continuity on the quality of assessments documented on daily encounter cards (DECs), a common form of WBA. The relationship between trainee performance and DEC quality was also examined. METHODS Daily encounter cards representing three differing degrees of supervisor-trainee continuity (low, intermediate, high) were scored by two raters using the Completed Clinical Evaluation Report Rating (CCERR), a previously published nine-item quantitative measure of DEC quality. An analysis of variance (anova) was performed to compare mean CCERR scores among the three groups. Linear regression analysis was conducted to examine the relationship between resident performance and DEC quality. RESULTS Differences in mean CCERR scores were observed between the three continuity groups (p = 0.02); however, the magnitude of the absolute differences was small (partial eta-squared = 0.03) and not educationally meaningful. Linear regression analysis demonstrated a significant inverse relationship between resident performance and CCERR score (p < 0.001, r2 = 0.18). This inverse relationship was observed in both groups representing on-service residents (p = 0.001, r2 = 0.25; p = 0.04, r2 = 0.19), but not in the Off-service group (p = 0.62, r2 = 0.05). CONCLUSIONS Supervisor-trainee continuity did not have an educationally meaningful influence on the quality of assessments documented on DECs. However, resident performance was found to affect assessor behaviours in the On-service group, whereas DEC quality remained poor regardless of performance in the Off-service group. The findings suggest that greater attention should be given to determining ways of improving the quality of assessments reported for off-service residents, as well as for those residents demonstrating appropriate clinical competence progression.
Collapse
Affiliation(s)
- Warren J Cheung
- Department of Emergency Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - Nancy L Dudek
- Division of Physical Medicine and Rehabilitation, Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - Timothy J Wood
- Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada
| | - Jason R Frank
- Department of Emergency Medicine, University of Ottawa, Ottawa, Ontario, Canada
- Royal College of Physicians and Surgeons of Canada, Ottawa, Ontario, Canada
| |
Collapse
|
38
|
Humphrey-Murto S, Wood TJ, Varpio L. When I say … consensus group methods. Med Educ 2017; 51:994-995. [PMID: 28247419 DOI: 10.1111/medu.13263] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/09/2016] [Revised: 11/29/2016] [Accepted: 12/15/2016] [Indexed: 06/06/2023]
|
39
|
Wood TJ, Chan J, Humphrey-Murto S, Pugh D, Touchie C. The influence of first impressions on subsequent ratings within an OSCE station. Adv Health Sci Educ Theory Pract 2017; 22:969-983. [PMID: 27848171 DOI: 10.1007/s10459-016-9736-z] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/12/2015] [Accepted: 11/03/2016] [Indexed: 05/25/2023]
Abstract
Competency-based assessment is placing increasing emphasis on the direct observation of learners. For this process to produce valid results, it is important that raters provide quality judgments that are accurate. Unfortunately, the quality of these judgments is variable and the roles of factors that influence the accuracy of those judgments are not clearly understood. One such factor is first impressions: that is, judgments about people we do not know, made quickly and based on very little information. This study explores the influence of first impressions in an OSCE. Specifically, the purpose is to begin to examine the accuracy of a first impression and its influence on subsequent ratings. We created six videotapes of history-taking performance. Each video was scripted from a real performance by six examinee residents within a single OSCE station. Each performance was re-enacted with six different actors playing the role of the examinees and one actor playing the role of the patient and videotaped. A total of 23 raters (i.e., physician examiners) reviewed each video and were asked to make a global judgment of the examinee's clinical abilities after 60 s (First Impression GR) by providing a rating on a six-point global rating scale and then to rate their confidence in the accuracy of that judgment by providing a rating on a five-point rating scale (Confidence GR). After making these ratings, raters then watched the remainder of the examinee's performance and made another global rating of performance (Final GR) before moving on to the next video. First impression ratings of ability varied across examinees and were moderately correlated to expert ratings (r = .59, 95% CI [-.13, .90]). There were significant differences in mean ratings for three examinees. Correlations ranged from .05 to .56 but were only significant for three examinees. Rater confidence in their first impression was not related to the likelihood of a rater changing their rating between the first impression and a subsequent rating. The findings suggest that first impressions could play a role in explaining variability in judgments, but their importance was determined by the videotaped performance of the examinees. More work is needed to clarify conditions that support or discourage the use of first impressions.
Collapse
Affiliation(s)
- Timothy J Wood
- Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa, RGN 2206, 451 Smyth Road, Ottawa, ON, K1H 8M5, Canada.
| | - James Chan
- Department of Medicine, University of Ottawa, 501 Smyth Road, Ottawa, ON, K1H 8L6, Canada
| | - Susan Humphrey-Murto
- Department of Medicine, University of Ottawa, 501 Smyth Road, Ottawa, ON, K1H 8L6, Canada
| | - Debra Pugh
- Department of Medicine, University of Ottawa, 501 Smyth Road, Ottawa, ON, K1H 8L6, Canada
| | - Claire Touchie
- Department of Medicine, University of Ottawa, 501 Smyth Road, Ottawa, ON, K1H 8L6, Canada
| |
Collapse
|
40
|
Humphrey-Murto S, Varpio L, Wood TJ, Gonsalves C, Ufholz LA, Mascioli K, Wang C, Foth T. The Use of the Delphi and Other Consensus Group Methods in Medical Education Research: A Review. Acad Med 2017; 92:1491-1498. [PMID: 28678098 DOI: 10.1097/acm.0000000000001812] [Citation(s) in RCA: 286] [Impact Index Per Article: 40.9] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/06/2023]
Abstract
PURPOSE Consensus group methods, such as the Delphi method and nominal group technique (NGT), are used to synthesize expert opinions when evidence is lacking. Despite their extensive use, these methods are inconsistently applied. Their use in medical education research has not been well studied. The authors set out to describe the use of consensus methods in medical education research and to assess the reporting quality of these methods and results. METHOD Using scoping review methods, the authors searched the Medline, Embase, PsycInfo, PubMed, Scopus, and ERIC databases for 2009-2016. Full-text articles that focused on medical education and the keywords Delphi, RAND, NGT, or other consensus group methods were included. A standardized extraction form was used to collect article demographic data and features reflecting methodological rigor. RESULTS Of the articles reviewed, 257 met the inclusion criteria. The Modified Delphi (105/257; 40.8%), Delphi (91/257; 35.4%), and NGT (23/257; 8.9%) methods were most often used. The most common study purpose was curriculum development or reform (68/257; 26.5%), assessment tool development (55/257; 21.4%), and defining competencies (43/257; 16.7%). The reporting quality varied, with 70.0% (180/257) of articles reporting a literature review, 27.2% (70/257) reporting what background information was provided to participants, 66.1% (170/257) describing the number of participants, 40.1% (103/257) reporting if private decisions were collected, 37.7% (97/257) reporting if formal feedback of group ratings was shared, and 43.2% (111/257) defining consensus a priori. CONCLUSIONS Consensus methods are poorly standardized and inconsistently used in medical education research. Improved criteria for reporting are needed.
Collapse
Affiliation(s)
- Susan Humphrey-Murto
- S. Humphrey-Murto is associate professor, Department of Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada. L. Varpio is associate professor, Department of Medicine, Uniformed Services University of the Health Sciences, Bethesda, Maryland. T.J. Wood is associate professor, Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada. C. Gonsalves is assistant professor, clinician educator, and clinical hematologist, Department of Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada. L.-A. Ufholz is a medical librarian, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada. K. Mascioli is psychiatrist, Department of Psychiatry, and education fellow, Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada. C. Wang is an internal medicine resident, Department of Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada. T. Foth is assistant professor, School of Nursing, Faculty of Health Sciences, University of Ottawa, Ottawa, Ontario, Canada
| | | | | | | | | | | | | | | |
Collapse
|
41
|
Amath A, Ambacher K, Leddy JJ, Wood TJ, Ramnanan CJ. Comparing alternative and traditional dissemination metrics in medical education. Med Educ 2017; 51:935-941. [PMID: 28719136 DOI: 10.1111/medu.13359] [Citation(s) in RCA: 55] [Impact Index Per Article: 7.9] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/22/2016] [Revised: 01/15/2017] [Accepted: 04/19/2017] [Indexed: 05/25/2023]
Abstract
CONTEXT The impact of academic scholarship has traditionally been measured using citation-based metrics. However, citations may not be the only measure of impact. In recent years, other platforms (e.g. Twitter) have provided new tools for promoting scholarship to both academic and non-academic audiences. Alternative metrics (altmetrics) can capture non-traditional dissemination data such as attention generated on social media platforms. OBJECTIVES The aims of this exploratory study were to characterise the relationships among altmetrics, access counts and citations in an international and pre-eminent medical education journal, and to clarify the roles of these metrics in assessing the impact of medical education academic scholarship. METHODS A database study was performed (September 2015) for all papers published in Medical Education in 2012 (n = 236) and 2013 (n = 246). Citation, altmetric and access (HTML views and PDF downloads) data were obtained from Scopus, the Altmetric Bookmarklet tool and the journal Medical Education, respectively. Pearson coefficients (r-values) between metrics of interest were then determined. RESULTS Twitter and Mendeley (an academic bibliography tool) were the only altmetric-tracked platforms frequently (> 50%) utilised in the dissemination of articles. Altmetric scores (composite measures of all online attention) were driven by Twitter mentions. For short and full-length articles in 2012 and 2013, both access counts and citation counts were most strongly correlated with one another, as well as with Mendeley downloads. By comparison, Twitter metrics and altmetric scores demonstrated weak to moderate correlations with both access and citation counts. CONCLUSIONS Whereas most altmetrics showed limited correlations with readership (access counts) and impact (citations), Mendeley downloads correlated strongly with both readership and impact indices for articles published in the journal Medical Education and may therefore have potential use that is complementary to that of citations in assessment of the impact of medical education scholarship.
Collapse
Affiliation(s)
- Aysah Amath
- Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - Kristin Ambacher
- Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - John J Leddy
- Faculty of Medicine, Department of Cellular and Molecular Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - Timothy J Wood
- Faculty of Medicine, Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada
| | - Christopher J Ramnanan
- Faculty of Medicine, Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada
| |
Collapse
|
42
|
Humphrey-Murto S, Varpio L, Gonsalves C, Wood TJ. Using consensus group methods such as Delphi and Nominal Group in medical education research . Med Teach 2017; 39:14-19. [PMID: 27841062 DOI: 10.1080/0142159x.2017.1245856] [Citation(s) in RCA: 223] [Impact Index Per Article: 31.9] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/23/2023]
Abstract
Consensus group methods are widely used in research to identify and measure areas where incomplete evidence exists for decision-making. Despite their widespread use, these methods are often inconsistently used and reported. Using examples from the three most commonly used methods, the Delphi, Nominal Group and RAND/UCLA; this paper and associated Guide aim to describe these methods and to highlight common weaknesses in methodology and reporting. The paper outlines a series of recommendations to assist researchers using consensus group methods in providing a comprehensive description and justification of the steps taken in their study.
Collapse
Affiliation(s)
| | - Lara Varpio
- b School of Medicine , Uniformed Services University of the Health Sciences , Bethesda , MD , USA
| | - Carol Gonsalves
- a Division of Rheumatology , University of Ottawa , Ottawa , Canada
| | - Timothy J Wood
- a Division of Rheumatology , University of Ottawa , Ottawa , Canada
| |
Collapse
|
43
|
Wood TJ. P Value: Pitfalls and Solutions. Acad Med 2016; 91:e22. [PMID: 27749307 DOI: 10.1097/acm.0000000000001445] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/06/2023]
Affiliation(s)
- Timothy J Wood
- associate professor, Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa
| |
Collapse
|
44
|
Moore CS, Wood TJ, Avery G, Balcam S, Needler L, Joshi H, Saunderson JR, Beavis AW. Automatic exposure control calibration and optimisation for abdomen, pelvis and lumbar spine imaging with an Agfa computed radiography system. Phys Med Biol 2016; 61:N551-N564. [DOI: 10.1088/0031-9155/61/21/n551] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
|
45
|
Pugh D, Halman S, Desjardins I, Humphrey-Murto S, Wood TJ. Done or Almost Done? Improving OSCE Checklists to Better Capture Performance in Progress Tests. Teach Learn Med 2016; 28:406-414. [PMID: 27700252 DOI: 10.1080/10401334.2016.1218337] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/06/2023]
Abstract
UNLABELLED Construct: The impact of using nonbinary checklists for scoring residents from different levels of training participating in objective structured clinical examination (OSCE) progress tests was explored. BACKGROUND OSCE progress tests typically employ similar rating instruments as traditional OSCEs. However, progress tests differ from other assessment modalities because learners from different stages of training participate in the same examination, which can pose challenges when deciding how to assign scores. In an attempt to better capture performance, nonbinary checklists were introduced in two OSCE progress tests. The purposes of this study were (a) to identify differences in the use of checklist options (e.g., done satisfactorily, attempted, or not done) by task type, (b) to analyze the impact of different scoring methods using nonbinary checklists for two OSCE progress tests (nonprocedural and procedural) for Internal Medicine residents, and (c) to determine which scoring method is better suited for a given task. APPROACH A retrospective analysis examined differences in scores (n = 119) for two OSCE progress tests (procedural and nonprocedural). Scoring methods (hawk, dove, and hybrid) varied in stringency in how they awarded marks for nonbinary checklist items that were rated as done satisfactorily, attempted, or not done. Difficulty, reliability (internal consistency), item-total correlations and pass rates were compared for each OSCE using the three scoring methods. RESULTS Mean OSCE scores were highest using the dove method and lowest using the hawk method. The hawk method resulted in higher item-total correlations for most stations, but there were differences by task type. Overall score reliability calculated using the three methods did not differ significantly. Pass-fail status differed as a function of scoring methods and exam type, with the hawk and hybrid methods resulting in higher failure rates for the nonprocedural OSCE and the dove method resulting in a higher failure rate for the procedural OSCE. CONCLUSION The use of different scoring methods for nonbinary OSCE checklists resulted in differences in mean scores and pass-fail status. The results varied with procedural and nonprocedural OSCEs.
Collapse
Affiliation(s)
- Debra Pugh
- a Department of Medicine , University of Ottawa, The Ottawa Hospital , Ottawa , Ontario , Canada
| | - Samantha Halman
- a Department of Medicine , University of Ottawa, The Ottawa Hospital , Ottawa , Ontario , Canada
| | - Isabelle Desjardins
- a Department of Medicine , University of Ottawa, The Ottawa Hospital , Ottawa , Ontario , Canada
| | - Susan Humphrey-Murto
- b Department of Medicine, University of Ottawa, The Ottawa Hospital , and Department of Innovation in Medical Education, University of Ottawa , Ottawa , Ontario , Canada
| | - Timothy J Wood
- b Department of Medicine, University of Ottawa, The Ottawa Hospital , and Department of Innovation in Medical Education, University of Ottawa , Ottawa , Ontario , Canada
| |
Collapse
|
46
|
Abstract
BACKGROUND Concerns over the quality of work-based assessment (WBA) completion has resulted in faculty development and rater training initiatives. Daily encounter cards (DECs) are a common form of WBA used in ambulatory care and shift work settings. A tool is needed to evaluate initiatives aimed at improving the quality of completion of this widely used form of WBA. OBJECTIVE The completed clinical evaluation report rating (CCERR) was designed to provide a measure of the quality of documented assessments on in-training evaluation reports. The purpose of this study was to provide validity evidence to support using the CCERR to assess the quality of DEC completion. METHODS Six experts in resident assessment grouped 60 DECs into 3 quality categories (high, average, and poor) based on how informative each DEC was for reporting judgments of the resident's performance. Eight supervisors (blinded to the expert groupings) scored the 10 most representative DECs in each group using the CCERR. Mean scores were compared to determine if the CCERR could discriminate based on DEC quality. RESULTS Statistically significant differences in CCERR scores were observed between all quality groups (P < .001). A generalizability analysis demonstrated the majority of score variation was due to differences in DECs. The reliability with a single rater was 0.95. CONCLUSIONS The CCERR is a reliable and valid tool to evaluate DEC quality. It can serve as an outcome measure for studying interventions targeted at improving the quality of assessments documented on DECs.
Collapse
Affiliation(s)
- Warren J. Cheung
- Corresponding author: Warren J. Cheung, MD, MMEd, FRCPC, University of Ottawa, Department of Emergency Medicine, F-Main, Room EM-206, 1053 Carling Avenue, Ottawa, Ontario K1Y 4E9 Canada, 613.798.5555, ext 17196, fax 613.761.5488,
| | | | | | | |
Collapse
|
47
|
Moore CS, Wood TJ, Cawthorne C, Hilton KL, Maher S, Saunderson JR, Archibald S, Beavis AW. A method to calibrate the RS 2000 x-ray biological irradiator for radiobiological flank irradiation of mice. Biomed Phys Eng Express 2016. [DOI: 10.1088/2057-1976/2/3/037001] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
|
48
|
Pugh D, Bhanji F, Cole G, Dupre J, Hatala R, Humphrey-Murto S, Touchie C, Wood TJ. Do OSCE progress test scores predict performance in a national high-stakes examination? Med Educ 2016; 50:351-358. [PMID: 26896020 DOI: 10.1111/medu.12942] [Citation(s) in RCA: 30] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/04/2015] [Revised: 09/01/2015] [Accepted: 09/21/2015] [Indexed: 06/05/2023]
Abstract
CONTEXT Progress tests, in which learners are repeatedly assessed on equivalent content at different times in their training and provided with feedback, would seem to lend themselves well to a competency-based framework, which requires more frequent formative assessments. The objective structured clinical examination (OSCE) progress test is a relatively new form of assessment that is used to assess the progression of clinical skills. The purpose of this study was to establish further evidence for the use of an OSCE progress test by demonstrating an association between scores from this assessment method and those from a national high-stakes examination. METHODS The results of 8 years' of data from an Internal Medicine Residency OSCE (IM-OSCE) progress test were compared with scores on the Royal College of Physicians and Surgeons of Canada Comprehensive Objective Examination in Internal Medicine (RCPSC IM examination), which is comprised of both a written and performance-based component (n = 180). Correlations between scores in the two examinations were calculated. Logistic regression analyses were performed comparing IM-OSCE progress test scores with an 'elevated risk of failure' on either component of the RCPSC IM examination. RESULTS Correlations between scores from the IM-OSCE (for PGY-1 residents to PGY-4 residents) and those from the RCPSC IM examination ranged from 0.316 (p = 0.001) to 0.554 (<.001) for the performance-based component and 0.305 (p = 0.002) to 0.516 (p < 0.001) for the written component. Logistic regression models demonstrated that PGY-2 and PGY-4 scores from the IM-OSCE were predictive of an 'elevated risk of failure' on both components of the RCPSC IM examination. CONCLUSIONS This study provides further evidence for the use of OSCE progress testing by demonstrating a correlation between scores from an OSCE progress test and a national high-stakes examination. Furthermore, there is evidence that OSCE progress test scores are predictive of future performance on a national high-stakes examination.
Collapse
Affiliation(s)
- Debra Pugh
- Department of Medicine, The Ottawa Hospital, Ottawa Hospital Research Institute, University of Ottawa, Ottawa, Ontario, Canada
| | - Farhan Bhanji
- Department of Pediatrics, McGill University, Montreal, Quebec, Canada
- Royal College of Physicians and Surgeons of Canada, Ottawa, Ontario, Canada
| | - Gary Cole
- Royal College of Physicians and Surgeons of Canada, Ottawa, Ontario, Canada
| | - Jonathan Dupre
- Royal College of Physicians and Surgeons of Canada, Ottawa, Ontario, Canada
| | - Rose Hatala
- Department of Medicine, University of British Columbia, Vancouver, British Columbia, Canada
| | - Susan Humphrey-Murto
- Department of Medicine, The Ottawa Hospital, Ottawa Hospital Research Institute, University of Ottawa, Ottawa, Ontario, Canada
- Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada
| | - Claire Touchie
- Department of Medicine, The Ottawa Hospital, Ottawa Hospital Research Institute, University of Ottawa, Ottawa, Ontario, Canada
- Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada
- Medical Council of Canada, Ottawa, Ontario, Canada
| | - Timothy J Wood
- Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada
| |
Collapse
|
49
|
MacEwan MJ, Dudek NL, Wood TJ, Gofton WT. Continued Validation of the O-SCORE (Ottawa Surgical Competency Operating Room Evaluation): Use in the Simulated Environment. Teach Learn Med 2016; 28:72-9. [PMID: 26787087 DOI: 10.1080/10401334.2015.1107483] [Citation(s) in RCA: 22] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/10/2023]
Abstract
UNLABELLED CONSTRUCT: The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) is a 9-item surgical evaluation tool designed to assess technical competence in surgical trainees using behavioral anchors. BACKGROUND The initial development of the O-SCORE produced evidence for valid results. Further work is required to determine if the use of a single surgeon or an unblinded rater introduces bias. In addition, the relationship of the O-SCORE to other currently used technical assessment tools should be explored to provide validity evidence related to the relationship to other measures. We have designed this project to provide continued validity evidence for the O-SCORE related to these two issues. APPROACH Nineteen residents and 2 staff Orthopedic Surgeons from the University of Ottawa volunteered to participate in a 2-part OSCE style station. Participants completed a written questionnaire followed by a videotaped 10-minute simulated open reduction and internal fixation of a midshaft radius fracture. Videos were rated individually by 2 blinded staff orthopedic surgeons using an Objective Structured Assessment of Technical Skills (OSATS) global rating scale, an OSATS checklist, and the O-SCORE in random order. RESULTS O-SCORE results appeared sensitive to surgical training level even when raters were blinded. In addition, strong agreement between two independent observers using the O-SCORE suggests that the measure captures a performance easily recognized by surgical observers. Ratings on the O-SCORE also were strongly associated with global ratings on the currently most validated technical evaluation tool (OSATS). Collectively, these results suggest that the O-SCORE generates accurate, reproducible, and meaningful results when used in a randomized and blinded fashion, providing continued validity evidence for using this tool to evaluate surgical trainee competence. CONCLUSIONS The O-SCORE was able to differentiate surgical trainee level using blinded raters providing further evidence of validity for the O-SCORE. There was strong agreement between two independent observers using the O-SCORE. Ratings on the O-SCORE also demonstrated equivalence to scores on the most validated technical evaluation tool (OSATS). These results suggest that the O-SCORE demonstrates accurate and reproducible results when used in a randomized and blinded fashion providing continued validity evidence for this tool in the evaluation of surgical competence in the trainees.
Collapse
Affiliation(s)
| | - Nancy L Dudek
- b Department of Medicine , Faculty of Medicine, University of Ottawa , Ontario , Canada
| | - Timothy J Wood
- c Department of Innovation in Medical Education , Faculty of Medicine, University of Ottawa , Ontario , Canada
| | - Wade T Gofton
- d Department of Surgery and Department of Innovation in Medical Education , Faculty of Medicine, University of Ottawa , Ontario , Canada
| |
Collapse
|
50
|
Abstract
THEORY The move to competency-based education has heightened the importance of direct observation of clinical skills and effective feedback. The Objective Structured Clinical Examination (OSCE) is widely used for assessment and affords an opportunity for both direct observation and feedback to occur simultaneously. For feedback to be effective, it should include direct observation, assessment of performance, provision of feedback, reflection, decision making, and use of feedback for learning and change. HYPOTHESES If one of the goals of feedback is to engage students to think about their performance (i.e., reflection), it would seem imperative that they can recall this feedback both immediately and into the future. This study explores recall of feedback in the context of an OSCE. Specifically, the purpose of this study was to (a) determine the amount and the accuracy of feedback that trainees remember immediately after an OSCE, as well as 1 month later, and (b) assess whether prompting immediate recall improved delayed recall. METHODS Internal medicine residents received 2 minutes of verbal feedback from physician examiners in the context of an OSCE. The feedback was audio-recorded and later transcribed. Residents were randomly allocated to the immediate recall group (immediate-RG; n = 10) or the delayed recall group (delayed-RG; n = 8). The immediate-RG completed a questionnaire prompting recall of feedback received immediately after the OSCE, and then again 1 month later. The delayed-RG completed a questionnaire only 1 month after the OSCE. The total number and accuracy of feedback points provided by examiners were compared to the points recalled by residents. Results comparing recall at 1 month between the immediate-RG and the delayed-RG were also studied. RESULTS Physician examiners provided considerably more feedback points (M = 16.3) than the residents recalled immediately after the OSCE (M = 2.61, p < .001). There was no significant difference between the number of feedback points recalled upon completion of the OSCE (2.61) compared to 1 month later (M = 1.96, p = .06, Cohen's d = .70). Prompting immediate recall did not improve later recall. The mean accuracy score for feedback recall immediately after the OSCE was 4.3/9 or "somewhat representative," and at 1 month the score dropped to 3.5/9 or "not representative" (ns). CONCLUSION Residents recall very few feedback points immediately after the OSCE and 1 month later. The feedback points that are recalled are neither very accurate nor representative of the feedback actually provided.
Collapse
Affiliation(s)
| | - Marika Mihok
- a Department of Medicine , University of Ottawa , Ottawa , Ontario , Canada
| | - Debra Pugh
- a Department of Medicine , University of Ottawa , Ottawa , Ontario , Canada
| | - Claire Touchie
- a Department of Medicine , University of Ottawa , Ottawa , Ontario , Canada
| | - Samantha Halman
- a Department of Medicine , University of Ottawa , Ottawa , Ontario , Canada
| | - Timothy J Wood
- a Department of Medicine , University of Ottawa , Ottawa , Ontario , Canada
| |
Collapse
|