1
|
Mee J, Pandian R, Wolczynski J, Morales A, Paniagua M, Harik P, Baldwin P, Clauser BE. An experimental comparison of multiple-choice and short-answer questions on a high-stakes test for medical students. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2023:10.1007/s10459-023-10266-3. [PMID: 37665413 DOI: 10.1007/s10459-023-10266-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/25/2023] [Accepted: 07/16/2023] [Indexed: 09/05/2023]
Abstract
Recent advances in automated scoring technology have made it practical to replace multiple-choice questions (MCQs) with short-answer questions (SAQs) in large-scale, high-stakes assessments. However, most previous research comparing these formats has used small examinee samples testing under low-stakes conditions. Additionally, previous studies have not reported on the time required to respond to the two item types. This study compares the difficulty, discrimination, and time requirements for the two formats when examinees responded as part of a large-scale, high-stakes assessment. Seventy-one MCQs were converted to SAQs. These matched items were randomly assigned to examinees completing a high-stakes assessment of internal medicine. No examinee saw the same item in both formats. Items administered in the SAQ format were generally more difficult than items in the MCQ format. The discrimination index for SAQs was modestly higher than that for MCQs and response times were substantially higher for SAQs. These results support the interchangeability of MCQs and SAQs. When it is important that the examinee generate the response rather than selecting it, SAQs may be preferred. The results relating to difficulty and discrimination reported in this paper are consistent with those of previous studies. The results on the relative time requirements for the two formats suggest that with a fixed testing time fewer SAQs can be administered, this limitation more than makes up for the higher discrimination that has been reported for SAQs. We additionally examine the extent to which increased difficulty may directly impact the discrimination of SAQs.
Collapse
|
2
|
Sam AH, Wilson R, Westacott R, Gurnell M, Melville C, Brown CA. Thinking differently - Students' cognitive processes when answering two different formats of written question. MEDICAL TEACHER 2021; 43:1278-1285. [PMID: 34126840 DOI: 10.1080/0142159x.2021.1935831] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
BACKGROUND Single-best answer questions (SBAQs) are common but are susceptible to cueing. Very short answer questions (VSAQs) could be an alternative, and we sought to determine if students' cognitive processes varied across question types and whether students with different performance levels used different methods for answering questions. METHODS We undertook a 'think aloud' study, interviewing 21 final year medical students at five UK medical schools. Each student described their thought processes and methods used for eight questions of each type. Responses were coded and quantified to determine the relative frequency with which each method was used, denominated on the number of times a method could have been used. RESULTS Students were more likely to use analytical reasoning methods (specifically identifying key features) when answering VSAQs. The use of test-taking behaviours was more common for SBAQs; students frequently used the answer options to help them reach an answer. Students acknowledged uncertainty more frequently when answering VSAQs. Analytical reasoning was more commonly used by high-performing students compared with low-performing students. CONCLUSIONS Our results suggest that VSAQs encourage more authentic clinical reasoning strategies. Differences in cognitive approaches used highlight the need for focused approaches to teaching clinical reasoning and dealing with uncertainty.
Collapse
Affiliation(s)
- Amir H Sam
- Imperial College School of Medicine, Imperial College London, London, UK
| | - Rebecca Wilson
- Imperial College School of Medicine, Imperial College London, London, UK
| | - Rachel Westacott
- College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
| | - Mark Gurnell
- Wellcome Trust-MRC Institute of Metabolic Science, University of Cambridge and NIHR Cambridge Biomedical Research Centre, Addenbrooke's Hospital, Cambridge, UK
| | - Colin Melville
- School of Medical Sciences, University of Manchester, Manchester, UK
| | - Celia A Brown
- Division of Health Sciences, University of Warwick, Coventry, UK
| |
Collapse
|
3
|
Schauber SK, Hautz SC, Kämmer JE, Stroben F, Hautz WE. Do different response formats affect how test takers approach a clinical reasoning task? An experimental study on antecedents of diagnostic accuracy using a constructed response and a selected response format. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2021; 26:1339-1354. [PMID: 33977409 PMCID: PMC8452553 DOI: 10.1007/s10459-021-10052-z] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/26/2018] [Accepted: 05/03/2021] [Indexed: 06/01/2023]
Abstract
The use of response formats in assessments of medical knowledge and clinical reasoning continues to be the focus of both research and debate. In this article, we report on an experimental study in which we address the question of how much list-type selected response formats and short-essay type constructed response formats are related to differences in how test takers approach clinical reasoning tasks. The design of this study was informed by a framework developed within cognitive psychology which stresses the importance of the interplay between two components of reasoning-self-monitoring and response inhibition-while solving a task or case. The results presented support the argument that different response formats are related to different processing behavior. Importantly, the pattern of how different factors are related to a correct response in both situations seem to be well in line with contemporary accounts of reasoning. Consequently, we argue that when designing assessments of clinical reasoning, it is crucial to tap into the different facets of this complex and important medical process.
Collapse
Affiliation(s)
- Stefan K Schauber
- Centre for Health Sciences Education, Faculty of Medicine, University of Oslo, Postboks 1161 Blindern, 0318, Oslo, Norway.
| | - Stefanie C Hautz
- Department of Emergency Medicine, Inselspital University Hospital, University of Berne, 3010, Freiburgstrasse, Berne, Switzerland
| | - Juliane E Kämmer
- Center for Adaptive Rationality (ARC), Max Planck Institute for Human Development, Lentzeallee 94, 14195, Berlin, Germany
- AG Progress Test Medizin, Charité Medical School Berlin, Hannoversche Straße 19, 10115, Berlin, Germany
| | - Fabian Stroben
- AG Progress Test Medizin, Charité Medical School Berlin, Hannoversche Straße 19, 10115, Berlin, Germany
- Office of the Vice Dean for Teaching and Learning, Charité Universitätsmedizin, Berlin, Germany
| | - Wolf E Hautz
- Department of Emergency Medicine, Inselspital University Hospital, University of Berne, 3010, Freiburgstrasse, Berne, Switzerland
- Centre for Educational Measurement (CEMO), University of Oslo, Postboks 1161 Blindern, 0318, Oslo, Norway
| |
Collapse
|
4
|
Harper BD, Pingree EW, Growdon AS, O'Donnell KA, Huang GC. How to ACTFAST But Think Deliberately: An Intervention to Teach Critical Thinking in the Pediatric Clerkship. J Pediatr 2020; 226:5-8.e2. [PMID: 34756202 DOI: 10.1016/j.jpeds.2020.02.073] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/26/2020] [Accepted: 02/26/2020] [Indexed: 11/29/2022]
Affiliation(s)
- Beth D Harper
- Department of Pediatrics, Boston Children's Hospital, Boston, MA; Department of Pediatrics, Harvard Medical School, Boston, MA.
| | - Elizabeth W Pingree
- Department of Pediatrics, Boston Children's Hospital, Boston, MA; Department of Pediatrics, Harvard Medical School, Boston, MA
| | - Amanda S Growdon
- Department of Pediatrics, Boston Children's Hospital, Boston, MA; Department of Pediatrics, Harvard Medical School, Boston, MA
| | - Katherine A O'Donnell
- Department of Pediatrics, Boston Children's Hospital, Boston, MA; Department of Pediatrics, Harvard Medical School, Boston, MA
| | - Grace C Huang
- Department of Medicine, Harvard Medical School, Boston, MA; Department of Medicine, Beth Israel Deaconess Medical Center, Boston, MA; Carl J. Shapiro Institute for Education and Research, Boston, MA
| |
Collapse
|
5
|
|
6
|
Huwendiek S, Reichert F, Duncker C, de Leng BA, van der Vleuten CPM, Muijtjens AMM, Bosse HM, Haag M, Hoffmann GF, Tönshoff B, Dolmans D. Electronic assessment of clinical reasoning in clerkships: A mixed-methods comparison of long-menu key-feature problems with context-rich single best answer questions. MEDICAL TEACHER 2017; 39:476-485. [PMID: 28281369 DOI: 10.1080/0142159x.2017.1297525] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/06/2023]
Abstract
BACKGROUND It remains unclear which item format would best suit the assessment of clinical reasoning: context-rich single best answer questions (crSBAs) or key-feature problems (KFPs). This study compared KFPs and crSBAs with respect to students' acceptance, their educational impact, and psychometric characteristics when used in a summative end-of-clinical-clerkship pediatric exam. METHODS Fifth-year medical students (n = 377) took a computer-based exam that included 6-9 KFPs and 9-20 crSBAs which assessed their clinical reasoning skills, in addition to an objective structured clinical exam (OSCE) that assessed their clinical skills. Each KFP consisted of a case vignette and three key features using a "long-menu" question format. We explored students' perceptions of the KFPs and crSBAs in eight focus groups and analyzed statistical data of 11 exams. RESULTS Compared to crSBAs, KFPs were perceived as more realistic and difficult, providing a greater stimulus for the intense study of clinical reasoning, and were generally well accepted. The statistical analysis revealed no difference in difficulty, but KFPs resulted more reliable and efficient than crSBAs. The correlation between the two formats was high, while KFPs correlated more closely with the OSCE score. CONCLUSIONS KFPs with long-menu exams seem to bring about a positive educational effect without psychometric drawbacks.
Collapse
Affiliation(s)
- Sören Huwendiek
- a Department of Assessment and Evaluation, Institute of Medical Education Bern , University of Bern , Bern, Switzerland
| | - Friedrich Reichert
- b Department of Pediatric Cardiology and Intensive Care Medicine , Klinikum Stuttgart, Stuttgart , Germany
| | - Cecilia Duncker
- c Clinic for Child and Adolescent Psychiatry, University Hospital Kiel , Germany
| | - Bas A de Leng
- d Institute of Medical Education (IfAS), Faculty of Medicine, University of Muenster, Münster , Germany
| | - Cees P M van der Vleuten
- e Department of Educational Development and Research , Maastricht University, Maastricht , the Netherlands
| | - Arno M M Muijtjens
- e Department of Educational Development and Research , Maastricht University, Maastricht , the Netherlands
| | - Hans-Martin Bosse
- f Clinic for General Paediatrics, Neonatology and Paediatric Cardiology, University Children's Hospital Düsseldorf, Düsseldorf , Germany
| | - Martin Haag
- g GECKO Institute of Medicine, Informatics & Economics, Heilbronn University, Heilbronn , Germany
| | - Georg F Hoffmann
- h Clinic I, University Children's Hospital Heidelberg, Heidelberg , Germany
| | - Burkhard Tönshoff
- h Clinic I, University Children's Hospital Heidelberg, Heidelberg , Germany
| | - Diana Dolmans
- e Department of Educational Development and Research , Maastricht University, Maastricht , the Netherlands
| |
Collapse
|
7
|
Power A, Lemay JF, Cooke S. Justify Your Answer: The Role of Written Think Aloud in Script Concordance Testing. TEACHING AND LEARNING IN MEDICINE 2017; 29:59-67. [PMID: 27662118 DOI: 10.1080/10401334.2016.1217778] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/06/2023]
Abstract
UNLABELLED Construct: Clinical reasoning assessment is a growing area of interest in the medical education literature. Script concordance testing (SCT) evaluates clinical reasoning in conditions of uncertainty and has emerged as an innovative tool in the domain of clinical reasoning assessment. SCT quantifies the degree of concordance between a learner and an experienced clinician and attempts to capture the breadth of responses of expert clinicians, acknowledging the significant yet acceptable variation in practice under situations of uncertainty. BACKGROUND SCT has been shown to be a valid and reliable clinical reasoning assessment tool. However, as SCT provides only quantitative information, it may not provide a complete assessment of clinical reasoning. APPROACH Think aloud (TA) is a qualitative research tool used in clinical reasoning assessment in which learners verbalize their thought process around an assigned task. This study explores the use of TA, in the form of written reflection, in SCT to assess resident clinical reasoning, hypothesizing that the information obtained from the written TA would enrich the quantitative data obtained through SCT. Ninety-one pediatric postgraduate trainees and 21 pediatricians from 4 Canadian training centers completed an online test consisting of 24 SCT cases immediately followed by retrospective written TA. Six of 24 cases were selected to gather TA data. These cases were chosen to allow all phases of clinical decision making (diagnosis, investigation, and treatment) to be represented in the TA data. Inductive thematic analysis was employed when systematically reviewing TA responses. RESULTS Three main benefits of adding written TA to SCT were identified: (a) uncovering instances of incorrect clinical reasoning despite a correct SCT response, (b) revealing sound clinical reasoning in the context of a suboptimal SCT response, and (c) detecting question misinterpretation. CONCLUSIONS Written TA can optimize SCT by demonstrating when correct examinee responses are based on guessing or uncertainty rather than robust clinical rationale. TA can also enhance SCT by allowing examinees to provide justification for responses that otherwise would have been considered incorrect and by identifying questions that are frequently misinterpreted to avoid including them in future examinations. TA also has significant value in differentiating between acceptable variations in expert clinician responses and deviance associated with faulty rationale or question misinterpretation; this could improve SCT reliability. A written TA protocol appears to be a valuable tool to assess trainees' clinical reasoning and can strengthen the quantitative assessment provided by SCT.
Collapse
Affiliation(s)
- Alyssa Power
- a Department of Pediatrics , Alberta Children's Hospital , Calgary , Alberta , Canada
| | - Jean-Francois Lemay
- a Department of Pediatrics , Alberta Children's Hospital , Calgary , Alberta , Canada
| | - Suzette Cooke
- a Department of Pediatrics , Alberta Children's Hospital , Calgary , Alberta , Canada
| |
Collapse
|
8
|
ALQahtani DA, Rotgans JI, Ahmed NE, Alalwan IA, Magzoub MEM. The Influence of Time Pressure and Case Complexity on Physicians׳ Diagnostic Performance. HEALTH PROFESSIONS EDUCATION 2016. [DOI: 10.1016/j.hpe.2016.01.006] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
|
9
|
Roots SA, Niven E, Moran RW. Osteopaths' clinical reasoning during consultation with patients experiencing acute low back pain: A qualitative case study approach. INT J OSTEOPATH MED 2016. [DOI: 10.1016/j.ijosm.2015.06.003] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
|
10
|
Heist BS, Gonzalo JD, Durning S, Torre D, Elnicki DM. Exploring Clinical Reasoning Strategies and Test-Taking Behaviors During Clinical Vignette Style Multiple-Choice Examinations: A Mixed Methods Study. J Grad Med Educ 2014; 6:709-14. [PMID: 26140123 PMCID: PMC4477567 DOI: 10.4300/jgme-d-14-00176.1] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/01/2014] [Revised: 06/18/2014] [Accepted: 07/14/2014] [Indexed: 11/06/2022] Open
Abstract
BACKGROUND Clinical vignette multiple-choice questions (MCQs) are widely used in medical education, but clinical reasoning (CR) strategies employed when approaching these questions have not been well described. OBJECTIVES The aims of the study were (1) to identify CR strategies and test-taking (TT) behaviors of physician trainees while solving clinical vignette MCQs; and (2) to examine the relationships between CR strategies and behaviors, and performance on a high-stakes clinical vignette MCQ examination. METHODS Thirteen postgraduate year-1 level trainees completed 6 clinical vignette MCQs using a think-aloud protocol. Thematic analysis employing elements of grounded theory was performed on data transcriptions to identify CR strategies and TT behaviors. Participants' CR strategies and TT behaviors were then compared with their US Medical Licensing Examination Step 2 Clinical Knowledge scores. RESULTS Twelve CR strategies and TT behaviors were identified. Individuals with low performance on Step 2 Clinical Knowledge demonstrated increased premature closure and increased faulty knowledge, and showed comparatively less ruling out of alternatives or admission of knowledge deficits. High performers on Step 2 Clinical Knowledge demonstrated increased ruling out of alternatives and admission of knowledge deficits, and less premature closure, faulty knowledge, or closure prior to reading the alternatives. CONCLUSIONS Different patterns of CR strategies and TT behaviors may be used by high and low performers during high-stakes clinical vignette MCQ examinations.
Collapse
|
11
|
Abstract
Each day, we generate hypotheses about our environment—our perceptions of people, our expectations of events, and our interpretation of images. These hypotheses provide a framework by which we interpret our experiences. The same is true for differential diagnosis by which health care practitioners develop hypotheses or diagnoses from a set of cues provided during an encounter with a patient. For clinicians to be successful at differential diagnosis, they must use a multidimensional and complex process involving nonanalytic and analytic cognitive processes and metacognition—thinking about thinking. Our conclusions, however, can lead to errors in diagnosis. Many of these errors are due to errors in cognition. The purpose of this article is to discuss this complex process, identify common errors in cognition, and offer strategies to prevent these common errors in differential diagnosis.
Collapse
Affiliation(s)
- Kristine Anne Scordo
- Kristine Anne Scordo is Professor and Director, Adult-Gero Acute Care Nurse Practitioner Program, College of Nursing, Wright State University, Dayton, OH 45324
| |
Collapse
|
12
|
Desjardins I, Touchie C, Pugh D, Wood TJ, Humphrey-Murto S. The impact of cueing on written examinations of clinical decision making: a case study. MEDICAL EDUCATION 2014; 48:255-261. [PMID: 24528460 DOI: 10.1111/medu.12296] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/22/2013] [Revised: 03/01/2013] [Accepted: 06/26/2013] [Indexed: 06/03/2023]
Abstract
CONTEXT Selected-response (SR) formats (e.g. multiple-choice questions) and constructed-response (CR) formats (e.g. short-answer questions) are commonly used to test the knowledge of examinees. Scores on SR formats are typically higher than scores on CR formats. This difference is often attributed to examinees being cued by options within an SR question, but there could be alternative explanations. The purpose of this study was to expand on previous work with regards to the cueing effect of SR formats by directly contrasting conditions that support cueing versus memory of previously seen questions. METHODS During an objective structured clinical examination, students (n = 144) completed two consecutive stations in which they were presented with the same written cases but in different formats. Group 1 students were presented with CR questions followed by SR questions. Group 2 students were presented with questions in reverse order. Participants were asked to describe their testing experience. RESULTS Selected-response scores (M = 4.21/10) were statistically higher than the CR scores (M = 3.82/10). However, there was no significant interaction between sequence and format (F(1,142) = 1.59, p = 0.21, ηp2 = 0.01) with scores increasing from 3.49/10 to 4.06/10 in the group that started with CR and decreasing (4.38/10-4.15/10) in the group that started with SR first. Correlations between SR scores and CR scores were high (CR first = 0.78, SR first = 0.89). Questionnaire results indicated that students felt the SR format was easier and led to cueing. CONCLUSION To better understand test performance, it is important to know how different response formats could influence results. Because SR scores were higher than CR scores, irrespective of the format seen first, the pattern is consistent with what would be expected if cueing rather than memory for prior questions led to higher SR scores. This could have implications for test designers, especially when selecting question formats.
Collapse
|
13
|
Pottier P, Hardouin JB, Hodges BD, Pistorius MA, Connault J, Durant C, Clairand R, Sebille V, Barrier JH, Planchon B. Exploring how students think: a new method combining think-aloud and concept mapping protocols. MEDICAL EDUCATION 2010; 44:926-935. [PMID: 20716103 DOI: 10.1111/j.1365-2923.2010.03748.x] [Citation(s) in RCA: 23] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/29/2023]
Abstract
OBJECTIVES A key element of medical competence is problem solving. Previous work has shown that doctors use inductive reasoning to progress from facts to hypotheses and deductive reasoning to move from hypotheses to the gathering of confirmatory information. No individual assessment method has been designed to quantify the use of inductive and deductive procedures within clinical reasoning. The aim of this study was to explore the feasibility and reliability of a new method which allows for the rapid identification of the style (inductive or deductive) of clinical reasoning in medical students and experts. METHODS The study included four groups of four participants. These comprised groups of medical students in Years 3, 4 and 5 and a group of specialists in internal medicine, all at a medical school with a 6-year curriculum in France. Participants were asked to solve four clinical problems by thinking aloud. The thinking expressed aloud was immediately transcribed into concept maps by one or two 'writers' trained to distinguish inductive and deductive links. Reliability was assessed by estimating the inter-writer correlation. The calculated rate of inductive reasoning, the richness score and the rate of exhaustiveness of reasoning were compared according to the level of expertise of the individual and the type of clinical problem. RESULTS The total number of maps drawn amounted to 32 for students in Year 4, 32 for students in Year 5, 16 for students in Year 3 and 16 for experts. A positive correlation was found between writers (R = 0.66-0.93). Richness scores and rates of exhaustiveness of reasoning did not differ according to expertise level. The rate of inductive reasoning varied as expected according to the nature of the clinical problem and was lower in experts (41% versus 67%). CONCLUSIONS This new method showed good reliability and may be a promising tool for the assessment of medical problem-solving skills, giving teachers a means of diagnosing how their students think when they are confronted with clinical problems.
Collapse
Affiliation(s)
- Pierre Pottier
- Department of Internal Medicine, Nantes University Hospital Centre, Nantes, FranceTeam for Biostatistics (EA 4572), Department of Clinical Research and Subjective Measures in Health Science, University of Nantes, Nantes, FranceWilson Centre for Research in Education, University of Toronto, Toronto, Ontario, CanadaDepartment of Psychiatry, University of Toronto, Toronto, Ontario, Canada
| | - Jean-Benoit Hardouin
- Department of Internal Medicine, Nantes University Hospital Centre, Nantes, FranceTeam for Biostatistics (EA 4572), Department of Clinical Research and Subjective Measures in Health Science, University of Nantes, Nantes, FranceWilson Centre for Research in Education, University of Toronto, Toronto, Ontario, CanadaDepartment of Psychiatry, University of Toronto, Toronto, Ontario, Canada
| | - Brian D Hodges
- Department of Internal Medicine, Nantes University Hospital Centre, Nantes, FranceTeam for Biostatistics (EA 4572), Department of Clinical Research and Subjective Measures in Health Science, University of Nantes, Nantes, FranceWilson Centre for Research in Education, University of Toronto, Toronto, Ontario, CanadaDepartment of Psychiatry, University of Toronto, Toronto, Ontario, Canada
| | - Marc-Antoine Pistorius
- Department of Internal Medicine, Nantes University Hospital Centre, Nantes, FranceTeam for Biostatistics (EA 4572), Department of Clinical Research and Subjective Measures in Health Science, University of Nantes, Nantes, FranceWilson Centre for Research in Education, University of Toronto, Toronto, Ontario, CanadaDepartment of Psychiatry, University of Toronto, Toronto, Ontario, Canada
| | - Jérome Connault
- Department of Internal Medicine, Nantes University Hospital Centre, Nantes, FranceTeam for Biostatistics (EA 4572), Department of Clinical Research and Subjective Measures in Health Science, University of Nantes, Nantes, FranceWilson Centre for Research in Education, University of Toronto, Toronto, Ontario, CanadaDepartment of Psychiatry, University of Toronto, Toronto, Ontario, Canada
| | - Cécile Durant
- Department of Internal Medicine, Nantes University Hospital Centre, Nantes, FranceTeam for Biostatistics (EA 4572), Department of Clinical Research and Subjective Measures in Health Science, University of Nantes, Nantes, FranceWilson Centre for Research in Education, University of Toronto, Toronto, Ontario, CanadaDepartment of Psychiatry, University of Toronto, Toronto, Ontario, Canada
| | - Renaud Clairand
- Department of Internal Medicine, Nantes University Hospital Centre, Nantes, FranceTeam for Biostatistics (EA 4572), Department of Clinical Research and Subjective Measures in Health Science, University of Nantes, Nantes, FranceWilson Centre for Research in Education, University of Toronto, Toronto, Ontario, CanadaDepartment of Psychiatry, University of Toronto, Toronto, Ontario, Canada
| | - Véronique Sebille
- Department of Internal Medicine, Nantes University Hospital Centre, Nantes, FranceTeam for Biostatistics (EA 4572), Department of Clinical Research and Subjective Measures in Health Science, University of Nantes, Nantes, FranceWilson Centre for Research in Education, University of Toronto, Toronto, Ontario, CanadaDepartment of Psychiatry, University of Toronto, Toronto, Ontario, Canada
| | - Jacques-Henri Barrier
- Department of Internal Medicine, Nantes University Hospital Centre, Nantes, FranceTeam for Biostatistics (EA 4572), Department of Clinical Research and Subjective Measures in Health Science, University of Nantes, Nantes, FranceWilson Centre for Research in Education, University of Toronto, Toronto, Ontario, CanadaDepartment of Psychiatry, University of Toronto, Toronto, Ontario, Canada
| | - Bernard Planchon
- Department of Internal Medicine, Nantes University Hospital Centre, Nantes, FranceTeam for Biostatistics (EA 4572), Department of Clinical Research and Subjective Measures in Health Science, University of Nantes, Nantes, FranceWilson Centre for Research in Education, University of Toronto, Toronto, Ontario, CanadaDepartment of Psychiatry, University of Toronto, Toronto, Ontario, Canada
| |
Collapse
|
14
|
Dory V, Gagnon R, Charlin B. Is case-specificity content-specificity? An analysis of data from extended-matching questions. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2010; 15:55-63. [PMID: 19496014 DOI: 10.1007/s10459-009-9169-z] [Citation(s) in RCA: 12] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/19/2008] [Accepted: 05/15/2009] [Indexed: 05/25/2023]
Abstract
Case-specificity, i.e., variability of a subject's performance across cases, has been a consistent finding in medical education. It has important implications for assessment validity and reliability. Its root causes remain a matter of discussion. One hypothesis, content-specificity, links variability of performance to variable levels of relevant knowledge. Extended-matching items (EMIs) are an ideal format to test this hypothesis as items are grouped by topic. If differences pertaining to content knowledge are the main cause of case-specificity, variability across topics should be high and variability across items within the same topic low. We used generalisability analysis on results of a written test composed of 159 EMIs sat by two cohorts of general practice trainees at one university. Two hundred and twenty-seven trainees took part. The variance component attributed to subjects was small. Variance attributed to topics was smaller than variance attributed to items. The main source of error was interaction between subjects and items, accounting for two-thirds of error. The generalisability D study revealed that for the same total number of items, increasing the number of topics results in a higher G coefficient than increasing the number of items per topic. Topical knowledge does not seem to explain case-specificity observed in our data. Structure of knowledge and reasoning strategy may be more important, in particular pattern-recognition which EMIs were designed to elicit. The causal explanations of case-specificity may be dependent on test format. Increasing the number of topics with fewer items each would increase reliability but also testing time.
Collapse
Affiliation(s)
- Valerie Dory
- Centre Academique de Medecine Generale, Universite catholique de Louvain, Avenue Emmanuel Mounier 53 (boite 5360), 1200, Brussels, Belgium.
| | - Robert Gagnon
- Centre de Pedagogie Appliquee aux Sciences de la Sante, Universite de Montreal, Montreal, Canada
| | - Bernard Charlin
- Centre de Pedagogie Appliquee aux Sciences de la Sante, Universite de Montreal, Montreal, Canada
| |
Collapse
|
15
|
Cook DA, Thompson WG, Thomas KG. Case-based or non-case-based questions for teaching postgraduate physicians: a randomized crossover trial. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2009; 84:1419-1425. [PMID: 19881436 DOI: 10.1097/acm.0b013e3181b6b36e] [Citation(s) in RCA: 13] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/28/2023]
Abstract
PURPOSE The comparative efficacy of case-based (CB) and non-CB self-assessment questions in Web-based instruction is unknown. The authors sought to compare CB and non-CB questions. METHOD The authors conducted a randomized crossover trial in the continuity clinics of two academic residency programs. Four Web-based modules on ambulatory medicine were developed in both CB (periodic questions based on patient scenarios) and non-CB (questions matched for content but lacking patient scenarios) formats. Participants completed two modules in each format (sequence randomly assigned). Participants also completed a pretest of applied knowledge for two modules (randomly assigned). RESULTS For the 130 participating internal medicine and family medicine residents, knowledge scores improved significantly (P < .0001) from pretest (mean: 53.5; SE: 1.1) to posttest (75.1; SE: 0.7). Posttest knowledge scores were similar in CB (75.0; SE: 0.1) and non-CB formats (74.7; SE: 1.1); the 95% CI was -1.6, 2.2 (P = .76). A nearly significant (P = .062) interaction between format and the presence or absence of pretest suggested a differential effect of question format, depending on pretest. Overall, those taking pretests had higher posttest knowledge scores (76.7; SE: 1.1) than did those not taking pretests (73.0; SE: 1.1; 95% CI: 1.7, 5.6; P = .0003). Learners preferred the CB format. Time required was similar (CB: 42.5; SE: 1.8 minutes, non-CB: 40.9; SE: 1.8 minutes; P = .22). CONCLUSIONS Our findings suggest that, among postgraduate physicians, CB and non-CB questions have similar effects on knowledge scores, but learners prefer CB questions. Pretests influence posttest scores.
Collapse
Affiliation(s)
- David A Cook
- Office of Education Research, Division of General Internal Medicine, College of Medicine, Mayo Clinic, Baldwin 4-A, 200 First Street SW, Rochester, MN 55905, USA.
| | | | | |
Collapse
|
16
|
Johnston JM, Schooling CM, Leung GM. A randomised-controlled trial of two educational modes for undergraduate evidence-based medicine learning in Asia. BMC MEDICAL EDUCATION 2009; 9:63. [PMID: 19785777 PMCID: PMC2761870 DOI: 10.1186/1472-6920-9-63] [Citation(s) in RCA: 30] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/28/2009] [Accepted: 09/29/2009] [Indexed: 05/08/2023]
Abstract
BACKGROUND As the overall evidence for the effectiveness of teaching of evidence based medicine (EBM) is not strong, and the impact of cultural and societal influences on teaching method is poorly understood, we undertook a randomised-controlled trial to test the effectiveness and learning satisfaction with two different EBM teaching methods (usual teaching vs. problem based learning (PBL)) for undergraduate medical students. METHODS A mixed methods study that included a randomised-controlled crossover trial with two intervention arms (usual teaching and PBL) and a nested qualitative study with focus groups to explore student perceptions of learning and to assess the effectiveness and utility of the two teaching methods.All 129 second-year medical students at the University of Hong Kong in 2007.The main outcomes measures were attitudes towards EBM; personal application and current use of EBM; EBM knowledge; future use of EBM. RESULTS PBL was less effective at imparting knowledge than usual teaching consisting of a lecture followed by a group tutorial. After usual teaching students showed improvement in scores for 'attitudes towards EBM', 'personal application and current use of EBM' and 'EBM knowledge, which were not evident after PBL. In contrast to the usual teaching, students found PBL difficult as they lacked the statistical knowledge necessary to support discussion, failed to understand core concepts, and lost direction. CONCLUSION The evidence presented here would suggest that the teaching of EBM within an Asian environment should adopt a format that facilitates both the acquisition of knowledge and encourages enquiry.
Collapse
Affiliation(s)
- Janice M Johnston
- School of Public Health, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Faculty of Medicine Building, 21 Sassoon Road, Pokfulam, Hong Kong, PR China
| | - C Mary Schooling
- School of Public Health, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Faculty of Medicine Building, 21 Sassoon Road, Pokfulam, Hong Kong, PR China
| | - Gabriel M Leung
- School of Public Health, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Faculty of Medicine Building, 21 Sassoon Road, Pokfulam, Hong Kong, PR China
| |
Collapse
|