1
|
Beaves M, Zoanetti N, Wallace EM, Palmer KR. Immediate and longer-term impacts of fetal surveillance education on workforce knowledge and cognitive skills. MEDEDPUBLISH 2023; 13:38. [PMID: 38779369 PMCID: PMC11109546 DOI: 10.12688/mep.19656.1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 05/22/2023] [Indexed: 05/25/2024] Open
Abstract
Background: Following the development of the Royal Australian College of Obstetricians and Gynaecologists Intrapartum Fetal Surveillance Guideline in 2003, an education program was developed to support guideline implementation and clinical practice. It was intended that improved clinician knowledge, particularly of cardiotocography, would reduce rates of intrapartum fetal morbidity and mortality. The program contains a multiple-choice assessment, designed to assess fetal surveillance knowledge and the application of that knowledge. We used the results of this assessment over time to evaluate the impact of the education program on clinicians' fetal surveillance knowledge and interpretive skills, in the immediate and longer-term. Methods: We undertook a retrospective analysis of the assessment results for all participants in the Fetal Surveillance Education Program, between 2004 and 2018. Classical Test Theory and Rasch Item Response Theory analysis were used to evaluate the statistical reliability and quality of the assessment, and the measurement invariance or stability of the assessments over time. Clinicians' assessment scores were then reviewed by craft group and previous exposure to the program. Results: The results from 64,430, broadly similar assessments, showed that participation in the education program was associated with an immediate improvement in clinician performance in the assessment. Performance improvement was sustained for up to 18 months following participation in the program and recurrent participation was associated with progressive improvements. These trends were observed for all craft groups (consultant obstetricians, doctors in training, general practitioners, midwives, student midwives). Conclusions: These findings suggest that the Fetal Surveillance Education Program has improved clinician knowledge and the associated cognitive skills over time. The stable difficulty of the assessment tool means any improvement in clinician's results, with ongoing exposure to the program, can be reliably assessed and demonstrated. Importantly this holds true for all craft groups involved in intrapartum care and the interpretation of cardiotocography.
Collapse
Affiliation(s)
- Mark Beaves
- Royal Australian and New Zealand College of Obstetricians and Gynaecologists, MELBOURNE, Victoria, 3004, Australia
- Dept of Obstetrics and Gynaecology, Monash University, Clayton, Victoria, 3800, Australia
| | - Nathan Zoanetti
- Research, Australian Council for Educational Research, Camberwell, Victoria, 3124, Australia
| | - Euan M Wallace
- Dept of Obstetrics and Gynaecology, Monash University, Clayton, Victoria, 3800, Australia
- Victorian Government, Department of Health, MELBOURNE, Victoria, 3000, Australia
| | - Kirsten R Palmer
- Dept of Obstetrics and Gynaecology, Monash University, Clayton, Victoria, 3800, Australia
- Obstetrics and Gynaecology, Monash Health, Clayton, Victoria, 3168, Australia
| |
Collapse
|
2
|
Al Zoubi F, Mayo N, Rochette A, Thomas A. Applying modern measurement approaches to constructs relevant to evidence-based practice among Canadian physical and occupational therapists. Implement Sci 2018; 13:152. [PMID: 30563550 PMCID: PMC6299597 DOI: 10.1186/s13012-018-0844-4] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2018] [Accepted: 11/28/2018] [Indexed: 12/16/2022] Open
Abstract
BACKGROUND Evidence-based practice (EBP) is a complex process. To quantify it, one has to also consider individual and contextual factors using multiple measures. Modern measurement approaches are available to optimize the measurement of complex constructs. This study aimed to develop a robust measurement approach for constructs around EBP including practice, individual (e.g. knowledge, attitudes, confidence, behaviours), and contextual factors (e.g. resources). METHODS One hundred eighty-one items arising from 5 validated EBP measures were subjected to an item analysis. Nominal group technique was used to arrive at a consensus about the content relevance of each item. Baseline questionnaire responses from a longitudinal study of the evolution of EBP in 128 new graduates of Canadian physical and occupational therapy programmes were analysed. Principles of Rasch Measurement Theory were applied to identify challenges with threshold ordering, item and person fit to the Rasch model, unidimensionality, local independence, and differential item functioning (DIF). RESULTS The nominal group technique identified 70/181 items, and modified Delphi approach identified 68 items that fit a formative model (2 related EBP domains: self-use of EBP (9 items) and EBP activities (7 items)) or a reflective model (4 related EBP domains: attitudes towards EBP (17 items), self-efficacy (9 items), knowledge (11 items) and resources (15 items)). Rasch analysis provided a single score for reflective construct. Among attitudes items, 65% (11/17) fit the Rasch model, item difficulties ranged from - 7.51 to logits (least difficult) to + 5.04 logits (most difficult), and person separation index (PSI) = 0.63. Among self-efficacy items, 89% (8/9) fit the Rasch model, item difficulties ranged from - 3.70 to + 4.91, and PSI = 0.80. Among knowledge items, 82% (9/11) fit the Rasch model, item difficulties ranged from - 7.85 to 4.50, and PSI = 0.81. Among resources items, 87% (13/15) fit the Rasch model, item difficulties ranged from - 3.38 to 2.86, and PSI = 0.86. DIF occurred in 2 constructs: attitudes (1 by profession and 2 by language) and knowledge (1 by language and 2 by profession) arising from poor wording in the original version leading to poor translation. CONCLUSIONS Rasch Measurement Theory was applied to develop a valid and reliable measure of EBP. Further modifications to the items can be done for subsequent waves of the survey.
Collapse
Affiliation(s)
- Fadi Al Zoubi
- McGill University, Montreal, QC, Canada.
- Centre for Interdisciplinary Research in Rehabilitation of Greater Montreal, Montreal, QC, Canada.
| | - Nancy Mayo
- McGill University, Montreal, QC, Canada
- Center for Outcomes Research and Evaluation (CORE), Research Institute of McGill University Health Center, Montreal, QC, Canada
| | - Annie Rochette
- Centre for Interdisciplinary Research in Rehabilitation of Greater Montreal, Montreal, QC, Canada
- Université de Montréal, Montreal, QC, Canada
| | - Aliki Thomas
- McGill University, Montreal, QC, Canada
- Centre for Interdisciplinary Research in Rehabilitation of Greater Montreal, Montreal, QC, Canada
- Centre for Medical Education, Montreal, Canada
| |
Collapse
|
3
|
Beaves M, Wallace E, Zoanetti N, Griffin P, Wu M. Graphical Item Maps: providing clearer feedback on professional exam performance. MEDEDPUBLISH 2018; 7:116. [PMID: 38074561 PMCID: PMC10699391 DOI: 10.15694/mep.2018.0000116.1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/23/2024] Open
Abstract
This article was migrated. The article was marked as recommended. Background: Structured feedback is an important component of learning and assessment and is highly valued by candidates. Unfortunately, item specific feedback is generally not feasible for high stakes professional assessments due to the high cost of item development and the need to maintain stable assessment performance characteristics. In a high stakes assessment of fetal surveillance knowledge, we sought to use graphical item mapping to allow informative candidate feedback without compromising the item bank. Methods: We developed Graphical Item Maps (GIMs) to display individual candidate performance in the Royal Australian and New Zealand College of Obstetricians and Gynaecologists (RANZCOG) Fetal Surveillance Education Program (FSEP) multiple-choice question assessment. GIMs use item and person parameter estimates from Item Response Theory (IRT) models to map the interaction between a test taker and assessment tasks of varying difficulty. Results: It is both feasible and relatively simple to provide GIMs for individual candidate feedback. Operational examples are presented from the RANZCOG FSEP assessment. This paper demonstrates how test takers and educators might use GIMs as a form of assessment feedback. Conclusions: Graphical Item Maps are a useful and insightful assessment feedback tool for clinical practitioners partaking in a high stakes professional education and assessment program. They might be usefully employed in similar healthcare professional assessments to inform directed learning.
Collapse
Affiliation(s)
- Mark Beaves
- Royal Australian and New Zealand College of Obstetricians and Gynaecologists
| | | | | | | | | |
Collapse
|
4
|
Thellesen L, Bergholt T, Hedegaard M, Colov NP, Christensen KB, Andersen KS, Sorensen JL. Development of a written assessment for a national interprofessional cardiotocography education program. BMC MEDICAL EDUCATION 2017; 17:88. [PMID: 28521768 PMCID: PMC5437628 DOI: 10.1186/s12909-017-0915-2] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/13/2016] [Accepted: 04/26/2017] [Indexed: 06/07/2023]
Abstract
BACKGROUND To reduce the incidence of hypoxic brain injuries among newborns a national cardiotocography (CTG) education program was implemented in Denmark. A multiple-choice question test was integrated as part of the program. The aim of this article was to describe and discuss the test development process and to introduce a feasible method for written test development in general. METHODS The test development was based on the unitary approach to validity. The process involved national consensus on learning objectives, standardized item writing, pilot testing, sensitivity analyses, standard setting and evaluation of psychometric properties using Item Response Theory models. Test responses and feedback from midwives, specialists and residents in obstetrics and gynecology, and medical and midwifery students were used in the process (proofreaders n = 6, pilot test participants n = 118, CTG course participants n = 1679). RESULTS The final test included 30 items and the passing score was established at 25 correct answers. All items fitted a loglinear Rasch model and the test was able to discriminate levels of competence. Seven items revealed differential item functioning in relation to profession and geographical regions, which means the test is not suitable for measuring differences between midwives and physicians or differences across regions. In the setting of pilot testing Cronbach's alpha equaled 0.79, whereas Cronbach's alpha equaled 0.63 in the setting of the CTG education program. This indicates a need for more items and items with a higher degree of difficulty in the test, and illuminates the importance of context when discussing validity. CONCLUSIONS Test development is a complex and time-consuming process. The unitary approach to validity was a useful and applicable tool for development of a CTG written assessment. The process and findings supported our proposed interpretation of the assessment as measuring CTG knowledge and interpretive skills. However, for the test to function as a high-stake assessment a higher reliability is required.
Collapse
Affiliation(s)
- Line Thellesen
- Department of Obstetrics, The Juliane Marie Centre for Children, Women and Reproduction, Rigshospitalet, University of Copenhagen, Blegdamsvej 9, DK-2100 Copenhagen, Denmark
| | - Thomas Bergholt
- Department of Obstetrics, The Juliane Marie Centre for Children, Women and Reproduction, Rigshospitalet, University of Copenhagen, Blegdamsvej 9, DK-2100 Copenhagen, Denmark
| | - Morten Hedegaard
- Department of Obstetrics, The Juliane Marie Centre for Children, Women and Reproduction, Rigshospitalet, University of Copenhagen, Blegdamsvej 9, DK-2100 Copenhagen, Denmark
| | - Nina Palmgren Colov
- Department of Obstetrics, The Juliane Marie Centre for Children, Women and Reproduction, Rigshospitalet, University of Copenhagen, Blegdamsvej 9, DK-2100 Copenhagen, Denmark
| | - Karl Bang Christensen
- Section of Biostatistics, Department of Public Health, University of Copenhagen, Oester Farimagsgade 5, Building 15.2.12, DK-1014 Copenhagen, Denmark
| | - Kristine Sylvan Andersen
- Department of Obstetrics, The Juliane Marie Centre for Children, Women and Reproduction, Rigshospitalet, University of Copenhagen, Blegdamsvej 9, DK-2100 Copenhagen, Denmark
| | - Jette Led Sorensen
- Department of Obstetrics, The Juliane Marie Centre for Children, Women and Reproduction, Rigshospitalet, University of Copenhagen, Blegdamsvej 9, DK-2100 Copenhagen, Denmark
| |
Collapse
|
5
|
Brown LD, Permezel M, Holberton JR, Whitehead CL. Neonatal outcomes after introduction of a national intrapartum fetal surveillance education program: a retrospective cohort study. J Matern Fetal Neonatal Med 2016; 30:1777-1781. [PMID: 27534984 DOI: 10.1080/14767058.2016.1224839] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
Abstract
OBJECTIVE To determine the impact of a multidisciplinary fetal surveillance education program (FSEP) on term neonatal outcomes. METHODS A retrospective cohort study of term neonatal outcomes before (1998-2004) and after (2005-2010) introduction of a FSEP. Clinical data was collected for all term infants admitted to a neonatal intensive care unit (NICU) in Australia between 1998 and 2010. Infants with congenital abnormalities were excluded. Neonatal mortality and severe neonatal morbidity (admission to a NICU, respiratory support, hypoxic encephalopathy) were compared before and after the FSEP was introduced. The rates of operative delivery during this time were assessed. RESULTS There were 3 512 596 live term births between 1998 and 2010. The intrapartum hypoxic death rate at term decreased from 2.02 to 1.07 per 10 000 total births. More neonates were admitted to NICU after 2005 (10.6 versus 14.6 per 10 000 live births), however fewer babies admitted to the neonatal unit had Apgar scores < 5 at five minutes (55.1-45.5%, RR 0.82, 95% CI 0.7-0.87); and rates of hypoxic ischemic encephalopathy fell from 36% to 30% (RR 0.83, 95% CI 0.76-0.90). There was no increase in rates of emergency in labour caesarean sections (11.7% pre versus 11.1% post, RR 0.95, 95% CI 0.95-0.96). CONCLUSIONS Introduction of a national FSEP was associated with increased neonatal admissions but a reduction in intrapartum hypoxia, without increasing emergency caesarean section rates.
Collapse
Affiliation(s)
- L D Brown
- a Department of Obstetrics and Gynaecology , Mercy Hospital for Women , Heidelberg , Victoria , Australia
| | - M Permezel
- a Department of Obstetrics and Gynaecology , Mercy Hospital for Women , Heidelberg , Victoria , Australia.,b Department of Obstetrics and Gynaecology , University of Melbourne, Mercy Hospital for Women , Heidelberg , Victoria , Australia , and
| | - J R Holberton
- c On behalf of the Australian and New Zealand Neonatal Network , Lismore , NSW , Australia
| | - C L Whitehead
- b Department of Obstetrics and Gynaecology , University of Melbourne, Mercy Hospital for Women , Heidelberg , Victoria , Australia , and
| |
Collapse
|
6
|
Zoanetti N, Beaves M, Griffin P, Wallace EM. Fixed or mixed: a comparison of three, four and mixed-option multiple-choice tests in a Fetal Surveillance Education Program. BMC MEDICAL EDUCATION 2013; 13:35. [PMID: 23453056 PMCID: PMC3599143 DOI: 10.1186/1472-6920-13-35] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 06/22/2011] [Accepted: 02/27/2013] [Indexed: 06/01/2023]
Abstract
BACKGROUND Despite the widespread use of multiple-choice assessments in medical education assessment, current practice and published advice concerning the number of response options remains equivocal. This article describes an empirical study contrasting the quality of three 60 item multiple-choice test forms within the Royal Australian and New Zealand College of Obstetricians and Gynaecologists (RANZCOG) Fetal Surveillance Education Program (FSEP). The three forms are described below. METHODS The first form featured four response options per item. The second form featured three response options, having removed the least functioning option from each item in the four-option counterpart. The third test form was constructed by retaining the best performing version of each item from the first two test forms. It contained both three and four option items. RESULTS Psychometric and educational factors were taken into account in formulating an approach to test construction for the FSEP. The four-option test performed better than the three-option test overall, but some items were improved by the removal of options. The mixed-option test demonstrated better measurement properties than the fixed-option tests, and has become the preferred test format in the FSEP program. The criteria used were reliability, errors of measurement and fit to the item response model. CONCLUSIONS The position taken is that decisions about the number of response options be made at the item level, with plausible options being added to complete each item on both psychometric and educational grounds rather than complying with a uniform policy. The point is to construct the better performing item in providing the best psychometric and educational information.
Collapse
Affiliation(s)
- Nathan Zoanetti
- Assessment Research Centre, Melbourne Graduate School of Education, University of Melbourne, Parkville, Australia
| | - Mark Beaves
- Royal Australian and New Zealand College of Obstetricians and Gynaecologists, East Melbourne, Australia
| | - Patrick Griffin
- Assessment Research Centre, Melbourne Graduate School of Education, University of Melbourne, Parkville, Australia
| | - Euan M Wallace
- Department of Obstetrics and Gynaecology, The Ritchie Centre, Monash Institute of Medical Research and Southern Clinical School, Monash University, Clayton, Australia
| |
Collapse
|