Test of integrated professional skills: objective structured clinical examination/simulation hybrid assessment of obstetrics-gynecology residents' skill integration

Perspectives and experiences of newly registered nurses surrounding the objective structured clinical examination in an obstetrics and gynecology hospital: A qualitative study in Shanghai, China

OBJECTIVES

Newly registered nurses in China are required to attend two years of standardized training programs after graduation, and an evaluation of the training program's effectiveness is critical. The objective structured clinical examination is a relatively new and objective approach to exploring the effectiveness of training programs and is increasingly being encouraged and used in clinics. However, the perspectives and experiences of newly registered nurses in obstetrics and gynecology regarding the objective structured clinical examination are unclear. Therefore, the objective of this study was to investigate newly registered nurses' perspectives and experiences of the objective structured clinical examination in an obstetrics and gynecology hospital.

DESIGN

This qualitative study was conducted using a phenomenological approach.

DATA SOURCES

Twenty-four newly registered nurses taking the objective structured clinical examination in a third-level obstetrics and gynecology hospital in Shanghai, China.

REVIEW METHODS

Semi-structured face-to-face interviews were conducted between July and August 2021. The Colaizzi seven-step framework was applied for data analysis.

RESULTS

Three main themes and six sub-themes emerged: 1) high satisfaction with the objective structured clinical examination; 2) gaining experience and growing as nurses; and 3) high pressure.

CONCLUSION

The objective clinical structured examination can be used to assess the competence of newly registered nurses after training in an obstetrics and gynecology hospital. The examination not only enables an objective and comprehensive evaluation of others and self-evaluation but also leads to positive psychological experiences in newly registered nurses. However, interventions are needed to relieve examination pressure and to provide effective support for participants. The objective clinical structured examination can be incorporated into the training assessment system; this study provides a basis for improving training programs and the training of newly registered nurses.

The OSPE

OBJECTIVES

Resident assessment tends to consist of multiple-choice examinations, even in nuanced areas, such as quality assurance. Internal medicine and many other specialties use objective structured clinical examinations, or OSCEs, to evaluate residents. We adapted the OSCE for pathology, termed the Objective Structured Pathology Examination (OSPE).

METHODS

The OSPE was used to evaluate first- and second-year residents over 2 years. The simulation included an anatomic pathology sign-out session, where the resident could be evaluated on diagnostic skills and knowledge of key information for cancer staging reports, as well as simulated frozen-section analysis, where the resident could be evaluated on communication skills with a "surgeon." The OSPE also included smaller cases with challenging quality issues, such as mismatched slides or gross description irregularities. All cases were scored based on the Pathology Milestones created by the Accreditation Council for Graduate Medical Education.

RESULTS

Using this OSPE, we were able to demonstrate that simulated experiences can be an appropriate tool for standardized evaluation of pathology residents.

CONCLUSIONS

Yearly evaluation using the OSPE could be used to track the progress of both individual residents and the residency program as a whole, identifying problem areas for which further educational content can be developed.

Evaluating shared decision-making in periviable counseling using objective structured clinical examinations

OBJECTIVE

To develop and test an Objective Structured Clinical Examination to evaluate the use of shared decision-making (SDM) in periviable counseling among fourth-year OB/GYN residents.

METHODS

Residents counseled a standardized patient presenting with preterm premature rupture of membranes at 23 weeks' gestation. Braddock's 9-item measure of SDM was adapted to a 10-item scoring rubric; rating each: 0 (absent), 1 (partial), or 2 (complete).

RESULTS

Twenty-six residents participated. All provided "complete" discussions of the clinical issue and "complete" or 'partial' ratings for informing the woman of her prognosis (62 and 38%, respectively) and addressing her role in decision-making (42 and 50%). Discussions of her goals and preferences were often absent (69 and 62%). Only 42% discussed uncertainties.

CONCLUSION

Critical elements of SDM related to a woman's values, goals and preferences were not explored when counseling about periviable delivery. Training in SDM is needed to advance communication skills for complex clinical decision-making.

The pediatric rheumatology objective structured clinical examination: progressing from a homegrown effort toward a reliable and valid national formative assessment

BACKGROUND

Of 37 pediatric rheumatology fellowship training programs in the United States, many have three or fewer fellows at a given time, making large-scale assessment of fellow performance difficult. An objective structured clinical examination (OSCE) is a scenario-based simulation method that assesses individual performance, thus indirectly measuring training program effectiveness. This study describes the development and implementation of two national pediatric rheumatology OSCEs and methods used for programmatic improvement.

METHODS

OSCEs for pediatric rheumatology fellows were held in 2009 and 2011 during national rheumatology meetings using scenarios and assessment forms originally developed by a fellowship program director. The seven scenarios tested medical knowledge, physical exam and interpersonal skills. Pediatric rheumatologist evaluators assessed fellows' performance using checklists and gave immediate feedback. Program directors were sent summaries of their fellows' performances. Fellows evaluated the OSCE, providing organizational and scenario improvement suggestions. Programmatic changes to the 2011 OSCE were based on 2009 performance data and program evaluation feedback.

RESULTS

Twenty-two fellows participated in 2009 and 19 in 2011. Performance scores in similar scenarios did not change considerably over the two iterations. In 2009, 85.7% of participants reported desire to change clinical behavior. Assessors' 2009 program evaluation data prompted changes in rating scales and removal of invalid or unreliable assessments. Negative evaluation data about individual stations decreased from 60% in 2009 to 15.4% in 2011. Fellows' ratings of the experience's overall value were similar in 2009 and 2011. The average experience ratings were lower among fellows who proposed scenario-specific improvements and higher among those who recommended organizational improvements.

CONCLUSIONS

The 2011 examination exhibited programmatic improvement via reduction in fellows' scenario-specific negative feedback. Fellows' overall satisfaction did not change. Further work in scenario selection, assessment validation and inter-rater reliability will improve future pediatric rheumatology OSCEs.

Assessment of Developmental Progress Using an Objective Structured Clinical Examination-Simulation Hybrid Examination for Obstetrics and Gynecology Residents

OBJECTIVE

The Test of Integrated Professional Skills (TIPS) is an objective structured clinical examination-simulation hybrid examination that assesses resident integration of technical, cognitive, and affective skills in Obstetrics and Gynecology (OBGYN) residents. The aim of this study was to analyze performance patterns and reactions of residents to the test to understand how it may fit within a comprehensive assessment program.

DESIGN

A retrospective, mixed methods review of the design and implementation of the examination, patterns of performance of trainees at different levels of training, focus group data, and description of use of TIPS results for resident remediation and curriculum development.

SETTING

OBGYN residents at New York University Langone Medical Center, a tertiary-care, urban academic health center.

PARTICIPANTS

OBGYN residents in all years of training, postgraduate year-1 through postgraduate year, all residents completing the TIPS examination and consenting to participate in focus groups were included.

RESULTS

In all, 24 residents completed the TIPS examination. Performance on the examination varied widely among individuals at each stage of training, and did not follow developmental trends, except for technical skills. Cronbach α for both standardized patient and faculty ratings ranged from 0.69 to 0.84, suggesting internal consistency. Focus group results indicated that residents respond to the TIPS examination in complex ways, ranging from anxiety about performance to mixed feelings about how to use the data for their learning.

CONCLUSION

TIPS assesses a range of attributes, and can support both formative and summative evaluation. Lack of clear developmental differences and wide variation in performance by learners at the same level of training support the argument for individualized learning plans and competency-based education.

Notes from the Field: Residents' Perceptions of Simulation-Based Skills Assessment in Obstetrics and Gynecology

Simulation in obstetrics and gynecology (OBGYN) training captures a range of interpersonal, cognitive, and technical skills. However, trainee perspectives on simulation-based assessment remain unexplored. After an observed structured clinical examination (OSCE) simulation hybrid exam, two focus groups of residents were conducted. Analysis grounded in a thematic coding guided the qualitative research process. Responses suggest a valuation of cognitive and technical skills over interpersonal skills. Realism was seen as critical and residents perceived the assessment as more valuable for the educator than the learner. Feedback was highly valued. Resident perspectives on this exam give insight into their perceptions of simulation-based assessment as well as their conceptions of their own learning through simulations.