[Systematic analysis of evaluations of Objective Structured Clinical Exam (OSCE) multiple circuits: Explanatory variables and inter-rater correlations]

INTRODUCTION

Objective Structured Clinical Examinations (OSCEs) assess professional performance in a simulated environment. Following their integration into the reform of the 2nd cycle of medical studies (R2C), this pedagogical modality was implemented in France. This study investigates the variability of students' OSCE scores, as well as their inter-rater reproducibility.

METHODS

This single-center retrospective study covered several sessions of evaluative OSCE circuits conducted between January 2022 and June 2023. Variables collected were: baseline situation family, competency domain, presence of a standardized participant for stations; gender and professional status for evaluators; scores (global, clinical and communication skills), number of previously completed OSCE circuits and faculty scores for students.

RESULTS

The variability of the overall score was explained mainly (79.7%, CI95% [77.4; 82.0]) by the station factor. The student factor and the circuit factor explained 7.5% [12.9; 20.2] and<0.01% [2.10-13; 2.10-9] respectively. The inter-rater intra-class correlation coefficient was 87.2% [86.4; 87.9] for the global score. Station characteristics (starting situation, domain) and evaluator characteristics (gender, status) were significantly associated with score variations.

CONCLUSION

This first study on the variability of OSCE circuit scores in France shows good reproducibility with influence of station characteristics. In order to standardize circuits, variability linked to the domain competency should be considered as well.

[Impact of sonography simulation in the training of midwifery students]

OBJECTIVE

During their studies, French midwives need to acquire basic ultrasound skills to measure the amniotic fluid index, fetal biometry and identify fetal lie and placental positions. They have a clinical ultrasound training during their curriculum to acquire these skills. Assess the contribution that a training on an obstetrical ultrasound simulator (OUS) could make to the competence of midwives and to evaluate the best time to integrate it into their curriculum.

METHODS

Forty students in their 4th and final year in midwifery were randomized into 2 groups. Group A learned with the traditional academic course, with a clinical ultrasound training which was followed by an evaluation on an OUS (E1A). Group B was evaluated on a simulator before the clinical ultrasound training (E1B). After these initial assessments, both groups were provided with learning and training sessions on OUS, and re-evaluated in the same exercise (E2A and E2B). Group B then completed its clinical ultrasound training and was evaluated one month later on the simulator (E3B). The evaluation consisted of scoring the biometry images according to previously published quality criteria, and of assessing their competence based on a published objective and structured assessment score of ultrasound skills (OSAUS).

RESULTS

Evaluation 1 found comparable OSAUS scores (2.375/5 vs. 2.24/5 P=0.52) between the two groups (E1A vs. E1B) but a significantly higher image quality score for Group A (P=0.02). In Evaluation 2, an improvement was noted for both groups with comparable OSAUS averages between the two groups (P=0.76). After the clinical training of Group B, their OSAUS score (E3B) was significantly better than that of Group A (E1A) after their clinical training (3.45/5 vs. 2.375/5 [P=0.00017]). The same was true for the quality of their images (12.67/16 vs. 9.95/16 [P=0.003]).

CONCLUSION

Ultrasound simulation-based training, as an adjunct to ultrasound clinical training, significantly improves obstetrical ultrasound skills. The best time to train on simulators seems to be prior to clinical training on real patients.

[Analysis of the discrimination of the final marks after the first computerized national ranking exam in Medicine in June 2016 in France]

INTRODUCTION

The first computerised national ranking exam (cNRE) in Medicine was introduced in June 2016 for 8214 students. It was made of 18 progressive clinical cases (PCCs) with multiple choice questions (MCQs), 120 independent MCQs and 2 scientific articles to criticize. A lack of mark discrimination grounded the cNRE reform. We aimed to assess the discrimination of the final marks after this first cNRE.

METHODS

A national Excel^® file gathering overall statistics and marks were transmitted to the medical faculties after the cNRE. The mean points deviation between two papers and the percentage of points ranking 75% of students allowed us to analyse marks' discrimination.

RESULTS

The national distribution sigmoid curve of the marks is superimposable with previous NRE in 2015. In PCCs, 72% of students were ranked in 1090 points out of 7560 (14%). In independents MCQs, 73% of students were ranked in 434 points out of 2160 (20%). In critical analysis of articles, 75% of students were ranked in 225 points out of 1080 (21%). The above percentages of students are on the plateau of each discrimination curve for PCCs, independent MCQs and critical analysis of scientific articles.

CONCLUSION

The cNRE reduced equally-ranked students compared to 2015, with a mean deviation between two papers of 0.28 in 2016 vs 0.04 in 2015. Despite the new format introduced by the cNRE, 75% of students are still ranked in a low proportion of points that is equivalent to previous NRE in 2015 (between 15 et 20% of points).

[Educational assessment of the first computerized national ranking exam in France in 2016: Opportunities for improvement]

In June 2016, 8124 medical students in their sixth year of graduation passed the first computerized national ranking exam (CNRE) in France after which they will have to choose what medical specialty they will be practicing all their life. We conducted the first educational assessment of this CNRE according to two criteria: the relevance of the questions and the cognitive domain mainly required to answer these questions. We propose two improvements for the future CNRE: promote student reasoning in the multiple choices questions, reduce to 10 the number of multiple choice questions in the progressive clinical cases and increase by 9 their total number (from 18 to 27), and use a majority of mini-clinical cases for isolated multiple choice questions in order to focus students on reasoning instead of simple knowledge restitution.

[Learning the gynecological examination on low-fidelity simulation: Impact on the feelings of medical students]

OBJECTIVES

The use of anatomic model is particularly suitable for teaching gynecological examination. Although simulation training was proved to be effective in improving medical students' technical skills, its interest in reducing their apprehension about pelvic examination was poorly evaluated. The aim of the present study was to evaluate low-fidelity simulation sessions to reduce the discomfort and apprehension of medical students towards gynecological examination. The secondary objective was to appreciate student satisfaction with these sessions.

METHODS

Descriptive analysis of simulation-based teaching of breast and pelvic examination on Ilumens platform at the university Paris 5 among 3rd and 4th year medical students. It consisted in a before/after evaluation, using an anonymous questionnaire of self-evaluation and satisfaction at the beginning and at the end of the session.

RESULTS

Four hundred and thirty-two students answered both questionnaires. Seventy-two percent apprehended the pelvic examination initially and 62% were uncomfortable with the idea of performing it. Among the students who had never performed pelvic examination, more than 80% felt less uncomfortable with the idea of performing pelvic examination and were less apprehensive about it at the end of the simulation session.

CONCLUSION

The benefits of low-fidelity simulation in gynecology extend to the well-being of medical students who are less apprehensive about performing gynecological examination.

[Script concordance test for knowledge assessment of lung cancer screening]

BACKGROUND

Annual screening for lung cancer using low-dose CT-scans is associated with decreased mortality. A survey conducted in Rhône-Alpes area in France found that clinicians need education and information on this topic. Script concordance tests (SCT) are a tool for assessing clinical reasoning in situations of uncertainty. They have not previously been used in France in the context of continuing medical education.

METHOD

We created a questionnaire with 5 multiple-choice questions (MCQ) and two SCT scenarios. The questionnaire was sent to all clinicians and residents who are members of French-Speaking Respiratory Society or the French Young Pulmonologist Association.

RESULTS

One hundred and ninety answers were analyzed. Seventy percent stated that decreasing mortality was the best criterion for assessing the effectiveness of a cancer screening policy, and 75% that low-dose CT scan was the best test to achieve this in lung cancer screening. Forty-five percent knew the eligibility criteria of the population, and 62% that low-dose CT scan should be performed annually. Participation in tumor boards and certification in oncology were significantly associated with a better score at MCQ and SCT. SCT and MCQ scores were significantly correlated (Spearman's Rho 0.339; P<0.0001).

CONCLUSION

SCT are feasible by electronic survey and seem relevant. Improving knowledge of clinicians on lung cancer screening is still critical.