Is ChatGPT reliable and accurate in answering pharmacotherapy-related inquiries in both Turkish and English?

INTRODUCTION

Artificial intelligence (AI), particularly ChatGPT, is becoming more and more prevalent in the healthcare field for tasks such as disease diagnosis and medical record analysis. The objective of this study is to evaluate the proficiency and accuracy of ChatGPT in different domains of clinical pharmacy cases and queries.

METHODS

The study NAPLEX® Review Questions, 4th edition, pertaining to 10 different chronic conditions compared ChatGPT's responses to pharmacotherapy cases and questions obtained from McGraw Hill's, alongside the answers provided by the book's authors. The proportion of correct responses was collected and analyzed using the Statistical Package for the Social Sciences (SPSS) version 29.

RESULTS

When tested in English, ChatGPT had substantially higher mean scores than when tested in Turkish. The average accurate score for English and Turkish was 0.41 ± 0.49 and 0.32 ± 0.46, respectively, p = 0.18. Responses to queries beginning with "Which of the following is correct?" are considerably more precise than those beginning with "Mark all the incorrect answers?" 0.66 ± 0.47 as opposed to 0.16 ± 0.36; p = 0.01 in English language and 0.50 ± 0.50 as opposed to 0.14 ± 0.34; p < 0.05in Turkish language.

CONCLUSION

ChatGPT displayed a moderate level of accuracy while responding to English inquiries, but it displayed a slight level of accuracy when responding to Turkish inquiries, contingent upon the question format. Improving the accuracy of ChatGPT in languages other than English requires the incorporation of several components. The integration of the English version of ChatGPT into clinical practice has the potential to improve the effectiveness, precision, and standard of patient care provision by supplementing personal expertise and professional judgment. However, it is crucial to utilize technology as an adjunct and not a replacement for human decision-making and critical thinking.

Publication Fate of Research Works Achieved During Master's Degree in Surgical Science in a French Survey

The aim of this study was to assess the publication fate of research projects performed during the research year by students enrolled in a Master's degree (MSc) of surgical science and to identify factors associated with subsequent publication. An anonymous online survey of 35 questions was sent to students enrolled in MSc of surgical science between 2013 and 2020. The questionnaire included student's characteristics, topic, and supervision of the research projects developed during the research year and dissemination of the research work. Data regarding publication was collected using PubMed database. Factors associated with publication were identified by univariate analysis. Among 361 students, 26% completed the survey. Among respondents, the publication rate of research projects was 53.7%. The median time interval between the end of the research year and the date of publication was 2 (1-3) years. The student was listed as a first author in 70.6% of publications. Factors associated with publication of the research work completed during the research year were student's previous publications (P = 0.041) and presentation of the research work in academic conferences (P = 0.005). The most mentioned cause for non-publication was the absence of completion of the research work. Among respondents, the publication rate of research works performed during the MSc was high, which emphasizes the quality of the work carried out by the students and their involvement. Significant efforts must be undertaken to encourage the enrollment of residents in scientific research.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-023-01973-y.

Impact of a university teaching of integrative medicine on the social representations of undergraduate medical students

Integrative medicine, need to be inoffensive, effective, and of quality (World Health Organization). In 2010, the American Society of Teachers of Family Medicine approved 19 competencies for teaching integrative medicine to residents. In 2018, the University of Rennes created a course: "Integrative Medicine and Complementary Therapies". Up until then, the only feedback from the courses was the students' opinions. We investigated the impact on medical students' social representation.We performed a sociological analysis of students' social representations before and after the course. The social representation is based on the way an individual creates his or her universe of beliefs and ideas. After hearing, "What word or group of words comes to mind when you hear people speak of integrative medicine and complementary therapies?", students were asked to provide 5 words/phrases, rank their importance, and show their attitude towards these words/phrases. The frequency and importance of these words/phrases were used to construct social representations (with central cores, and primary and secondary peripheries) before and after the course.Among the 101 students registered, 59 provided complete responses before and 63 after the course. Before, the central core comprised "hypnosis" and "alternative medicine", while after: "complementary care" and "global care". We only identified first periphery before the course: "acupuncture" and "homeopathy". 4 new contrasting elements: "integration with conventional treatment", "patient's choice", "personalisation of care", and "caring relationship of trust".This teaching course positively affected students' social representation of integrative medicine, and might promote their use during future practices.

Translation and validation of the Persian version of the perception to care in acute situations (PCAS-P) scale in novice nurses

BACKGROUND

Novice nurses providing care in acute conditions should have satisfactory performance. Accurate and appropriate evaluation of the performance of novice nurses in providing care in acute situations is essential for planning interventions to improve the quality of patient care. This study was conducted to translate and evaluate the psychometric properties of the Persian version of the Perception to Care in Acute Situations (PCAS-P) scale in novice nurses.

METHODS

In this methodological study, 236 novice nurses were selected by the convenience sampling method. 17-item scale PCAS-P was translated into Persian by the forward-backward process. Then, this version was used for psychometric evaluation. For this purpose, face validity, content validity, and construct validity were assessed using confirmatory factor analysis. Internal consistency and stability reliability were calculated. The data were analyzed using SPSS and AMOS software.

RESULTS

The PCAS-P scale maintained the meaning of the original English version and was clear, explicit, and understandable for novice nurses. Confirmatory factor analysis showed that this Persian version is consistent with the proposed model and confirmed the fit of the three-factor model. The values of Cronbach's alpha coefficient, McDonald's omega, Coefficient H, and average inter-item correlation were excellent for the overall scale and its dimensions, and the three latent factors had good convergent and discriminant validity. Additionally, the average measurement size was 0.944 ICC (95% CI 0.909 to 0.969).

CONCLUSION

The PCAS-P scale is valid and reliable for measuring novice nurses' perception of acute situations.

Preliminary Development of a Program ABSITE Dashboard (PAD) to Guide Curriculum Innovation

PURPOSE

Medical Knowledge for general surgery residents' is assessed by the American Board of Surgery In- Training Examination (ABSITE). ASBITE score reports contain many metrics residency directors can utilize to assess resident progress and perform program evaluation. The purpose of this study was to develop a framework to evaluate program effectiveness in teaching specific subtest and subtopic areas of the ABSITE, using ABSITE score reports as an indicator. The aim is to demonstrate the identification of topic areas of weakness in program-wide performance on the ABSITE to guide proposed modification of the general surgery residency program curriculum, and to initiate development of a data visualizing dashboard to communicate these metrics.

METHODS

A single institution retrospective study was performed utilizing ABSITE score reports from general surgery residents at a large academic training program from 2017 to 2020. ABSITE performance metrics from 320 unique records were entered into a database; statistical analysis for linear trends and variance were conducted for standard scores, subtest standard scores, and incorrect subtest topics. Deviation from national average scores were calculated by subtracting the national average score from each subtest score for each trainee. Data were displayed as medians or proportions and are displayed to optimize visualization as a proof-of-concept for the development of a program dashboard.

RESULTS

Trends and variance in general surgery program and cohort performance on various elements of the ABSITE were visualized using figures and tables that represent a prototype for a program dashboard. Figure A1 demonstrates one example, in which a heatmap displays the median deviation from national average scores for each subtest by program year. Boxplots show the distribution of the deviation from national average, range for national average scores, and the recorded scores for each subtest by program year. Trends in median deviation from the national average scores are displayed for each program year paneled by subtest or for each exam year paneled by cohort. Median change in overall test scores from one program year to another in a cohort is visualized as a table. Bar graphs show the most often missed topics across all program years and heatmaps were generated showing the proportion of times each topic was missed for each subtest and exam year.

CONCLUSIONS

We demonstrate use of ABSITE reports to identify specific thematic areas of opportunities for curriculum modification and innovation as an element of program evaluation. In this study we demonstrate, through data analysis and visualization, feasibility for the creation of a Program ABSITE Dashboard (PAD) that enhances the use of ABSITE reports for formative program evaluation and can guide modifications to surgery program curriculum and educational practices.

Gender differences in self-assessed performance and stress level during training of basic interventional radiology maneuvers

OBJECTIVES

Gender differences have been reported to influence medical training. We investigated gender differences encountered during training in interventional radiology maneuvers.

METHODS

Catheter handling was analyzed under standardized conditions in 64 participants naïve to endovascular procedures (26 women, 38 men). Objective (e.g., catheter pathway, catheter movements, required time) and subjective parameters (stress level) were recorded. The NASA-Task Load Index (NASA-TLX; 1-20 points) was used to assess participants' stress levels and perceived workload.

RESULTS

In the easier tasks, no significant differences between male and female participants regarding catheter handling were observed. In the most complex task, female participants took themselves more time (688 ± 363 vs. 501 ± 230 s; p = 0.02), asked for help more frequently (n = 19 vs. n = 8) and earlier than men (203 ± 94 vs. 305 ± 142 s; p = 0.049), whereas men stood out by more agitated catheter handling (6.0 ± 1.8 vs. 4.8 ± 1.6 movements/s; p = 0.005). Overall, female participants perceived tasks to be more difficult (11.5 ± 4.2 vs. 9.6 ± 3.3; p = 0.016), perceived higher stress levels (8.9 ± 4.9 vs. 6.3 ± 4.4; p = 0.037), and rated their own performance lower (9.12 ± 3.3 vs. 11.3 ± 3.3; p = 0.009). However, female participants were able to correlate self-assessed with objective parameters correctly (r between -0.555 and -0.469; p = 0.004-0.018), whereas male participants failed to correctly rate their performance (p between 0.34 and 0.73). Stress levels correlated with objective parameters in males (r between 0.4 and 0.587; p < 0.005), but not in female participants.

CONCLUSION

Perceived stress levels, self-evaluation skills, and catheter handling differ greatly between untrained male and female participants trying to solve interventional radiological tasks. These gender-specific differences should be considered in interventional radiology training.

CLINICAL RELEVANCE STATEMENT

As psychological aspects may influence individual working strategies gender-specific differences in self-perception while learning interventional radiology maneuvers could be essential regarding success in teaching and treatment outcomes.

KEY POINTS

• After performing standardized training, 38 male and 26 female volunteers showed significant differences regarding objective and self-assessed performance, as well as in perceived workload while performing simulated endovascular catheter maneuvers. • After solving simulated endovascular radiological tasks, female participants were able to self-assess their objective performance much more accurately than male participants. • Women took more time to solve simulated endovascular tasks and asked earlier and more frequently for help than men.

Modeling item-level heterogeneous treatment effects: A tutorial with the glmer function from the lme4 package in R

Recent advancements in education scholarship have introduced Item Response Theory (IRT) models to address treatment heterogeneity at the assessment item level. These models for item-level heterogeneous treatment effects (IL-HTE) enable detailed analyses of treatments that may have varying impacts on individual items within an assessment. This article offers a comprehensive tutorial for applied researchers interested in implementing IL-HTE analysis in R, utilizing the lme4 package. Using empirical data from a second-grade reading comprehension assessment as a running example, this tutorial emphasizes model-building strategies, interpretation techniques, visualization methods, and extensions. By following this tutorial, researchers will gain practical insights into utilizing IL-HTE analysis for enhanced understanding and interpretation of treatment effects at the item level.

Item difficulty index, discrimination index, and reliability of the 26 health professions licensing examinations in 2022, Korea: a psychometric study

PURPOSE

This study presents item analysis results of the 26 health personnel licensing examinations managed by the Korea Health Personnel Licensing Examination Institute (KHPLEI) in 2022.

METHODS

The item difficulty index, item discrimination index, and reliability were calculated. The item discrimination index was calculated using a discrimination index based on the upper and lower 27% rule and the item-total correlation.

RESULTS

Out of 468,352 total examinees, 418,887 (89.4%) passed. The pass rates ranged from 27.3% for health educators level 1 to 97.1% for oriental medical doctors. Most examinations had a high average difficulty index, albeit to varying degrees, ranging from 61.3% for prosthetists and orthotists to 83.9% for care workers. The average discrimination index based on the upper and lower 27% rule ranged from 0.17 for oriental medical doctors to 0.38 for radiological technologists. The average item-total correlation ranged from 0.20 for oriental medical doctors to 0.38 for radiological technologists. The Cronbach α, as a measure of reliability, ranged from 0.872 for health educators-level 3 to 0.978 for medical technologists. The correlation coefficient between the average difficulty index and average discrimination index was -0.2452 (P=0.1557), that between the average difficulty index and the average item-total correlation was 0.3502 (P=0.0392), and that between the average discrimination index and the average item-total correlation was 0.7944 (P<0.0001).

CONCLUSION

This technical report presents the item analysis results and reliability of the recent examinations by the KHPLEI, demonstrating an acceptable range of difficulty index and discrimination index values, as well as good reliability.

Performance of ChatGPT, Bard, Claude, and Bing on the Peruvian National Licensing Medical Examination: a cross-sectional study

PURPOSE

We aimed to describe the performance and evaluate the educational value of justifications provided by artificial intelligence chatbots, including GPT-3.5, GPT-4, Bard, Claude, and Bing, on the Peruvian National Medical Licensing Examination (P-NLME).

METHODS

This was a cross-sectional analytical study. On July 25, 2023, each multiple-choice question (MCQ) from the P-NLME was entered into each chatbot (GPT-3, GPT-4, Bing, Bard, and Claude) 3 times. Then, 4 medical educators categorized the MCQs in terms of medical area, item type, and whether the MCQ required Peru-specific knowledge. They assessed the educational value of the justifications from the 2 top performers (GPT-4 and Bing).

RESULTS

GPT-4 scored 86.7% and Bing scored 82.2%, followed by Bard and Claude, and the historical performance of Peruvian examinees was 55%. Among the factors associated with correct answers, only MCQs that required Peru-specific knowledge had lower odds (odds ratio, 0.23; 95% confidence interval, 0.09-0.61), whereas the remaining factors showed no associations. In assessing the educational value of justifications provided by GPT-4 and Bing, neither showed any significant differences in certainty, usefulness, or potential use in the classroom.

CONCLUSION

Among chatbots, GPT-4 and Bing were the top performers, with Bing performing better at Peru-specific MCQs. Moreover, the educational value of justifications provided by the GPT-4 and Bing could be deemed appropriate. However, it is essential to start addressing the educational value of these chatbots, rather than merely their performance on examinations.

Diversity, contradictions, and distinction - A scoping review of honours programmes in nursing

BACKGROUND

There is a long-standing tradition of honours education in the field of nursing, dating back to the early 1960s in the United States. However, its adoption in European and particularly Scandinavian egalitarian educational contexts is relatively recent.

PURPOSE

This scoping review aims to provide an analysis of the global utilisation and distribution of honours education within the field of nursing.

METHOD

In this scoping review, we conducted an extensive examination of the existing literature to assess the worldwide implementation of honours education in nursing. We employed a systematic approach to identify key trends, patterns, and commonalities in the use of the honours concept across different regions.

RESULTS

Our review reveals three predominant approaches to honours education, primarily concentrated in the Anglo-Saxon world: distinction programmes, add-on-year programmes, and embedded programmes. Regardless of the approach, our findings highlight a consistent lack of robust theoretical foundations, limited documentation supporting the educational impact, and a noticeable absence of standardisation. Instead, honours education appears to serve a symbolic and distinct purpose rather than a purely pedagogical one.

CONCLUSION

As the prevalence of honours education continues to rise within continental education systems, it becomes imperative to prioritize further research to ensure the optimal allocation of resources. Addressing the lack of evidence, especially in terms of educational value and theoretical foundations, is crucial for refining and maximizing the potential benefits of honours education in nursing. A more strategic and cohesive approach to developing honours programmes is essential to enhance their effectiveness and alignment with global educational goals.

Adaptation of the professionalism mini-evaluation exercise instrument into Turkish: a validity and reliability study

BACKGROUND

There is an ongoing search for standardized scales appropriate for each culture to evaluate professionalism, which is one of the basic competencies of a physician. The Professionalism Mini-evaluation Exercise (P-MEX) instrument was originally developed in Canada to meet this need. In this study, it was aimed to adapt the P-MEX to Turkish and to evaluate the validity and reliability of the Turkish version.

METHODS

A total of 58 residents at Bakirkoy Dr. Sadi Konuk Training and Research Hospital were assessed with the Turkish version of P-MEX by 24 raters consisting of faculty members, attending physicians, peer residents, and nurses during patient room visits, outpatient clinic and group practices. For construct validity, the confirmatory factor analysis was performed. For reliability, Cronbach's alpha scores were calculated. Generalizibility and decision studies were undertaken to predict the reliability of the validated tool under different conditions. After the administration of P-MEX was completed, the participants were asked to provide feedback on the acceptability, feasibility, and educational impact of the instrument.

RESULTS

A total of 696 forms were obtained from the administration of P-MEX. The content validity of P-MEX was found to be appropriate by the faculty members. In the confirmatory factor analysis of the original structure of the 24-item Turkish scale, the goodness-of-fit parameters were calculated as follows: CFI = 0.675, TLI = 0.604, and RMSEA = 0.089. In the second stage, the factors on which the items loaded were changed without removing any item, and the model was modified. For the modified model, the CFI, TLI, and RMSEA values were calculated as 0.857, 0.834, and 0.057, respectively. The decision study on the results obtained from the use of P-MEX in a Turkish population revealed the necessity to perform this evaluation 18 times to correctly evaluate professionalism with this instrument. Cronbach's alpha score was 0.844. All the faculty members provided positive feedback on the acceptability, feasibility, and educational impact of the adapted P-MEX.

CONCLUSION

The findings of this study showed that the Turkish version of P-MEX had sufficient validity and reliability in assessing professionalism among residents. Similarly, the acceptability and feasibility of the instrument were found to be high, and it had a positive impact on education.

TRIAL REGISTRATION

2020/249, Bakirkoy Dr. Sadi Konuk Training and Research Hospital.

Educational implications of assessing learning outcomes with multiple choice questions and short essay questions

PURPOSE

This study investigates the characteristics of different item types to assess learning outcomes and explore the educational implications that can be obtained from the results of learning outcome assessments.

METHODS

Forty-five second-year premedical students participated in this study. Multiple choice question (MCQ) and short essay question (SEQ) scores and pass rates for 10 learning outcomes were analyzed. Descriptive statistics and correlation analysis were used to analyze the data.

RESULTS

The correlation analysis indicated that there was a significant correlation between SEQs and pass rate but there was no significant correlation between MCQs and pass rate. Some students with identical scores on the MCQs had different scores on the SEQs or on the learning outcomes.

CONCLUSION

This study showed that students' achievement of learning outcomes can be assessed using various types of questions in outcome-based education.

Influence of educational intervention on nurse-midwives' knowledge and management practices of Intimate Partner Violence (IPV) in healthcare Facilities in Ekiti-State, Nigeria

OBJECTIVE

To assess the influence of an educational intervention on midwives' knowledge, detection and management of intimate partner violence (IPV).

METHODS

A quasi-experimental study involving 158 midwives from two districts in Ekiti State, Nigeria. The sample was divided into experimental and control groups (79 midwives per group). Data were collected using a questionnaire and an observation checklist. A customized educational training program on IPV detection and management was conducted in the experimental group. Measurement was performed before the intervention, immediately after and 6 weeks later. Data were analyzed using descriptive and inferential statistics (Chi-square and binary logistic regression) with a level of significance set at p < 0.05.

RESULTS

Prior knowledge regarding IPV detection and management among midwives in both groups was poor, with only 16.5 % of the experimental group and 17.7 % of the control group having good knowledge in the pre-intervention phase. The experimental group had a significant improvement in knowledge of IPV screening and management, with 82.1 % having good knowledge immediately after the intervention and 92.0 % at 6 weeks after the intervention (p = 0.001). Observed practice of IPV detection and management improved significantly from 21.9 % satisfactory practice before the intervention to 63.5 % after the intervention (p = 0.001) in the experimental group, with no appreciable improvement in practice detected in the control group (21.9% versus 36.5%; p = 0.682).

CONCLUSION

The use of a customized educational training program improved midwives' knowledge and practice in the detection and management of intimate partner violence.

Causal attributions of failure among post graduate medical residents in exit fellowship examination in Pakistan: A qualitative study

Background and Objective

Causal attributions are reasons given to certain events in life including failure and success. The objective of this qualitative study was to explore the attributions perceived as the cause of failure by post graduate residents, failing the final clinical exit examination of Fellowship of College of Physicians and Surgeons Pakistan (FCPS).

Methods

This exploratory study was conducted from July 2021 to July 2022, at Jinnah Hospital, Lahore. Study population was selected by purposeful maximal variation sampling. A total of ten post graduate residents from four specialties, failing in FCPS part two clinical examination were included. After written informed consent, semi structured face to face, in depth interviews were conducted. Data saturation was achieved after eight interviews after which two more interviews were carried out. All interviews were recorded and transcribed verbatim. Thematic analysis of the data was done by synthesizing codes into categories and themes.

Results

A total of 17 attributions noted were grouped into five categories namely 'Inadequate exam preparation', 'Personal factors,' 'Exam related factors,' 'Training related factors and 'Luck'. Two main themes of External and Internal factors emerged from these. Common attributions were; lack of effort (8/10), inadequate knowledge (8/10), family commitments (7/10), luck (8/10) and examiner's attitude (5/10).

Conclusions

Most of the residents attributed internal, unstable and controllable factors like inadequate knowledge and lack of effort. In addition, external uncontrollable factors of bad luck and harsh attitude of examiners were considered as contributory factors towards failure.

Analysis of Spanish vocational radiographers' education through clinical training and perceptions of core subject teachers: A survey

INTRODUCTION

This work presents a prospective analysis of the key aspects involved in the Spanish radiographer's perception of the weaknesses of the current educational curriculum in terms of teaching staff's qualifications and composition in the clinical training and core subjects. The goal is to show clinical training and professional's perception of the teaching quality and to characterise the weakness detected within the context of the European radiographer's academic system.

METHODS

The perception of the quality of the training received by the professionals was collected through an anonymous survey. A total of 758 valid responses were received, and were analysed to three hypotheses: the variability of the teachers' qualifications in the core subjects, the variability in terms of internships time carried out by the students, and assessment about teaching quality of the teachers.

RESULTS

The results prove there is a huge variability and little academic affinity of the teachers' degrees with the core subjects. On the other hand, the results shows there is a scarcity of clinical training hours in Spain, especially compared to European standards. It was demonstrated that teachers with a specific radiographer degree obtained the best scores.

CONCLUSION

It is mandatory to adjust the criteria for selecting clinical imaging teachers to improve the teaching quality in Spain and increase the clinical training period of the Spanish radiographers to match their European counterparts.

IMPLICATION FOR PRACTICE

Improving the training of Spanish radiographers will contribute to standardising the training quality of the whole European profession.

Training and implementation of handheld ultrasound technology at Georgetown Public Hospital Corporation in Guyana: a virtual learning cohort study

A virtual point-of-care ultrasound (POCUS) education program was initiated to introduce handheld ultrasound technology to Georgetown Public Hospital Corporation in Guyana, a low-resource setting. We studied ultrasound competency and participant satisfaction in a cohort of 20 physicians-in-training through the urology clinic. The program consisted of a training phase, where they learned how to use the Butterfly iQ ultrasound, and a mentored implementation phase, where they applied their skills in the clinic. The assessment was through written exams and an objective structured clinical exam (OSCE). Fourteen students completed the program. The written exam scores were 3.36/5 in the training phase and 3.57/5 in the mentored implementation phase, and all students earned 100% on the OSCE. Students expressed satisfaction with the program. Our POCUS education program demonstrates the potential to teach clinical skills in low-resource settings and the value of virtual global health partnerships in advancing POCUS and minimally invasive diagnostics.

The Debriefing Assessment in Real Time (DART) tool for simulation-based medical education

BACKGROUND

Debriefing is crucial for enhancing learning following healthcare simulation. Various validated tools have been shown to have contextual value for assessing debriefers. The Debriefing Assessment in Real Time (DART) tool may offer an alternative or additional assessment of conversational dynamics during debriefings.

METHODS

This is a multi-method international study investigating reliability and validity. Enrolled raters (n = 12) were active simulation educators. Following tool training, the raters were asked to score a mixed sample of debriefings. Descriptive statistics are recorded, with coefficient of variation (CV%) and Cronbach's α used to estimate reliability. Raters returned a detailed reflective survey following their contribution. Kane's framework was used to construct validity arguments.

RESULTS

The 8 debriefings (μ = 15.4 min (SD 2.7)) included 45 interdisciplinary learners at various levels of training. Reliability (mean CV%) for key components was as follows: instructor questions μ = 14.7%, instructor statements μ = 34.1%, and trainee responses μ = 29.0%. Cronbach α ranged from 0.852 to 0.978 across the debriefings. Post-experience responses suggested that DARTs can highlight suboptimal practices including unqualified lecturing by debriefers.

CONCLUSION

The DART demonstrated acceptable reliability and may have a limited role in assessment of healthcare simulation debriefing. Inherent complexity and emergent properties of debriefing practice should be accounted for when using this tool.

Investigating medical students' satisfaction towards video-based learning versus face-to-face lectures: a Jordanian tertiary teaching hospital experience

PURPOSE

We aimed to evaluate the disparity between video-based learning and lecture-based learning on Jordanian medical students' satisfaction.

METHODS

We conducted this cross-sectional study using a web-based questionnaire adapted from Student Evaluation of Educational Quality survey. Using convenience sampling, medical students studying at the University of Jordan and Jordan University Hospital were recruited. Participants in either clinical or basic-science years that have completed the entire survey were included in the final analysis.

RESULTS

We surveyed a total 487 participants among which male to female ratio was 1.19:1. Participants perceived greater benefit in terms of learning, instructor enthusiasm, content organization, breadth of teaching, and quality and number of assignments when using video-based learning (all p&lt;0.01). In contrast, face-to-face learning was associated with significantly higher benefits in terms of group interactions (p&lt;0.01) and capacity for rapport building (p&lt;0.01). There was no significant difference in perceived examination performance between the two learning modalities (p=0.11).

CONCLUSION

Video-based learning is the preferred learning modality among Jordanian medical students. Despite its dominance across multiple domains, it should be implemented as an adjunct to traditional classroom teaching for it is vital in the development of good communication skills and building rapport in medical students.

Are ChatGPT’s knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination?: a descriptive study

This study aimed to compare the knowledge and interpretation ability of ChatGPT, a language model of artificial general intelligence, with those of medical students in Korea by administering a parasitology examination to both ChatGPT and medical students. The examination consisted of 79 items and was administered to ChatGPT on January 1, 2023. The examination results were analyzed in terms of ChatGPT’s overall performance score, its correct answer rate by the items’ knowledge level, and the acceptability of its explanations of the items. ChatGPT’s performance was lower than that of the medical students, and ChatGPT’s correct answer rate was not related to the items’ knowledge level. However, there was a relationship between acceptable explanations and correct answers. In conclusion, ChatGPT’s knowledge and interpretation ability for this parasitology examination were not yet comparable to those of medical students in Korea.

Impact of service-learning placement on pharmacy student knowledge, attitudes, and confidence in helping patients navigate Medicare Part D

INTRODUCTION

Pharmacists are in a prime position to empower patients to navigate Medicare Part D. This study aimed to determine if service-learning has a place in pharmacy student Medicare education. The primary objective of the study was to assess knowledge, attitudes, and confidence of first-year pharmacy students at Medicare focused service-learning sites compared to students at alternative sites.

METHODS

First-year pharmacy students at the University of Pittsburgh School of Pharmacy were assessed on their knowledge, attitudes, and confidence of Medicare Part D after a semester of service-learning either at a State Health Insurance Assistance Program (SHIP) or at an alternative site not focused on Medicare (control). All students attended a four-part lecture series on Medicare before starting service-learning. Students were surveyed at baseline and after their service-learning experience.

RESULTS

A total of 110 (94.8%) students successfully completed both the pre- and post-survey. Knowledge improved significantly in the SHIP group (P = .01) and did not increase significantly in the control group (P = .06). Attitudes toward Medicare Part D, assessed on a Likert scale, became less favorable in the control group (-0.40, P < .001). Student confidence in the ability to counsel patients on Medicare part D improved in the SHIP group (0.42, P < .001) and decreased in the control group (-0.80, P < .001).

CONCLUSIONS

Combining a didactic lecture series on Medicare Part D with service-learning involving Medicare counseling may solidify student knowledge of Medicare as well as students' confidence in helping patients navigate Medicare Part D.

Factors associated with medical students’ scores on the National Licensing Exam in Peru: a systematic review

PURPOSE

This study aimed to identify factors that have been studied for their associations with National Licensing Examination (ENAM) scores in Peru.

METHODS

A search was conducted of literature databases and registers, including EMBASE, SciELO, Web of Science, MEDLINE, Peru’s National Register of Research Work, and Google Scholar. The following key terms were used: “ENAM” and “associated factors.” Studies in English and Spanish were included. The quality of the included studies was evaluated using the Medical Education Research Study Quality Instrument (MERSQI).

RESULTS

In total, 38,500 participants were enrolled in 12 studies. Most (11/12) studies were cross-sectional, except for one case-control study. Three studies were published in peer-reviewed journals. The mean MERSQI was 10.33. A better performance on the ENAM was associated with a higher-grade point average (GPA) (n=8), internship setting in EsSalud (n=4), and regular academic status (n=3). Other factors showed associations in various studies, such as medical school, internship setting, age, gender, socioeconomic status, simulations test, study resources, preparation time, learning styles, study techniques, test-anxiety, and self-regulated learning strategies.

CONCLUSION

The ENAM is a multifactorial phenomenon; our model gives students a locus of control on what they can do to improve their score (i.e., implement self-regulated learning strategies) and faculty, health policymakers, and managers a framework to improve the ENAM score (i.e., design remediation programs to improve GPA and integrate anxiety-management courses into the curriculum).

Possibility of independent use of the yes/no Angoff and Hofstee methods for the standard setting of the Korean Medical Licensing Examination written test: a descriptive study

PURPOSE

This study aims to apply the yes/no Angoff and Hofstee methods to actual Korean Medical Licensing Examination (KMLE) 2022 written examination data to estimate cut scores for the written KMLE.

METHODS

Fourteen panelists gathered to derive the cut score of the 86th KMLE written examination data using the yes/no Angoff method. The panel reviewed the items individually before the meeting and shared their respective understanding of the minimum-competency physician. The standard setting process was conducted in 5 rounds over a total of 800 minutes. In addition, 2 rounds of the Hofstee method were conducted before starting the standard setting process and after the second round of yes/no Angoff.

RESULTS

For yes/no Angoff, as each round progressed, the panel’s opinion gradually converged to a cut score of 198 points, and the final passing rate was 95.1%. The Hofstee cut score was 208 points out of a maximum 320 with a passing rate of 92.1% at the first round. It scored 204 points with a passing rate of 93.3% in the second round.

CONCLUSION

The difference between the cut scores obtained through yes/no Angoff and Hofstee methods did not exceed 2% points, and they were within the range of cut scores from previous studies. In both methods, the difference between the panelists decreased as rounds were repeated. Overall, our findings suggest the acceptability of cut scores and the possibility of independent use of both methods.

Evaluation research in Korean medical education: a systematic review

The purpose of this study aims to analyze research trends related to 'evaluation' in Korean medical education through a systematic review. This study used a systematic review method, which is a research methodology for research trends and 'literature analysis.' Researchers searched the Korean journal literature published until the end of December 2020 in the Korean research database with keywords related to medicine and evaluation. Thus, 5,205 cases were identified. Based on these data, 143 papers were selected through a logical screening process, requiring 1 month to complete the data search and analysis process. In terms of publications, medical journals overwhelmingly outnumbered nonmedical journals until 2015; however, after 2016, the number of papers published in nonmedical journals increased, and the number of published papers was similar to that of medical journals. In terms of evaluation-related research, research on student and program evaluations has been very active compared to that on accreditation. As the number of evaluation studies has gradually decreased over the past 10 years, preparing a plan to revitalize them in Korean medical education is necessary. Considering that the role of evaluation in education has been emphasized in recent years, research on reestablishing the concept of evaluation; developing evaluation indicators; analyzing the status of student evaluation, program evaluation, and accreditation; and deriving measures to improve medical education through evaluation is required.

Equal Z standard-setting method to estimate the minimum number of panelists for a medical school’s objective structured clinical examination in Taiwan: a simulation study

PURPOSE

Undertaking a standard-setting exercise is a common method for setting pass/fail cut scores for high-stakes examinations. The recently introduced equal Z standard-setting method (EZ method) has been found to be a valid and effective alternative for the commonly used Angoff and Hofstee methods and their variants. The current study aims to estimate the minimum number of panelists required for obtaining acceptable and reliable cut scores using the EZ method.

METHODS

The primary data were extracted from 31 panelists who used the EZ method for setting cut scores for a 12-station of medical school’s final objective structured clinical examination (OSCE) in Taiwan. For this study, a new data set composed of 1,000 random samples of different panel sizes, ranging from 5 to 25 panelists, was established and analyzed. Analysis of variance was performed to measure the differences in the cut scores set by the sampled groups, across all sizes within each station.

RESULTS

On average, a panel of 10 experts or more yielded cut scores with confidence more than or equal to 90% and 15 experts yielded cut scores with confidence more than or equal to 95%. No significant differences in cut scores associated with panel size were identified for panels of 5 or more experts.

CONCLUSION

The EZ method was found to be valid and feasible. Less than an hour was required for 12 panelists to assess 12 OSCE stations. Calculating the cut scores required only basic statistical skills.

Acceptability of the 8-case objective structured clinical examination of medical students in Korea using generalizability theory: a reliability study

PURPOSE

This study investigated whether the reliability was acceptable when the number of cases in the objective structured clinical examination (OSCE) decreased from 12 to 8 using generalizability theory (GT).

METHODS

This psychometric study analyzed the OSCE data of 439 fourth-year medical students conducted in the Busan and Gyeongnam areas of South Korea from July 12 to 15, 2021. The generalizability study (G-study) considered 3 facets—students (p), cases (c), and items (i)—and designed the analysis as p×(i:c) due to items being nested in a case. The acceptable generalizability (G) coefficient was set to 0.70. The G-study and decision study (D-study) were performed using G String IV ver. 6.3.8 (Papawork, Hamilton, ON, Canada).

RESULTS

All G coefficients except for July 14 (0.69) were above 0.70. The major sources of variance components (VCs) were items nested in cases (i:c), from 51.34% to 57.70%, and residual error (pi:c), from 39.55% to 43.26%. The proportion of VCs in cases was negligible, ranging from 0% to 2.03%.

CONCLUSION

The case numbers decreased in the 2021 Busan and Gyeongnam OSCE. However, the reliability was acceptable. In the D-study, reliability was maintained at 0.70 or higher if there were more than 21 items/case in 8 cases and more than 18 items/case in 9 cases. However, according to the G-study, increasing the number of items nested in cases rather than the number of cases could further improve reliability. The consortium needs to maintain a case bank with various items to implement a reliable blueprinting combination for the OSCE.

Possibility of using the yes/no Angoff method as a substitute for the percent Angoff method for estimating the cutoff score of the Korean Medical Licensing Examination: a simulation study

PURPOSE

The percent Angoff (PA) method has been recommended as a reliable method to set the cutoff score instead of a fixed cut point of 60% in the Korean Medical Licensing Examination (KMLE). The yes/no Angoff (YNA) method, which is easy for panelists to judge, can be considered as an alternative because the KMLE has many items to evaluate. This study aimed to compare the cutoff score and the reliability depending on whether the PA or the YNA standard-setting method was used in the KMLE.

METHODS

The materials were the open-access PA data of the KMLE. The PA data were converted to YNA data in 5 categories, in which the probabilities for a “yes” decision by panelists were 50%, 60%, 70%, 80%, and 90%. SPSS for descriptive analysis and G-string for generalizability theory were used to present the results.

RESULTS

The PA method and the YNA method counting 60% as “yes,” estimated similar cutoff scores. Those cutoff scores were deemed acceptable based on the results of the Hofstee method. The highest reliability coefficients estimated by the generalizability test were from the PA method and the YNA method, with probabilities of 70%, 80%, 60%, and 50% for deciding “yes,” in descending order. The panelist’s specialty was the main cause of the error variance. The error size was similar regardless of the standard-setting method.

CONCLUSION

The above results showed that the PA method was more reliable than the YNA method in estimating the cutoff score of the KMLE. However, the YNA method with a 60% probability for deciding “yes” also can be used as a substitute for the PA method in estimating the cutoff score of the KMLE.

Pilot study of the DART tool - an objective healthcare simulation debriefing assessment instrument

BACKGROUND

Various rating tools aim to assess simulation debriefing quality, but their use may be limited by complexity and subjectivity. The Debriefing Assessment in Real Time (DART) tool represents an alternative debriefing aid that uses quantitative measures to estimate quality and requires minimal training to use. The DART is uses a cumulative tally of instructor questions (IQ), instructor statements (IS) and trainee responses (TR). Ratios for IQ:IS and TR:[IQ + IS] may estimate the level of debriefer inclusivity and participant engagement.

METHODS

Experienced faculty from four geographically disparate university-affiliated simulation centers rated video-based debriefings and a transcript using the DART. The primary endpoint was an assessment of the estimated reliability of the tool. The small sample size confined analysis to descriptive statistics and coefficient of variations (CV%) as an estimate of reliability.

RESULTS

Ratings for Video A (n = 7), Video B (n = 6), and Transcript A (n = 6) demonstrated mean CV% for IQ (27.8%), IS (39.5%), TR (34.8%), IQ:IS (40.8%), and TR:[IQ + IS] (28.0%). Higher CV% observed in IS and TR may be attributable to rater characterizations of longer contributions as either lumped or split. Lower variances in IQ and TR:[IQ + IS] suggest overall consistency regardless of scores being lumped or split.

CONCLUSION

The DART tool appears to be reliable for the recording of data which may be useful for informing feedback to debriefers. Future studies should assess reliability in a wider pool of debriefings and examine potential uses in faculty development.

Rating criteria to evaluate student performance in digital wax-up training using multi-purpose software

PURPOSE

The aim of this study was to introduce rating criteria to evaluate student performance in a newly developed, digital wax-up preclinical program for computer-aided design (CAD) of full-coverage crowns and preliminarily investigate the reliability and internal consistency of the rating system.

MATERIALS AND METHODS

This study, conducted in 2017, enrolled 47 fifth-year dental students of Okayama University Dental School. Digital wax-up training included a fundamental practice using computer graphics (CG), multipurpose CAD software programs, and an advanced practice to execute a digital wax-up of the right mandibular second molar (#47). Each student’s digital wax-up work (stereolithography data) was evaluated by two instructors using seven qualitative criteria. The total qualitative score (0-90) of the criteria was calculated. The total volumetric discrepancy between each student’s digital wax-up work and a reference prepared by an instructor was automatically measured by the CAD software. The inter-rater reliability of each criterion was analyzed using a weighted kappa index. The relationship between the total volume discrepancy and the total qualitative score was analyzed using Spearman’s correlation.

RESULTS

The weighted kappa values for the seven qualitative criteria ranged from 0.62 - 0.93. The total qualitative score and the total volumetric discrepancy were negatively correlated (ρ = -0.27, P = .09, respectively); however, this was not statistically significant.

CONCLUSION

The established qualitative criteria to evaluate students’ work showed sufficiently high inter-rater reliability; however, the digitally measured volumetric discrepancy could not sufficiently predict the total qualitative score.

Medical Residency Milestones: Competencies in Informatics, Library, and Evidence-Based Practice

The Accreditation Council for Graduate Medical Education (ACGME) sets standards known as Milestones and monitors the progress of medical residents as they advance toward medical practice in their specialties. Health sciences librarians need to train medical residents in certain competency areas to help reach the Milestone standards. This project analyzed the Milestones related to informatics, library, and evidence-based practice (EBP) skills to identify core and optional library-related curricular elements that can be integrated into different medical specialty residencies. The authors collected key competency documents from ACGME and from those specialties representing 2% or more of the residencies in the United States. Then, they compared and contrasted those Milestones related to informatics, library, and EBP competency skills. Most relevant Milestones were categorized under the fifth broad ACGME competency area of "Practice Based Learning and Improvement." The Milestones followed developmental patterns, reflecting residents' increased sophistication in meeting these competencies as they advanced in their specialties. The curriculum was designed to meet the residents' learning needs at each progressive Milestone.

Communication skills preparedness for practice: Is there a key ingredient in undergraduate curricula design?

OBJECTIVE

To characterize medical interns' experience regarding communication skills education and to explore potential associations with preparedness for practice.

METHODS

Two hundred sixty-six medical interns answered an original questionnaire specifically developed to explore how well they feel their undergraduate training had prepared them in key aspects of medical communication. Instrument's psychometric properties were tested. Medical schools' curricula were considered and associations explored using non-parametric tests.

RESULTS

The questionnaire reliability was high, with Cronbach's alphas ranging from 0.89 to 0.94 on all the factors. Core communication skills were highly rated. Perceived preparedness was lower in aspects concerning dealing with emotion, breaking bad news and communicating with speech impaired patients. Better preparedness was associated with a longitudinal integration of communication skills throughout the curriculum, simulation with standardized patients and real patient interviewing with feedback on communication skills.

CONCLUSIONS

Integrated programs, standing on a strong experimental component, particularly combining patient-simulation strategies with continuous supervision and learner centred feedback, were associated with higher preparedness. These results support the expansion of an educational model based on simulation strategies and structured longitudinally throughout the undergraduate medical curriculum.

PRACTICE IMPLICATIONS

This study intends to inform educational background and to support further development of communication skills curricula.

Ensuring competence in ultrasound-guided procedures-a validity study of a newly developed assessment tool

OBJECTIVES

To investigate the validity of the Interventional Ultrasound Skills Evaluation (IUSE) tool for assessment of procedural competence in ultrasound-guided procedures in a clinical environment, including a pass/fail score.

METHODS

Novices and experienced radiologists were recruited from four hospitals and were observed and assessed while performing ultrasound-guided procedures. Performances were assessed using the IUSE tool by two independent raters. Validity evidence was gathered in accordance with Messick's framework: response process was ensured by standardisation of written rater instructions. Internal structure was explored using Cronbach's alpha for internal consistency reliability; inter-rater reliability was calculated as Pearson's r independently across all ratings, and test-retest reliability was reported using Cronbach's alpha. Relationship to other variables was investigated by comparing performances of the participants in each group. Consequences evidence was explored by calculating a pass/fail standard using the contrasting groups method.

RESULTS

Six novices and twelve experienced radiologists were enrolled. The IUSE tool had high internal consistency (Cronbach's alpha = 0.96, high inter-rater reliability (Pearson's r = 0.95), and high test-retest reliability (Cronbach's alpha = 0.98), and the mean score was 33.28 for novices and 59.25 for experienced with a highly significant difference (p value < 0.001). The pass/fail score was set at 55 resulting in no false positives or false negatives.

CONCLUSIONS

Validity evidence from multiple sources supports the use of the IUSE tool for assessment of competence in ultrasound-guided procedures in a clinical environment and its use in high-stakes assessment such as certification. A credible pass/fail criterion was established to inform decision-making.

KEY POINTS

• A multi-site validity investigation established that the Interventional Ultrasound Skills Evaluation (IUSE) tool can be used to assess procedural competence in ultrasound-guided procedures. • Validity evidence was gathered according to Messick's framework validity from the following sources: response process, internal structure, relationship to other variables, and consequences evidence. • The IUSE tool can be used for both formative and summative assessment, and a credible pass/fail score was established to help inform decision-making such as certification.

Providing a model for validation of the assessment system of internal medicine residents based on Kane's framework

BACKGROUND

Kane's validity framework examines the validity of the interpretation of a test at the four levels of scoring, generalization, extrapolation, and implications. No model has been yet proposed to use this framework particularly for a system of assessment. This study provided a model for the validation of the internal medicine residents' assessment system, based on the Kane's framework.

MATERIALS AND METHODS

Through a five stages study, first, by reviewing the literature, the methods used, and the study challenges, in using Kane's framework, were extracted. Then, possible assumptions about the design and implementation of residents' tests and the proposed methods for their validation at each of their four inferences of Kane's validity were made in the form of two tables. Subsequently, in a focus group session, the assumptions and proposed validation methods were reviewed. In the fourth stage, the opinions of seven internal medicine professors were asked about the results of the focus group. Finally, the assumptions and the final validation model were prepared.

RESULTS

The proposed tables were modified in the focus group. The validation table was developed consisting of tests, used at each Miller's pyramid level. The results were approved by five professors of the internal medicine. The final table has five rows, respectively, as the levels of Knows and Knows How, Shows How, Shows, Does, and the fifth one for the final scores of residents. The columns of the table demonstrate the necessary measures for validation at the four levels of inferences of Kane's framework.

CONCLUSION

The proposed model ensures the validity of the internal medicine specialty residency assessment system based on Kane's framework, especially at the implication level.

Clinical assessment during a global pandemic - Transitioning to a COVID safe hybrid OSCE

Objective structured clinical examinations (OSCEs) are often used to assess the clinical competence of students in preprofessional osteopathy training. During the COVID-19 global pandemic, the final year OSCE in the RMIT University osteopathy program was redeveloped leveraging online technologies within COVID-19 guidelines such as hygiene and occupancy limitations. Final year osteopathy students were assessed using a hybrid ten station OSCE, comprising both online and face-to-face components. The examination was led by a pre-recorded narrated PowerPoint video. The video contained instructions, case information for five cases and prompts for the practical stations. A student model stepped into the room as needed for practical stations. The examiner assessed students from another room via video streaming, with limited interaction with students. The hybrid OSCE was conducted safely during Stage 4 restrictions adhering to COVID Safe guidelines, allowing robust competency assessment of final year students, enabling timely graduation and transition to practice. Institutional support, technology infrastructure, clear communication and stakeholder collaboration are key to successful implementation. The hybrid OSCE format offers a potential solution for institutions delivering high-stakes assessment in the continuing challenges of clinical assessment in the post COVID landscape. Adopting hybrid assessment formats may facilitate remote assessment of students in clinical placements.

Filipino nurses' perspectives of the clinical and language competency requirements for nursing registration in England: A qualitative exploration

AIM

To understand Filipino nurses' experiences of the 'Test of Competence' process, alongside the additional competency requirements of their sponsor Trust.

BACKGROUND

The Philippines has been a significant and sustained source of foreign trained nurses to the UK over the past twenty years. Since October 2014, the Nursing and Midwifery Council has required that all non-UK nurses pass a 'Test of Competence' prior to acquiring registration. However, there is limited evidence exploring how overseas nurses experience this process and the tests which comprise it.

DESIGN

A qualitative study informed by pragmatism, using focus group discussions.

METHODS

Focus groups were held with pre-departure nurses in Manila, the Philippines and with post-arrival Filipino nurses in the Southeast of England. A total of 21 male and female Filipino nurses participated. Qualitative data were analysed using Reflexive Thematic Analysis.

RESULTS

Three themes were generated in relation to the competency process. Firstly, inappropriate preparation, Second, a competency/practice disparity and finally intergenerational and intra-professional support.

CONCLUSION

Some aspects of the 'Test of Competence' process were seen as unnecessarily difficult, contradictory, repetitious and/or of little relevance to practice. Lack of support and insufficient information relating to how best prepare for the tests were mitigated to some extent by individual learning and informal support mechanisms. An evaluation of the impact that repeated changes to the Test of Competence process may have on the quality of support offered by NHS Trusts to overseas nurses is recommended.

Comparing the cut score for the borderline group method and borderline regression method with norm-referenced standard setting in an objective structured clinical examination in medical school in Korea

PURPOSE

Setting standards is critical in health professions. However, appropriate standard setting methods do not always apply to the set cut score in performance assessment. The aim of this study was to compare the cut score when the standard setting is changed from the norm-referenced method to the borderline group method (BGM) and borderline regression method (BRM) in an objective structured clinical examination (OSCE) in medical school.

METHODS

This was an explorative study to model of the BGM and BRM. A total of 107 fourth-year medical students attended the OSCE at seven stations with encountering standardized patients (SPs) and one station with performing skills on a manikin on 15 July 2021. Thirty-two physician examiners evaluated the performance by completing a checklist and global rating scales.

RESULTS

The cut score of the norm-referenced method was lower than that of the BGM (p<0.01) and BRM (p<0.02). There was no significant difference in the cut score between the BGM and BRM (p=0.40). The station with the highest standard deviation and the highest proportion of the borderline group showed the largest cut score difference in standard setting methods.

CONCLUSION

Prefixed cut scores by the norm-referenced method without considering station contents or examinee performance can vary due to station difficulty and content, affecting the appropriateness of standard setting decisions. If there is an adequate consensus on the criteria for the borderline group, standard setting with the BRM could be applied as a practical and defensible method to determine the cut score for OSCE.

Translation, cultural adaptation, and validation of the Brazilian Portuguese version of the End-of-Life Professional Caregiver Survey

OBJECTIVES

The aim of this study was to translate, culturally adapt, and psychometrically evaluate the Brazilian version of the "End-of-Life Professional Caregiver Survey" (BR-EPCS).

METHOD

This is an observational cross-sectional study. The sample was composed of 285 Brazilian healthcare professionals who work or worked in the palliative care area. A minimum number of 280 participants were established, following the recommendation of 10 subjects for each instrument item. The European Organisation for Research and Treatment of Cancer - Quality of Life Group Translation Procedure protocol was used for the translation and the cultural adaptation. For the precise/reliable evaluation of factors measured by the BR-EPCS, Cronbach's alpha (α) and composite reliability coefficients were used. The factorial analyses were made by means of the exploratory structural equation modeling methods and confirmatory factor analysis. We have conducted a multiple linear regression analysis to evaluate the sociodemographic variables' capabilities in the result prediction measured by BR-EPCS factors.

RESULTS

The factorial analysis showed the relevance of two factors: Factor 1 - "Given care effectiveness" (18 items; Cronbach's α = 0.94; Composite Reliability = 0.95) and Factor 2 - "Mourning and ethical and cultural values" (10 items; Cronbach's α = 0.89; Composite Reliability = 0.88). Multiple linear regression analyses revealed that the working time, sex, palliative care training, and its own advance directives are predictors of the constructs assessed by the BR-EPCS.

SIGNIFICANCE OF RESULTS

The BR-EPCS is a reliable, valid, and culturally appropriate tool to identify the educational needs of healthcare professionals who work with palliative care. This instrument can be used for educational and research reasons.

Self-assessment of Competence in Endoscopy: Challenges and Insights

BACKGROUND

Endoscopists use self-assessment to monitor the development and maintenance of their skills. The accuracy of these self-assessments, which reflects how closely one's own rating corresponds to an external rating, is unclear.

METHODS

In this narrative review, we critically examine the current literature on self-assessment in gastrointestinal endoscopy with the aim of informing training and practice and identifying opportunities to improve the methodological rigor of future studies.

RESULTS

In the seven included studies, the evidence regarding self-assessment accuracy was mixed. When stratified by experience level, however, novice endoscopists were least accurate in their self-assessments and tended to overestimate their performance. Studies examining the utility of video-based interventions using observation of expert benchmark performances show promise as a mechanism to improve self-assessment accuracy among novices.

CONCLUSIONS

Based on the results of this review, we highlight problematic areas, identify opportunities to improve the methodological rigor of future studies on endoscopic self-assessment and outline potential avenues for further exploration.

Effects of a curriculum integrating critical thinking on medical students’ critical thinking ability in Iran: a quasi-experimental study

PURPOSE

Improving physicians’ critical thinking abilities could have meaningful impacts on various aspects of routine medical practice, such as choosing treatment plans, making an accurate diagnosis, and reducing medical errors. The present study aimed to measure the effects of a curriculum integrating critical thinking on medical students’ skills at Tehran University of Medical Sciences, Iran.

METHODS

A 1-group pre-test, post-test quasi-experimental design was used to assess medical students’ critical thinking abilities as they progressed from the first week of medical school to middle of the third year of the undergraduate medical curriculum. Fifty-six participants completed the California Critical Thinking Skills Test twice from 2016 to 2019.

RESULTS

Medical students were asked to complete the California Critical Thinking Skills Test the week before their first educational session. The post-test was conducted 6 weeks after the 2 and half-year program. Out of 91 medical students with a mean age of 20±2.8 years who initially participated in the study, 56 completed both the pre- and post-tests. The response rate of this study was 61.5%. The analysis subscale showed the largest change. Significant changes were found in the analysis (P=0.03), evaluation (P=0.04), and inductive reasoning (P<0.0001) subscales, but not in the inference (P=0.28), and deductive reasoning (P=0.42) subscales. There was no significant difference according to gender (P=0.77).

CONCLUSION

The findings of this study show that a critical thinking program had a substantial effect on medical students’ analysis, inductive reasoning, and evaluation skills, but negligible effects on their inference and deductive reasoning scores.

Malaysian pharmacy students' perspectives on the virtual objective structured clinical examination during the coronavirus disease 2019 pandemic

PURPOSE

This study investigated pharmacy students' perceptions of various aspects of virtual objective structured clinical examinations (vOSCEs) conducted during the coronavirus disease 2019 pandemic in Malaysia.

METHODS

This cross-sectional study involved third- and fourth-year pharmacy students at the International Islamic University Malaysia. A validated self-administered questionnaire was distributed to students who had taken a vOSCE a week before.

RESULTS

Out of the 253 students who were approached, 231 (91.3%) completed the questionnaire. More than 75% of the participants agreed that the instructions and preparations were clear and helpful in familiarizing them with the vOSCE flow. It was found that 53.2% of the respondents were satisfied with the flow and conduct of the vOSCE. However, only approximately one-third of the respondents believed that the tasks provided in the vOSCE were more convenient, less stressful, and easier to perform than those in the conventional OSCE. Furthermore, 49.7% of the students favored not having a vOSCE in the future when conducting a conventional OSCE becomes feasible again. Internet connection was reported as a problem hindering the performance of the vOSCE by 51.9% of the participants. Students who were interested in clinical pharmacy courses were more satisfied than other students with the preparation and operation of the vOSCE, the faculty support, and the allocated time.

CONCLUSION

Students were satisfied with the organization and operation of the vOSCE. However, they still preferred the conventional OSCE over the vOSCE. These findings might indicate a further need to expose students to telehealthcare models.

The practical examination types (spot test and slide test) of gross anatomy course in faculty of medicine: a simultaneous evaluation of the aspect of student success

PURPOSE

This study aimed to compare the students' scores of the spot (spotter/classical/traditional/tag/ring/bell-ringer) test (3D environment) performed in the laboratory with the slide test (gross anatomy images) (2D environment) in the class. The observation of our department regarding both types for practical examination was reported, in terms of exam marks of the students. Both are preferred as the practical examination types for gross anatomy course our in medical faculty.

METHODS

The 29 blocks' scores in 5 years (2013/2014-2017/2018) belonging to first- and second-year medical students' spot tests and slide tests are evaluated retrospectively and statistically compared. Correlations of the spot tests and the slide tests, besides the correlations between theoretical examinations and the practical examination types, are calculated.

RESULTS

Spot test scores were significantly higher (p < 0.05) in nine blocks, while slide test scores were higher significantly (p < 0.05) in fourteen. There was no statistically significant difference between the practical examination types (spot/slide) in six blocks. There were correlations between the spot test and the slide test in all blocks (p < 0.001).

CONCLUSION

It is considered that the spot test reflects the success/ability in a 3D environment, while the slide test reflects it in the 2D environment. In conclusion, neither of these two types of examinations stands out absolutely. Both types of examinations have their own features in areas, such as assessment power, applicability, and effect on success.

Development of a Competency-Based Curriculum for undergraduate education in Pediatric Dentistry: A systematic approach

The changing paradigm of dental education in India has led its way to the development of Competency-Based Curriculum (CBC). This article describes the process of developing CBC in the specialty of Pediatric Dentistry under the initiative of Dental Council of India. Rationale behind CBC development is to bring uniform system of education for improving oral health outcomes of the society in long term. The process of CBC development was a collaborative teamwork, planned meticulously with predefined outline, tasks and timelines. Workflow involved identification of curricular content, defining program goals, outlining competencies, assigning them domains / levels of clinical competence, priority, educational strategies, assessment practices, integration and numbers needed for certification in selected competencies. Early clinical exposure was introduced in CBC. The final content was validated and submitted to the Council. CBC output can be summarized as competencies with fair share of all domains, levels, prioritization and integration. It is characterized by flexibility for choosing educational strategies and assessment practices. It opens up ways for global competition. However, it still has some inherent weaknesses like diverging learning paths, time constraints and number chasing. CBC can further have more academic flexibility and develop toward an outcome-based approach. Faculty preparedness and acceptability shall be the biggest challenges in CBC implementation besides resources' availability, support from leadership and acceptability from our primary stakeholders, our learners. CBC is the beginning of evidence-based delivery of education in dentistry. An effective implementation of CBC in current form would result in increased numbers of competent oral healthcare professionals for the society.

Image-based assessment in undergraduate dermatology curriculum: A step toward competency-based medical education

Background

Dermatology being a visual branch, there is a need to add a visual element in learning and assessment of dermatology. This study compares the utility of image-based assessment (IBA) as a new tool compared to routinely used semi-structured viva (SSV) in dermatology formative assessment at undergraduate level.

Methods

Comparison was made between batches of students in year 2018 who underwent clinical posting term ending assessment by IBA with the retrospective cohort of batch of students in year 2015 who underwent assessment by SSV. The students' marks in this assessment and their attendance were collected. Feedback was taken from batch of students who had undergone IBA assessment. Faculty feedback was also taken.

Results

Correlation of attendance with marks was higher in IBA batch compared to SSV. IBA is better able to assess the diagnostic skills which requires visual element and prescription writing skill. SSV can do an authentic assessment of clinical reasoning skills. IBA had higher variability in marks allotted to students suggesting that it was more objective tool whereas with narrow range of marks SSV was found to be more subjective. Both IBA and SSV had similar acceptability by students and faculty. IBA was more resource intensive at preparation stage while SSV was so in conduction stage. IBA had better educational impact, as it promoted learning through exposure to actual patients.

Conclusion

IBA fared better in terms of validity, reliability, acceptability, and educational impact. In terms of feasibility IBA and SSV had differing challenges.

Medical knowledge and clinical productivity: independently correlated metrics during radiology residency

Objective

To determine the association between medical knowledge relevant to radiology practice (as measured by the Dutch radiology progress test [DRPT]) and clinical productivity during radiology residency.

Methods

This study analyzed the results of 6 DRPTs and time period–matched clinical production points of radiology residents affiliated to a tertiary care academic medical center between 2013 and 2016. The Spearman correlation analysis was performed to determine the association between DRPT percentile scores and average daily clinical production points. Linear regression analyses were performed to determine the association of DRPT percentile scores with average daily clinical production points, adjusted for age and gender of the radiology resident, and postgraduate year.

Results

Eighty-four DRPTs with time period–matched clinical production points were included. These 84 DRPTs were made by 29 radiology residents (18 males and 11 females) with a median age of 31 years (range: 26–38 years). The Spearman correlation coefficient between DRPT percentile scores and average daily clinical production points was 0.550 (95% confidence interval: 0.381–0.694) (p < 0.001), indicating a significant moderate positive association. On multivariate analysis, average daily clinical production points (β coefficient of 0.035, p = 0.003), female gender of the radiology resident (β coefficient of 12.690, p = 0.001), and postgraduate year (β coefficient of 10.179, p < 0.001) were significantly associated with DRPT percentile scores. These three independent variables achieved an adjusted R² of 0.527.

Conclusion

Clinical productivity is independently associated with medical knowledge relevant to radiology practice during radiology residency. These findings indicate that clinical productivity of a resident could be a potentially relevant metric in a radiology training program.

Key Points

• There is a significant moderate correlation between medical knowledge relevant to radiology practice and clinical productivity during radiology residency.

• Medical knowledge relevant to radiology practice remains independently associated with clinical productivity during radiology residency after adjustment for postgraduate year and gender.

• Clinical productivity of a resident may be regarded as a potentially relevant metric in a radiology training program.

Evaluation of competence in ultrasound-guided procedures-a generic assessment tool developed through the Delphi method

OBJECTIVES

To develop a generic and objective tool for assessing competence in percutaneous ultrasound-guided procedures.

METHODS

Interventional ultrasound experts from the Nordic countries were invited to participate in a three-round Delphi process. A steering committee was formed to manage the process. In round 1, the experts were asked to suggest all aspects to consider when assessing competence in US-guided procedures. Suggestions were analyzed and condensed into assessment items. In round 2, the expert panel rated these items on a 1-5 scale and suggested new items. Items with a mean rating of ≤ 3.5 were excluded. In round 3, the expert panel rated the list items and suggested changes to the items.

RESULTS

Twenty-five experts were invited, and response rates in the three rounds were 68% (17 out of 25), 100% (17 out of 17), and 100% (17 out of 17). The three-round Delphi process resulted in a 12-item assessment tool, using a five-point rating scale. The final assessment tool evaluates pre-procedural planning, US technique, procedural technique, patient safety, communication, and teamwork.

CONCLUSIONS

Expert consensus was achieved on a generic tool for assessment of competence in percutaneous ultrasound-guided procedures-the Interventional Ultrasound Skills Evaluation (IUSE). This is the initial step in ensuring a valid and reliable method for assessment of interventional US skill.

KEY POINTS

• Through a Delphi process, expert consensus was achieved on the content of an assessment tool for percutaneous ultrasound-guided procedures-the Interventional Ultrasound Skills Evaluation (IUSE) tool. • The IUSE tool is comprehensive and covers pre-procedural planning, US technique, procedural technique, patient safety, communication, and teamwork. • This is an important step in ensuring valid and reliable assessment of interventional US skills.

Accuracy of performance-test linking based on a many-facet Rasch model

Performance assessments, in which human raters assess examinee performance in practical tasks, have attracted much attention in various assessment contexts involving measurement of higher-order abilities. However, difficulty persists in that ability measurement accuracy strongly depends on rater and task characteristics such as rater severity and task difficulty. To resolve this problem, various item response theory (IRT) models incorporating rater and task parameters, including many-facet Rasch models (MFRMs), have been proposed. When applying such IRT models to datasets comprising results of multiple performance tests administered to different examinees, test linking is needed to unify the scale for model parameters estimated from individual test results. In test linking, test administrators generally need to design multiple tests such that raters and tasks partially overlap. The accuracy of linking under this design is highly reliant on the numbers of common raters and tasks. However, the numbers of common raters and tasks required to ensure high accuracy in test linking remain unclear, making it difficult to determine appropriate test designs. We therefore empirically evaluate the accuracy of IRT-based performance-test linking under common rater and task designs. Concretely, we conduct evaluations through simulation experiments that examine linking accuracy based on a MFRM while changing numbers of common raters and tasks with various factors that possibly affect linking accuracy.

Prospective validation of a core curriculum progress assimilation instrument for radiation oncology residentship

Objectives

To develop a tool that could assess residents' knowledge beyond simple information gathering and evaluate its reliability.

Methods

An assessment tool of 40 objective questions and at least one essay-based question was developed to assess residents' comprehension of general radiation oncology accordingly to validated training curricula beyond level 2 in the Bloom scale. With randomized content, questions were developed such as to be classified as at least 2 in the Bloom scale, so that reasoning and not only information gathering could be assessed. Criteria validation was made using the Classical Test Theory to describe difficulty and discrimination of each item. Reliability was tested by internal consistency using the Cronbach alpha test.

Results

Between 2016 and 2019, 24 residents were assessed. Six different versions of the test were made with a total of 240 objective questions and 18 essay-based questions. Five of the six versions were deemed valid and reliable. Comparisons between 1st (PGY-1) and 3rd (PGY-3) year residents were made. Consistently, PGY-3 residents had scores 150% higher than PGY-1 residents. Only two different PGY-3 reached the most complex level of answers in the essay-based questions. The results demonstrated that the major part of the acquired knowledge and retention occurs in the first six months of training rather than in all the following period.

Conclusion

The instrument can be considered valid. This developed instrument also raised the hypothesis that residents may not assess and analyze their acquired knowledge beyond the application level.

The compatibility principle: on philosophies in the assessment of clinical competence

The array of different philosophical positions underlying contemporary views on competence, assessment strategies and justification have led to advances in assessment science. Challenges may arise when these philosophical positions are not considered in assessment design. These can include (a) a logical incompatibility leading to varied or difficult interpretations of assessment results, (b) an "anything goes" approach, and (c) uncertainty regarding when and in what context various philosophical positions are appropriate. We propose a compatibility principle that recognizes that different philosophical positions commit assessors/assessment researchers to particular ideas, assumptions and commitments, and applies ta logic of philosophically-informed, assessment-based inquiry. Assessment is optimized when its underlying philosophical position produces congruent, aligned and coherent views on constructs, assessment strategies, justification and their interpretations. As a way forward we argue that (a) there can and should be variability in the philosophical positions used in assessment, and these should be clearly articulated to promote understanding of assumptions and make sense of justifications; (b) we focus on developing the merits, boundaries and relationships within and/or between philosophical positions in assessment; (c) we examine a core set of principles related to the role and relevance of philosophical positions; (d) we elaborate strategies and criteria to delineate compatible from incompatible; and (f) we articulate a need to broaden knowledge/competencies related to these issues. The broadened use of philosophical positions in assessment in the health professions affect the "state of play" and can undermine assessment programs. This may be overcome with attention to the alignment between underlying assumptions/commitments.

Differences in clerkship development between public and private Brazilian medical schools: an overview

BACKGROUND

Around the world, it is very expensive to become a physician. Although public medical schools are less expensive than private medical schools, tuition fees are charged at public medical schools in the majority of countries. In Brazil, public medical schools, with the exception of municipal schools, are free. There has been little investigation of any differences in conditions offered by paid or free medical schools or what occurs in public and private clerkships in Brazil. We investigated the clerkship conditions offered to the students in both public and private Brazilian medical schools by gathering the opinions of clerkship coordinators and others responsible for clerkships.

METHODS

A cross-sectional, descriptive, analytical study using an electronic questionnaire was answered by clerkship coordinators to compare the clerkships of 30 public and 38 private Brazilian medical schools from all regions of the country. The questionnaires covered various aspects of the clinical environments, student supervision, faculty development, student assessments, rotation evaluations and extracurricular activities developed by students.

RESULTS

We observed significant differences between public and private medical schools in several aspects investigated. Based on the opinions of the clerkship coordinators, with the exception of access to university hospitals, which was predominantly offered by public medical schools, private medical schools offer better clerkship conditions. The main differences were related to the number of positions, infrastructure, clinical learning environments, faculty development, student assessments, rotation evaluations and students' extracurricular activities.

CONCLUSION

This is the first study comparing Brazilian medical clerkships in private and public medical schools and provides a general vision of these programmes. It is necessary to further investigate clerkship development in the Brazilian medical school system and to study the differences between private and public medical schools globally.