Validation of the General Medicine in-Training Examination Using the Professional and Linguistic Assessments Board Examination Among Postgraduate Residents in Japan

Hospital Use of a Web-Based Clinical Knowledge Support System and In-Training Examination Performance Among Postgraduate Resident Physicians in Japan: Nationwide Observational Study

Background

The relationship between educational outcomes and the use of web-based clinical knowledge support systems in teaching hospitals remains unknown in Japan. A previous study on this topic could have been affected by recall bias because of the use of a self-reported questionnaire.

Objective

We aimed to explore the relationship between the use of the Wolters Kluwer UpToDate clinical knowledge support system in teaching hospitals and residents' General Medicine In-Training Examination (GM-ITE) scores. In this study, we objectively evaluated the relationship between the total number of UpToDate hospital use logs and the GM-ITE scores.

Methods

This nationwide cross-sectional study included postgraduate year-1 and -2 residents who had taken the examination in the 2020 academic year. Hospital-level information was obtained from published web pages, and UpToDate hospital use logs were provided by Wolters Kluwer. We evaluated the relationship between the total number of UpToDate hospital use logs and residents' GM-ITE scores. We analyzed 215 teaching hospitals with at least 5 GM-ITE examinees and hospital use logs from 2017 to 2019.

Results

The study population consisted of 3013 residents from 215 teaching hospitals with at least 5 GM-ITE examinees and web-based resource use log data from 2017 to 2019. High-use hospital residents had significantly higher GM-ITE scores than low-use hospital residents (mean 26.9, SD 2.0 vs mean 26.2, SD 2.3; P=.009; Cohen d=0.35, 95% CI 0.08-0.62). The GM-ITE scores were significantly correlated with the total number of hospital use logs (Pearson r=0.28; P<.001). The multilevel analysis revealed a positive association between the total number of logs divided by the number of hospital physicians and the GM-ITE scores (estimated coefficient=0.36, 95% CI 0.14-0.59; P=.001).

Conclusions

The findings suggest that the development of residents' clinical reasoning abilities through UpToDate is associated with high GM-ITE scores. Thus, higher use of UpToDate may lead physicians and residents in high-use hospitals to increase the implementation of evidence-based medicine, leading to high educational outcomes.

Distribution of internal medicine rotations among resident physicians in Japan: a nationwide, multicenter, cross-sectional study

BACKGROUND

In Japan, postgraduate clinical training encompasses a 2-year residency program, including at least 24 weeks of internal medicine (IM) rotations. However, the fragmented structure of these rotations can compromise the training's quality and depth. For example, a resident might spend only a few weeks in cardiology before moving to endocrinology, without sufficient time to deepen their understanding or have clinical experience. This study examined current patterns and lengths of IM rotations within the Japanese postgraduate medical system. It scrutinized the piecemeal approach-whereby residents may engage in multiple short-term stints across various subspecialties without an overarching, integrated experience-and explored potential consequences for their clinical education.

METHODS

This nationwide, multicenter, cross-sectional study used data from self-reported questionnaires completed by participants in the 2022 General Medicine In-Training Examination (GM-ITE). Data of 1,393 postgraduate year (PGY) one and two resident physicians who participated in the GM-ITE were included. We examined the IM rotation duration and number of IM subspecialties chosen by resident physicians during a 2-year rotation.

RESULTS

Approximately half of the participants chose IM rotation periods of 32-40 weeks. A significant proportion of participants rotated in 5-7 internal medicine departments throughout the observation period. Notable variations in the distribution of rotations were observed, characterized by a common pattern where resident physicians typically spend 4 weeks in each department before moving to the next. This 4-week rotation is incrementally repeated across different subspecialties without a longer, continuous period in any single area. Notably, 39.7% of participants did not undertake general internal medicine rotations. These results suggest a narrowed exposure to medical conditions and patient care practices.

CONCLUSIONS

Our study highlights the need to address the fragmented structure of IM rotations in Japan. We suggest that short, specialized learning periods may limit the opportunity to gain broad in-depth knowledge and practical experience. To improve the efficacy of postgraduate clinical education, we recommend fostering more sustained and comprehensive learning experiences.

Awareness and Perception of Dental Undergraduates in Belagavi District Towards the Newly Proposed National Exit Test for India

BACKGROUND

 The National Dental Commission Bill of 2023 introduced the National Exit Test (NExT) as a common final examination for all dental graduates in India, granting them the license to practice. This study evaluated dental undergraduates' awareness and perceptions of the newly proposed NExT.

METHODS

 A self-administered questionnaire in English comprising 23 close-ended questions was used to assess the awareness and perception of the students. A pilot study was conducted to determine the sample size, and 510 students were selected using a simple random sampling technique. The survey was administered to students ranging from 1st to 4th-year undergraduates and interns from two dental colleges in the Belagavi District, India. The questionnaire demonstrated good reliability (Cronbach's alpha coefficient = 0.86) and a content validity ratio 0.82.

RESULTS

 Interns had the highest mean awareness (39.56 ± 8.99) and perception (40.87 ± 5.56) scores, whereas first-year students had the lowest, with statistically significant differences among the groups (p ≤ 0.001). Although 81% of students were aware of NExT in India, only 17.3% found it student-friendly. A positive correlation was seen between the perception and awareness scores (r = + 0.242; p ≤ 0.001). The dependence of awareness and perception scores on predictors such as age, gender, and year of study were 16.7% and 15.3%, respectively.

CONCLUSION

 Interns displayed a positive perception and higher awareness of NExT, whereas first and second-year dental students exhibited lower awareness and apprehensive perceptions. The introduction of NExT promises to enhance the overall quality of dental education on a national scale by providing high-quality care to patients.

Development of a Clinical Simulation Video to Evaluate Multiple Domains of Clinical Competence: Cross-Sectional Study

BACKGROUND

Medical students in Japan undergo a 2-year postgraduate residency program to acquire clinical knowledge and general medical skills. The General Medicine In-Training Examination (GM-ITE) assesses postgraduate residents' clinical knowledge. A clinical simulation video (CSV) may assess learners' interpersonal abilities.

OBJECTIVE

This study aimed to evaluate the relationship between GM-ITE scores and resident physicians' diagnostic skills by having them watch a CSV and to explore resident physicians' perceptions of the CSV's realism, educational value, and impact on their motivation to learn.

METHODS

The participants included 56 postgraduate medical residents who took the GM-ITE between January 21 and January 28, 2021; watched the CSV; and then provided a diagnosis. The CSV and GM-ITE scores were compared, and the validity of the simulations was examined using discrimination indices, wherein ≥0.20 indicated high discriminatory power and >0.40 indicated a very good measure of the subject's qualifications. Additionally, we administered an anonymous questionnaire to ascertain participants' views on the realism and educational value of the CSV and its impact on their motivation to learn.

RESULTS

Of the 56 participants, 6 (11%) provided the correct diagnosis, and all were from the second postgraduate year. All domains indicated high discriminatory power. The (anonymous) follow-up responses indicated that the CSV format was more suitable than the conventional GM-ITE for assessing clinical competence. The anonymous survey revealed that 12 (52%) participants found the CSV format more suitable than the GM-ITE for assessing clinical competence, 18 (78%) affirmed the realism of the video simulation, and 17 (74%) indicated that the experience increased their motivation to learn.

CONCLUSIONS

The findings indicated that CSV modules simulating real-world clinical examinations were successful in assessing examinees' clinical competence across multiple domains. The study demonstrated that the CSV not only augmented the assessment of diagnostic skills but also positively impacted learners' motivation, suggesting a multifaceted role for simulation in medical education.

Association between mentorship and mental health among junior residents: A nationwide cross-sectional study in Japan

Background

Mentorship is a dynamic, reciprocal relationship in which an advanced careerist (mentor) encourages the growth of a novice (mentee). Mentorship may protect the mental health of residents at risk for depression and burnout, yet despite its frequent use and known benefits, limited reports exist regarding the prevalence and mental effects of mentorship on residents in Japan.

Methods

We conducted a cross-sectional study involving postgraduate year 1 and 2 (PGY-1 and PGY-2) residents in Japan who took the General Medicine In-Training Examination (GM-ITE) at the end of the 2021 academic year. Data on mentorship were collected using surveys administered immediately following GM-ITE completion. The primary outcome was the Patient Health Questionaire-2 (PHQ-2), which consisted depressed mood and loss of interest. A positive response for either item indicated PHQ-2 positive. We examined associations between self-reported mentorship and PHQ-2 by multi-level analysis.

Results

Of 4929 residents, 3266 (66.3%) residents reported having at least one mentor. Compared to residents without any mentor, those with a mentor were associated with a lower likelihood of a positive PHQ-2 response (adjusted odds ratio [aOR] 0.75; 95% confidence interval [95% CI] 0.65-0.86). Mentor characteristic significantly associated with negative PHQ-2 response was a formal mentor (aOR; 0.68; 95% CI 0.55-0.84).

Conclusions

A mentor-based support system was positively associated with residents' mental health. Further research is needed to determine the quality of mentorship during clinical residency in Japan.

Performance Comparison of ChatGPT-4 and Japanese Medical Residents in the General Medicine In-Training Examination: Comparison Study

BACKGROUND

The reliability of GPT-4, a state-of-the-art expansive language model specializing in clinical reasoning and medical knowledge, remains largely unverified across non-English languages.

OBJECTIVE

This study aims to compare fundamental clinical competencies between Japanese residents and GPT-4 by using the General Medicine In-Training Examination (GM-ITE).

METHODS

We used the GPT-4 model provided by OpenAI and the GM-ITE examination questions for the years 2020, 2021, and 2022 to conduct a comparative analysis. This analysis focused on evaluating the performance of individuals who were concluding their second year of residency in comparison to that of GPT-4. Given the current abilities of GPT-4, our study included only single-choice exam questions, excluding those involving audio, video, or image data. The assessment included 4 categories: general theory (professionalism and medical interviewing), symptomatology and clinical reasoning, physical examinations and clinical procedures, and specific diseases. Additionally, we categorized the questions into 7 specialty fields and 3 levels of difficulty, which were determined based on residents' correct response rates.

RESULTS

Upon examination of 137 GM-ITE questions in Japanese, GPT-4 scores were significantly higher than the mean scores of residents (residents: 55.8%, GPT-4: 70.1%; P<.001). In terms of specific disciplines, GPT-4 scored 23.5 points higher in the "specific diseases," 30.9 points higher in "obstetrics and gynecology," and 26.1 points higher in "internal medicine." In contrast, GPT-4 scores in "medical interviewing and professionalism," "general practice," and "psychiatry" were lower than those of the residents, although this discrepancy was not statistically significant. Upon analyzing scores based on question difficulty, GPT-4 scores were 17.2 points lower for easy problems (P=.007) but were 25.4 and 24.4 points higher for normal and difficult problems, respectively (P<.001). In year-on-year comparisons, GPT-4 scores were 21.7 and 21.5 points higher in the 2020 (P=.01) and 2022 (P=.003) examinations, respectively, but only 3.5 points higher in the 2021 examinations (no significant difference).

CONCLUSIONS

In the Japanese language, GPT-4 also outperformed the average medical residents in the GM-ITE test, originally designed for them. Specifically, GPT-4 demonstrated a tendency to score higher on difficult questions with low resident correct response rates and those demanding a more comprehensive understanding of diseases. However, GPT-4 scored comparatively lower on questions that residents could readily answer, such as those testing attitudes toward patients and professionalism, as well as those necessitating an understanding of context and communication. These findings highlight the strengths and limitations of artificial intelligence applications in medical education and practice.

Association between regional quota program in medical schools and practical clinical competency based on General Medicine In-Training Examination score: a nationwide cross-sectional study of resident physicians in Japan

PURPOSE

A regional quota program (RQP) was introduced in Japan to ameliorate the urban-rural imbalance of physicians. Despite concerns about the low learning abilities of RQP graduates, the relationship between the RQP and practical clinical competency after initiating clinical residency has not been evaluated.

METHODS

We conducted a nationwide cross-sectional study to assess the association between the RQP and practical clinical competency based on General Medicine In-Training Examination (GM-ITE) scores. We compared the overall and category GM-ITE results between RQP graduates and other resident physicians. The relationship between the RQP and scores was examined using multilevel linear regression analysis.

RESULTS

There were 4978 other resident physicians and 1119 RQP graduates out of 6097 participants from 593 training hospitals. Being younger; preferring internal, general, or emergency medicine; managing fewer inpatients; and having fewer ER shifts were all characteristics of RQP graduates. In multilevel multivariable linear regression analysis, there was no significant association between RQP graduates and total GM-ITE scores (coefficient: 0.26; 95% confidence interval: -0.09, 0.61; P = .15). The associations of RQP graduates with GM-ITE scores in each category and specialty were not clinically relevant. However, in the same multivariable model, the analysis did reveal that total GM-ITE scores demonstrated strong positive associations with younger age and GM preference, both of which were significantly common in RQP graduates.

CONCLUSION

Practical clinical competency evaluated based on the GM-ITE score showed no clinically relevant differences between RQP graduates and other resident physicians. Key messages What is already known on this topic Many countries offer unique admission processes to medical schools and special undergraduate programs to increase the supply of physicians in rural areas. Concerns have been raised about the motivation, learning capabilities, and academic performance of the program graduates. What this study adds This nationwide cross-sectional study in Japan revealed clinical competency based on the scores from the General Medicine In-Training Examination showed no clinically relevant differences between graduates of regional quota programs and other resident physicians. How this study might affect research, practice, or policy The study provides evidence to support the Japanese regional quota program from the perspective of clinical competency after initiating clinical practice.

Association between prolonged weekly duty hours and self-study time among residents: a cross-sectional study

PURPOSE

In 2024, the Japanese government will enforce a maximum 80-hour weekly duty hours (DHs) regulation for medical residents. Although this reduction in weekly DHs could increase the self-study time (SST) of these residents, the relationship between these two variables remains unclear. The aim of the study was to investigate the relationship between the SST and DHs of residents in Japan.

METHODS

In this nationwide cross-sectional study, the subjects were candidates of the General Medicine In-Training Examination in the 2020 academic year. We administered questionnaires and categorically asked questions regarding daily SST and weekly DHs during the training period. To account for hospital variability, proportional odds regression models with generalized estimating equations were used to analyse the association between SST and DHs.

RESULTS

Of the surveyed 6117 residents, 32.0% were female, 49.1% were postgraduate year-1 residents, 83.8% were affiliated with community hospitals, and 19.9% worked for ≥80 hours/week. Multivariable analysis revealed that residents working ≥80 hours/week spent more time on self-study than those working 60-70 hours/week. Conversely, residents who worked <50 hours/week spent less time on self-study than those who worked 60-70 hours/week. The factors associated with longer SST were sex, postgraduate year, career aspiration for internal medicine, affiliation with community hospitals, academic involvement, and well-being.

CONCLUSION

Residents with long DHs had longer SSTs than residents with short DHs. Future DH restrictions may not increase but rather decrease resident SST. Effective measures to encourage self-study are required, as DH restrictions may shorten SST.

Empathy competence and future specialty among medical residents in Japan: a nationwide cross-sectional study

Empathy is essential for physicians to provide patient-centered care. Nevertheless, the degree to which empathy varies among medical residents based on their desired future specialty remains undetermined. This nationwide cross-sectional study compared empathy levels (Jefferson Scale of Empathy, JSE) of 824 year one and two postgraduate residents in Japan by intended medical specialty, individual characteristics, and training and working environment characteristics. Empathy levels were compared with applicants for general medicine, which emphasizes patient-centeredness. The highest mean JSE and the highest percentage of women residents were observed in general medicine (M = 109.74; SD = 14.04), followed by dermatology (M = 106.64; SD = 16.90), obstetrics and gynecology (M = 106.48; SD = 14.31), and pediatrics (106.02; SD 12.18). Residents interested in procedure-centered departments (e.g. ophthalmology, orthopedics) garnered lower JSE scores. Multivariate regression revealed that future general medicine candidates achieved the highest JSE scores ([Formula: see text] = 6.68, 95% CI 2.39-10.9, p = 0.002). Women achieved significantly higher JSE scores than men ([Formula: see text] = 2.42, 95% CI 0.11-4.73, p = 0.041). The results have implications for empathy training and postgraduate education strategy in different clinical specialties.

Medical resident's pursuing specialty and differences in clinical proficiency among medical residents in Japan: a nationwide cross-sectional study

IMPORTANCE

Standardized examinations assess both learners and training programs within the medical training system in Japan. However, it is unknown if there is an association between clinical proficiency as assessed by the General Medicine In-Training Examination (GM-ITE) and pursuing specialty.

OBJECTIVE

To determine the relative achievement of fundamental skills as assessed by the standardized GM-ITE based on pursuing career specialty among residents in the Japanese training system.

DESIGN

Nationwide cross-sectional study.

SETTING

Medical residents in Japan who attempted the GM-ITE in their first or second year were surveyed.

PARTICIPANTS

A total of 4,363 postgraduate years 1 and 2 residents who completed the GM-ITE were surveyed between January 18 and March 31, 2021.

MAIN MEASURES

GM-ITE total scores and individual scores in each of four domains assessing clinical knowledge: 1) medical interview and professionalism, 2) symptomatology and clinical reasoning, 3) physical examination and treatment, and 4) detailed disease knowledge.

RESULTS

When compared to the most pursued specialty, internal medicine, only those residents who chose general medicine achieved higher GM-ITE scores (coefficient 1.38, 95% CI 0.08 to 2.68, p = 0.038). Conversely, the nine specialties and "Other/Not decided" groups scored significantly lower. Higher scores were noted among residents entering general medicine, emergency medicine, and internal medicine and among those who trained in community hospitals with higher numbers of beds, were more advanced in their training, spent more time working and studying, and cared for a moderate but not an extreme number of patients at a time.

CONCLUSIONS

Levels of basic skill achievement differed depending on respective chosen future specialties among residents in Japan. Scores were higher among those pursuing careers in general medical fields and lower among those pursuing highly specialized careers. Residents in training programs devoid of specialty-specific competition may not possess the same motivations as those in competitive systems.

Impact of duty hours on competency-related knowledge acquisition among community hospital residents

Background

The effect of duty hour (DH) restrictions on postgraduate residents' acquisition of clinical competencies is unclear. We evaluated the relationship between DHs and competency-related knowledge acquisition using the General Medicine In-training Examination (GM-ITE).

Methods

We conducted a multicenter, cross-sectional study of community hospital residents among 2019 GM-ITE examinees. Self-reported average DHs per week were classified into five DH categories and the competency domains were classified into four areas: symptomatology and clinical reasoning (CR), physical examination and clinical procedure (PP), medical interview and professionalism (MP), and disease knowledge (DK). The association between these scores and DHs was examined using random-intercept linear models with and without adjustment for confounding factors.

Results

We included 4753 participants in the analyses. Of these, 31% were women, and 49.1% were in the postgraduate year (PGY) 2. Mean CR and MP scores were lower among residents in Category 1 (<50 h) than in residents in Category 3 (≥60 and <70 h; reference group). Mean DK scores were lower among residents in Categories 1 and 2 (≥50 and <60 h) than in the reference group. PGY-2 residents in Categories 1 and 2 had lower CR scores than those in Category 3; however, PGY-1 residents in Category 5 showed higher scores.

Conclusions

The relationship between DHs and each competency area is not strictly linear. The acquisition of knowledge of physical examination and clinical procedures skills in particular may not be related to DHs.

Relationship between COVID-19 care and burnout among postgraduate clinical residents in Japan: a nationwide cross-sectional study

OBJECTIVE

The relationship between the care of patients with COVID-19 and mental health among resident physicians in Japan is imperative for ensuring appropriate care of patients with COVID-19 and should be clarified. We herein assessed the relationship between the care of patients with COVID-19 and mental health among postgraduate year 1 (PGY-1) and PGY-2 resident physicians and factors associated with mental health.

DESIGN

This nationwide cross-sectional study analysed data obtained using the clinical training environment self-reported questionnaire.

SETTING

An observational study across Japan among resident physicians (PGY-1 and PGY-2) from 583 teaching hospitals.

PARTICIPANTS

Examinees who took the general medicine in-training examination of academic year 2020.

PRIMARY AND SECONDARY OUTCOME MEASURES

The Patient Health Questionnaire and Mini-Z 2.0 were used to assess mental health, and experience of caring for patients with COVID-19 was divided into three groups (none, 1-10 and ≥11). The prevalence of mental conditions in the three groups was compared using the 'modified' Poisson generalised estimating equations by adjusting for prefecture-level, hospital-level and resident-level variables.

RESULTS

Of the 5976 participants analysed, 50.9% were PGY-1. The prevalence of burnout was 21.4%. Moreover, 47.0% of all resident physicians had no experience in the care of patients with COVID-19. The well-experienced group accounted for only 7.9% of the total participants. A positive association was found between the number of caring patients with COVID-19 and burnout (prevalence ratio 1.25; 95% CI 1.02 to 1.53). Moreover, the shortage of personal protective equipment was identified as a major contributor to burnout (prevalence ratio 1.60; 95% CI 1.36 to 1.88).

CONCLUSIONS

Resident physicians who experienced more care of patients with COVID-19 had slightly greater burnout prevalence than those who did not. Approximately half of resident physicians did not participate in the care of patients with COVID-19, which posed a challenge from an educational perspective.

Association between mental health and duty hours of postgraduate residents in Japan: a nationwide cross-sectional study

The new duty hour (DH) limit for doctors in Japan will begin in 2024, setting the maximum DHs for postgraduate residents at approximately 80 h weekly. To set appropriate limits, understanding the association between DHs and psychological health is necessary. Thus, we assessed the relationship between residents' psychological health and DHs. We conducted a cross-sectional study involving examinees of the General Medicine In-training Examination 2020. Mental health outcomes were measured dichotomously using the Patient Health Questionnaire-2 for depression and Mini-Z 2.0, for burnout, stress, and satisfaction. Weekly DHs were measured in seven categories at 10-h intervals. The prevalence ratios (PRs) between the DH categories were estimated for each outcome. Of the 6045 residents who provided data on DHs and psychological outcomes, 37.3% showed signs of depression, 21.6% experienced burn out, and 39.2% were highly stressed. In contrast, 62.3% were highly satisfied with their training. Proportions of burnout were higher among residents in Category 6 (≥ 90 and < 100 h; PR 1.36; 95% CI 1.11-1.66) and Category 7 (≥ 100 h; PR 1.36; 95% CI 1.10-1.68) compared with residents in Category 3 (≥ 60 and < 70 h; reference). The results partially support the weekly 80-h DH limit in terms of resident well-being.