Variation in Transcript Reports Among Residency Applicants: An Anesthesia Program's Perspective

Background

With recent changes made to move USMLE Step 1 and COMLEX Level 1 scores to Pass/Fail, it becomes necessary to find other metrics to evaluate residency candidates. One conserved metric included in all residency applications is medical school transcripts. This study aims to highlight the highly varied transcript reporting in a new era of holistic applicant review.

Methods

Medical school transcripts were extracted from the Electronic Residency Application Service applications to our anesthesiology residency program for the 2021-2022 application cycle. All personally identifiable information was removed. Results were categorized and tallied by 2 independent reviewers. Overall, we assessed transcript information from 156 allopathic and osteopathic medical schools. Transcript data were separated into 9 different categories.

Results

The most common grading system for allopathic medical schools was Pass/Fail. The most common grading system for osteopathic medical schools was Pass/Fail and Letter Grades. There were several medical schools that had unique grading systems and many of those did not provide a grading key for interpretation. Less than half of the allopathic and osteopathic schools offered Honors or High Pass in their grading systems, often with little information provided as to how these grades were earned.

Conclusion

The information provided on medical school transcripts is extremely variable. Although many schools reported grades as Pass/Fail, there was no majority or consistent presentation among the transcripts. Much of the information provided on transcripts required interpretation by its reviewer and made the process of holistic applicant review more difficult.

"I Was Told to Think Like a Middle-Aged White Woman": A Survey on Identity and the Association of Social Work Boards Exam

The release of the 2022 Association of Social Work Boards (ASWB) exam passage rate report confirmed what many test takers who failed their exam believe. The ASWB exams are biased, with differential passage rates based on the test taker's race, age, and "English as a second language" status. However, the report only offered basic descriptive statistics and lacked insight into the test takers' experience. The present study addresses this gap. Results are from a 2022 survey of individuals who had taken the ASWB master's level licensing exam (N = 1,045) highlighting test taker identity and experience. Thirty percent of the study sample identified as neurodivergent, 29.1 percent as primary caregivers, and 27.1 percent as Black, Indigenous, or persons of color. White respondents had the highest first-attempt passing rates (95.7 percent), followed by Latinx and Black respondents (84.9 percent and 78.2 percent, respectively). Forty-four respondents reported taking the test three or more times before passing. Among this group, 52 percent identified as Black, 25 percent as White, and 18 percent as Latinx. Respondents were asked how they felt their identity impacted their experience, and three interrelated themes emerged: privilege, challenges, and critique of the exam. Respondents discussed the impact of having or not having privilege; the emotional, physical, and financial challenges of preparing for the exam; and the ways in which they experienced the exam as biased.

Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study

BACKGROUND

The potential of artificial intelligence (AI)-based large language models, such as ChatGPT, has gained significant attention in the medical field. This enthusiasm is driven not only by recent breakthroughs and improved accessibility, but also by the prospect of democratizing medical knowledge and promoting equitable health care. However, the performance of ChatGPT is substantially influenced by the input language, and given the growing public trust in this AI tool compared to that in traditional sources of information, investigating its medical accuracy across different languages is of particular importance.

OBJECTIVE

This study aimed to compare the performance of GPT-3.5 and GPT-4 with that of medical students on the written German medical licensing examination.

METHODS

To assess GPT-3.5's and GPT-4's medical proficiency, we used 937 original multiple-choice questions from 3 written German medical licensing examinations in October 2021, April 2022, and October 2022.

RESULTS

GPT-4 achieved an average score of 85% and ranked in the 92.8th, 99.5th, and 92.6th percentiles among medical students who took the same examinations in October 2021, April 2022, and October 2022, respectively. This represents a substantial improvement of 27% compared to GPT-3.5, which only passed 1 out of the 3 examinations. While GPT-3.5 performed well in psychiatry questions, GPT-4 exhibited strengths in internal medicine and surgery but showed weakness in academic research.

CONCLUSIONS

The study results highlight ChatGPT's remarkable improvement from moderate (GPT-3.5) to high competency (GPT-4) in answering medical licensing examination questions in German. While GPT-4's predecessor (GPT-3.5) was imprecise and inconsistent, it demonstrates considerable potential to improve medical education and patient care, provided that medically trained users critically evaluate its results. As the replacement of search engines by AI tools seems possible in the future, further studies with nonprofessional questions are needed to assess the safety and accuracy of ChatGPT for the general population.

Fellowship of the Australian College of Rural & Remote Medicine (FACRRM) Assessment: a review of the first 12 years

This article was migrated. The article was marked as recommended. This paper provides an overview of the first 12 years of the formal assessment program of the Australian College of Rural and Remote Medicine (ACRRM). The ACRRM Fellowship represents the world's first and only Fellowship exam in Rural Medicine. The ACRRM assessment program is mapped to its Rural Generalist curriculum, based on the principles of programmatic assessment. ACRRM offers candidates the opportunity to participate in assessment in or close to their home location. The ACRRM Rural Generalist Curriculum defines the scope and standards for independent general practice anywhere in Australia, with a focus on rural and remote settings. The program was initially developed in 2006 and has evolved during delivery from 2008 onwards, utilising the following modalities: •Multi Source Feedback (MSF)•Multiple Choice Questions (MCQ)•Mini Clinical Evaluation Exercise (Mini-CEX)•Case Based Discussion (CBD)•Procedural Skills Logbook•Structured Assessment using Multiple Patient Scenarios (StAMPS) StAMPS is a unique examination, blending the formats of an Objective Structured Clinical Examination and a traditional viva vocè. The program has an emphasis on formative assessment. Over the past 12 years there has been considerable work in developing resources for candidates, governance structures and quality assurance processes. ACRRM's Fellowship requirements represent a customised bespoke assessment tailored to ACRRM's curriculum and the Australian rural and remote context. ACRRM's assessment program has grown substantially with 649 Fellowships being awarded from 2008 - 2019, with considerable experience gained in rural and remote assessment. It now represents a mature firmly-established process as a vocational endpoint in Rural and Remote Medicine. ACRRM has continued to offer its 'tele-assessment' program throughout the COVID-19 pandemic, with candidates and examiners participating in assessment by use of distance technology while remaining in or near their home community. This model may provide some insights for other medical Colleges and educational institutions facing challenges in the current environment.