Levels of evidence have increased for musculoskeletal trauma questions on the orthopaedic in-training examination

Comparison of Artificial Intelligence to Resident Performance on Upper-Extremity Orthopaedic In-Training Examination Questions

Purpose

Currently, there is a paucity of prior investigations and studies examining applications for artificial intelligence (AI) in upper-extremity (UE) surgical education. The purpose of this investigation was to assess the performance of a novel AI tool (ChatGPT) on UE questions on the Orthopaedic In-Training Examination (OITE). We aimed to compare the performance of ChatGPT to the examination performance of hand surgery residents.

Methods

We selected questions from the 2020-2022 OITEs that focused on both the hand and UE as well as the shoulder and elbow content domains. These questions were divided into two categories: those with text-only prompts (text-only questions) and those that included supplementary images or videos (media questions). Two authors (B.K.F. and G.S.M.) converted the accompanying media into text-based descriptions. Included questions were inputted into ChatGPT (version 3.5) to generate responses. Each OITE question was entered into ChatGPT three times: (1) open-ended response, which requested a free-text response; (2) multiple-choice responses without asking for justification; and (3) multiple-choice response with justification. We referred to the OITE scoring guide for each year in order to compare the percentage of correct AI responses to correct resident responses.

Results

A total of 102 UE OITE questions were included; 59 were text-only questions, and 43 were media-based. ChatGPT correctly answered 46 (45%) of 102 questions using the Multiple Choice No Justification prompt requirement (42% for text-based and 44% for media questions). Compared to ChatGPT, postgraduate year 1 orthopaedic residents achieved an average score of 51% correct. Postgraduate year 5 residents answered 76% of the same questions correctly.

Conclusions

ChatGPT answered fewer UE OITE questions correctly compared to hand surgery residents of all training levels.

Clinical relevance

Further development of novel AI tools may be necessary if this technology is going to have a role in UE education.

Fifty Most-Cited Research Articles in Elbow Surgery: A Modern Reading List

Purpose

Bibliometric analysis is a common method for evaluating current trends within a scientific field. The primary aim of this study was to define and analyze the 50 most frequently cited articles in the field of elbow surgery, both of all time and those published during the 21st century.

Methods

We searched the Journal Citation Report to identify articles related to elbow surgery within academic journals. Articles were sorted by total citations. The overall top 50 articles and those published since 2000 were identified, and data were collected, including title, journal of publication, publication year, country of publication, citation density, level of evidence, article type, institution, and sex of the lead and senior authors, and inclusion on the reference list for the Orthopaedic In-Training Examination within the last 5 years. Descriptive statistics were reported, and correlation analysis was performed using Spearman test.

Results

For the most-cited elbow surgery articles, "fracture" was overall the most reported topic, whereas "lateral epicondylosis" and "fracture" were equal for those published since 2000. The United States was the most represented overall and for articles published since 2000. Women comprised 1/50 (2%) of lead authors overall, increasing to 8/50 (16%) for articles published during the 21st century. Most articles in during both periods contained level IV evidence, with level I evidence appearing infrequently (4%). Six percent of the most-cited articles of all time had appeared on the reference list of the Orthopaedic In-Training Examination within the past 5 years.

Conclusions

The top 50 most-cited elbow surgery articles often assess fracture and lateral epicondylosis, most commonly originating from the United States. Level IV retrospective series comprises over half of the articles on this list. Women remain underrepresented as authors.

Clinical Relevance

This study provides a modern reading list for upper-extremity surgeons about impactful elbow surgery articles.

Analysis of Questions in Sections of the Orthopaedic In-Training Examination: A Scoping Review

PURPOSE

The purpose of this review was to assess all available studies that analyzed the types of questions in individual sections of the Orthopaedic In-Training Examination, which may be used as a reference for residents studying for their examination.

METHODS

Following the Providing Innovative Service Models and Assessment extension for Scoping Reviews guidelines, a systematic review was conducted on studies that report on sections or question categories of the Orthopaedic In-Training Examination using PubMed, MEDLINE, and Web of Science databases. Two reviewers and an arbitrator reviewed and extracted relevant data from 20 included studies which made up the systematic review.

RESULTS

All 20 studies in the review reported the mean number of questions per section, with the highest coming from musculoskeletal trauma (18.9% to 19.0%). 18 studies reported the Buckwalter taxonomic classification; 42.0% of questions were T1, 18.2% were T2, and 39.5% were T3 with a wide range from section to section. Primary sources were nearly three times more likely to be cited when compared with textbook sources. There were 12 journals that were commonly cited with the most being the Journal of Bone and Joint Surgery: American Volume (17/18).

DISCUSSION

This study accurately portrays the characteristics of each section of the Orthopaedic In-Training Examination over the past 10 years. These data suggest that orthopaedic residents may be inclined to focus on musculoskeletal trauma, topics related to clinical management, and primary journal sources for studying. In addition, residency programs may choose to focus on higher yield sources or material to prepare their residents for the examination.

Orthopaedic In-Training Examination: History, Perspective, and Tips for Residents

Introduced in 1963, the orthopaedic in-training examination (OITE) is a standardized, national test administered annually to orthopaedic residents by the American Academy of Orthopaedic Surgeons. The examination consists of 275 multiple-choice questions that cover 11 domains of orthopaedic knowledge, including basic science, foot and ankle, hand, hip and knee, oncology, pediatrics, shoulder and elbow, spine, sports medicine, trauma, and practice management. The OITE has been validated and is considered predictive of success in both orthopaedic surgery residency and on the American Board of Orthopaedic Surgery part I examination. This article provides a historical overview of the OITE, details its current structure and scoring system, and reviews currently available study materials. For examination preparation, the residents are encouraged to (1) start the examination preparation early, (2) practice on old OITE or self-assessment examination questions, (3) focus on the questions answered incorrectly, (4) focus on comprehension over memorization, and (5) recognize and avoid burnout. Finally, the residents should have a systemic way of approaching each multiple-choice question, both during practice and on the actual examination.

An Analysis of the Most Commonly Tested Topics and Their Taxonomy From Recent Self-Assessment Examinations

OBJECTIVE

The purpose of this study is to determine the most commonly tested topics and the question taxonomy of the American Academy of Orthopaedic Surgeons Self-Assessment Examinations (SAE) from 2009 through 2014.

DESIGN

All SAEs were analyzed from 2009 through 2014. The SAEs were separated by subject and the questions of each SAE were analyzed for topic, taxonomic classification, and question type.

RESULTS

A total of 2107 questions were reviewed from 10 different subjects. In all, 6 of the 9 subjects had roughly 1/3 of their questions composed of the 3 most commonly tested topics. Each subject had at least 1 trauma-related question within its top 5 most commonly tested topics. Almost half (47%) of all questions were of taxonomy 1 classification and 29% were taxonomy 3. The Basic Science SAEs had the greatest percentage of taxonomy 1 questions of any subject (83%) whereas Trauma contained the highest percentage of taxonomy 3 questions (47%).

CONCLUSIONS

Certain topics within each subject are consistently tested more often than other topics. In general, the 3 most commonly tested topics comprise about one-third of total questions and orthopedic surgeons should be very familiar with these topics in order to best prepare for standardized examinations.

Evaluation of Pediatric Questions on the Orthopaedic In-Training Examination-An Update

BACKGROUND

Pediatric orthopaedics is tested frequently on the Orthopaedic In-Training Examination (OITE). The most recent data on the pediatrics section of the OITE were generated from content 10 years old. The purpose of this study is to assess the pediatric orthopaedic questions on the 2011 to 2014 OITE, and to compare question categories and cognitive taxonomy with previous data.

METHODS

Four years (2011 to 2014) of OITE questions, answers, and references were reviewed. The number of pediatric questions per year was recorded, as well as presence of a clinical photo or imaging modality. Each question was categorized and assigned a cognitive taxonomy level. Categories included: knowledge; knowledge-treatment modalities; diagnosis; diagnosis/recognition of associated conditions; diagnosis/further studies; and diagnosis/treatment. Cognitive taxonomy levels included: simple recall, interpretation of data, and advanced problem-solving.

RESULTS

The 3 most commonly covered topics were upper extremity trauma (17.4%), scoliosis (10.1%), and developmental dysplasia of the hip (5.7%). Compared with previous data, the percentage of pediatric questions was constant (13% vs. 14%). Categorically, the more recent OITE examinations contained significantly fewer questions testing simple knowledge (19% vs. 39%, P=0.0047), and significantly more questions testing knowledge of treatment modalities (17% vs. 9%, P=0.016) and diagnosis with associated conditions (19% vs. 9%, P=0.0034). Regarding cognitive taxonomy, there was a significant increase in the average number of questions that required advanced problem-solving (57% vs. 46%, P=0.048). Significantly more questions utilized clinical photographs and imaging studies (62% vs. 48%, P=0.012). The most common reference materials provided to support correct responses included Lovell and Winter's Pediatric Orthopaedics (25.7%) and the Journal of Pediatric Orthopaedics (23.4%).

CONCLUSIONS

Although the percentage of pediatric questions on the OITE has remained essentially constant, the percentage of questions requiring advanced problem-solving or interpretation of images has increased significantly in the past 10 years. Knowledge of question type and content may be helpful for those involved in resident education and in the development of didactic pediatric orthopaedic curricula.

LEVEL OF EVIDENCE

Level IV.

Levels of Evidence for Hand Questions on the Orthopaedic In-Training Examination

Background: Although analyses of the Orthopaedic In-Training Examination (OITE) subspecialty content domains have been performed, few studies have analyzed the levels of evidence (LoEs) for journal articles used as references to create OITE questions. We present an analysis of reference characteristics and question taxonomy for the hand surgery content domain on the OITE. We aim to determine whether level of evidence (LoE) for hand surgery questions have increased over a 15-year period. Methods: All questions and references in the hand surgery content domain on the OITE from 1995-1997 and 2010-2012 were reviewed. The taxonomic classification was determined for each question. Publication characteristics were defined for each reference, and each primary journal article was assigned a LoE. Results: A total of 129 questions containing 222 references met inclusion criteria: 76 questions from 1995-1997 and 53 from 2010-2012. The Journal of Hand Surgery American and European Volumes, Journal of Bone and Joint Surgery American and British Volumes, and the Journal of the American Academy of Orthopaedic Surgeons were the most frequently cited journals overall. Recent examinations were more likely to have Buckwalter T3 complex clinical management questions. There was a statically significant increase in the LoE used to create hand questions on the 2010-2012 compared with the 1995-1997 OITE. Conclusions: Primary journal articles cited on the hand surgery content domain of the OITE frequently included recent publications from both general and subspecialty journals. More recent examination questions appear to test clinical management scenarios. LoE for hand questions has increased over a 15-year period. Our results can be used as a guide to help prepare orthopedic residents for the OITE.