Predicting the technical competence of surgical residents

I describe an approach to predicting competence in technical skills for the purposes of resident selection. To demonstrate a predictive relationship, it is necessary to use measures that exhibit variation, reliability, and validity. There is little evidence that such measures are routinely used in the process of selecting residents. I argue that the selection of assessment instruments in the predictor domain must be guided by relevant theoretical considerations, while assessment in the surgical domain must make use of more objective and reliable instruments than is currently the practice. I present a brief summary of research on predicting operative technical competence.

The need for research training in orthopaedic residency education

Orthopaedic surgery residents should be exposed during their clinical training to the processes of creativity and innovation that are the basis of research. The definition of a research experience for surgery residents should be broad and include not only traditional bench research in a basic science environment but also translational and clinical research to move innovation from bench to bedside and validate its value in a scientific manner. Additionally, there are enormous opportunities for surgeons to study healthcare delivery and policy and to develop new approaches to educating colleagues, other medical personnel, and patients. The question that must be addressed is how can the knowledge and human resources residing in orthopaedic surgery best be used to meet the challenges future residents will face as healthcare undergoes profound changes? How these issues are managed in a rapidly changing environment is the critical issue and the challenge faced by surgical training programs wishing to remain viable and provide trainees with the opportunity to adapt and be successful in the future. What is state of the art today will not be tomorrow and unless trainees are encouraged and taught to be creative and innovative they risk becoming surgical dinosaurs.

Important demographic variables impact the musculoskeletal knowledge and confidence of academic primary care physicians

BACKGROUND

Although most musculoskeletal illness is managed by primary care providers, and not by surgeons, evidence suggests that primary care physicians may receive inadequate training in musculoskeletal medicine. We evaluated the musculoskeletal knowledge and self-perceived confidence of fully trained, practicing academic primary care physicians and tested the following hypotheses: (1) a relationship exists between a provider's musculoskeletal knowledge and self-perceived confidence, (2) demographic variables are associated with differences in the knowledge-confidence relationship, and (3) specific education or training affects a provider's musculoskeletal knowledge and clinical confidence.

METHODS

An examination of basic musculoskeletal knowledge and a 10-point Likert scale assessing self-perceived confidence were administered to family practice, internal medicine, and pediatric faculty at a large, regional, academic primary care institution serving both rural and urban populations across a five-state region. Subspecialty physicians were excluded. Individual examination scores and self-reported confidence scores were correlated and compared with demographic variables.

RESULTS

One hundred and five physicians participated. Ninety-two physicians adequately completed the musculo-skeletal knowledge examination. Fifty-nine (64%) of the ninety-two physicians scored < 70%. Higher examination scores were associated with male gender (p = 0.01) and participation in a musculoskeletal course (p = 0.009). Practitioners who took elective courses demonstrated higher scores compared with those who took required courses (p = 0.014). Greater musculoskeletal confidence was associated with the number of years in clinical practice (p = 0.045), male gender (p = 0.01), residency training in family practice (p < 0.00001), and prior participation in a musculoskeletal course (p = 0.0004). Physicians demonstrated greater confidence with medical issues than with musculoskeletal issues (mean confidence scores, 8.3 and 5.1, respectively; p < 0.00001). Higher scores for musculoskeletal knowledge correlated significantly with increasing levels of musculoskeletal confidence (r = 0.416, p < 0.0001).

CONCLUSIONS

Although a large proportion of primary care visits are for musculoskeletal symptoms, the majority of primary care providers tested at a large, regional, academic primary care institution failed to demonstrate adequate musculoskeletal knowledge and confidence. Further characterization of the relationship between knowledge and confidence and its association with demographic variables might benefit the education of musculoskeletal providers in the future.

Determining the reliability of the Graf classification for hip dysplasia

We sought to establish the levels of interrater reliability and intrarater reliability of the Graf classification among orthopaedic surgeons in their final training year and who learned the method by instructed teaching or self study. Using standard teaching material developed by Graf, two groups of senior orthopaedic residents at the same training level received structured teaching sessions (Group A, n = 2) or performed self study (Group B, n = 2). Interrater reliability and intrarater reliability were determined (Cohen's weighted kappa). Proportions of correctly rated sonograms were compared between groups, implications of misclassifications were analyzed, and sensitivity analyses were performed. Interrater reliability was 0.59 (95% CI = 0.32-0.85) for Group A, and 0.47 (95% CI = 0.14-0.79) for Group B. Intrarater reliability showed an overall kappa of 0.57 (95% CI = 0.35-0.78) in Group A, and 0.47 (95% CI = 0.19-0.75) in Group B. The proportion of correctly rated sonograms between groups was similar in the original dataset and in the sensitivity analysis. Misclassifications influencing treatment were infrequent; one patient would have received unwarranted treatment and three patients would not have received warranted treatment. The Graf classification showed moderate reliability. Using self study, it can be learned almost, but not quite as effectively as by a structured program.

Confidence in spine training among senior neurosurgical and orthopedic residents

STUDY DESIGN

A cross-sectional survey of senior neurosurgical and orthopedic residents.

OBJECTIVE

To evaluate the confidence of senior orthopedic and neurosurgery residents in performing spinal surgical procedures and their need for further training. The content and exposure to spine training as well as anticipated practice profile were characterized.

SUMMARY OF BACKGROUND DATA

Spinal surgery is performed by specialists with backgrounds in orthopedic surgery and neurosurgery. As this subspecialty evolves, the need to modify training programs to keep up with technological and medical advances becomes increasingly clear. The primary objective of this study was to evaluate the self-assessed confidence and perceived need for further training of senior orthopedic and neurosurgical residents in performing a number of spinal surgical procedures.

METHODS

An evaluation of self-assessed surgical competence of senior orthopedic and neurosurgery residents in Canada was undertaken by mail-out questionnaire. A follow-up questionnaire was mailed to nonresponders 3 months later. Survey results were summarized using SPSS statistical software, and descriptive and comparative analyses were performed.

RESULTS

Significant differences in time and exposure to spine training differentiated the neurosurgical and orthopedic residencies (37% and 16% of total residency time devoted to spine, respectively). Neurosurgical residents reported significantly higher levels of confidence for all 25 surgical procedures. Of those residents anticipating incorporating spine into their practice, 29% of neurosurgery residents planned on entering a spine fellowship compared with 17% of their orthopedic colleagues.

CONCLUSIONS

Training in spine surgery constitutes a considerably larger proportion of neurosurgery residency than orthopedic residency. Neurosurgery residents graduate with significantly higher levels of confidence to perform spine surgery, while orthopedic residents report significantly higher need for additional training in spine surgery. The majority of neurosurgery graduates report that they will include spine in their clinical practice, while most orthopedic graduates will exclude it.

Orthopaedic in-training examination scores: a correlation with USMLE results

BACKGROUND

Both the United States Medical Licensing Examination and the Orthopaedic In-Training Examination measure factual recall as well as interpretative and problem-solving skills. The former examination is used to a variable degree by postgraduate programs in resident selection. Orthopaedic In-Training Examination scores are one measure of the medical knowledge of residents and are used by all American orthopaedic residency programs on a yearly basis. This investigation was performed to retrospectively review Orthopaedic In-Training Examination scores of orthopaedic residents who took the examination in our program. In addition, we sought to determine whether a relationship existed between performance on the Orthopaedic In-Training Examination and the United States Medical Licensing Examinations taken while in medical school.

METHODS

The records of each orthopaedic resident who took the examination from November 1993 through November 2000 were reviewed. Correlation coefficients and 95% confidence intervals were calculated to assess the relationship, if any, between the Orthopaedic In-Training Examination percentiles and the three-digit scores on the Step-1 and Step-2 United States Medical Licensing Examination. In addition, examination scores were evaluated longitudinally from year-in-training 1 through 4.

RESULTS

A significant moderate-sized correlation was found between United States Medical Licensing Examination Step-2 scores and Orthopaedic In-Training Examination score percentiles (p < 0.05); however, with the numbers available, no correlation was seen between United States Medical Licensing Examination Step-1 scores and Orthopaedic In-Training Examination scores. The mean Orthopaedic In-Training Examination scores were in the 66th percentile for year-in-training 1, the 53rd percentile for year 2, the 57th percentile for year 3, and the 50th percentile for year 4. Residents in the laboratory for one year scored in the 88th percentile while in the laboratory (year 0), in the 86th percentile in year 1, and in the 48th percentile in year 4.

CONCLUSION

Although Step-1 United States Medical Licensing Examination scores have been used by our department as a major factor in resident selection historically, our data failed to reveal a significant correlation with performance on the Orthopaedic In-Training Examination. The decrease in Orthopaedic In-Training Examination scores over time for our residents who worked in the laboratory is most likely attributable to multiple factors, including clinical workload hours.

Hunter's legacy and surgical training and competence in the 21st century

John Hunter's many contributions to surgery include the development of the scientific approach and possibly the first use of evidence-based medicine. This oration, concentrates on two other areas ? first, some of his contributions to orthopaedics and secondly past, present and future surgical training and competence.

Virtual reality techniques. Application to anatomic visualization and orthopaedics training

Surgical training systems using virtual reality simulation techniques offer a cost-effective alternative to traditional training methods. In this sense, techniques for interactive visualization and virtual reality surgery have been one of the very important research areas. We describe various techniques we have used in developing a virtual reality system for anatomic visualization and training arthroscopic knee surgeons. Virtual models used in our systems are constructed from the Visible Human Project and Chinese Visible Human data sets. We present our various developments in segmentation, personal-computer-based real-time volume visualization, soft tissue deformation with topological change in real-time using finite element analysis, and soft tissue cutting with tactile feedback.

Virtual reality simulation of fluoroscopic navigation

Surgical navigation systems assist surgeons by tracking targets attached to the patient's bones and to the surgical tools. Fluoroscopic navigation is a technology that adds "live" imaging to surgical navigation. In order to identify the limitations and potential sources of errors in fluoroscopic navigation, a virtual environment is created in which all the key steps of fluoroscopic navigation can be rehearsed. This provides a low-cost, highly portable, and radiation-free training environment with the ability of instant accuracy validation and reliable measurement of trainee progress. This virtual environment also is suitable for comparing different fluoroscopic registration techniques and protocols. Although the imaging component of the navigation is simulated, physical interaction with the environment can be modeled at different levels; from a fully virtual environment, to miniature models, to life-size dummies and cadavers. Each of these options satisfies different specific training needs, such as the need for portability and easy setup to the need for a completely realistic simulation of the entire clinical environment and operating procedure. In this study, we present the example of a system based on a scaled-down model with a specific focus on analyzing and training bony landmark localization using fluoroscopic navigation.

Report of a group developing a virtual reality simulator for arthroscopic surgery of the knee joint

Apprenticeship training of surgical skills is time consuming and can lead to surgical errors. Our group is developing an arthroscopic virtual reality knee simulator for training orthopaedic residents in arthroscopic surgery before live-patient operating room experience. The simulator displays realistic human knee anatomy derived from the Visible Human Dataset developed by the National Library of Medicine and incorporates active force-feedback haptic technology. Our premise is that postgraduate year 2 residents completing a formal virtual education program who are trained to reach a proficiency standard in the techniques and protocol for an arthroscopic knee examination will complete a diagnostic arthroscopy on an actual patient in less time with greater accuracy, less iteration of movement of the arthroscope, and less damage to the patient's tissue compared with residents in the control group learning and practicing the arthroscopic knee examination procedures through the residency program's established education and training program. The validation study, done at eight orthopaedic residency programs, will commence in early 2006 and will take one year to complete. We anticipate that proficiency obtained on the simulator will transfer to surgical skills in the operating room.

Design and evaluation of International Video Teleconference (iVTC) for orthopedic trauma education

This poster describes the design and evaluation of an International Video Teleconference (iVTC) system for orthopedic trauma case studies. Three medical facilities in the United States and one in Australia participated in monthly sessions where past and ongoing military and civilian cases were discussed. Participant feedback indicated that iVTC fully met their expectations as an educational tool and that remote participation did not adversely impact their ability to engage in discussion.

Maintaining standards in surgical training

In order to assess the efficacy of inspection and accreditation by the Specialist Advisory Committee for higher surgical training in orthopaedic surgery and trauma, seven training regions with 109 hospitals and 433 Specialist Registrars were studied over a period of two years.

There were initial deficiencies in a mean of 14.8% of required standards (10.3% to 19.2%). This improved following completion of the inspection, with a mean residual deficiency in 8.9% (6.5% to 12.7%.) Overall, 84% of standards were checked, 68% of the units improved and training was withdrawn in 4%.

Most units (97%) were deficient on initial assessment. Moderately good rectification was achieved but the process of follow-up and collection of data require improvement. There is an imbalance between the setting of standards and their implementation. Any major revision of the process of accreditation by the new Post-graduate Medical Education and Training Board should recognise the importance of assessment of training by direct inspection on site, of the relationship between service and training, and the advantage of defining mandatory and developmental standards.