Crowdsourced Assessment of Surgical Skill Proficiency in Cataract Surgery

Fluoroscopic image-based behavior analysis can objectively explain subjective expert assessment of wire navigation skill

Psychomotor skill and decision-making efficiency in surgical wire navigation can be objectively evaluated by analysis of intraoperative fluoroscopic image sequences. Prior work suggests that such image-based behavior analysis of operating room (OR) performance can predict performer experience level (R2  = 0.62) and agree with expert opinion (the current standard) on the quality of a final implant construct (R2  = 0.59). However, it is unclear how objective image-based evaluation compares with expert assessments for entire technical OR performances. This study examines the relationships between three key variables: (1) objective image-based criteria, (2) expert opinions, and (3) performing surgeon experience level. A paired-comparison survey of seven experts, based upon eight OR fluoroscopic wire navigation image sequences, shows that the experts' preferences are best explained by objective metrics that reflect psychomotor and decision-making behaviors which are counter-productive to successful implant placement, like image count (R2  = 0.83) and behavior tally (R2  = 0.74). One such behavior, adjustments away from goal, uniquely correlated well with all three key variables: a fluoroscopic image-based analysis composite score (R2  = 0.40), expert consensus (R2  = 0.76), and performer experience (R2  = 0.41). These results confirm that experts view less efficient technical behavior as indicative of lesser technical proficiency. While expert assessments of technical skill were reliable and consistent, neither individual nor consensus expert opinion appears to correlate with performer experience (R2  = 0.11).

Using the language of surgery to enhance ophthalmology surgical education

Background

Currently, surgical education utilizes a combination of the apprentice model, wet-lab training, and simulation, but due to reliance on subjective data, the quality of teaching and assessment can be variable. The "language of surgery," an established concept in engineering literature whose incorporation into surgical education has been limited, is defined as the description of each surgical maneuver using quantifiable metrics. This concept is different from the traditional notion of surgical language, generally thought of as the qualitative definitions and terminology used by surgeons.

Methods

A literature search was conducted through April 2023 using MEDLINE/PubMed using search terms to investigate wet-lab, virtual simulators, and robotics in ophthalmology, along with the language of surgery and surgical education. Articles published before 2005 were mostly excluded, although a few were included on a case-by-case basis.

Results

Surgical maneuvers can be quantified by leveraging technological advances in virtual simulators, video recordings, and surgical robots to create a language of surgery. By measuring and describing maneuver metrics, the learning surgeon can adjust surgical movements in an appropriately graded fashion that is based on objective and standardized data. The main contribution is outlining a structured education framework that details how surgical education could be improved by incorporating the language of surgery, using ophthalmology surgical education as an example.

Conclusion

By describing each surgical maneuver in quantifiable, objective, and standardized terminology, a language of surgery can be created that can be used to learn, teach, and assess surgical technical skill with an approach that minimizes bias.

Key message

The "language of surgery," defined as the quantification of each surgical movement's characteristics, is an established concept in the engineering literature. Using ophthalmology surgical education as an example, we describe a structured education framework based on the language of surgery to improve surgical education.

Classifications

Surgical education, robotic surgery, ophthalmology, education standardization, computerized assessment, simulations in teaching.

Competencies

Practice-Based Learning and Improvement.

Ophthalmology Residency in the United States: The Case for a National Curriculum

To identify strategies for effective curriculum development and implementation in United States (US) ophthalmology residency training programs. A literature review was conducted for all English-language PubMed/Medline articles relating to ophthalmology residency education or curriculum/curricula. Despite ACGME-defined program requirements outlining curricular goals for US ophthalmology residency training programs, there is no comprehensive, national curriculum with detailed plans for instruction of necessary topics within the 36-month residency training period. Several articles identify a need for detailed curricula on various topics, propose ideas on how residency programs could create curricula, and explore ways of assessing resident competence. There is a paucity of literature evaluating how ophthalmology residents best learn various ophthalmology topics. We need to develop an intentional, comprehensive, and timely national curriculum for ophthalmology residency programs in the US, with detailed plans on how to meet curricular objectives and consideration of the most effective teaching strategies for different ophthalmology concepts.

Dedicated Chalazion Clinic as a Tool for Early Surgical Education in Ophthalmology Residency

Objective  Ophthalmology residency programs aim to improve resident surgical teaching through increased surgical exposure over a 4-year period. Resident-centric surgical clinics across various surgical specialties have been established to help develop surgical autonomy and experience. We present the first demonstration of a resident-centric chalazion incision and drainage clinic (chalazion clinic) in an ophthalmology residency with the goal of increasing early surgical exposure to residents. Design  The chalazion clinic was founded in July 2019. It is a once weekly procedure clinic conducted by an ophthalmology resident and supervised by an ophthalmology attending. Patients with chalazia were referred directly to this clinic for evaluation and management, rather than the oculoplastics clinic as they were in the past. Retrospective review of Accreditation Council for Graduate Medical Education (ACGME) case logs of all residents per academic year before and after establishment of the chalazion clinic was performed in order to assess the impact on residents' chalazion procedures numbers per academic year. Setting  The study involved a single academic ophthalmology department. Participants  Ophthalmology residents of all years were present. Results  A resident of any year performed an average of 3.0 chalazion procedures per year in the 2018 to 2019 academic year, 3.8 in 2019 to 2020, and 8.4 in the 2020 to 2021, which represents a 180% increase in procedure numbers per resident. Among post-graduate-year 2s (PGY)2s, the average number of chalazion procedures increased from 2.1 procedure per year to 22.3 per year (961.9% increase). Conclusion  To the best of our knowledge, this is the first description of a dedicated resident-centric chalazion clinic in an ophthalmology residency program. PGY2s demonstrated the largest increase in procedural numbers. While chalazion incision and drainage is a minor procedure, increased exposure to surgical procedures early in training could help improve residents' skills and confidence. This clinic provides a proof of concept of a dedicated minor procedure clinic for ophthalmology residents to increase early procedural volume.

A Virtual Reading Center Model Using Crowdsourcing to Grade Photographs for Trachoma: Validation Study (Preprint)

BACKGROUND

As trachoma is eliminated, skilled field graders become less adept at correctly identifying active disease (trachomatous inflammation-follicular [TF]). Deciding if trachoma has been eliminated from a district or if treatment strategies need to be continued or reinstated is of critical public health importance. Telemedicine solutions require both connectivity, which can be poor in the resource-limited regions of the world in which trachoma occurs, and accurate grading of the images.

OBJECTIVE

Our purpose was to develop and validate a cloud-based "virtual reading center" (VRC) model using crowdsourcing for image interpretation.

METHODS

The Amazon Mechanical Turk (AMT) platform was used to recruit lay graders to interpret 2299 gradable images from a prior field trial of a smartphone-based camera system. Each image received 7 grades for US $0.05 per grade in this VRC. The resultant data set was divided into training and test sets to internally validate the VRC. In the training set, crowdsourcing scores were summed, and the optimal raw score cutoff was chosen to optimize kappa agreement and the resulting prevalence of TF. The best method was then applied to the test set, and the sensitivity, specificity, kappa, and TF prevalence were calculated.

RESULTS

In this trial, over 16,000 grades were rendered in just over 60 minutes for US $1098 including AMT fees. After choosing an AMT raw score cut point to optimize kappa near the World Health Organization (WHO)-endorsed level of 0.7 (with a simulated 40% prevalence TF), crowdsourcing was 95% sensitive and 87% specific for TF in the training set with a kappa of 0.797. All 196 crowdsourced-positive images received a skilled overread to mimic a tiered reading center and specificity improved to 99%, while sensitivity remained above 78%. Kappa for the entire sample improved from 0.162 to 0.685 with overreads, and the skilled grader burden was reduced by over 80%. This tiered VRC model was then applied to the test set and produced a sensitivity of 99% and a specificity of 76% with a kappa of 0.775 in the entire set. The prevalence estimated by the VRC was 2.70% (95% CI 1.84%-3.80%) compared to the ground truth prevalence of 2.87% (95% CI 1.98%-4.01%).

CONCLUSIONS

A VRC model using crowdsourcing as a first pass with skilled grading of positive images was able to identify TF rapidly and accurately in a low prevalence setting. The findings from this study support further validation of a VRC and crowdsourcing for image grading and estimation of trachoma prevalence from field-acquired images, although further prospective field testing is required to determine if diagnostic characteristics are acceptable in real-world surveys with a low prevalence of the disease.