Physician Rating Scales Do Not Accurately Rate Physicians

Using Sentiment Analysis to Understand What Patients Are Saying About Hand Surgeons Online

BACKGROUND

Physician review websites have influence on a patient's selection of a provider. Written reviews are subjective and difficult to quantitatively analyze. Sentiment analysis of writing can quantitatively assess surgeon reviews to provide actionable feedback for surgeons to improve practice. The objective of this study is to quantitatively analyze large subset of written reviews of hand surgeons using sentiment analysis and report unbiased trends in words used to describe the reviewed surgeons and biases associated with surgeon demographic factors.

METHODS

Online written and star-rating reviews of hand surgeons were obtained from healthgrades.com and webmd.com. A sentiment analysis package was used to calculate compound scores of all reviews. Mann-Whitney U tests were performed to determine the relationship between demographic variables and average sentiment score of written reviews. Positive and negative word and word-pair frequency analysis was also performed.

RESULTS

A total of 786 hand surgeons' reviews were analyzed. Analysis showed a significant relationship between the sentiment scores and overall average star-rated reviews (r2 = 0.604, P ≤ .01). There was no significant difference in review sentiment by provider sex; however, surgeons aged 50 years and younger had more positive reviews than older (P < .01). The most frequently used bigrams used to describe top-rated surgeons were associated with good bedside manner and efficient pain management, whereas those with the worst reviews are often characterized as rude and unable to relieve pain.

CONCLUSIONS

This study provides insight into both demographic and behavioral factors contributing to positive reviews and reinforces the importance of pain expectation management.

Building better pediatric surgeons: A sentiment analysis of online physician review websites

PURPOSE

Physician review websites are a heavily utilized patient tool for finding, rating, and reviewing surgeons. Natural language processing such as sentiment analysis provides a comprehensive approach to better understand the nuances of patient perception. This study utilizes sentiment analysis to examine how specific patient sentiments correspond to positive and negative experiences in online reviews of pediatric orthopedic surgeons.

METHODS

The online written reviews and star ratings of pediatric surgeons belonging to the Pediatric Orthopaedic Society of North America were obtained from healthgrades.com. A sentiment analysis package obtained compound scores of each surgeon's reviews. Inferential statistics analyzed relationships between demographic variables and star/sentiment scores. Word frequency analyses and multiple logistic regression analyses were performed on key terms.

RESULTS

A total of 749 pediatric surgeons (3830 total online reviews) were included. 80.8% were males and 33.8% were below 50 years of age. Male surgeons and younger surgeons had higher mean star ratings. Surgeon attributes including "confident" (p < 0.01) and "comfortable" (p < 0.01) improved the odds of positive reviews, while "rude" (p < 0.01) and "unprofessional" (p < 0.01) decreased these odds. Comments regarding "pain" lowered the odds of positive reviews (p < 0.01), whereas "pain-free" increased these odds (p < 0.01).

CONCLUSION

Pediatric surgeons who were younger, communicated effectively, eased pain, and curated a welcoming office setting were more likely to receive positively written online reviews. This suggests that a spectrum of interpersonal and ancillary factors impact patient experience and perceptions beyond surgical skill. These outcomes can advise pediatric surgeons on behavioral and office qualities that patients and families prioritize when rating/recommending surgeons online.

LEVEL OF EVIDENCE

IV.

Patient Satisfaction Scores Are a Poor Metric of Orthopedic Care

Patient satisfaction scores are a popular metric used to evaluate orthopedic care. There is little consistency with how satisfaction is described in the orthopedic literature. Online physician reviews are a growing trend that directly and indirectly affect a surgeon's reputation. There is little correlation of higher satisfaction with improved surgical outcomes, so rating surgical care may be misguided and possibly dangerous. Patient satisfaction is an important part of the patient-centered care model, so rating systems should directly reflect quality. More research is needed to determine the relationship between patient satisfaction and the delivery of quality care. [Orthopedics. 2022;45(3):e127-e133.].

Patient Utilization of Online Information and its Influence on Orthopedic Surgeon Selection: Cross-sectional Survey of Patient Beliefs and Behaviors

Background

Patient attitudes and behavior are critical to understand owing to the increasing role of patient choice. There is a paucity of investigation into the perceived credibility of online information and whether such information impacts how patients choose their surgeons.

Objective

The purpose of this study was to explore the attitudes and behavior of patients regarding online information and orthopedic surgeon selection. Secondary purposes included gaining insight into the relative importance of provider selection factors, and their association with patient age and education level.

Methods

This was a cross-sectional study involving five multispecialty orthopedic surgery groups. A total of 329 patients who sought treatment by six different orthopedic surgeons were asked to anonymously answer a questionnaire consisting of 25 questions. Four questions regarded demographic information, 10 questions asked patients to rate the importance of specific criteria regarding the selection of their orthopedic surgeon (on a 4-point Likert scale), and 6 questions were designed to determine patient attitude and behaviors related to online information.

Results

Patient-reported referral sources included the emergency room (29/329, 8.8%), friend (42/329, 12.8%), insurance company (47/329, 14.3%), internet search/website (28/329, 8.5%), primary care physician (148/329, 45.0%), and other (34/329, 10.3%). Among the 329 patients, 130 (39.5%) reported that they searched the internet for information before their first visit. There was a trend of increased belief in online information to be accurate and complete in younger age groups (P=.02). There was an increased relative frequency in younger groups to perceive physician rating websites to be unbiased (P=.003), provide sufficient patient satisfaction information (P=.01), and information about physician education and training (P=.03). There was a significant trend for patients that found a surgeon’s website to be useful (P<.001), with the relative frequency increased in younger age groups.

Conclusions

This study shows that insurance network, physician referrals, appointment availability, and office location are important to patients, whereas advertising and internet reviews by other patients were considered to be not as helpful in choosing an orthopedic surgeon. Future studies may seek to identify obstacles to patients in integrating online resources for decision-making and strategies to improve health-seeking behaviors.

Comparison of Internal Patient Satisfaction Scores at a Cancer Center With Star Ratings on Online Physician-Rating Websites

PURPOSE

Patients have been increasingly using physician-rating websites (PRWs); however, few studies have analyzed the validity of star ratings on PRWs. We aimed to compare PRW patient satisfaction scores with internally generated patient satisfaction scores (internal scores) of physicians at a large quaternary cancer center.

METHODS

We collected internal scores and PRW scores for physicians at MD Anderson Cancer Center. Internal scores were based on patient responses to the Clinician and Group Consumer Assessment of Healthcare Providers and Systems patient experience (CG-CAHPS) survey. Only physicians with an internal score on the basis of ≥ 30 patient reviews were included. The median numbers of reviews and median scores were compared between internal data and four PRWs (Google, HealthGrades, Vitals, and WebMD). Both internally and on PRWs, possible scores ranged from 1 (least satisfied) to 5 (most satisfied).

RESULTS

Of 640 physicians with an internal score, 510 (79.7%) met the inclusion criteria. For these 510 physicians, the median (IQR) number of internal reviews was 49.5 (30-93) and the median (IQR) internal score was 4.89 (4.81-4.93); the median number of reviews on PRWs ranged from 2 to 7, and the median score on PRWs ranged from 4.40 to 5.00. No physician had an internal score < 4, but the proportions with score < 4 on PRWs ranged from 16% to 30%.

CONCLUSION

Internal patient satisfaction scores were higher and calculated from more reviews than PRW patient satisfaction scores and correlated weakly with PRW scores. Given that patients rely on PRWs when evaluating potential physicians, we recommend publishing internal scores online to give patients more complete information regarding physician performance.

Is There an Association Between Physician Review Websites and Press Ganey Survey Results in a Neurosurgical Outpatient Clinic?

OBJECTIVE

Recent studies suggest a poor association between physician review websites and the validated metrics used by the Centers for Medicare and Medicaid Services. The purpose of this study was to evaluate the association between online and outpatient Press Ganey (PG) measures of patient satisfaction in a neurosurgical department.

METHODS

We obtained PG survey results from one large academic institution's outpatient neurosurgery clinic. Popular physician review websites were searched for each of the faculty captured in the PG data. Average physician rating and percent Top Box scores were calculated for each physician. PG data were separated into new and established clinic visits for subset analysis. Spearman's rank correlation coefficients were calculated to determine associations.

RESULTS

Twelve neurosurgeons were included. Established patients demonstrated greater PG scores as compared with new patients, with an average physician rating increase of 0.55 and an average Top Box increase of 12.5%. Online physician ratings were found to demonstrate strong agreement with PG scores for the entire PG population, new patient subset, and established patient subset (ρ = 0.77-0.79, P < 0.05). Online Top Box scores demonstrated moderate agreement with overall PG Top Box scores (ρ = 0.59, P = 0.042), moderate agreement with the new patient population Top Box scores (ρ = 0.56, P = 0.059), and weak agreement with established patient population Top Box scores (ρ = 0.38, P = 0.217).

CONCLUSIONS

Our findings demonstrated a strong agreement between PG ratings and online physician ratings and a poorer correlation when comparing PG Top Box scores with online physician Top Box scores, particularly in the established patient population.