Assessing the Quality, Content, and Readability of Freely Available Online Information for Patients Regarding Age-Related Macular Degeneration

Appropriateness and Readability of ChatGPT-4-Generated Responses for Surgical Treatment of Retinal Diseases

OBJECTIVE

To evaluate the appropriateness and readability of the medical knowledge provided by ChatGPT-4, an artificial intelligence-powered conversational search engine, regarding common vitreoretinal surgeries for retinal detachments (RDs), macular holes (MHs), and epiretinal membranes (ERMs).

DESIGN

Retrospective cross-sectional study.

SUBJECTS

This study did not involve any human participants.

METHODS

We created lists of common questions about the definition, prevalence, visual impact, diagnostic methods, surgical and nonsurgical treatment options, postoperative information, surgery-related complications, and visual prognosis of RD, MH, and ERM, and asked each question 3 times on the online ChatGPT-4 platform. The data for this cross-sectional study were recorded on April 25, 2023. Two independent retina specialists graded the appropriateness of the responses. Readability was assessed using Readable, an online readability tool.

MAIN OUTCOME MEASURES

The "appropriateness" and "readability" of the answers generated by ChatGPT-4 bot.

RESULTS

Responses were consistently appropriate in 84.6% (33/39), 92% (23/25), and 91.7% (22/24) of the questions related to RD, MH, and ERM, respectively. Answers were inappropriate at least once in 5.1% (2/39), 8% (2/25), and 8.3% (2/24) of the respective questions. The average Flesch Kincaid Grade Level and Flesch Reading Ease Score were 14.1 ± 2.6 and 32.3 ± 10.8 for RD, 14 ± 1.3 and 34.4 ± 7.7 for MH, and 14.8 ± 1.3 and 28.1 ± 7.5 for ERM. These scores indicate that the answers are difficult or very difficult to read for the average lay person and college graduation would be required to understand the material.

CONCLUSIONS

Most of the answers provided by ChatGPT-4 were consistently appropriate. However, ChatGPT and other natural language models in their current form are not a source of factual information. Improving the credibility and readability of responses, especially in specialized fields, such as medicine, is a critical focus of research. Patients, physicians, and laypersons should be advised of the limitations of these tools for eye- and health-related counseling.

FINANCIAL DISCLOSURE(S)

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Effective health communication for age-related macular degeneration: An exploratory qualitative study

PURPOSE

Age-related macular degeneration (AMD) is a major cause of vision loss globally. Patients with AMD may not always understand or retain the information about AMD communicated by their eyecare practitioner. This study aims to explore the characteristics of effective health communication for AMD, from both patients' and eyecare practitioners' perspectives. The purpose is to provide a foundation for understanding how health communication for AMD could potentially be improved in the future.

METHODS

A total of 10 focus groups involving 17 patients with AMD and 17 optometrists were conducted via web conferencing. Each session was audio-recorded, transcribed and analysed using the Grounded Theory Methodology.

RESULTS

The five themes identified are as follows: (1) materials' quality, (2) materials' relevance, (3) contextualising for the individual, (4) contextualising for the disease and (5) support network. Participants expressed concern about the unrealistic yet common depiction of vision loss in AMD as a black patch overlying common visual scenes. They also preferred education materials tailored to a specific disease stage and the regular opportunity to ask or answer questions. Longer appointment durations and peer support (from family, friends or others with AMD) were also valued.

CONCLUSION

Optometrists are encouraged to focus on three over-arching dimensions when counselling patients with AMD in routine clinical practice: (1) curating and using disease and stage-specific, impactful education materials, (2) their chairside verbal communication techniques and (3) AMD-specific opportunities for care coordination among patient family and friends, peers and other multidisciplinary members of the care support team.

Content, Readability, and Accountability of Online Health Information for Patients Regarding Blue Light and Impact on Ocular Health

Objective To evaluate the quality and readability of online health content regarding the ocular health effects of blue light. Methods Five commercial and five non-commercial websites with content regarding the ocular effect of blue light were examined. Quality evaluations were conducted using a 14-question assessment composed by the authors and the 16-question DISCERN instrument. Website accountability was evaluated via the Journal of the American Medical Association (JAMA) benchmarks. Readability was determined using an online tool (Readable). Correlational and comparative analyses were conducted where appropriate. Results The average questionnaire score was 84 (standard deviation [SD] ± 17.89, 95% confidence interval [CI] 77.32-90.68) out of 136 points (61.8%). Significant differences in quality were identified between websites (p = 0.02), with Healthline achieving the highest score. Compared to commercial websites, non-commercial websites trended toward having significantly higher median questionnaire scores (p = 0.06). Zero websites achieved all four JAMA benchmarks. The average reading grade level of content was 10.43 (SD ± 1.15, 95% CI 9.60 - 11.25), with differences between websites trending toward significance (p = 0.09). There was no correlation between resource readability and quality (ρ = 0.28; p = 0.43) or accountability (ρ = 0.47; p = 0.17). Conclusions There remain substantial deficiencies in the quality, accountability, and readability of online content concerning the effect of blue light on ocular health. Clinicians and patients must recognize such issues when recommending and consuming these resources.

Assessing the quality and readability of online content on shock wave therapy for erectile dysfunction

Patients are becoming increasingly reliant on online platforms for obtaining health information. Previous research has shown that the quality of information available on the internet regarding novel medical therapies is generally poor and frequently misleading. Shock wave therapy represents a novel restorative therapy for erectile dysfunction (ED) that has recently gained attention. We hypothesised that online sources regarding shock wave therapy for ED would be fraught with misleading claims and unreliable health information. Our objective was to evaluate the quality and readability of online medical information on shock wave therapy as a treatment for ED. Websites were generated using a Google search of 'shock wave therapy for erectile dysfunction' with location filters disabled. Readability was analysed using the Readable software (Readable.com, Horsham, United Kingdom). Quality was assessed independently by three reviewers using the DISCERN tool. Articles were subdivided into those from private clinic websites and those from universities or news media websites. Statistical analysis was conducted using the Student's t test. Nine articles that resulted from the Google search had mean readability scores as follows: Flesch-Kincaid grade level (10.8), Gunning-Fog Index (13.67), Coleman-Liau Index (12.74), Simple Measure of Gobbledygook (SMOG) Index (13.33), FORCAST Grade Level (11.33), and Automated Readability Index (11.08). The mean Flesch Reading Ease score was 46.4. The articles had a mean DISCERN score of 3.1, suggesting 'moderate quality' content. Articles from universities (n = 2) or news sources (n = 3) had significantly higher DISCERN scores than articles from private medical practices (n = 4). There was no difference in readability scores between the groups. Articles from private clinics are just as readable as those from universities or news media, but they are significantly more biased and misleading. The current online material relating to shock wave therapy for ED may not adequately inform patients in their medical decisions making, thereby necessitating closer collaboration between the sources disseminating information and urologists.

What Glaucoma Patients Are Reading on the Internet: A Systematic Analysis of Online Glaucoma Content

PURPOSE

To systematically assess the content, quality, accuracy, navigability, and readability of the highest ranked online information regarding glaucoma.

DESIGN

Internet-based, cross-sectional study.

PARTICIPANTS

Not applicable.

METHODS

The top 15 Google websites that resulted from the search term "glaucoma" were independently evaluated by 3 glaucoma specialists for content, accuracy (1-4 scale), and navigability (1-3 scale) in March 2020. Content was evaluated by answers to 25 questions regarding information most relevant to glaucoma patients (0-2 scale). Readability (Flesch-Kincaid grade level) and quality (Quality Component Scoring System) were also analyzed.

MAIN OUTCOME MEASURES

Overall content, accuracy, navigability, quality component scores, and reading grade level as described. Additional analyses included type of websites, presence and type of graphic content.

RESULTS

The mean (standard deviation) content score for all websites was 28 (7.6) of 50 possible points (range, 15-45). Inter-rater reliability for content score was good (0.619, 0.746, 0.872 for 3 pairwise comparisons between graders). Mean accuracy score was 9.8 (2.2) of 12 possible points (range, 5-12). Mean navigability score was 7.5 (1.4) of 9 possible points (range, 5-9). Mean reading grade level was 9.3 (1.7) with a range of 7-13 reading grades. Mean quality component score was 7.7 (2.6) of 13 possible points (range, 3-11). Six websites (40%) attempted to visually simulate glaucoma symptoms as tunnel vision (n = 5) and black spots (n = 1). Google rank did not correlate with any of the measures.

CONCLUSIONS

The most readily accessible online information about glaucoma varies in content, quality, accuracy, navigability, and readability. This systematic analysis identifies potential areas of improvement.

Assessment of patient education materials for age-related macular degeneration

Purpose

Age‐related macular degeneration (AMD) is a leading cause of vision loss. It is helpful for patients living with AMD to understand the prognosis, risk factors and management of their condition. Online education materials are a popular and promising channel for conveying this knowledge to patients with AMD. However, the quality of these materials—particularly with respect to qualities such as ‘understandability’ and ‘actionability’—is not yet known. This study assessed a collection of online materials about AMD based on these qualities of ‘understandability’ and ‘actionability’.

Methods

Online education materials about AMD were sourced through Google from six English‐speaking nations: Australia, New Zealand, USA, UK, Ireland and Canada. Three Australian/New Zealand trained and registered optometrists participated in the grading of the ‘understandability’ and ‘actionability’ of online education materials using the Patient Education Materials Assessment Tool (PEMAT).

Results

This study analysed a total of 75 online materials. The mean ‘understandability’ score was 74% (range: 38%–94%). The ‘understandability’ PEMAT criterion U11 (calling for a summary of the key points) scored most poorly across all materials. The mean ‘actionability’ score was 49% (range: 0%–83%). The ‘actionability’ PEMAT criterion A26 (using ‘visual aids’ to make instructions easier to act on) scored most poorly across all materials.

Conclusion

Most education materials about AMD are easy to understand, but difficult to act on, because of a lack of meaningful visual aids. We propose future enhancements to AMD education materials—including the use of summaries, visual aids and a habit tracker—to help patients with AMD improve their understanding of disease prognosis, risk factors and eye assessment schedule requirements.