Quantitative readability analysis of websites providing information on traumatic brain injury and epilepsy: A need for clear communication

Assessing AI Simplification of Medical Texts: Readability and Content Fidelity

INTRODUCTION

The escalating complexity of medical literature necessitates tools to enhance readability for patients. This study aimed to evaluate the efficacy of ChatGPT-4 in simplifying neurology and neurosurgical abstracts and patient education materials (PEMs) while assessing content preservation using Latent Semantic Analysis (LSA).

METHODS

A total of 100 abstracts (25 each from Neurosurgery, Journal of Neurosurgery, Lancet Neurology, and JAMA Neurology) and 340 PEMs (66 from the American Association of Neurological Surgeons, 274 from the American Academy of Neurology) were transformed by a GPT-4.0 prompt requesting a 5th grade reading level. Flesch-Kincaid Grade Level (FKGL) and Flesch Reading Ease (FKRE) scores were used before/after transformation. Content fidelity was validated via LSA (ranging 0-1, 1 meaning identical topics) and by expert assessment (0-1) for a subset (n = 40). Pearson correlation coefficient compared assessments.

RESULTS

FKGL decreased from 12th to 5th grade for abstracts and 13th to 5th for PEMs (p < 0.001). FKRE scores showed similar improvement (p < 0.001). LSA confirmed high content similarity for abstracts (mean cosine similarity 0.746) and PEMs (mean 0.953). Expert assessment indicated a mean topic similarity of 0.775 for abstracts and 0.715 for PEMs. The Pearson coefficient between LSA and expert assessment of textual similarity was 0.598 for abstracts and -0.167 for PEMs. Segmented analysis of similarity correlations revealed a correlation of 0.48 (p = 0.02) below 450 words and a -0.20 (p = 0.43) correlation above 450 words.

CONCLUSION

GPT-4.0 markedly improved the readability of medical texts, predominantly maintaining content integrity as substantiated by LSA and expert evaluations. LSA emerged as a reliable tool for assessing content fidelity within moderate-length texts, but its utility diminished for longer documents, overestimating similarity. These findings support the potential of AI in combating low health literacy, however, the similarity scores indicate expert validation is crucial. Future research must strive to improve transformation precision and develop validation methodologies.

Assessing the Quality, Readability, and Acceptability of AI-Generated Information in Plastic and Aesthetic Surgery

INTRODUCTION

Within plastic surgery, a patient's most commonly used first point of information before consulting a surgeon is the internet. Free-to-use artificial intelligence (AI) websites like ChatGPT (Generative Pre-trained Transformers) are attractive applications for patient information due to their ability to instantaneously answer almost any query. Although relatively new, ChatGPT is now one of the most popular artificial intelligence conversational software tools. The aim of this study was to evaluate the quality and readability of information given by ChatGPT-4 on key areas in plastic and reconstructive surgery.

METHODS

The ten plastic and aesthetic surgery topics with the highest worldwide search volume in the 15 years were identified. These were rephrased into question format to create nine individual questions. These questions were then input into ChatGPT-4. The response quality was assessed using the DISCERN. The readability and grade reading level of the responses were calculated using the Flesch-Kincaid Reading Ease Index and Coleman-Liau Index. Twelve physicians working in a plastic and reconstructive surgery unit were asked to rate the clarity and accuracy of the answers on a scale of 1-10 and state 'yes or no' if they would share the generated response with a patient.

RESULTS

All answers were scored as poor or very poor according to the DISCERN tool. The mean DISCERN score for all questions was 34. The responses also scored low in readability and understandability. The mean FKRE index was 33.6, and the CL index was 15.6. Clinicians working in plastics and reconstructive surgery rated the questions well in clarity and accuracy. The mean clarity score was 7.38, and the accuracy score was 7.4.

CONCLUSION

This study found that according to validated quality assessment tools, ChatGPT-4 produced low-quality information when asked about popular queries relating to plastic and aesthetic surgery. Furthermore, the information produced was pitched at a high reading level. However, the responses were still rated well in clarity and accuracy, according to clinicians working in plastic surgery. Although improvements need to be made, this study suggests that language models such as ChatGPT could be a useful starting point when developing written health information. With the expansion of AI, improvements in content quality are anticipated.

Oral cancer: analysis of the clarity of publications in Instagram profiles of official health agencies in Brazil

This article aims to analyze the clarity and engagement measures of publications on oral cancer in the Instagram profiles of official health agencies in Brazil. An infodemiological study was conducted with 81 profiles. Data collected concerned content classification, account and media, manner of addressing the topic, number of posts, likes, comments, views and hashtags. The clarity of the educational publications was assessed with the Brazilian version of the Clear Communication Index (BR-CDC-CCI). Data analysis involved Spearman's correlation and the Mann-Whitney test (α = 5%). A total of 775 publications on oral cancer were found. The average BR-CDC-CCI score was 69.8 (SD = 15.5). The clarity of the information was adequate in 9.5% of the educational publications. Positive correlations were found between the number of likes and engagement (comments [r = 0.49], views [r = 0.96]), number of hashtags (r = 0.13) and year of publication (r = 0.21). Publications from the Health Ministry had a significantly higher BR-CDC-CCI score compared to the other profiles. Publications on oral cancer were correlated with engagement, year of publication and number of hashtags. Public agencies increased publications to reach the population, but the clarity of the content was low.

The Birth of the Contextual Health Education Readability Score in an Examination of Online Influenza Patient Education Materials

Introduction Influenza is a major global health concern, with its rapid spread and mutation rate posing significant challenges in public health education and communication. Effective patient education materials (PEMs) are crucial for informed decision-making and improved health outcomes. This study evaluates the efficacy of online influenza PEMs using traditional readability tools and introduces the Contextual Health Education Readability Score (CHERS) to address the limitations of existing methods that do not capture the diverse array of visual and thematic means displayed. Materials and methods A comprehensive search was conducted to select relevant online influenza PEMs. This involved looking through Google's first two pages of results sorted by relevance, for a total of 20 results. These materials were evaluated using established readability tools (e.g., Flesch Reading Ease, Flesch-Kincaid Grade Level) and the Patient Education Materials Assessment Tool (PEMAT) for understandability and actionability. The study also involved the creation of CHERS, integrating factors such as semantic complexity, cultural relevance, and visual aid effectiveness. The development of CHERS included weighting each component based on its impact on readability and comprehension. Results The traditional readability tools demonstrated significant variability in the readability of the selected materials. The PEMAT analysis revealed general trends toward clarity in purpose and use of everyday language but indicated a need for improvement in summaries and visual aids. The CHERS formula was calculated as follows: CHERS = (0.4 × Average Sentence Length) + (0.3 × Average Syllables per Word) + (0.15 × Semantic Complexity Score) + (0.1 × Cultural Relevance Score) + (0.05 × Visual Aid Effectiveness Score), integrating multiple dimensions beyond traditional readability metrics. Discussion The study highlighted the limitations of traditional readability tools in assessing the complexity and cultural relevance of health information. The introduction of CHERS addressed these gaps by incorporating additional dimensions crucial for understanding in a healthcare context. The recommendations provided for creating effective influenza PEMs focused on language simplicity, cultural sensitivity, and actionability. This may enable further research into evaluating current PEMs and clarifying means of creating more effective content in the future. Conclusions The study underscores the need for comprehensive readability assessments in PEMs. The creation of CHERS marks a significant advancement in this field, providing a more holistic approach to evaluating health literacy materials. Its application could lead to the development of more inclusive and effective educational content, thereby improving public health outcomes and reducing the global burden of influenza. Future research should focus on further validating CHERS and exploring its applicability to other health conditions.

Characterizing Health Literacy and Its Correlates Among Individuals With Traumatic Brain Injury (TBI): A TBI Model Systems Study

OBJECTIVE

To characterize health literacy among individuals with traumatic brain injury (TBI) at least a year postinjury and to explore its relationship to sociodemographic variables, injury severity, and cognition.

SETTING

Community following discharge from inpatient rehabilitation.

PARTICIPANTS

In total, 205 individuals with complicated mild to severe TBI who completed follow-up as part of a national longitudinal study of TBI and completed a web-based health literacy measure.

DESIGN

Multicenter, cross-sectional, observational study.

MAIN MEASURES

Health Literacy Assessment Using Talking Touchscreen Technology.

RESULTS

Thirty-one percent of the sample demonstrated marginal/inadequate health literacy; 69% demonstrated adequate health literacy. A higher proportion of non-Hispanic White adults had adequate health literacy than non-Hispanic Black and Hispanic adults. Individuals with greater than a high school education were more likely to have adequate health literacy than those with a high school education or less. Better executive functioning performance was related to adequate health literacy. Better episodic memory performance was related to adequate health literacy, but only for those with complicated mild to moderate injury.

CONCLUSIONS

A substantial proportion of individuals with TBI have marginal/inadequate health literacy, which may impact their understanding, appreciation, and use of health-related information and recommendations. While low health literacy may be preexisting, directly related to TBI, or a combination of both, it should be screened and considered by professionals when communicating with persons with TBI. Healthcare providers should tailor their communication approaches and presentation of health information, particularly for those with low health literacy.

The Relationship of Health Literacy to Health Outcomes Among Individuals With Traumatic Brain Injury: A Traumatic Brain Injury Model Systems Study

OBJECTIVE

To examine the associations between health literacy and health outcomes among individuals with traumatic brain injury (TBI) at least a year post-injury.

SETTING

Community following discharge from inpatient rehabilitation.

PARTICIPANTS

A total of 205 individuals with complicated mild to severe TBI who completed a TBI Model Systems National Database follow-up interview and a web-based health literacy measure.

DESIGN

A multicenter, cross-sectional, observational study.

MAIN MEASURES

The Health Literacy Assessment Using Talking Touchscreen Technology (Health LiTT), number of comorbid conditions (Medical and Mental Health Comorbidities Interview [MMHCI]), perceived physical and mental health (PROMIS Global Physical and Mental Health subscales), Patient Health Questionnaire-9 (PHQ-9), and Generalized Anxiety Disorder-7 (GAD-7).

RESULTS

After controlling for sociodemographic, injury, cognition, and time post-injury, adequate health literacy was associated with higher odds of greater perceived physical health compared with participants with marginal/inadequate health literacy (odds ratio = 4.10; CI = 1.52-11.70]. Participants with inadequate/marginal health literacy had 3.50 times greater odds of depression (PHQ-9 ≥ 10) compared with those with adequate health literacy. Participants 45 years and older reported a greater number of MMHCI physical health conditions, but fewer MMHCI mental health conditions and GAD-7 anxiety symptoms compared with those who were younger. Non-Hispanic White participants and those with mild/moderate TBI were more likely to report a greater number of MMHCI mental health conditions compared with non-Hispanic Black participants or those with severe TBI. Greater time post-injury was associated with greater number of chronic physical and mental health conditions, and less odds of good-to-excellent perceived global mental health.

CONCLUSIONS

Inadequate health literacy is associated with worse perceived physical health and greater depressive symptoms among adults with TBI. Greater efforts are needed to explore the mechanisms by which health literacy influences chronic disease management and mental health after TBI to improve postinjury health status and outcomes, particularly among those with limited health literacy skills.

A Readability Study of Carpal Tunnel Release in 2023

BACKGROUND

The National Institutes of Health (NIH) and the American Medical Association (AMA) recommend a sixth-grade reading level for patient-directed content. This study aims to quantitatively evaluate the readability of online information sources related to carpal tunnel surgery using established readability indices.

METHODS

Web searches for "carpal tunnel release" and "carpal tunnel decompression surgery" queries were performed using Google, and the first 20 websites were identified per query. WebFX online software tools were utilized to determine readability. Indices included Flesch Kincaid Reading Ease, Flesch Kincaid Grade Level, Coleman Liau Index, Automated Readability Index, Gunning Fog Score, and the Simple Measure of Gobbledygook Index. Health-specific clickthrough rate (CTR) data were used in order to select the first 20 search engine results page from each query.

RESULTS

"Carpal tunnel release" had a mean readability of 8.46, and "carpal tunnel decompression surgery" had a mean readability of 8.70. The range of mean readability scores among the indices used for both search queries was 6.17 to 14.0. The total mean readability for carpal tunnel surgery information was found to be 8.58. This corresponds to approximately a ninth-grade reading level in the United States.

CONCLUSION

The average readability of carpal tunnel surgery online content is three grade levels above the recommended sixth-grade level for patient-directed materials. This discrepancy indicates that existing online materials related to carpal tunnel surgery are more difficult to understand than the standards set by NIH and AMA.

Evaluating the performance of the language model ChatGPT in responding to common questions of people with epilepsy

OBJECTIVE

People with epilepsy desire to acquire accurate information about epilepsy and actively engage in its management throughout the long journey of living with seizures. ChatGPT is a large language model and we aimed to assess the accuracy and consistency of ChatGPT in responding to the common concerns of people with epilepsy and to evaluate its ability to provide emotional support.

METHODS

Questions were collected from the International League against Epilepsy and the China Association against Epilepsy. The responses were independently assessed by two board-certified epileptologists from the China Association against Epilepsy, and a third reviewer resolved disagreements. The reviewers assessed its ability to provide emotional support subjectively.

RESULTS

A total of 378 questions related to epilepsy and 5 questions related to emotional support were included. ChatGPT provided "correct and comprehensive" answers to 68.4% of the questions. The model provided reproducible answers for 82.3% questions. The model performed poorly in answering prognostic questions, with only 46.8% of the answers rated as comprehensive. When faced with questions requiring emotional support, the model can generate natural and understandable responses.

SIGNIFICANCE

ChatGPT provides accurate and reliable answers to patients with epilepsy and is a valuable source of information. It also provides partial emotional support, potentially assisting those experiencing emotional distress. However, ChatGPT may provide incorrect responses, leading users to inadvertently accept incorrect and potentially dangerous advice. Therefore, the direct use of ChatGPT for medical guidance is not recommended and its primary use at present is in patients education.

Readability and content gaps in online epilepsy surgery materials as potential health literacy and shared-decision-making barriers

OBJECTIVE

Epilepsy surgery is an effective albeit underused treatment for refractory epilepsy, and online materials are vital to patient understanding of the complex process. Our goal is to analyze the readability and content inclusion of online patient health education materials designed for epilepsy surgery.

METHODS

A private browser setting was used on Google and Bing to identify the top 100 search results for the terms "epilepsy+surgery". Scientific papers, insurance pages, pay-wall access sites, and non-text content were excluded. The website text was reformatted to exclude graphics, contact information, links, and headers. Readability metrics were calculated using an online tool. Text content was analyzed for inclusion of important concepts (pre-surgical evaluation, complications, risks of continued seizures, types of surgery, complimentary diagrams/audiovisual material). Comparison of readability and content inclusion was performed as a function of organization types (epilepsy center, community health organization, pediatric-specific) and location (region, country).

RESULTS

Browser search yielded 82 distinct websites with information regarding epilepsy surgery, with 98.7% of websites exceeding the recommended 6th-grade reading level for health information. Epilepsy centers had significantly worse readability (Flesch-Kincaid Grade Level (FKGL) P < 0.01 and Flesch Reading Ease (FRE) P < 0.05). Content analysis showed that only 37% of websites discuss surgical side effects and only 23% mention the risks of continued seizures. Epilepsy centers were less likely to report information on surgical side effects (P < 0.001). UK-based websites had better readability (FKGL P < 0.01 and FRE P < 0.01) and were more likely to discuss side effects (P = 0.01) compared to US-based websites.

SIGNIFICANCE

The majority of online health content is overly complex and relatively incomplete in multiple key areas important to health literacy and understanding of surgical candidacy. Our findings suggest academic organizations, including level 4 epilepsy centers, need to simplify and broaden online education resources. More comprehensive, publicly accessible, and readable information may lead to better-shared decision-making.

Evaluating a Public Health Information Service According to Users' Socioeconomic Position and Health Status: Protocol for a Cross-Sectional Study

BACKGROUND

The increasing use of information technology in the field of health is supposed to promote users' empowerment but can also reinforce social inequalities. Some health authorities in various countries have developed mechanisms to offer accurate and relevant information to health care system users, often through health websites. However, the evaluation of these sociotechnical tools is inadequate, particularly with respect to differences and inequalities in use by social groups.

OBJECTIVE

Our study aims to evaluate the access, understanding, appraisal, and use of the French website Santé.fr by users according to their socioeconomic position and perceived health status.

METHODS

This cross-sectional study involves the entire French population to which Santé.fr is offered. Data will be collected through mixed methods, including a web-based questionnaire for quantitative data and interviews and focus groups for qualitative data. Collected data will cover users' access, understanding, appraisal, and use of Santé.fr, as well as sociodemographic and socioeconomic characteristics, health status, and digital health literacy. A validation of the dimensions of access, understanding, appraisal, and use of Santé.fr will be conducted, followed by principal component analysis and ascendant hierarchical classification based on the 2 main components of principal component analysis to characterize homogeneous users' profiles. Regression models will be used to investigate the relationships between each dimension and socioeconomic position and health status variables. NVivo 11 software (Lumivero) will be used to categorize interviewees' comments into preidentified themes or themes emerging from the discourse and compare them with the comments of various types of interviewees to understand the factors influencing people's access, understanding, appraisal, and use of Santé.fr.

RESULTS

Recruitment is scheduled to begin in January 2024 and will conclude when the required number of participants is reached. Data collection is expected to be finalized approximately 7 months after recruitment, with the final data analysis programmed to be completed around December 2024.

CONCLUSIONS

This study would be the first in France and in Europe to evaluate a public health information service, in this case the Santé.fr website (the official website of the French Ministry of Health), according to users' socioeconomic position and health status. The study could discover issues related to inequalities in access to, and the use of, digital technologies for obtaining health information on the internet. Given that access to health information on the internet is crucial for health decision-making and empowerment, inequalities in access may have subsequent consequences on health inequalities among social categories. Therefore, it is important to ensure that all social categories have access to Santé.fr.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/51123.

Worth the paper they're printed on? Findings from an independent evaluation of how understandable patient information leaflets for antiseizure medications are

OBJECTIVE

The Patient Information Leaflet (PIL) is an authoritative document that all people with epilepsy (PWE) in the European Union receive when prescribed anti-seizure medication (ASM). We undertook the first independent, comprehensive assessment to determine how understandable they are. Regulators state that when patients are asked comprehension questions about them, ≥80% should answer correctly. Also recommended PILs have a maximum reading requirement of US Grade 8.

METHODS

Study 1: Obtained 140 current ASM PILs written in English. 'Readability' assessed using 4 tests, with and without adjustment for influence of familiar, polysyllabic words. 179 online materials on epilepsy also assessed. Study 2: Two PILs from Study 1 randomly selected (Pregabalin Focus; Inovelon) and shown to 35 people from UK epilepsy population. Their comprehension was assessed. Study 3: To understand whether student population provides accessible alternative population for future examination of ASM PILs, Study 3 was completed. Used same methods as Study 2 but participants were 262 UK university students.

RESULTS

Study 1: No PIL had a reading level of Grade 8. Median was 11. Adjusting for context, the PILs were still at Grade 10.5. PILs for branded ASMs were most readable. PILs were no more readable than (unregulated) online materials. Study 2: Users struggled to comprehend the PILs key messages. The 8 questions asked about pregabalin were typically answered correctly by 54%. For Inovelon it was 62%. Study 3: Most student participants comprehend the PILs key messages. The questions about Inovelon were answered correctly by 90%; for pregabalin it was 86%.

SIGNIFICANCE

This is the first independent and comprehensive examination of ASM PILs. Found PILs being used fail to meet recommendations and regulatory requirements and risk not being understandable to substantial proportion of users. In finding that people from epilepsy population differ markedly in comprehension of PILs compared to students, study highlights importance of completing user testing with the target population.

Current Status of Websites Offering Information to Patients with Traumatic Brain Injury and Caregivers: Time for Reform?

OBJECTIVE

Traumatic brain injury (TBI) is a global public health problem, causing long-term burden to patients and caregivers. Patients and their families often resort to seeking online information regarding TBI management while awaiting formal health care consultations. Although this information is accessible and immediately available, little is known about the quality of online resources. We evaluated the accessibility, relevance, and readability of information regarding TBI from major online search engines.

METHODS

TBI-related search terms were entered into 2 online search engines (Google and Yahoo), and the first 30 websites per search were assessed for eligibility. Quality (DISCERN score, JAMA Benchmark score) and readability (Flesch-Kincaid Grade Level, Flesch Reading Ease Score) were assessed. Associations between search ranking, quality, and readability were evaluated.

RESULTS

In total, 202 websites were evaluated with mean DISCERN score 36.5 ± 9.9/80, signifying poor global quality, and mean JAMA Benchmark score 2.8 ± 1.1/4. The majority required 9-12 years of education (113/202; 55.9%) according to Flesch-Kincaid Grade Level and categorized as "Difficult" on Flesch Reading Ease Score (94/202; 46.5%). Website quality was not associated with search ranking or readability.

CONCLUSIONS

There is a paucity of high-quality online resources for patients with TBI. Herein, we highlight: 1) the importance of guidance from healthcare professionals regarding online-information seeking; and 2) recommendations for the most useful online resources available.

A cross-sectional survey among physicians on internet use for epilepsy-related information

OBJECTIVE

We investigated views towards the Internet in a sample of Italian healthcare specialists involved in epilepsy field, to identify factors associated with the attitude of being influenced by information found on the Internet.

METHODS

This study was a self-administered survey conducted in a group of members of the Italian Chapter of the International League Against Epilepsy (ILAE) in January 2018.

RESULTS

184 questionnaires were analyzed. 97.8 % of responders reported to seek online information on epilepsy. The Internet was most frequently searched to obtain new information (69.9 %) or to confirm a diagnostic or therapeutic decision (37.3 %). The influence of consulting the Internet on clinical practice was associated with registration to social network(s) (OR: 2.94; 95 %CI: 1.28-6.76; p = 0.011), higher frequency of Internet use (OR: 3.66; 95 %CI: 1.56-9.21; p = 0.006) and higher confidence in reliability of online information (OR: 2.61; 95 %CI: 1.09-6.26; p = 0.031). No association was found with age, sex, years in epilepsy practice or easiness to find online information.

CONCLUSION

Internet is frequently used among healthcare professionals involved in the epilepsy to obtain information about this disease. The attitude of being influenced by the Internet for diagnostic and/or therapeutic decisions in epilepsy is independent on age and years of experience in epilepsy, and probably reflects an individual approach towards the Web.