Assessing the Understandability and Actionability of Online Neurosurgical Patient Education Materials

Readability of Neurosurgical Patient Education Resources by the American Association of Neurological Surgeons

BACKGROUND

The importance of patient education has become increasingly apparent in recent years. A prominent patient education tool in neurosurgery is the Neurosurgical Conditions and Treatments page provided by the American Association of Neurological Surgeons (AANS). This study aimed to investigate the readability of this resource page as many new articles have been incorporated in the past decade.

METHODS

Flesch-Kincaid Grade Level and Flesch-Kincaid Reading Ease scores were calculated for each article. One-way analysis of variance and Scheffé, Tukey-Kramer, Bonferroni, Fisher least significant difference, and Dunnett test post hoc analyses were conducted to determine differences between each category with respect to their mean grade level and reading ease scores.

RESULTS

Overall mean (SD) Flesch-Kincaid Reading Ease score was 40.2 (12.24), and overall mean (SD) Flesch-Kincaid Grade Level score was 7.48 (1.26). Significant differences were found between mean reading ease scores between each categorization by the AANS (P = 0.014). No significant differences were found between mean grade level score for each categorization (analysis of variance, P = 0.154).

CONCLUSIONS

As compared to a single previous investigation conducted 10 years ago, the readability of articles has changed modestly, and the reading grade level remains well above the recommendations by the American Medical Association and National Institutes of Health. The 6 new articles introduced in the past decade have demonstrated similar readability, presenting a persistent challenge in the realm of patient education in neurosurgery.

An evaluation of readability and understandability of online education materials for breast cancer survivors

PURPOSE

We aimed to determine the availability of existing web-based educational materials on breast cancer survivorship and assess their readability and understandability.

METHODS

We identified materials eligible for review in two ways: (1) reviews of websites of major cancer-related organizations (e.g., American Cancer Society); (2) Google searches for breast cancer survivorship, breast cancer, breast cancer follow-up care, and cancer survivorship. We measured Flesch-Kincaid and New Dale Readability of existing breast cancer and breast cancer survivorship materials. Readability grade levels 5 to 8 were considered ideal to acceptable. We used the Patient Education Materials Assessment Tool (PEMAT) to measure the understandability of 53 videos and 152 written materials, such as booklets and manuals. A resource was considered understandable and/or actionable if it scored ≥ 70% on either the understandability section or the actionability section of the PEMAT.

RESULTS

We identified a total of 205 existing materials including brochures, booklets, facts sheets, websites, and videos in English. The average Flesch-Kincaid grade score of written educational materials was 9.7 (range 3.5-16.4), which translates to a 9th grade reading level. According to the New Dale-Chall readability assessment, most of the materials were in the 9 to 10 grade level range. The average PEMAT score was 88.6% (range 56-100%).

CONCLUSION

Patient educational materials are available online as printable, written materials, and videos and they focus on a wide selection of survivorship-related topics. Most of the breast cancer educational materials that are available online were above an 8th grade reading level. The PEMAT results, however, suggest that materials are easy to understand regarding word choice and style, use of numbers, organization, layout and design, and use of visual aids.

IMPLICATIONS FOR CANCER SURVIVORS

Understandable patient education materials are essential for guiding breast cancer survivors towards improving their health outcomes and optimizing their quality of life.

Critical Appraisal of Content Quality of YouTube Videos on the Whipple Pancreaticoduodenectomy

INTRODUCTION

The coronavirus pandemic has demonstrated profound issues with using the Internet to research health information. For patients recommended a complex operation, such as the Whipple pancreaticoduodenectomy, the quality of health information online has not been appraised. The objective of this study was to define the readability and content quality of YouTube search results for the Whipple pancreaticoduodenectomy.

METHODS

The first 100 search results for "whipple procedure" less than 10 min long in English with audio and or text were transcribed. The Flesch-Kincaid Grade defined the reading grade level. High content quality videos were accredited by YouTube in accordance with principles specified by the National Academy of Medicine or mentioned the standard components for a surgical consent. The Anderson-Lau score is a composite of these consent criteria out of a maximum of 8/8. The simplicity of videos for patient education was defined by the DISCERN tool.

RESULTS

The reading level of 23% of the top 100 search results met the American average (8th grade). Accreditation was present for 45% and associated with an earlier median search ranking (36 versus 68, P = 0.002) and more 5th-8th grade level material (70% versus 38%, P = 0.014). The median Anderson-Lau score was 3/8 (range = 0/8-7/8) with only 5% achieving 7/8. Only 4% were high quality per DISCERN.

CONCLUSIONS

Although accredited videos were more readable, most videos, especially those targeting patients, were beyond the comprehension of the average American. Simpler and higher quality educational materials are needed to inform patients on Whipple pancreaticoduodenectomy beyond their date of clinical diagnosis or surgical consenting.

Addressing Current Deficits in Patient Education Materials Through Crowdsourcing

BACKGROUND

Patient education materials are commonly reported to be difficult to understand.

OBJECTIVES

We aimed to use crowdsourcing to improve patient education materials at our institution.

METHODS

This was a department-wide quality improvement project to increase organizational health literacy. There are 6 phases of this pilot study: (1) evaluating preexisting patient education materials, (2) evaluating online patient education materials at the society (the American Society of Plastic Surgeon) and government level (Medline Plus), (3) redesigning our patient education material and reevaluating the education material, (4) crowdsourcing to evaluate understandability of the new patient education material, (5) data analysis, and (6) incorporating crowdsourcing suggestions to the patient education material.

RESULTS

Breast-related patient education materials are not easy to read at the institution level, the society level, and the government level. Our new implant-based breast reconstruction patient education material is easy to read as demonstrated by the crowdsourcing evaluation. More than 90% of the participants reported our material is "very easy to understand" or "easy to understand." The crowdsourcing process took 1.5 days, with 700 workers responding to the survey. The total cost was $9. After incorporating participants' feedback into the finalized material, the readability of the material is at the recommended reading level. The material also had the recommended length (between 400 and 800 words).

DISCUSSION

Our study demonstrated a pathway for clinicians to efficiently obtain a large amount of feedback to improve patient education materials. Crowdsourcing is an effective tool to improve organizational health literacy.

Health literacy in rotator cuff repair: a quantitative assessment of the understandability of online patient education material

Background

The American Medical Association and National Institutes of Health recommend online health information be written at a 6th grade or lower reading level for clear understanding. While syntax reading grade level has previously been utilized, those analyses do not determine whether readers are processing key information (understandability) or identifying available actions to take (actionability). The Patient Education Materials Assessment Tool (PEMAT-P) is a method to measure the understandability and actionability of online patient education materials. The purpose of this study was to evaluate online resources regarding rotator cuff repair utilizing measures of readability, understandability, and actionability.

Methods

The search term "rotator cuff surgery" was used in two independent online searches to obtain the top 50 search results. The readability of included resources was quantified using valid objective algorithms: Flesch-Kincaid Grade-Level, Simple Measure of Gobbledygook grade, Coleman-Liau Index, and Gunning Fog Index. The PEMAT-P form was used to assess actionability and understandability.

Results

A total of 49 unique websites were identified to meet our inclusion criteria and were included in our analysis. The mean Flesch-Kincaid Grade Level graded materials at a 10.6 (approximately a 10th grade reading level), with only two websites offering materials at a 6th grade reading level or below. The remaining readability studies graded the mean reading level at high school or greater, with the Gunning Fog Index scoring at a collegiate reading level. Mean understandability and actionability scores were 64.6% and 29.5%, respectively, falling below the 70% PEMAT score threshold for both scales. Fourteen (28.6%) websites were above the threshold for understandability, while no website (0%) scored above the 70% threshold for actionability. When comparing source categories, commercial health publishers provided websites that scored higher in understandability (P < .05), while private practice materials scored higher in actionability (P < .05). Resources published by academic institutions or organizations scored lower in both understandability and actionability than private practice and commercial health publishers (P < .05). No readability, understandability, or actionability score was significantly associated with search result rank.

Conclusion

Overall, online patient education materials related to rotator cuff surgery scored poorly with respect to readability, understandability, and actionability. Only two (4.1%) of the patient education websites scored at the American Medical Association and National Institutes of Health recommended reading level. Fourteen (28.6%) scored above the 70% PEMAT score for understandability; however, no website met the threshold for actionability.

Qualitative assessment of available online patient resources for pulmonary embolism and deep vein thrombosis

OBJECTIVE

This study seeks to evaluate the quality and readability of freely available online patient information resources for deep vein thrombosis (DVT) and pulmonary embolism (PE).

METHODS

Internet searches were performed for five DVT and PE search terms in July 2020 across three search engines and two metasearch engines. Qualitative content analysis was performed. Readability was assessed using four validated instruments.

RESULTS

Two hundred fifty websites were identified of which 62 websites met inclusion criteria.Website structure and content were satisfactory (>50% overall score), accountability was mixed between DVT (47%) and PE (56%) sites, while interactivity was poor (<30%). On qualitative content analysis, anticoagulation (95.2%) was the most discussed treatment while the most discussed procedures were IVC filter placement for DVT and thrombolysis for PE. Overall readability was difficult with median level suitable for ages 14-18 years.

CONCLUSION

Freely available online DVT and PE patient information resources publish appropriate content but have very variable accountability and poor readability for the average patient.

Assessing Arabic youtube videos on herbal cancer treatment: Absence of health information quality

This study assesses the quality of the health information in Arabic YouTube videos regarding herbal cancer treatment. It also provides an overview of how the quality of video content shapes user awareness by assessing the users' engagement indicators. A simple Python tool was developed using YouTube API V3 to automate the YouTube search based on the recommendation of Google Trends. After applying inclusion and exclusion criteria, 110 YouTube videos were selected, of which 95% were uploaded by non-experts and had a total of 8,633,569 views. The analyzed videos presented more than 40 different herbals as sources of cancer treatment; for example, Ephedra, garden cress, Green tea, Ginseng, Rosemary, and Thyme. 32.7% of the videos provided information about a single herb, 41% about mixing herbals, and 26.3% provided testimonials and success stories without pointing to specific herbs. The videos were assessed by two experts using two reliable tools, DISCERN and PEMAT, which were produced mainly for assessing health information quality. DISCERN has evaluated the reliability and quality of health information. PEMAT has assessed the understandability and actionability. The qualitative and quantitative analyses of the videos represent bias and poor health information quality, with a total score of 27 out of 80 for DISCERN and 31 out of 100 for the PEMAT. The results also showed weak users' awareness regarding the content of videos with no association between user engagement indicators (likes, dislikes, VPI, views, comments) and the dimensions of the two tools. The study concludes that it is evident that YouTube, in its current form, is an inadequate Arabic source for herbal cancer treatment information. To overcome this, this study proposed the GAP framework for social media that integrated Governance, Awareness, and Proficiency.

Health Literacy in Neurosurgery: A Scoping Review

OBJECTIVE

Low health literacy is prevalent and associated with suboptimal health outcomes. In neurosurgery, social determinants of health are increasingly recognized as factors underpinning outcomes, as well as access to and use of care. We conducted a scoping review to delineate the scope of existing literature regarding health literacy in the field and facilitate future research.

METHODS

A scoping review was conducted using the PubMed, Embase, and Scopus databases. Titles and abstracts were screened for relevance. Studies meeting prespecified inclusion criteria underwent full text review. Relevant data were extracted.

RESULTS

Of 5056 resultant articles, 57 manuscripts were included. Thirty-seven studies (64.9%) investigated personal health literacy, while the remaining 20 (35.1%) investigated organizational health literacy. Domains of health literacy investigated were science (36, 63.2%), fundamental (20, 35.1%), and civic (1, 1.7%). No studies investigated numeracy. Recall among patients after discussions with neurosurgeons is low. Patient perspectives are often erroneous. Patient informational needs are often unmet. Written patient educational materials are written at a level too complex for the average patients. Videos are mostly of poor quality. Multimodal audiovisual interventions, eBooks, models, and virtual reality are shown to be effective methods for promoting recall.

CONCLUSIONS

Studies examining health literacy in neurosurgery primarily focus on the topic indirectly, most often via written educational materials and recall after educational interventions. Increasing awareness of health literacy among neurosurgeons, assessing health literacy, and incorporating health literacy-informed counseling approaches are warranted to improve patient care.

Academics versus the Internet: Evaluating the readability of patient education materials for cerebrovascular conditions from major academic centers

Background:

Health literacy profoundly impacts patient outcomes as patients with decreased health literacy are less likely to understand their illness and adhere to treatment regimens. Patient education materials supplement in-person patient education, especially in cerebrovascular diseases that may require a multidisciplinary care team. This study aims to assess the readability of online patient education materials related to cerebrovascular diseases and to contrast the readability of those materials produced by academic institutions with those of non-academic sources.

Methods:

The readability of online patient education materials was analyzed using Flesch-Kincaid Grade Level (FKGL) and Flesch Reading Ease (FRE) assessments. Readability of academic-based online patient education materials was compared to nonacademic online patient education materials. Online patient education materials from 20 academic institutions and five sources from the web were included in the analysis.

Results:

Overall median FKGL for neurovascular-related patient online education documents was 11.9 (95% CI: 10.8–13.1), reflecting that they are written at a 12th grade level, while the median FRE was 40.6 (95% CI: 34.1–47.1), indicating a rating as “difficult” to read. When comparing academic-based online patient education materials to other internet sources, there was no significant difference in FRE and FKGL scores (P = 0.63 and P = 0.26 for FKGL and FRE, respectively).

Conclusion:

This study demonstrates that online patient education materials pertaining to cerebrovascular diseases from major academic centers and other nonacademic internet sites are difficult to understand and written at levels significantly higher than that recommended by national agencies. Both academic and nonacademic sources reflect this finding equally. Further study and implementation are warranted to investigate how improvements can be made.

Patient Education Materials Found via Google Search for Shoulder Arthroscopy Are Written at Too-High of a Reading Level

Purpose

To evaluate the quality and correlation of readability on actionability and understandability of shoulder arthroscopy-related patient education materials (PEMs) found via a routine Google search.

Methods

Two independent authors performed an online Google search with the term “shoulder arthroscopy.” The first 5 pages of search results were then screened for PEMs. Journal articles, news articles, nontext materials, and unrelated websites were excluded. The readability of included resources was calculated using objective metrics: Flesch–Kincaid Grade Score, Simple Measure of Gobbledygook index, Coleman–Liau Index, and the Gunning Fog Index. Patient Education Material Assessment Tool for Printed Materials assessed for understandability and actionability. Associations between readability and actionability and understandability were determined using Spearman correlation and linear regression.

Results

The searches returned 53 websites related to shoulder arthroscopy. A total of 34 (64%) met inclusion criteria. A high school reading level or greater was required to read the average PEM according to all scales used. The average PEM received a Patient Education Material Assessment Tool for Printed Materials score of 61.33 in understandability (range 18.75-89.47) and 55.59 points in actionability (range 16.67-83.33). An easily understood or actionable article would score at least 70 points. A moderate correlation was observed between readability and actionability on three of the scales used (r = 0.5, r = 0.59, r = 0.61).

Conclusions

Most shoulder arthroscopy PEMs identified on Google are not written at a level that the average patient can read, understand, or act on (actionability).

Clinical Relevance

Orthopaedic surgeons should be aware of the resources that patients use to obtain medical information. More accessible PEMs should be developed for patients undergoing shoulder arthroscopy to enhance comprehension of their condition and improve shared decision-making.

Can Patients Read, Understand, and Act on Online Resources for Anterior Cruciate Ligament Surgery?

Background:

Patients undergoing elective procedures often utilize online educational materials to familiarize themselves with the surgical procedure and expected postoperative recovery. While the Internet is easily accessible and ubiquitous today, the ability of patients to read, understand, and act on these materials is unknown.

Purpose:

To evaluate online resources about anterior cruciate ligament (ACL) surgery utilizing measures of readability, understandability, and actionability.

Study Design:

Cross-sectional study; Level of evidence, 4.

Methods:

Using the term “ACL surgery,” 2 independent searches were performed utilizing a public search engine (Google.com). Patient education materials were identified from the top 50 results. Audiovisual materials, news articles, materials intended for advertising or medical professionals, and materials unrelated to ACL surgery were excluded. Readability was quantified using the Flesch Reading Ease, Flesch-Kincaid Grade Level, Simple Measure of Gobbledygook, Coleman-Liau Index, Automated Readability Index, and Gunning Fog Index. The Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P) was utilized to assess the actionability and understandability of materials. For each online source, the relationship between its Google search rank (from first to last) and its readability, understandability, and actionability was calculated utilizing the Spearman rank correlation coefficient (ρ_S).

Results:

Overall, we identified 68 unique websites, of which 39 met inclusion criteria. The mean Flesch-Kincaid Grade Level was 10.08 ± 2.34, with no website scoring at or below the 6th-grade level. Mean understandability and actionability scores were 59.18 ± 10.86 (range, 33.64-79.17) and 34.41 ± 22.31 (range, 0.00-81.67), respectively. Only 5 (12.82%) and 1 (2.56%) resource scored above the 70% adequate PEMAT-P threshold mark for understandability and actionability, respectively. Readability (lowest P value = .103), understandability (ρ_S = –0.13; P = .441), and actionability (ρ_S = 0.28; P = .096) scores were not associated with Google rank.

Conclusion:

Patient education materials on ACL surgery scored poorly with respect to readability, understandability, and actionability. No online resource scored at the recommended reading level of the American Medical Association or National Institutes of Health. Only 5 resources scored above the proven threshold for understandability, and only 1 resource scored above it for actionability.

Health Literacy and Web-Based Audiovisual Multimedia in Pituitary and Endoscopic Skull Base Surgery

Introduction The internet presents a rich milieu of multimedia options relating to pituitary and endoscopic skull base surgery (ESBS). Misinformation can create discordance between patient and provider expectations. The purpose of this study is to analyze the understandability and actionability of available ESBS and pituitary surgery audiovisual information on YouTube and Google.

Methods The top 50 videos generated by searching “pituitary surgery/transsphenoidal surgery” and “endoscopic skull base surgery” in both YouTube and Google were sorted by relevance. Two independent reviewers evaluated each for understandability and actionability based on the Patient Education Materials Assessment Tool for audiovisual material (PEMAT-A/V). Source, authorship, audience, and education/advertisement variables were collected. Chi-square test followed by univariate and multivariate regression analyses assessed the association between these variables and quality.

Results A total of 85 videos (52 YouTube and 33 Google) met inclusion criteria for analysis. There was no significant difference in the presence of the aforementioned variables between YouTube and Google (p < 0.05). Also, 72% of videos targeted patients and 28% targeted surgeons. Academic institutions uploaded 58% of videos. Surgeon-targeted videos were more educational (p = 0.01) and patient-targeted videos involved more advertisement (p = 0.01). Understandability and actionability scores were below the 70% threshold for both YouTube (65 ± 15, 38 ± 33, p = 0.65) and Google (66 ± 12, 38 ± 26, p = 0.94). Patient-targeted videos (p = 0.002) were more understandable, while surgeon- (p < 0.001) and education-focused videos (p < 0.001) were more actionable.

Conclusion Understandability and actionability of YouTube and Google audiovisual patient information on ESBS and pituitary surgery is poor. Consideration should be given to the formation of a standardized patient information resource.

Assessing Communicative Effectiveness of Public Health Information in Chinese: Developing Automatic Decision Aids for International Health Professionals

Effective multilingual communication of authoritative health information plays an important role in helping to reduce health disparities and inequalities in developed and developing countries. Health information communication from the World Health Organization is governed by key principles including health information relevance, credibility, understandability, actionability, accessibility. Multilingual health information developed under these principles provide valuable benchmarks to assess the quality of health resources developed by local health authorities. In this paper, we developed machine learning classifiers for health professionals with or without Chinese proficiency to assess public-oriented health information in Chinese based on the definition of effective health communication by the WHO. We compared our optimized classifier (SVM_F5) with the state-of-art Chinese readability classifier (Chinese Readability Index Explorer CRIE 3.0), and classifiers adapted from established English readability formula, Gunning Fog Index, Automated Readability Index. Our optimized classifier achieved statistically significant higher area under the receiver operator curve (AUC of ROC), accuracy, sensitivity, and specificity than those of SVM using CRIE 3.0 features and SVM using linguistic features of Gunning Fog Index and Automated Readability Index (ARI). The statistically improved performance of our optimized classifier compared to that of SVM classifiers adapted from popular readability formula suggests that evaluation of health communication effectiveness as defined by the principles of the WHO is more complex than information readability assessment. Our SVM classifier validated on health information covering diverse topics (environmental health, infectious diseases, pregnancy, maternity care, non-communicable diseases, tobacco control) can aid effectively in the automatic assessment of original, translated Chinese public health information of whether they satisfy or not the current international standard of effective health communication as set by the WHO.

Quality of Carpal Tunnel Syndrome Patient Education Handouts Available on the Internet: A Systematic Analysis of Content and Design

OBJECTIVE

To evaluate the quality of carpal tunnel syndrome (CTS) patient education handouts and identify the best resources for patients and clinicians.

DESIGN

A document content analysis of handouts identified through a systematic internet search using 8 search terms on Google and Bing and a hand search of professional association websites.

SETTING

Not applicable.

PARTICIPANTS

Documents (N=56) were identified from the top 50 search results across 16 individual searches. Included documents provided general patient education for CTS; descriptive websites, videos, and research studies were excluded.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURES

Content analysis was conducted using the Information Score (IS) tool to evaluate completeness of information (0%-100%) and misleading treatment recommendations were identified. Design analysis was conducted using the Patient Material Assessment Tool for Printable Materials (PEMAT-P) (0%-100%) and 2 widely used readability formulas, Flesch Reading Ease and Flesch Kincaid Grade Level. Using these results, all handouts were rated with a summative 12-point scale.

RESULTS

Of 805 unique search results, we included 56 CTS handouts. The average IS was 74.6%±17.9%, and 78.6% of the handouts mentioned non-evidence-based treatment recommendations. The average PEMAT-P score was 70.2%±10.9%, and the average readability grade level was 7.7±1.7. Only 3 handouts were identified as high quality based on the 12-point summative scores, 22 handouts had mixed quality, and 17 handouts had low quality on both content and design.

CONCLUSIONS

Findings of this study suggest a lack of high-quality and easily understandable CTS patient education handouts. Most handouts contained unreliable treatment information. Improvements are needed to ensure patients' ability to understand and manage this condition.

Predicting Writing Styles of Web-Based Materials for Children's Health Education Using the Selection of Semantic Features: Machine Learning Approach

BACKGROUND

Medical writing styles can have an impact on the understandability of health educational resources. Amid current web-based health information research, there is a dearth of research-based evidence that demonstrates what constitutes the best practice of the development of web-based health resources on children's health promotion and education.

OBJECTIVE

Using authoritative and highly influential web-based children's health educational resources from the Nemours Foundation, the largest not-for-profit organization promoting children's health and well-being, we aimed to develop machine learning algorithms to discriminate and predict the writing styles of health educational resources on children versus adult health promotion using a variety of health educational resources aimed at the general public.

METHODS

The selection of natural language features as predicator variables of algorithms went through initial automatic feature selection using ridge classifier, support vector machine, extreme gradient boost tree, and recursive feature elimination followed by revision by education experts. We compared algorithms using the automatically selected (n=19) and linguistically enhanced (n=20) feature sets, using the initial feature set (n=115) as the baseline.

RESULTS

Using five-fold cross-validation, compared with the baseline (115 features), the Gaussian Naive Bayes model (20 features) achieved statistically higher mean sensitivity (P=.02; 95% CI -0.016 to 0.1929), mean specificity (P=.02; 95% CI -0.016 to 0.199), mean area under the receiver operating characteristic curve (P=.02; 95% CI -0.007 to 0.140), and mean macro F1 (P=.006; 95% CI 0.016-0.167). The statistically improved performance of the final model (20 features) is in contrast to the statistically insignificant changes between the original feature set (n=115) and the automatically selected features (n=19): mean sensitivity (P=.13; 95% CI -0.1699 to 0.0681), mean specificity (P=.10; 95% CI -0.1389 to 0.4017), mean area under the receiver operating characteristic curve (P=.008; 95% CI 0.0059-0.1126), and mean macro F1 (P=.98; 95% CI -0.0555 to 0.0548). This demonstrates the importance and effectiveness of combining automatic feature selection and expert-based linguistic revision to develop the most effective machine learning algorithms from high-dimensional data sets.

CONCLUSIONS

We developed new evaluation tools for the discrimination and prediction of writing styles of web-based health resources for children's health education and promotion among parents and caregivers of children. User-adaptive automatic assessment of web-based health content holds great promise for distant and remote health education among young readers. Our study leveraged the precision and adaptability of machine learning algorithms and insights from health linguistics to help advance this significant yet understudied area of research.

A health literacy analysis of the consumer-oriented COVID-19 information produced by ten state health departments

Objective:

The COVID-19 pandemic highlights the public's need for quality health information that is understandable. This study aimed to identify (1) the extent to which COVID-19 messaging by state public health departments is understandable, actionable, and clear; (2) whether materials produced by public health departments are easily readable; (3) relationships between material type and understandability, actionability, clarity, and reading grade level; and (4) potential strategies to improve public health messaging around COVID-19.

Methods:

Based on US Centers for Disease Control and Prevention statistics from June 30, 2020, we identified the ten states with the most COVID-19 cases and selected forty-two materials (i.e., webpages, infographics, and videos) related to COVID-19 prevention according to predefined eligibility criteria. We applied three validated health literacy tools (i.e., Patient Education Materials Assessment Tool, CDC Clear Communication Index, and Flesch-Kincaid Grade Level) to assess material understandability, actionability, clarity, and readability. We also analyzed correlations between scores on the three health literacy tools and material types.

Results:

Overall, COVID-19 materials had high understandability and actionability but could be improved in terms of clarity and readability. Material type was significantly correlated with understandability, actionability, and clarity. Infographics and videos received higher scores on all tools.

Conclusions:

Based on our findings, we recommend public health entities apply a combination of these tools when developing health information materials to improve their understandability, actionability, and clarity. We also recommend using infographics and videos when possible, taking a human-centered approach to information design, and providing multiple modes and platforms for information delivery.

Evaluation of rapid drug safety communication materials for patients in Japan

Since 2011, pharmaceutical companies in Japan have been required to issue two types of documents regarding severe adverse drug reactions reported post-marketing, namely the Rapid Safety Communication Materials for Patients and the Related Materials. However, the adequacy of these documents has not yet been systematically assessed. The aim of this study was to evaluate the adequacy of these two types of materials. The Rapid Safety Communications for Patients were obtained from the Pharmaceuticals and Medical Devices Agency (PMDA) website. The Related Materials were obtained from pharmaceutical companies or the PMDA website. Three assessors independently scored the Rapid Safety Communication for Patients and the Related Materials using the Centers for Disease Control and Prevention Clear Communication Index (CCI). In addition, the contents and descriptions of the materials were analyzed. In total, 13 materials for seven drugs were assessed. Almost all materials contained the "main message" and "call to action". However, the average CCI scores for the Rapid Safety Communication for Patients and Related Materials for Patients were 68.8 and 74.3 (out of 100), respectively. Further, none of the evaluated materials were scored above the CCI threshold score (i.e., ≥ 90%). Descriptions regarding "language", "state of science", and "risk" were not adequate. In particular, the terminology used in materials seemed difficult for patients to understand. In conclusion, the Japanese Rapid Communication Materials for Patients require improvement. Furthermore, a system for evaluating these materials prior to publication should be established.

Neuromodulation, Deep Brain Stimulation, and Spinal Cord Stimulation on YouTube: A Content-Quality Analysis of Search Terms

BACKGROUND

YouTube is one of the most widely accessed platforms for patients seeking information across all medical specialties. More than 50% of patients engage with online information before seeing a physician. There has been no qualitative analysis of neuromodulation content on YouTube.

METHODS

By evaluating the search interest of neuromodulation terms via Google Trends, the authors selected 3 search terms to assess on YouTube. For each term, the top 20 videos were selected, 10 sorting by view count and 10 by relevance. DISCERN criteria were used to evaluate each video. Statistical analysis was performed with R software to discover associations among video characteristics and DISCERN scores (DSs).

RESULTS

Neuromodulation, deep brain stimulation, and spinal cord stimulation terms were searched. There were 52 unique and 8 duplicate videos, which were excluded. The videos received 6,984,743 combined views (134,321 average). Neurosurgeons were the most common authors, producing 22 videos, followed by anesthesiologists with 10 videos. Statistical analysis showed that videos that were sponsored (DS 44.79 vs. 34.53; P < 0.011), had board-certified physicians (DS 42.88 vs. 31.25; P < 0.004), and had credentials of the video producer (DS 41.82 vs. 34.42; P < 0.044) were more likely to have higher DSs. Spinal cord stimulation videos had the lowest quality of the search terms (DS 30.9 vs. 39.2, 42.6; P < 0.043).

CONCLUSIONS

Certain characteristics of YouTube videos are associated with higher quality. Video producers should aim to use these characteristics and follow DISCERN criteria when preparing online videos.

Development, testing and generalizability of a standardized evaluation form for the assessment of patient-directed reports in the new final medical licensing examination in Germany

Background: As doctors often fail to explain diagnoses and therapies to patients in an understandable and appropriate way, the improvement of doctor-patient communication is essential. The current medical training and examinations are focused on verbal rather than on written communication. Following the premise of "assessment drives learning", the final medical licensing examination in Germany has been further developed by the German National Institute for state examinations in Medicine, Pharmacy and Psychotherapy (IMPP). As part of the discharge management the candidates have to prepare a report for the patient that is understandable and provides them with all important information about their stay in hospital. Aim: A standardized evaluation form for formative and summative feedback has been developed and tested with regard to applicability and the assurance of test quality criteria, especially the reliability to assess the written communication skills of the students. Methodology: In an expert consensus procedure, a draft for a standardized evaluation form was developed. This form was revised after an initial trial run on patient-directed reports written by students in their last year of medical studies. Afterwards twenty-one patient-directed reports were evaluated by fourteen different examiners. The reliability was tested by calculating the generalizability-coefficient and by analysing the inter-rater reliability. Results: The first test on the evaluation of the patient-directed reports indicated the practicability of the application and the usefulness of the evaluation form as an instrument for assessing the written communication skills of students. The analyses of the inter-rater reliability showed that the degree of agreement in the evaluations was partly different between two groups of examiners. The calculated G-coefficient indicates a high reliability. The content validity of the evaluation form was given through the comprehensive medical expertise in the development process. Conclusion: Assessing written patient-directed communication is a benefit of the newly developed last part of the medical licensing examination in Germany. Continuous formative assessment and feedback based on the evaluation form is intended to improve the written communication skills of future doctors. Furthermore, a better understanding of their diagnosis and treatment as well as a trusting relationship with their doctor may empower patients in the medical decision process and lead to fewer dismissal errors in the future. For consistent use of the evaluation form a standardized training of examiners should be implemented.

Readability, understandability, and quality of retinopathy of prematurity information on the web

BACKGROUND

The purpose of this study is to assess the readability, understandability, and quality of information on retinopathy of prematurity presented at websites frequently visited by parents.

METHODS

A total of 220 websites were assessed, which were recruited by searching for "retinopathy of prematurity" at the Google search engine. The readability of each web page was assessed by Flesch Reading Ease Score, Gobbledygook's Gunning Frequency, Flesch Kincaid Grade Level, Coleman Liau score, The Simple Measure of Gobbledygook, Fry Graph Readability Formula, and Automated readability score. The understandability of the web pages included in the study was measured by using the Patient Education Materials Assessment Tool. Quality was evaluated using Health in Net code and JAMA. The ALEXA traffic tool was used to reference the domains' popularity and visibility.

RESULTS

Sixty-four websites were included to the study. The average Flesch Reading Ease Score was 50.1 ± 11.4, Gunning Frequency of Gobbledygook level was 13.4 ± 2.5, The Flesch-Kincaid Grade level was 10.7 ± 2.2, Coleman Liau level was 10.8 ± 1.7, Simple Measure of Gobbledygook level was 10.0 ± 1.9, and Fry Graph Readability Formula 11.9 ± 2.7, Automated readability score 10.4 ± 2.5. The average understandability score for all website-based patient education materials was 76.9 ± 15.2. Total JAMA Benchmark score is 2.27 ± 1.14 (range from 1 to 4). The quality of information at most websites were determined by our chosen assessments to not to be good.

CONCLUSION

Websites addressed to parents for retinopathy of prematurity had found to have high understandability. It was concluded based on this study that readability and quality of presented written materials at online sources need to be improved.

Patient Education in Neurosurgery: Part 1 of a Systematic Review

BACKGROUND

Patient education has become increasingly important in neurosurgery. However, little is known regarding how to comprehensively educate neurosurgical patients. In the first part of a 2-part systematic review, we identify baseline patient understanding and educational needs, examine existing patient education materials, and characterize shortcomings in neurosurgical patient education practices. Our findings may guide neurosurgeons, departments, and professional associations in improving communication with patients.

METHODS

A 2-part systematic review was conducted using the PubMed, Embase, and Scopus databases. Titles and abstracts were read and selected for full text review. Studies meeting prespecified inclusion criteria were reviewed in full and analyzed for study design, aim, population, interventions, and outcomes.

RESULTS

Of 1617 resultant articles, 41 were included. Patient comprehension was low. Patient informational needs differed across patients and between patients and neurosurgeons. Patient informational needs were often unmet, promoting dissatisfaction. Written educational materials from government agencies, professional associations, neurosurgery departments, neurosurgeons, or widely read Web sites were written at a reading level above the recommended reading level, with complex topics written at a higher level. Information found on hospital and university Web sites was poor quality because of self-marketing and lack of reference to peer-reviewed literature. Educational videos created by universities and hospitals were poor quality.

CONCLUSIONS

Current in-clinic discussions and education materials show shortcomings, promoting poor comprehension. Neurosurgeons, departments, and professional organizations can act to improve the effectiveness of patient education initiatives. This policy will better inform patients, increase rapport between neurosurgeons and patients, and improve patient decision making and satisfaction.

Osteosarcoma Online Resources: A Quantitative Assessment of the Understandability and Readability of Web-based Patient Education Material (Preprint)

Background

Patients often turn to web-based resources following the diagnosis of osteosarcoma. To be fully understood by average American adults, the American Medical Association (AMA) and National Institutes of Health (NIH) recommend web-based health information to be written at a 6th grade level or lower. Previous analyses of osteosarcoma resources have not measured whether text is written such that readers can process key information (understandability) or identify available actions to take (actionability). The Patient Education Materials Assessment Tool (PEMAT) is a validated measurement of understandability and actionability.

Objective

The purpose of this study was to evaluate web-based osteosarcoma resources using measures of readability, understandability, and actionability.

Methods

Using the search term “osteosarcoma,” two independent Google searches were performed on March 7, 2020 (by AGS), and March 11, 2020 (by TRG). The top 50 results were collected. Websites were included if they were directed at providing patient education on osteosarcoma. Readability was quantified using validated algorithms: Flesh-Kincaid Grade Ease (FKGE), Flesch-Kincaid Grade-Level (FKGL). A higher FKGE score indicates that the material is easier to read. All other readability scores represent the US school grade level. Two independent PEMAT assessments were performed with independent scores assigned for both understandability and actionability. A PEMAT score of 70% or below is considered poorly understandable or poorly actionable. Statistical significance was defined as P≤.05.

Results

Two searches yielded 53 unique websites, of which 37 (70%) met the inclusion criteria. The mean FKGE and FKGL scores were 40.8 (SD 13.6) and 12.0 (SD 2.4), respectively. No website scored within the acceptable NIH or AHA recommended reading level. Only 4 (11%) and 1 (3%) website met the acceptable understandability and actionability threshold. Both understandability and actionability were positively correlated with FKGE (ρ=0.55, P<.001; ρ=0.60, P<.001), but were otherwise not significantly associated with other readability scores. There were no associations between readability (P=.15), understandability (P=.20), or actionability (P=.31) scores and Google rank.

Conclusions

Overall, web-based osteosarcoma patient educational materials scored poorly with respect to readability, understandability, and actionability. None of the web-based resources scored at the recommended reading level. Only 4 achieved the appropriate score to be considered understandable by the general public. Authors of patient resources should incorporate PEMAT and readability criteria to improve web-based resources to support patient understanding.

Health literacy among neurosurgery and other surgical subspecialties: Readability of online patient materials found with Google

OBJECTIVE

To both determine whether the most high-yield online patient materials for surgical specialties meet the 6th grade readability level recommended by the National Institutes of Health (NIH) and American Medical Association (AMA), and to discover differences in readability across specialties. We hypothesize average readability scores will exceed an 11th grade level.

METHODS

The top five most common procedures for each of seven surgical specialties (neurological, orthopedic, plastic, general, thoracic, pediatric, and vascular) were searched using an incognito Google query to minimize location bias. The text from the top five patient-relevant links per procedure, excluding Wikipedia, journal articles, and videos, was extracted and inserted into Readability Studio Software for analysis.

RESULTS

The combined average grade level of materials (± standard deviation) was: 10.47 ± 2.51 Flesh-Kincaid Grade Level (FKGL), 11-12 New Dale-Chall (NDC), 10.09 ± 1.97 Simple Measure of Gobbledygook (SMOG), 12 Fry Graph (FG). Thoracic, neurologic, vascular, plastic, and orthopedic were least readable (grade level 10+ by all metrics).

CONCLUSIONS

High readability of procedure materials for patients is not unique to neurosurgery: all specialties exceeded the recommended 6th grade level by three or more grades. Online patient education materials related to surgical subspecialties must be written more comprehensibly.

Literacy assessment of preimplantation genetic patient education materials exceed national reading levels

PURPOSE

In vitro fertilization with preimplantation genetic testing (IVF+PGT-M) reduces the risk of having a child affected by a heritable condition, yet only one-third of eligible patients are aware of this reproductive option. Access to education materials written at appropriate literacy levels could raise patients' awareness, but there is a mismatch between patient reading ability and the literacy demand of most materials. This study aimed to systematically identify written education materials on IVF+PGT-M and evaluate their literacy levels. We hypothesized that materials would fail to meet standards set by the Joint Commission and Centers for Disease Control and Prevention (CDC).

METHODS

To identify patient education materials about IVF+PGT-M from academic databases and public-facing sources, an environmental scan was performed. Materials were analyzed using three validated scales: Simple Measure of Gobbledygook, Patient Education Materials Assessment Tool, and Clear Communication Index.

RESULTS

Seventeen patient education materials about IVF+PGT-M were identified from patient education databases, a consumer search engine, and professional organizations. The median reading level was 14.5 grade, median understandability was 74.2%, and median comprehensibility was 73.3%.

CONCLUSIONS

For most American adults, materials about IVF+PGT-M are not readable, understandable, or clear. The Joint Commission requires patient education materials be written at or below 5th grade reading level and the CDC recommends a 90% minimum score for comprehensibility. No evaluated material met these guidelines. Patient education materials that exceed average American literacy skills may perpetuate disparities in the utilization of IVF+PGT-M. Materials that communicate this complicated subject at an understandable level are needed.