Interrater reliability of the Patient Education Materials Assessment Tool (PEMAT)

AI-Enhanced evaluation of YouTube content on post-surgical incontinence following pelvic cancer treatment

Background

Several pelvic area cancers exhibit high incidence rates, and their surgical treatment can result in adverse effects such as urinary and fecal incontinence, significantly impacting patients' quality of life. Post-surgery incontinence is a significant concern, with prevalence rates ranging from 25 to 45% for urinary incontinence and 9-68% for fecal incontinence. Cancer survivors are increasingly turning to YouTube as a platform to connect with others, yet caution is warranted as misinformation is prevalent.

Objective

This study aims to evaluate the information quality in YouTube videos about post-surgical incontinence after pelvic area cancer surgery.

Methods

A YouTube search for "Incontinence after cancer surgery" yielded 108 videos, which were subsequently analyzed. To evaluate these videos, several quality assessment tools were utilized, including DISCERN, GQS, JAMA, PEMAT, and MQ-VET. Statistical analyses, such as descriptive statistics and intercorrelation tests, were employed to assess various video attributes, including characteristics, popularity, educational value, quality, and reliability. Also, artificial intelligence techniques like PCA, t-SNE, and UMAP were used for data analysis. HeatMap and Hierarchical Clustering Dendrogram techniques validated the Machine Learning results.

Results

The quality scales presented a high level of correlation one with each other (p < 0.01) and the Artificial Intelligence-based techniques presented clear clustering representations of the dataset samples, which were reinforced by the Heat Map and Hierarchical Clustering Dendrogram.

Conclusions

YouTube videos on "Incontinence after Cancer Surgery" present a "High" quality across multiple scales. The use of AI tools, like PCA, t-SNE, and UMAP, is highlighted for clustering large health datasets, improving data visualization, pattern recognition, and complex healthcare analysis.

A health literacy analysis of online patient-directed educational materials about mycobacterium avium complex

Introduction

Mycobacterium avium complex (MAC) is a complex lung infection requiring multi-disciplinary approach and management. Due to limited clinician-patient interactions, clinicians may refer patients to online resources to learn about the diagnosis, prognosis, and treatment of MAC. The American Medical Association (AMA) recommends educational materials be written at a sixth-grade reading level and the National Institutes of Health (NIH) recommends that patient education materials be written at an eighth-grade reading level; however, several evaluations found these materials inaccessible due to high literacy levels. To date, there has never been a health literacy assessment of MAC patient education materials. The study aims to assess the health literacy of online patient education materials about MAC.

Methods

The patient education materials were evaluated for readability, actionability, understandability and clarity. Readability was assessed through the Flesch-Kincaid Grade Level Scale (FkGL), SMOG Index, Coleman Liau Index (CLI), Gunning Fog Index (GFI), and Automated Readability Index (ARI). Actionability and understandability was evaluated using the Patient Education Materials Assessment Tool (PEMAT). The Centers for Disease Control (CDC) Clear Communication Index (CCI) was used to assess clarity.

Results

Ten patient education resources were evaluated: CDC, Cleveland Clinic, Mayo Clinic, JAMA, American Thoracic Society (ATS), National Jewish Health, UpToDate, CHEST, WebMD, and Medline. The mean readability scores were as follows: FkGL (9.76), SMOG index (9.82), CLI (13.54), GFI (11.66), ARI (9.15). Four patient education materials were written at a sixth-grade reading level and eight patient education materials were written at an eighth-grade reading level. The majority of the materials received a passing score for understandability but failed to achieve a passing score for actionability. Cleveland Clinic, JAMA, and ATS all received a passing clarity score, indicating that they are easy to read. No patient education materials were available on UpToDate.

Conclusion

Most patient education materials scored poorly for actionability and clarity while scoring highly for readability and understandability. This study should serve as a guide for clinicians interested in offering online education materials to their patients. Increasing access to readable MAC educational materials should be a priority for those working at the intersection of public health, clinical care, and communications.

Exploring Web-Based Information and Resources That Support Adolescents and Young Adults With Cancer to Resume Study and Work: Environmental Scan Study

BACKGROUND

Adolescents and young adults (AYAs) diagnosed with cancer experience physical, cognitive, and psychosocial effects from cancer treatment that can negatively affect their ability to remain engaged in education or work through cancer treatment and in the long term. Disengagement from education or work can have lasting implications for AYAs' financial independence, psychosocial well-being, and quality of life. Australian AYAs with cancer lack access to adequate specialist support for their education and work needs and report a preference for web-based support that they can access from anywhere, in their own time. However, it remains unclear what web-based resources exist that are tailored to support AYAs with cancer in reaching their educational or work goals.

OBJECTIVE

This study aimed to determine what web-based resources exist for Australian AYAs with cancer to (1) support return to education or work and (2) identify the degree to which existing resources are age-specific, cancer-specific, culturally inclusive, and evidence-based; are co-designed with AYAs; use age-appropriate language; and are easy to find.

METHODS

We conducted an environmental scan by searching Google with English search terms in August 2022 to identify information resources about employment and education for AYAs ever diagnosed with cancer. Data extraction was conducted in Microsoft Excel, and the following were assessed: understandability and actionability (using the Patient Education and Materials Tool), readability (using the Sydney Health Literacy Laboratory Health Literacy Editor), and whether the resource was easy to locate, evidence-based, co-designed with AYAs, and culturally inclusive of Aboriginal and Torres Strait Islander peoples. The latter was assessed using 7 criteria previously developed by members of the research team.

RESULTS

We identified 24 web-based resources, comprising 22 written text resources and 12 video resources. Most resources (21/24, 88%) were published by nongovernmental organizations in Australia, Canada, the United States, and the United Kingdom. A total of 7 resources focused on education, 8 focused on work, and 9 focused on both education and work. The evaluation of resources demonstrated poor understandability and actionability. Resources were rarely evidence-based or co-designed by AYAs, difficult to locate on the internet, and largely not inclusive of Aboriginal and Torres Strait Islander populations.

CONCLUSIONS

Although web-based resources for AYAs with cancer are often available through the websites of hospitals or nongovernmental organizations, this environmental scan suggests they would benefit from more evidence-based and actionable resources that are available in multiple formats (eg, text and audio-visual) and tailored to be age-appropriate and culturally inclusive.

The Birth of the Contextual Health Education Readability Score in an Examination of Online Influenza Patient Education Materials

Introduction Influenza is a major global health concern, with its rapid spread and mutation rate posing significant challenges in public health education and communication. Effective patient education materials (PEMs) are crucial for informed decision-making and improved health outcomes. This study evaluates the efficacy of online influenza PEMs using traditional readability tools and introduces the Contextual Health Education Readability Score (CHERS) to address the limitations of existing methods that do not capture the diverse array of visual and thematic means displayed. Materials and methods A comprehensive search was conducted to select relevant online influenza PEMs. This involved looking through Google's first two pages of results sorted by relevance, for a total of 20 results. These materials were evaluated using established readability tools (e.g., Flesch Reading Ease, Flesch-Kincaid Grade Level) and the Patient Education Materials Assessment Tool (PEMAT) for understandability and actionability. The study also involved the creation of CHERS, integrating factors such as semantic complexity, cultural relevance, and visual aid effectiveness. The development of CHERS included weighting each component based on its impact on readability and comprehension. Results The traditional readability tools demonstrated significant variability in the readability of the selected materials. The PEMAT analysis revealed general trends toward clarity in purpose and use of everyday language but indicated a need for improvement in summaries and visual aids. The CHERS formula was calculated as follows: CHERS = (0.4 × Average Sentence Length) + (0.3 × Average Syllables per Word) + (0.15 × Semantic Complexity Score) + (0.1 × Cultural Relevance Score) + (0.05 × Visual Aid Effectiveness Score), integrating multiple dimensions beyond traditional readability metrics. Discussion The study highlighted the limitations of traditional readability tools in assessing the complexity and cultural relevance of health information. The introduction of CHERS addressed these gaps by incorporating additional dimensions crucial for understanding in a healthcare context. The recommendations provided for creating effective influenza PEMs focused on language simplicity, cultural sensitivity, and actionability. This may enable further research into evaluating current PEMs and clarifying means of creating more effective content in the future. Conclusions The study underscores the need for comprehensive readability assessments in PEMs. The creation of CHERS marks a significant advancement in this field, providing a more holistic approach to evaluating health literacy materials. Its application could lead to the development of more inclusive and effective educational content, thereby improving public health outcomes and reducing the global burden of influenza. Future research should focus on further validating CHERS and exploring its applicability to other health conditions.

Don't Throw Away the Handout: Efficacy of Patient Education in Pediatric Orthopaedic Trauma

BACKGROUND

Printed educational materials (PEMs) have been used for patient education in various settings. The purpose of this study was to determine the readability, understandability, and actionability of trauma-related educational material from the Pediatric Orthopaedic Society of North America (POSNA, Orthokids), as well as determine its efficacy in educating pediatric orthopaedic trauma patients and caregivers.

METHODS

The readability, understandability and actionability of PEMs was assessed using the Patient Education materials Assessment Tool (PEMAT). Five reviewers ranging in experience independently evaluated the educational materials. The efficacy of PEMs was assessed prospectively by randomizing patients into 2 groups. The first group (Education) received the OrthoKids educational material related to the patient's fracture. The second group (No Education) did not receive the educational material. At the first follow-up visit, parents/guardians in both groups completed surveys. Statistical analyses included descriptive and univariate statistics.

RESULTS

The understandability of PEMs was similar (68% to 74%); however, the educational materials had varying actionability scores ranging from 20% for femoral shaft fractures to 60% for elbow fractures. In total, 101 patients were randomized to assess the efficacy of educational materials (Education=51, No Education=50). There were no significant differences in sex, age, race/ethnicity, and level of education between caregivers in both groups ( P > 0.05). Only 61% (31/51) participants in the Education group reported using the educational material; however, 67% to 68% of participants in either group reported wanting PEMs. Participants in the group that did not receive PEMs were significantly more likely to use the internet to find more information (74% vs. 51%, P < 0.05).

CONCLUSIONS

This study suggests that participants that did not receive PEMs were significantly more likely to search the internet for more information. Improving the quality and actionability of educational resources on electronic platforms is needed to improve patient education. A multi-modal approach using PEMs that includes a list of high-quality online sources would likely be most effective in educating pediatric trauma patients and caregivers.

LEVEL OF EVIDENCE

I.

Quality and Readability of Spanish-Language Online Information for Aortic Aneurysm and Its Endovascular Treatment

BACKGROUND

Aortic aneurysms represent a chronic degenerative disease with life-threatening implications. In order for patients to comprehend health related information, it must be written at a level that can be readily understood.

STUDY DESIGN

In January 2023, we searched "aneurisma aorta" and "endoprótesis aorta" terms on Google, Yahoo and Microsoft/Bing. The 31th websites provided by each search engine were analyzed. Four readability measures were used to evaluate websites regarding aortic aneurysm and their endovascular treatment in Spanish language: Flesch Index, Flesch-Szigriszt Index, Fernández-Huerta Index and grading Inflesz scale. The quality on information was evaluated by the HONcode seal, the DISCERN instrument and the JAMA benchmark criteria.

RESULTS

180 websites containing the terms "aneurisma aorta" and "endoprótesis aorta" were analyzed. Among the websites retrieved, the mean Flesh index score (53.12 ± 6.09 and 47.48 ± 7.12, respectively; P = .019), the Flesch-Szigriszt index (56.39 ± 5.72 and 48.10 ± 8.33; P = .000), and the Fernández Huerta index (61.30 ± 5.59 and 53.19 ± 8.21; P = .000), corresponding to a "somewhat difficult" readability level. In addition, the Inflesz scale (2.62 ± .59 and 2.07 ± .61; P = .000) reported a "somewhat difficult" readability, higher for the websites regarding aortic aneurysm. The HONcode seal was only presented in websites regarding aortic aneurysm (16.7%), whereas none of the websites relating to aortic endoprostheses presented it (0%) (P = .000). Websites that presented the HONcode seal obtained higher DISCERN score (P = .000, 95% CI = 6.42-16.84) and JAMA score (P = .000, 95% CI = 3.44-11.32).

CONCLUSION

Internet information on aortic aneurysms and aortic endoprostheses is too difficult to read for the general Spanish-speaking population and is lacking in quality.

Assessing the Quality of YouTube's Incontinence Information after Cancer Surgery: An Innovative Graphical Analysis

BACKGROUND

Prostate and colorectum cancers rank among the most common cancers, and incontinence is a significant postsurgical issue affecting the physical and psychological well-being of cancer survivors. Social media, particularly YouTube, has emerged as a vital source of health information. While YouTube offers valuable content, users must exercise caution due to potential misinformation.

OBJECTIVE

This study aims to assess the quality of publicly available YouTube videos related to incontinence after pelvic cancer surgery.

METHODS

A search on YouTube related to "Incontinence after cancer surgery" was performed, and 108 videos were analyzed. Multiple quality assessment tools (DISCERN, GQS, JAMA, PEMAT, and MQ-VET) and statistical analyses (descriptive statistics and intercorrelation tests) were used to evaluate the characteristics and popularity, educational value, quality, and reliability of these videos, relying on novel graphical representation techniques such as Sankey and Chord diagrams.

RESULTS

Strong positive correlations were found among quality rating scales, emphasizing agreement. The performed graphical analysis reinforced the reliability and validity of quality assessments.

CONCLUSIONS

This study found strong correlations among five quality scales, suggesting their effectiveness in assessing health information quality. The evaluation of YouTube videos consistently revealed "high" quality content. Considering the source is mandatory when assessing quality, healthcare and academic institutions are reliable sources. Caution is advised with ad-containing videos. Future research should focus on policy improvements and tools to aid patients in finding high-quality health content.

The Readability and Quality of Web-Based Patient Information on Nasopharyngeal Carcinoma: Quantitative Content Analysis

BACKGROUND

Nasopharyngeal carcinoma (NPC) is a rare disease that is strongly associated with exposure to the Epstein-Barr virus and is characterized by the formation of malignant cells in nasopharynx tissues. Early diagnosis of NPC is often difficult owing to the location of initial tumor sites and the nonspecificity of initial symptoms, resulting in a higher frequency of advanced-stage diagnoses and a poorer prognosis. Access to high-quality, readable information could improve the early detection of the disease and provide support to patients during disease management.

OBJECTIVE

This study aims to assess the quality and readability of publicly available web-based information in the English language about NPC, using the most popular search engines.

METHODS

Key terms relevant to NPC were searched across 3 of the most popular internet search engines: Google, Yahoo, and Bing. The top 25 results from each search engine were included in the analysis. Websites that contained text written in languages other than English, required paywall access, targeted medical professionals, or included nontext content were excluded. Readability for each website was assessed using the Flesch Reading Ease score and the Flesch-Kincaid grade level. Website quality was assessed using the Journal of the American Medical Association (JAMA) and DISCERN tools as well as the presence of a Health on the Net Foundation seal.

RESULTS

Overall, 57 suitable websites were included in this study; 26% (15/57) of the websites were academic. The mean JAMA and DISCERN scores of all websites were 2.80 (IQR 3) and 57.60 (IQR 19), respectively, with a median of 3 (IQR 2-4) and 61 (IQR 49-68), respectively. Health care industry websites (n=3) had the highest mean JAMA score of 4 (SD 0). Academic websites (15/57, 26%) had the highest mean DISCERN score of 77.5. The Health on the Net Foundation seal was present on only 1 website, which also achieved a JAMA score of 3 and a DISCERN score of 50. Significant differences were observed between the JAMA score of hospital websites and the scores of industry websites (P=.04), news service websites (P<.048), charity and nongovernmental organization websites (P=.03). Despite being a vital source for patients, general practitioner websites were found to have significantly lower JAMA scores compared with charity websites (P=.05). The overall mean readability scores reflected an average reading age of 14.3 (SD 1.1) years.

CONCLUSIONS

The results of this study suggest an inconsistent and suboptimal quality of information related to NPC on the internet. On average, websites presented readability challenges, as written information about NPC was above the recommended reading level of sixth grade. As such, web-based information requires improvement in both quality and accessibility, and healthcare providers should be selective about information recommended to patients, ensuring they are reliable and readable.

Content Quality of YouTube Videos About Metastatic Breast Cancer in Young Women: Systematic Assessment

BACKGROUND

Young women with metastatic breast cancer (MBC) are part of a digitally connected generation yet are underserved in terms of information needs. YouTube is widely used to find and identify health information. The accessibility of health-related content on social media together with the rare and marginalized experiences of young women with MBC and the digital media practices of younger generations imply a considerable likelihood that young women with MBC will seek information and community on the internet.

OBJECTIVE

This study aims to assess the content quality of MBC YouTube videos and to identify themes in the experiences of young women with MBC based on YouTube videos.

METHODS

A systematic assessment of MBC YouTube videos using the search term "metastatic breast cancer young" was conducted in August 2021. The search was performed in an incognito browser and with no associated YouTube or Google account. Search results were placed in order from most to least views. Title, date uploaded, length, poster identity, number of likes, and number of comments were collected. Understandability and actionability were assessed using the Patient Education Materials Assessment Tool (PEMAT); information reliability and quality were assessed with DISCERN. Themes, sponsorships, and health care professionals' and patients' narratives were reported.

RESULTS

A total of 101 videos were identified. Of these, 78.2% (n=79) included sponsorships. The mean PEMAT scores were 78.8% (SD 15.3%) and 43.1% (SD 45.2%) for understandability and actionability, respectively. The mean DISCERN score was 2.44 (SD 0.7) out of 5. Identified themes included treatment (n=67, 66.3%), family relationship (n=46, 45.5%), and motherhood (n=38, 37.6%).

CONCLUSIONS

YouTube videos about young women with MBC are highly understandable but demonstrate moderate rates of actionability, with low reliability and quality scores. Many have a commercial bias. While web-based materials have limitations, their potential to provide patient support is not fully developed. By acknowledging their patients' media habits, health care professionals can further develop a trusting bond with their patients, provide a space for open and honest discussions of web-based materials, and avoid any potential instances of confusion caused by misleading, inaccurate, or false web-based materials.

Health literacy in rotator cuff repair: a quantitative assessment of the understandability of online patient education material

Background

The American Medical Association and National Institutes of Health recommend online health information be written at a 6th grade or lower reading level for clear understanding. While syntax reading grade level has previously been utilized, those analyses do not determine whether readers are processing key information (understandability) or identifying available actions to take (actionability). The Patient Education Materials Assessment Tool (PEMAT-P) is a method to measure the understandability and actionability of online patient education materials. The purpose of this study was to evaluate online resources regarding rotator cuff repair utilizing measures of readability, understandability, and actionability.

Methods

The search term "rotator cuff surgery" was used in two independent online searches to obtain the top 50 search results. The readability of included resources was quantified using valid objective algorithms: Flesch-Kincaid Grade-Level, Simple Measure of Gobbledygook grade, Coleman-Liau Index, and Gunning Fog Index. The PEMAT-P form was used to assess actionability and understandability.

Results

A total of 49 unique websites were identified to meet our inclusion criteria and were included in our analysis. The mean Flesch-Kincaid Grade Level graded materials at a 10.6 (approximately a 10th grade reading level), with only two websites offering materials at a 6th grade reading level or below. The remaining readability studies graded the mean reading level at high school or greater, with the Gunning Fog Index scoring at a collegiate reading level. Mean understandability and actionability scores were 64.6% and 29.5%, respectively, falling below the 70% PEMAT score threshold for both scales. Fourteen (28.6%) websites were above the threshold for understandability, while no website (0%) scored above the 70% threshold for actionability. When comparing source categories, commercial health publishers provided websites that scored higher in understandability (P < .05), while private practice materials scored higher in actionability (P < .05). Resources published by academic institutions or organizations scored lower in both understandability and actionability than private practice and commercial health publishers (P < .05). No readability, understandability, or actionability score was significantly associated with search result rank.

Conclusion

Overall, online patient education materials related to rotator cuff surgery scored poorly with respect to readability, understandability, and actionability. Only two (4.1%) of the patient education websites scored at the American Medical Association and National Institutes of Health recommended reading level. Fourteen (28.6%) scored above the 70% PEMAT score for understandability; however, no website met the threshold for actionability.

Psychometric Testing of the Turkish Version of the Patient Education Materials Assessment Tool

AIM

The aim of this study was to test the psychometric properties of the Turkish version of the Patient Education Materials Assessment Tool METHODS: The study was carried out with intern nurses receiving education at a Nursing Faculty in Turkey between July 2019 and June 2020. 248 intern nurses participated in PEMAT-P and 223 in PEMAT-A/V in this study. Five audio-visual and five printed materials were used. The Patient Education Materials Assessment Tool and the Patient Education Materials Assessment Tool User's Guide were translated into Turkish and culturally adapted using standard guidelines. Content and construct validity analyses were performed for the validity of the Turkish version of the Patient Education Materials Assessment Tool, and internal consistency, invariability, and external consistency analyses were performed for its reliability.

RESULTS

The content validity index was 1.00 for all the scale items. At the conclusion of the factor analyses, the scale has composed a twodimension construct of understandability and actionability. The Cronbach's alpha coefficients were observed to be 0.901 of the Turkish version of the Patient Education Materials Assessment Tool for Printable Materials and 0.897 of the Turkish version of the Patient Education Materials Assessment Tool for Audiovisual Materials. The Pearson Product-Moment Correlation Coefficient (rxy) was high (Turkish version of the Patient Education Materials Assessment Tool for Printable Materials rxy=0.815, Turkish version of the Patient Education Materials Assessment Tool for Audiovisual Materials rxy=0.804). Cohen's kappa coefficients were obtained as 0.736 of the Turkish version of the Patient Education Materials Assessment Tool for Printable Materials and 0.781 of the Turkish version of the Patient Education Materials Assessment Tool for Audiovisual Materials.

CONCLUSION

The Turkish version of the Patient Education Materials Assessment Tool is a valid and reliable measurement tool in the evaluation of both printed and audiovisual patient education materials.

Appraising publicly available online resources to support patients considering decisions about medical assistance in dying in Canada: an environmental scan

BACKGROUND

Medical assistance in dying (MAiD) was legalized in Canada in 2016, with legislation updated in 2021. It is unclear whether resources are available to help patients make this difficult decision; therefore, we sought to identify and quality appraise Canadian MAiD resources for supporting patients making this decision.

METHODS

We conducted an environmental scan by searching Canadian websites for online MAiD resources that were published after the 2016 MAiD legislation, patient targeted, publicly accessible and able to inform decisions about MAiD in Canada. We excluded resources that targeted health care professionals or policy-makers, service protocols and personal narratives. Two authors appraised resources using the International Patient Decision Aids Standards (IPDAS) criteria and the Patient Education Materials Assessment Tool (PEMAT) for health literacy. Descriptive analysis was conducted. We defined resources as patient decision aids if 7 IPDAS defining criteria were met, and we rated resources as adequate for understandability or actionability if the PEMAT score was 70% or greater.

RESULTS

We identified 80 MAiD resources. As of March 2023, 62 resources (90%) provided eligibility according to the 2021 legislation and 11 did not discuss any eligibility criteria. The median IPDAS score was 3 out of 7; 52% discussed alternative options and none provided benefits or harms. Of 80 resources, 59% were adequate for understandability and 29% were adequate for actionability.

INTERPRETATION

Although many resources on MAiD were updated with 2021 legislation, few were adequate to support patients with lower health literacy. There is a need to determine whether a patient decision aid would be appropriate for people in Canada considering MAiD.

Clinical utility of shared book reading on YouTube: An analysis of metadata, evidence-based content, understandability, and actionability

PURPOSE

Digital content may increase parental access to training as well as the diversity of the training programs. YouTube videos may provide a platform from which professionals may adopt content for such programs. This study provides data regarding the clinical utility of YouTube videos about shared book reading (SBR) found on the United States (USA) and India channels.

METHOD

A cross-sectional analysis of videos about SBR was conducted to examine metadata, evidence-based content, understandability, and actionability. Descriptive statistics were used to analyse metadata and content. Health literacy concepts of understandability and actionability were measured, and descriptive statistics are reported.

RESULT

Of 800 videos identified, 10 included information related to storybook selection, dialogic reading, extra-textual strategies, and print-referencing strategies. These met adequate levels of understandability and actionability.

CONCLUSION

Results provide some direction and recommendations for parent-training programs in the USA and India. The study highlights the use of YouTube videos and suggests its utility as a supplement source within parent-training programs.

Assessing Arabic youtube videos on herbal cancer treatment: Absence of health information quality

This study assesses the quality of the health information in Arabic YouTube videos regarding herbal cancer treatment. It also provides an overview of how the quality of video content shapes user awareness by assessing the users' engagement indicators. A simple Python tool was developed using YouTube API V3 to automate the YouTube search based on the recommendation of Google Trends. After applying inclusion and exclusion criteria, 110 YouTube videos were selected, of which 95% were uploaded by non-experts and had a total of 8,633,569 views. The analyzed videos presented more than 40 different herbals as sources of cancer treatment; for example, Ephedra, garden cress, Green tea, Ginseng, Rosemary, and Thyme. 32.7% of the videos provided information about a single herb, 41% about mixing herbals, and 26.3% provided testimonials and success stories without pointing to specific herbs. The videos were assessed by two experts using two reliable tools, DISCERN and PEMAT, which were produced mainly for assessing health information quality. DISCERN has evaluated the reliability and quality of health information. PEMAT has assessed the understandability and actionability. The qualitative and quantitative analyses of the videos represent bias and poor health information quality, with a total score of 27 out of 80 for DISCERN and 31 out of 100 for the PEMAT. The results also showed weak users' awareness regarding the content of videos with no association between user engagement indicators (likes, dislikes, VPI, views, comments) and the dimensions of the two tools. The study concludes that it is evident that YouTube, in its current form, is an inadequate Arabic source for herbal cancer treatment information. To overcome this, this study proposed the GAP framework for social media that integrated Governance, Awareness, and Proficiency.

Quality of patient decision aids to support the public making COVID-19 decisions: An online environmental scan

OBJECTIVE

To identify and appraise the quality of COVID-19 patient decision aids (PtDAs).

METHODS

We conducted an environmental scan of online publicly available COVID-19 PtDAs. Two reviewers independently searched and extracted data. We calculated median International Patient Decision Aid Standards (IPDAS) scores and proportion scoring > 70% on Patient Education Materials Information Tool (PEMAT) adequate for understandability and actionability.

RESULTS

Of 876 resources identified, 12 were PtDAs. Decisions focused on initial COVID-19 vaccination series (n = 9), location of care for elderly (n = 2), and social distancing (n = 1). All 12 PtDAs were written materials and two had accompanying videos. The median IPDAS score minimizing risk of biased decisions was 4 of 6 items (IQR 1, range 2-4). For PEMAT, 92% had adequate for understandability and none for actionability.

CONCLUSIONS

We identified few online publicly available COVID-19 PtDAs and none were about COVID-19 vaccination boosters or treatment. PtDAs scored poorly on actionability and none met all IPDAS criteria for minimizing risk of biased decisions.

PRACTICE IMPLICATIONS

PtDA developers for COVID-19 and future pandemics should ensure their PtDAs meet all IPDAS criteria for minimizing risk of bias, have adequate scores for actionability, and are disseminated in the A to Z inventory.

The Chinese Version of the Patient Education Materials Assessment Tool for Printable Materials: Translation, Adaptation, and Validation Study

BACKGROUND

Providing people with understandable and actionable health information can considerably promote healthy behaviors and outcomes. To this end, some valid and reliable scales assessing the patient-friendliness of health education materials, like the PEMAT-P (Patient Education Materials Assessment Tool for printable materials), have been well developed in English-speaking countries. However, the English version of the PEMAT-P has not been translated and adapted into simplified Chinese and validated in mainland China.

OBJECTIVE

This study sought to translate the PEMAT-P tool into a simplified Chinese (Mandarin) version (C-PEMAT-P, a Chinese version of the Patient Education Materials Assessment Tool for printable materials) and verify its validity and reliability for assessing the comprehensibility and actionability of health education resources written in simplified Chinese. As a result, the validated C-PEMAT-P could be used to guide health researchers and educators to design more comprehensible and actionable materials for more tailored and targeted health education and interventions.

METHODS

We translated the PEMAT-P into simplified Chinese in the following three steps: (1) forward-translating the PEMAT-P into simplified Chinese, (2) back-translating the simplified Chinese version into English, and (3) testing translation equivalence linguistically and culturally by examining the original English version of the PEMAT-P and the back-translated English version of the tool. Any discrepancies between the original English tool and the back-translated English tool were resolved through a panel discussion among the research team of all authors to produce a revised forward-translated Chinese version (C-PEMAT-P). We then evaluated the clarity of construction and wording as well as the content relevance of the C-PEMAT-P using a 4-point ordinal scale to determine its content validity. After that, 2 native Chinese speakers (health educators) used the C-PEMAT-P to rate 15 health education handouts concerning air pollution and health to validate their reliability. We calculated the Cohen coefficient and Cronbach α to determine the interrater agreement and internal consistency of the C-PEMAT-P, respectively.

RESULTS

We finalized the translated Chinese tool after discussing the differences between the 2 English versions (original and back-translated) of the PEMAT-P, producing the final Chinese version of the PEMAT-P (C-PEMAT-P). The content validity index of the C-PEMAT-P version was 0.969, the Cohen coefficient for the interrater scoring agreement was 0.928, and the Cronbach α for internal consistency was .897. These values indicated the high validity and reliability of the C-PEMAT-P.

CONCLUSIONS

The C-PEMAT-P has been proven valid and reliable. It is the first Chinese scale for assessing the comprehensibility and actionability of Chinese health education materials. It can be used as an assessment tool to evaluate health education materials currently available and a guide to help health researchers and educators design more comprehensible and actionable materials for more tailored and targeted health education and interventions.

Hip and Knee Total Joint Arthroplasty Online Resources for Patients and Health Care Professionals: A Canadian Environmental Scan

Purpose:

To appraise the quality of publicly available online Canadian resources for patients with hip or knee osteoarthritis considering total joint arthroplasty (TJA) and health care professionals participating in TJA decision-making processes.

Method:

An environmental scan. Two independent authors appraised: a) patient resources against the International Patient Decision Aids Standards (IPDAS) criteria and the Patient Education Material Evaluation Tool (PEMAT); and b) health care professional resources against six appropriateness criteria for TJA and eight elements of shared decision-making. Analysis was descriptive.

Results:

Of 84 included resources, 71 were for patients, 11 for health care professionals, and 2 for both. For patient resources, the median number of IPDAS defining criteria met was 2 of 7, median PEMAT understandability score was 83%, and median PEMAT actionability score was 60%. For health care professional resources, the median number of appropriateness criteria was 3 of 6, and the median number of shared decision-making elements was 3 of 8.

Conclusions:

Only four of 73 patient resources were structured to help patients consider their options and reach a decision based on their preferences. Health care professional resources were limited to traditional criteria for determining TJA appropriateness (evidence of osteoarthritis, use of conservative treatments) and poorly met key elements of shared decision-making.

Readability of Online Patient Education Materials for Graft-Versus-Host Disease

Graft-versus-host disease (GVHD) is a complex systemic diagnosis which is associated with significant symptom distress in patients. Patient education has shown to mitigate uncertainty and distress, but to our knowledge, no studies have evaluated patient education materials on GVHD. We characterized the readability and understandability of patient education materials on GVHD available online. We conducted a Google search of the top 100 non-sponsored search results, selecting for full-text patient education that is not peer-reviewed or a news article. We evaluated the text of the eligible search results against the Flesch-Kincaid Reading Ease, Flesch Kincaid Grade Level, Gunning Fog, Automated Readability, Linsear Write Formula, Coleman-Liau Index, Smog Index, and Patient Education Materials Assessment Tool (PEMAT) for understandability. Among 52 included Web results, 17 (32.7%) were provider-authored and 15 (28.8%) were hosted on university Web sites. The total average scores on validated readability tools were Flesch-Kincaid Reading Ease (46.4), Flesch Kincaid Grade Level (11.6), Gunning Fog (13.6), Automated Readability (12.3), Linsear Write Formula (12.6), Coleman-Liau Index (12.3), Smog Index (10.0), and PEMAT Understandability (65.5). Provider-authored links scored poorer than non-provider-authored links on all metrics, with significant differences for the Gunning Fog index (p < 0.05). University-hosted links scored better than non-university-hosted links on all metrics. Evaluation of online patient education materials for GVHD demonstrates the need for more readable and understandable resources to mitigate the distress and uncertainty that patients may feel upon being diagnosed with GVHD.

Patient education information material assessment criteria: A scoping review

BACKGROUND

Patient education information material (PEIM) is an essential component of patient education programs in increasing patients' ability to cope with their diseases. Therefore, it is essential to consider the criteria that will be used to prepare and evaluate these resources.

OBJECTIVE

This paper aims to identify these criteria and recognize the tools or methods used to evaluate them.

METHODS

National and international databases and indexing banks, including PubMed, Scopus, Web of Science, ProQuest, the Cochrane Library, Magiran, SID and ISC, were searched for this review. Original or review articles, theses, short surveys, and conference papers published between January 1990 and June 2022 were included.

RESULTS

Overall, 4688 documents were retrieved, of which 298 documents met the inclusion criteria. The criteria were grouped into 24 overarching criteria. The most frequently used criteria were readability, quality, suitability, comprehensibility and understandability.

CONCLUSION

This review has provided empirical evidence to identify criteria, tools, techniques or methods for developing or evaluating a PEIM. The authors suggest that developing a comprehensive tool based on these findings is critical for evaluating the overall efficiency of PEIM using effective criteria.

Video tools in pediatric goals of care communication: A systematic review

•Goals of Care resources are primarily in written format and for adult patients.•Video tools can support families of pediatric patients facing prognostic uncertainty.•Videos represent an effective but underutilized Goals of Care communication tool.

ASSESSMENT OF AVAILABLE ONLINE EDUCATIONAL RESOURCES FOR PATIENTS WITH ATRIAL FIBRILLATION

Background

A diagnosis of atrial fibrillation (AF) often leads patients to search online for information, which can expose them to information of varied quality.

Objective

We conducted a qualitative systematic review of websites that contain useful information regarding AF.

Methods

The following terms were searched on 3 search engines (Google/Yahoo/Bing): (Atrial fibrillation for patients), (What is atrial fibrillation), (Atrial fibrillation patient information), (Atrial fibrillation educational resources). Inclusion criteria included websites with comprehensive AF information and information about treatment options. The Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P) and PEMAT for Audiovisual Materials assessed understandability and actionability (score range 0-100). Those with a mean PEMAT-P score of >70, meaning acceptable understandability and actionability, underwent DISCERN score assessment of information content quality and reliability (score range 16-80).

Results

The search yielded 720 websites that underwent full review. After exclusions, 49 underwent full scoring. The mean overall PEMAT-P score was 69.3 ± 17.2. The mean PEMAT-AV score was 63.4 ± 13.6. Of the websites that scored >70% on the PEMAT-P, 23 (46%) underwent DISCERN scoring. The mean DISCERN score was 54.7 ± 4.6.

Conclusions

There is a wide variation in the understandably, actionability, and quality of websites, many not providing patient-level materials. Knowledge of quality websites could provide an important adjunct for improving patients understanding of AF.

Translation, Cross-Cultural Adaptation, and Validation of the Japanese Version of the Patient Education Materials Assessment Tool (PEMAT)

BACKGROUND

The Patient Education Materials Assessment Tool (PEMAT) systematically evaluates the understandability and actionability of patient education materials. This study aimed to develop a Japanese version of PEMAT and verify its reliability and validity.

METHODS

After assessing content validation, experts scored healthcare-related leaflets and videos according to PEMAT to verify inter-rater reliability. In validation testing with laypeople, the high-scoring material group (n = 800) was presented with materials that received high ratings on PEMAT, and the low-scoring material group (n = 799) with materials that received low ratings. Both groups responded to the understandability and actionability of the materials and perceived self-efficacy for the recommended actions.

RESULTS

The Japanese version of PEMAT showed strong inter-rater reliability (PEMAT-P: % agreement = 87.3, Gwet's AC1 = 0.83. PEMAT-A/V: % agreement = 85.7, Gwet's AC1 = 0.80). The high-scoring material group had significantly higher scores for understandability and actionability than the low-scoring material group (PEMAT-P: understandability 6.53 vs. 5.96, p < 0.001; actionability 6.04 vs. 5.49, p < 0.001; PEMAT-A/V: understandability 7.65 vs. 6.76, p < 0.001; actionability 7.40 vs. 6.36, p < 0.001). Perceived self-efficacy increased more in the high-scoring material group than in the low-scoring material group.

CONCLUSIONS

Our study showed that materials rated highly on Japanese version of PEMAT were also easy for laypeople to understand and action.

The Impact of Patient Education Level on Split-Dose Colonoscopy Bowel Preparation for CRC Prevention

INTRODUCTION

AGA guidelines emphasize split-dose bowel preparation (BP) to ensure high-quality colonoscopy for the prevention of colorectal cancer (CRC). Split dose results in higher-quality preparation, but understanding instructions might be more difficult. Lower education levels may negatively influence BP quality. The confounding role of education level on BP quality was investigated.

METHODS

This was a cross-sectional study of 60 patients given split-dose BP. Patients consented and were asked three Likert scale questions based on BP instructions before the procedure. Compliance was self-reported. BP adequacy and the number of adenomas were recorded. BP was characterized as adequate (excellent, good) or inadequate (fair, poor). Data was analyzed with chi-square, odds ratio, Mann-Whitney, and regression analysis.

RESULTS

Thirty-one (52%) patients were high school graduates, 21 (38%) completed some college, and 6 (10%) were college graduates. College-educated patients had adequate BP (72%) more often than high school graduates (51%) (p = 0.02). Adenoma findings were not significantly different. The Likert scale mean ranks for patient understanding and reviewing of instructions were comparable between the two groups. Patient rating of scheduler explanations of the importance of following instructions was significantly better in the college group (mean ranks 2.59 and 1.83, respectively; p = 0.018).

DISCUSSION

Patient education level significantly affected the success of BP. Split BP can be more complex to comprehend, and instructions should consider patient education level. Specific intervention programs should be implemented to advise patients with less education that poor preparation may result in missed advanced neoplasias and subsequent procedures.

Can Patients Read, Understand, and Act on Online Resources for Anterior Cruciate Ligament Surgery?

Background:

Patients undergoing elective procedures often utilize online educational materials to familiarize themselves with the surgical procedure and expected postoperative recovery. While the Internet is easily accessible and ubiquitous today, the ability of patients to read, understand, and act on these materials is unknown.

Purpose:

To evaluate online resources about anterior cruciate ligament (ACL) surgery utilizing measures of readability, understandability, and actionability.

Study Design:

Cross-sectional study; Level of evidence, 4.

Methods:

Using the term “ACL surgery,” 2 independent searches were performed utilizing a public search engine (Google.com). Patient education materials were identified from the top 50 results. Audiovisual materials, news articles, materials intended for advertising or medical professionals, and materials unrelated to ACL surgery were excluded. Readability was quantified using the Flesch Reading Ease, Flesch-Kincaid Grade Level, Simple Measure of Gobbledygook, Coleman-Liau Index, Automated Readability Index, and Gunning Fog Index. The Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P) was utilized to assess the actionability and understandability of materials. For each online source, the relationship between its Google search rank (from first to last) and its readability, understandability, and actionability was calculated utilizing the Spearman rank correlation coefficient (ρ_S).

Results:

Overall, we identified 68 unique websites, of which 39 met inclusion criteria. The mean Flesch-Kincaid Grade Level was 10.08 ± 2.34, with no website scoring at or below the 6th-grade level. Mean understandability and actionability scores were 59.18 ± 10.86 (range, 33.64-79.17) and 34.41 ± 22.31 (range, 0.00-81.67), respectively. Only 5 (12.82%) and 1 (2.56%) resource scored above the 70% adequate PEMAT-P threshold mark for understandability and actionability, respectively. Readability (lowest P value = .103), understandability (ρ_S = –0.13; P = .441), and actionability (ρ_S = 0.28; P = .096) scores were not associated with Google rank.

Conclusion:

Patient education materials on ACL surgery scored poorly with respect to readability, understandability, and actionability. No online resource scored at the recommended reading level of the American Medical Association or National Institutes of Health. Only 5 resources scored above the proven threshold for understandability, and only 1 resource scored above it for actionability.

Readability, understandability and language accessibility of Swedish websites about the coronavirus disease 2019: a cross-sectional study

BACKGROUND

The COVID-19 pandemic has caused significant morbidity and mortality. To mitigate its spread, members in the general population were prompted to apply significant behavioral changes. This required an effective dissemination of understandable information accessible for people with a wide range of literacy backgrounds. The aim of this study was to investigate the readability, understandability and language accessibility of Swedish consumer-oriented websites containing information about COVID-19.

METHODS

Websites were identified through systematic searches in Google.se (n = 76), and were collected in May 2020 when the pandemic spread started in Sweden. Readability and understandability were assessed with the Readability Index, the Ensuring Quality Information for Patients (EQIP) tool, and the Patient Education Materials Assessment Tool Understandability subscale (PEMAT-PU).

RESULTS

The median total sample score for Readability Index was 42.0, with the majority of scores being classified as moderate (n = 30, 39%) or difficult (n = 43, 57%). Median total sample scores were for EQIP 54.0% (IQR = 17.0, Range = 8-75) and for PEMAT-PU 60.0% (IQR = 14.75, Range = 12-87). The majority of the websites did not have any texts or links containing information in an alternative language (n = 58, 76%).

CONCLUSIONS

Swedish websites contained information of difficult readability and understandability at the beginning of the coronavirus disease 2019 pandemic, with few providing information available in alternative languages. It is possible that these deficits contributed to the spread and impact of the virus. There is a need for studies investigating methods aiming to enhance the readability, understandability and language accessibility of web-based information at the beginning of an epidemic or pandemic.

Assessment of patient education materials for age-related macular degeneration

Purpose

Age‐related macular degeneration (AMD) is a leading cause of vision loss. It is helpful for patients living with AMD to understand the prognosis, risk factors and management of their condition. Online education materials are a popular and promising channel for conveying this knowledge to patients with AMD. However, the quality of these materials—particularly with respect to qualities such as ‘understandability’ and ‘actionability’—is not yet known. This study assessed a collection of online materials about AMD based on these qualities of ‘understandability’ and ‘actionability’.

Methods

Online education materials about AMD were sourced through Google from six English‐speaking nations: Australia, New Zealand, USA, UK, Ireland and Canada. Three Australian/New Zealand trained and registered optometrists participated in the grading of the ‘understandability’ and ‘actionability’ of online education materials using the Patient Education Materials Assessment Tool (PEMAT).

Results

This study analysed a total of 75 online materials. The mean ‘understandability’ score was 74% (range: 38%–94%). The ‘understandability’ PEMAT criterion U11 (calling for a summary of the key points) scored most poorly across all materials. The mean ‘actionability’ score was 49% (range: 0%–83%). The ‘actionability’ PEMAT criterion A26 (using ‘visual aids’ to make instructions easier to act on) scored most poorly across all materials.

Conclusion

Most education materials about AMD are easy to understand, but difficult to act on, because of a lack of meaningful visual aids. We propose future enhancements to AMD education materials—including the use of summaries, visual aids and a habit tracker—to help patients with AMD improve their understanding of disease prognosis, risk factors and eye assessment schedule requirements.

Readability of patient educational materials in venous thrombosis. Analysis of the 2021 ESVS guidelines and comparison with other medical societies information

BACKGROUND

In order for patients to comprehend health related information, it must be written at a level that can be readily understood by the intended population. During 2021 the European Society for Vascular Surgery (ESVS) published a sub-section about information for patients into its Guidelines on the Management of Venous Thrombosis.

METHODS

Nine readability measures were used to evaluate the patient educational material regarding venous thrombosis published by seven medical societies: ESVS, Society for Vascular Medicine (SVM), Society for Vascular Surgery (SVS), Vascular Society for Great Britain and Ireland (VS), Australia and New Zealand Society for Vascular Surgery (ANZSVS), Canadian Society for Vascular Surgery (CSVS) and American Heart Association (AHA).

RESULTS

The mean reading grade level (RGL) for all the 58 recommendations was 10.61 (range 6.4-14.5) and the mean Flesch Reading Ease (FRE) was 56.10 (51.3-62.9), corresponding to a "fairly difficult" reading level. The mean RGL of the ESVS recommendations (11.45, 95% CI, 9.90-13.00) was significantly higher than the others. Post-hoc analysis determined a significant difference between the ESVS and the SVS (10.86, 95% CI, 9.84-11.91) recommendations (p=0.005). All the patient's education information published by the medical societies presented a RGL higher than recommended. The fifteen sub-sections of the information for patients included into the ESVS clinical guidelines presented a mean RGL above 9.5 points, revealing that no one (0%) was written at or below the recommended GRL. The mean FRE was 47.63 (28.2-61.6), corresponding to a "difficult" reading level.

CONCLUSIONS

Venous thrombosis patient educational materials produced by leading medical societies have readability scores that are above the recommended levels. The innovative patient's information included into the ESVS venous thrombosis guidelines represents an important advance in the amelioration of the medical information for patients but their readability should be improved to adapt the understanding to the general-population.

An examination of the quality of late-life depression websites on the Internet (Preprint)

Background

The internet can increase the accessibility of mental health information and improve the mental health literacy of older adults. The quality of mental health information on the internet can be inaccurate or biased, leading to misinformation.

Objective

This study aims to evaluate the quality, usability, and readability of websites providing information concerning depression in later life.

Methods

Websites were identified through a Google search and evaluated by assessing quality (DISCERN), usability (Patient Education Materials Assessment Tool), and readability (Simple Measure of Gobbledygook).

Results

The overall quality of late-life depression websites (N=19) was adequate, and the usability and readability were poor. No significant relationship was found between the quality and readability of the websites.

Conclusions

The websites can be improved by enhancing information quality, usability, and readability related to late-life depression. The use of high-quality websites may improve mental health literacy and shared treatment decision-making for older adults.

Total Knee Arthroplasty: A Quantitative Assessment of Online Patient Education Resources

Background

Patients often turn to the online resources to learn about orthopedic procedures. As the rate of joint arthroplasty is projected to increase, the corresponding interest in relevant online education material will increase as well. The American Medical Association (AMA) and National Institutes of Health (NIH) recommend that publicly available online health information be written at the 6th grade or lower reading level to be fully understood by the average adult in the United States. Additionally, educational resources should be written such that readers can process key information (understandability) or identify available actions to take (actionability). The purpose of this study was to quantify the readability, understandability, and actionability of online patient educational materials regarding total knee arthroplasty (TKA).

Methods

The most common Google™ search term utilized by the American public was determined to be "knee replacement". Subsequently two independent online searches (Google.com) were performed. From the top 50 search results, websites were included if directed at educating patients regarding TKA. Non-text websites (audiovisual), articles (news/research/industry), and unrelated resources were excluded. Readability was quantified using the following valid objective algorithms: Flesch-Kincaid Grade-Level (FKGL), Simple Measure of Gobbledygook (SMOG) grade, Coleman-Liau Index (CLI), and Gunning-Fog Index (GFI). PEMAT was utilized to assess understandability and actionability (0-100%; score ≥70% indicates acceptable scoring). The relationship between search rank with FKGL and PEMAT scores was quantified.

Results

A total of 34 (68%) unique websites met inclusion criteria. The mean FKGL, SMOG, CLI, and GFI was 11.8±1.6, 11.1±1.2, 11.9±1.4, and 14.7±1.6, respectively. None of the websites scored within the acceptable NIH/AMA recommended reading levels. Mean understandability and actionability scores were 54.9±12.1 and 30.3±22.0. Only 5.9% (n=2) and 9.2% (n=1) of websites met the ≥70% threshold for understandability and actionability. Only 29.4% (n=10) sources used common language and only 26.9% (n=9) properly defined complicated medical terms. Based on website type, the mean understandability scores for academic institution, private practice, and health information publisher websites were 57.2±8.8%, 52.6±11.1%, and 54.3±15.3% (p=0.67). Readability (rho: -0.07; p=0.69), understandability (rho: -0.02; p=0.93), and actionability (rho: -0.22; p=0.23) scores were not associated with Google™ search rank.

Conclusion

TKA materials scored poorly with respect to readability, understandability, and actionability. None of the resources scored within the recommended AMA/NIH reading levels. Only 5.9% scored adequately on understandability measures. Substantial efforts are needed to improve online resources to optimize patient comprehension and facilitate informed decision-making. Level of Evidence: III.

Automatic Diagnosis of Mental Healthcare Information Actionability: Developing Binary Classifiers

We aimed to develop a quantitative instrument to assist with the automatic evaluation of the actionability of mental healthcare information. We collected and classified two large sets of mental health information from certified mental health websites: generic and patient-specific mental healthcare information. We compared the performance of the optimised classifier with popular readability tools and non-optimised classifiers in predicting mental health information of high actionability for people with mental disorders. sensitivity of the classifier using both semantic and structural features as variables achieved statistically higher than that of the binary classifier using either semantic (p < 0.001) or structural features (p = 0.0010). The specificity of the optimized classifier was statistically higher than that of the classifier using structural variables (p = 0.002) and the classifier using semantic variables (p = 0.001). Differences in specificity between the full-variable classifier and the optimised classifier were statistically insignificant (p = 0.687). These findings suggest the optimised classifier using as few as 19 semantic-structural variables was the best-performing classifier. By combining insights of linguistics and statistical analyses, we effectively increased the interpretability and the diagnostic utility of the binary classifiers to guide the development, evaluation of the actionability and usability of mental healthcare information.

Radiotherapy or Surgery? Comparative, Qualitative Assessment of Online Patient Education Materials on Prostate Cancer

As multiple different treatment options are available for prostate cancer (PCa) and YouTube is commonly used as a source for medical information, we performed a systematic and comparative assessment of available videos guiding patients on their choice for the optimal treatment. An independent search for surgical therapy or radiotherapy of PCa on YouTube was performed and the 40 most viewed videos of both groups were analyzed. The validated DISCERN questionnaire and PEMAT were utilized to evaluate their quality and misinformation. The median overall quality of the videos was found to be low for surgery videos, while radiotherapy videos results reached a moderate quality. The median PEMAT understandability score was 60% (range 0-100%) for radiotherapy and 75% (range 40-100) for surgery videos. The radiotherapy videos contained less misinformation and were judged to be of higher quality. Summarized, the majority of the provided videos offer insufficient quality of content and are potentially subject to commercial bias without reports on possible conflict of interest. Thus, most of available videos on YouTube informing PCa patients about possible treatment methods are not suited for a balanced patient education or as a basis for the patient's decision.

A health literacy analysis of the consumer-oriented COVID-19 information produced by ten state health departments

Objective:

The COVID-19 pandemic highlights the public's need for quality health information that is understandable. This study aimed to identify (1) the extent to which COVID-19 messaging by state public health departments is understandable, actionable, and clear; (2) whether materials produced by public health departments are easily readable; (3) relationships between material type and understandability, actionability, clarity, and reading grade level; and (4) potential strategies to improve public health messaging around COVID-19.

Methods:

Based on US Centers for Disease Control and Prevention statistics from June 30, 2020, we identified the ten states with the most COVID-19 cases and selected forty-two materials (i.e., webpages, infographics, and videos) related to COVID-19 prevention according to predefined eligibility criteria. We applied three validated health literacy tools (i.e., Patient Education Materials Assessment Tool, CDC Clear Communication Index, and Flesch-Kincaid Grade Level) to assess material understandability, actionability, clarity, and readability. We also analyzed correlations between scores on the three health literacy tools and material types.

Results:

Overall, COVID-19 materials had high understandability and actionability but could be improved in terms of clarity and readability. Material type was significantly correlated with understandability, actionability, and clarity. Infographics and videos received higher scores on all tools.

Conclusions:

Based on our findings, we recommend public health entities apply a combination of these tools when developing health information materials to improve their understandability, actionability, and clarity. We also recommend using infographics and videos when possible, taking a human-centered approach to information design, and providing multiple modes and platforms for information delivery.

Breast Implant-Associated Anaplastic Large Cell Lymphoma Online Education Tools: A Multimetric Analysis

PURPOSE

In light of highly publicized media coverage on breast implant recalls and Food and Drug Administration hearings on breast implant safety, online searches of these topics have surged. It is thus critical to determine whether such searches are providing meaningful information for those who use them. Patient/laywomen-directed online education materials on breast implant-associated anaplastic large cell lymphoma (BIA-ALCL) must not only be comprehensible but also accurate, actionable, and culturally concordant, especially as more diverse populations turn to the internet for breast implant-related information. This study assessed the overall suitability of BIA-ALCL patient-directed educational text and video-based materials online.

METHODS

This was a cross-sectional, multimetric investigation of online text- and video-based patient-directed educational materials for BIA-ALCL using multiple validated measures. Two reviewers assessed each resource. Kruskal-Wallis and Fisher exact analyses were used as appropriate to compare text- and video-based online resources.

RESULTS

In total, 30 websites and 15 videos were evaluated, more than half (56%) of which were from academic/physician or governmental/nonprofit sources. Overall, website and video content, as well as quality, varied by source. Academic/physician or governmental/nonprofit sources tended to be more comprehensive and have higher-quality information than commercial/media outlet sources. Median actionability of websites was 38%, well below the threshold of 70% used in the literature to identify actionable resources. The median suitability score for BIA-ALCL websites was 57%, which is considered "adequate" based on published thresholds. The mean overall Cultural Sensitivity Assessment Tool score for websites was 2.4; Cultural Sensitivity Assessment Tool scores higher than 2.5 are generally regarded as culturally sensitive. In general, videos were more understandable than websites. Substantial interrater reliability across the validated tools used in this study was noted using Pearson correlation coefficients.

CONCLUSIONS

Online resources varied in content and quality by source. As BIA-ALCL becomes an increasingly salient topic among both providers and patients, it is important to empower women with accurate information about this implant-associated cancer. Of available resources, providers should refer patients or those seeking more information to websites from governmental/academic organizations (".gov" or ".org" domains) and videos from academic/physician or governmental sources, given that among high-quality resources, these were most clear/comprehensible. Overall, there is a need for improved online content on this topic.

Online Decision Aids for Knee Arthroplasty: An Environmental Scan

BACKGROUND

Knee arthroplasty is a high-risk, resource-intensive procedure that should be reserved for patients in whom the benefit will outweigh the risks. The provision of high-quality, publicly available decision aids can help patients to balance the benefits against the harms of treatments and to assist informed decision-making. The aim of this study was to identify and evaluate the content and readability of freely available knee arthroplasty decision aids.

METHODS

A systematic search using an environmental scan methodology of publicly available online materials was performed in December 2018. Included materials were assessed for quality using the International Patient Decision Aid Standards instrument (IPDASi), understandability and actionability from a patient's perspective using the Patient Education Materials Assessment Tool (PEMAT), and readability grade level using the Flesch-Kincaid Grade Level and the Gunning Fog Index.

RESULTS

Of 761 online materials screened, 26 decision aids were identified. Only 18 (69%) may be considered to meet criteria to be defined as a decision aid and 4 (15%) met criteria suggesting that the material did not introduce potential harmful bias according to the IPDASi. The mean score (and standard deviation) for all decision aids was 74% ± 12% for understandability and 44% ± 19% for actionability using the PEMAT. Readability indices indicated a mean minimum reading level of Grade 10 (10.8 ± 2.5). No decision aid included a wait-and-see option.

CONCLUSIONS

Few high-quality decision aids exist for patients considering knee arthroplasty and none include a wait-and-see option. Many do not provide actionable options and are pitched at above-average reading levels. Developers need to address these issues to encourage high-quality decision-making, especially for those with low health literacy.

Developing Strategies for Targeted Improvement of Perioperative Education for Postbariatric Surgery Body-Contouring Patients

BACKGROUND

The quality of perioperative patient education impacts surgical outcomes, patient experiences, and resources needed to address patient concerns and unplanned visits. We examined patient inquiries and education materials to assess the quality of perioperative education and identify areas of targeted improvement for postbariatric surgery body-contouring procedures.

METHODS

We examined 100 consecutive postbariatric procedures at an academic center. Themes of patient-generated calls, e-mails, and electronic medical record portal messages during the perioperative period were identified via qualitative analysis. Understandability and actionability of perioperative educational resources were assessed using the Patient Education Materials Assessment Tool (PEMAT).

RESULTS

Among 212 communications identified, 167 (79%) were postoperative. Common themes were concerns regarding the surgical site (38%), medications (10%), and activity restrictions (10%). One hundred thirty inquiries were resolved through patient re-education (57%), but 36 (16%) required in-person evaluation including 4 unplanned emergency department visits and 3 readmissions for surgical-site concerns. The PEMAT scores for institutional materials were fair for understandability (69%) and actionability (60%). American Society of Plastic Surgeons materials were more understandable (84%) but less actionable (40%).

CONCLUSIONS

Patient queries can be leveraged as a source of qualitative data to identify gaps in perioperative education. High-yield topics, such as education regarding the surgical site and medications, can be targeted for quality improvement through better communication and potentially reduce the number of unnecessary visits. Using the PEMAT, we also identified how directly the education materials can be revised. Improving perioperative education can promote mutual understanding between patients and surgeons, better outcomes, and efficient resource utilization.

Practical Advice Regarding the Reliability of the Patient Educational Materials Assessment Tool for Health Educators

The quality of patient education materials is an important issue for health educators, clinicians, and community health workers. We describe a challenge achieving reliable scores between coders when using the Patient Educational Materials Assessment Tool (PEMAT) to evaluate farmworker health materials in spring 2020. Four coders were unable to achieve reliability after three attempts at coding calibration. Further investigation identified improvements to the PEMAT codebook and evidence of the difficulty of achieving traditional interrater reliability in the form of Krippendorff's alpha. Our solution was to use multiple raters and average ratings to achieve an acceptable score with an intraclass correlation coefficient. Practitioners using the PEMAT to evaluate materials should consider averaging the scores of multiple raters as PEMAT results otherwise may be highly sensitive to who is doing the rating. Not doing so may inadvertently result in the use of suboptimal patient education materials.

Health Literacy in Clubfoot: A Quantitative Assessment of the Readability, Understandability and Actionability of Online Patient Education Material

BACKGROUND

Parents often access online resources to educate themselves when a child is diagnosed with clubfoot and/ or prior to treatment initiation. In order to be fully understood by the average adult American, online health information must be written at an elementary school reading level. It was hypothesized that current available online resources regarding clubfoot would score poorly on objective measures of readability (syntax reading grade-level), understandability (ability to process key messages), and actionability (providing actions the reader may take). Additionally, it was hypothesized that the outcomes measured would not correlate with the order of listed search results.

METHODS

Patient education materials were identified utilizing two independent online searches (Google.com) of the term "Clubfoot". From the top 50 search results, websites were included if directed at educating patients and their families regarding clubfoot. News articles, non-text material (video), research and journal articles, industry websites, and articles not related to clubfoot were excluded. The readability of included resources was quantified using the Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Simple Measure of Gobbledygook (SMOG) Grade, Coleman-Liau Index (CLI), Gunning-Fog Index( GFI) and Automated Reading Index (ARI). The Patient Education Materials Assessment Tool (PEMAT) was used to assess actionability and understandability using a 0-100% scale for both measures of interest.

RESULTS

Of the 55 unique websites, 37 websites (65.2%) met inclusion criteria. The mean FKGL was 9.2 (+/- 2.1) with only three websites (7.32%) having a reading level ≤6. Mean understandability and actionability scores were 67.2±12.6 and 25.4±25.2, respectively. Thirteen (35%) websites met the understandability threshold of ≥70% but no websites met the actionability criteria. No readability statistics were statistically associated with Google™ search rank (p=0.07). There was no association between readability (p=0.94) nor actionability (p=0.18) scores and Google™ rank. However, understandability scores did correlate with Google™ rank (p=0.02).

CONCLUSION

Overall, online clubfoot educational materials scored poorly with respect to readability, understandability, and actionability. There is an association with Google™ search rank for understandability of clubfoot materials. However, readability and actionability are not significantly associated with search rank. In the era of shared decision-making, efforts should be made by medical professionals to improve the readability, understandability, and actionability of online resources in order to optimize parent understanding and facilitate effective outcomes.Level of Evidence.

Osteosarcoma Online Resources: A Quantitative Assessment of the Understandability and Readability of Web-based Patient Education Material (Preprint)

Background

Patients often turn to web-based resources following the diagnosis of osteosarcoma. To be fully understood by average American adults, the American Medical Association (AMA) and National Institutes of Health (NIH) recommend web-based health information to be written at a 6th grade level or lower. Previous analyses of osteosarcoma resources have not measured whether text is written such that readers can process key information (understandability) or identify available actions to take (actionability). The Patient Education Materials Assessment Tool (PEMAT) is a validated measurement of understandability and actionability.

Objective

The purpose of this study was to evaluate web-based osteosarcoma resources using measures of readability, understandability, and actionability.

Methods

Using the search term “osteosarcoma,” two independent Google searches were performed on March 7, 2020 (by AGS), and March 11, 2020 (by TRG). The top 50 results were collected. Websites were included if they were directed at providing patient education on osteosarcoma. Readability was quantified using validated algorithms: Flesh-Kincaid Grade Ease (FKGE), Flesch-Kincaid Grade-Level (FKGL). A higher FKGE score indicates that the material is easier to read. All other readability scores represent the US school grade level. Two independent PEMAT assessments were performed with independent scores assigned for both understandability and actionability. A PEMAT score of 70% or below is considered poorly understandable or poorly actionable. Statistical significance was defined as P≤.05.

Results

Two searches yielded 53 unique websites, of which 37 (70%) met the inclusion criteria. The mean FKGE and FKGL scores were 40.8 (SD 13.6) and 12.0 (SD 2.4), respectively. No website scored within the acceptable NIH or AHA recommended reading level. Only 4 (11%) and 1 (3%) website met the acceptable understandability and actionability threshold. Both understandability and actionability were positively correlated with FKGE (ρ=0.55, P<.001; ρ=0.60, P<.001), but were otherwise not significantly associated with other readability scores. There were no associations between readability (P=.15), understandability (P=.20), or actionability (P=.31) scores and Google rank.

Conclusions

Overall, web-based osteosarcoma patient educational materials scored poorly with respect to readability, understandability, and actionability. None of the web-based resources scored at the recommended reading level. Only 4 achieved the appropriate score to be considered understandable by the general public. Authors of patient resources should incorporate PEMAT and readability criteria to improve web-based resources to support patient understanding.

Consulting "Dr. YouTube": an objective evaluation of hypospadias videos on a popular video-sharing website

INTRODUCTION

Parents who make decisions about hypospadias repair for their child may seek information from online platforms such as YouTube.

OBJECTIVE

The purpose of this study is to evaluate the health literacy demand of hypospadias videos on YouTube using the Patient Education Materials Assessment Tool for Audiovisual Materials (PEMAT-A/V).

STUDY DESIGN

We performed a YouTube search using the term "hypospadias," limiting results to the first 100 videos. We excluded videos that were <1 min or >20 min and videos that were not in English or did not include subtitles. Two evaluators independently examined videos and determined PEMAT-A/V scores for understandability and actionability (i.e., ability to identify actions the viewer can take). Videos with scores >70% are understandable or actionable. The inter-rater reliability (kappa) and intraclass correlation coefficient (ICC) of PEMAT scores were calculated. Bivariate and multivariable linear regression models assessed the association of video characteristics with respective scores.

RESULTS

Of the 100 videos that were identified on YouTube, 47 (47%) were excluded leaving 53 for analysis: 14 were >20 min, 14 were <1 min, 9 had no audio or subtitles, 7 were not in English, 1 was a duplicate, 1 was unrelated to hypospadias, and 1 was deleted at the time of data analysis. Three (5.6%) were understandable (mean score 54.5%, standard deviation (SD) 14.9) and eight (15.1%) were actionable (mean score 21.8%, SD 16.6) (Extended Summary Figure). Kappa values ranged from 0.4 to 1. The ICC's were 0.55 and 0.33 for understandability and actionability, respectively. In the bivariate analysis, mean understandability scores were significantly higher for English language videos (p = 0.04), videos with animation (p = 0.002), and those produced by industry (p = 0.02). In the multivariable analysis, mean understandability scores were significantly higher for "expert testimonial" or "other" video types after adjusting for graphics type and overall tone (p = 0.04). Mean understandability scores were also significantly higher for videos with animation after adjusting for video type and overall tone (p = 0.01). Mean actionability scores were significantly higher for videos with a negative tone (p = 0.01).

DISCUSSION

The vast majority of hypospadias-related YouTube content is not appropriate for users with low health literacy although certain types of videos, such those with animation and expert testimonials, scored higher on understandability than other types.

CONCLUSION

Due to the lack of sufficient online informational content regarding hypospadias, we plan to engage parents of sons with hypospadias in the development of high-quality patient educational materials about hypospadias.

Implementation of an organisation-wide health literacy approach to improve the understandability and actionability of patient information and education materials: A pre-post effectiveness study

OBJECTIVE

Limited examples exist globally of coordinated, organisation-wide health literacy approaches to systematically improve the understandability and actionability of patient health information. Even fewer have been formally evaluated. The aim of this study was to use the Patient Education Materials Assessment Tool (PEMAT) to evaluate the effectiveness of an organisation-wide, evidence-based approach to improve the understandability and actionability of patient information materials in regional health service in New South Wales, Australia.

METHODS

Two independent raters (blinded to the document version) evaluated pre- and post-implementation versions of 50 randomly-selected patient information materials using the PEMAT, with differences in understandability and actionability analysed using paired samples tests.

RESULTS

Mean (±SD) overall scores for understandability increased significantly by 5% (95% CI 2-8; p = 0.002) up to 77%±10%, and mean actionability (±SD) increased significantly by 4% (95% CI 0-8; p = 0.046) up to 56%±22%.

CONCLUSION

These results demonstrate that organisation-wide approaches with standardised processes for staff to prepare, review and store written patient information and education materials can be successfully implemented to address the impacts and risks of low health literacy.

PRACTICE IMPLICATIONS

The success of this approach provides a framework for other health organisations to work in partnership with patients to make health information more understandable and actionable.

Exploring the Most Visible German Websites on Melanoma Immunotherapy: A Web-Based Analysis

Background

Patients diagnosed with melanoma frequently search the internet for treatment information, including novel and complex immunotherapy. However, health literacy is limited among half of the German population, and no assessment of websites on melanoma treatment has been performed so far.

Objective

The aim of this study was to identify and assess the most visible websites in German language on melanoma immunotherapy.

Methods

In accordance with the common Web-based information-seeking behavior of patients with cancer, the first 20 hits on Google, Yahoo, and Bing were searched for combinations of German synonyms for “melanoma” and “immunotherapy” in July 2017. Websites that met our predefined eligibility criteria were considered for assessment. Three reviewers independently assessed their quality by using the established DISCERN tool and by checking the presence of quality certification. Usability and reliability were evaluated by the LIDA tool and understandability by the Patient Education Materials Assessment Tool (PEMAT). The Flesch Reading Ease Score (FRES) was calculated to estimate the readability. The ALEXA and SISTRIX tools were used to investigate the websites’ popularity and visibility. The interrater agreement was determined by calculating Cronbach alpha. Subgroup differences were identified by t test, U test, or one-way analysis of variance.

Results

Of 480 hits, 45 single websites from 30 domains were assessed. Only 2 website domains displayed a German quality certification. The average assessment scores, mean (SD), were as follows: DISCERN, 48 (7.6); LIDA (usability), 40 (2.0); LIDA (reliability), 10 (1.6); PEMAT, 69% (16%); and FRES, 17 (14), indicating mediocre quality, good usability, and understandability but low reliability and an even very low readability of the included individual websites. SISTRIX scores ranged from 0 to 6872 and ALEXA scores ranged from 17 to 192,675, indicating heterogeneity of the visibility and popularity of German website domains providing information on melanoma immunotherapy.

Conclusions

Optimization of the most accessible German websites on melanoma immunotherapy is desirable. Especially, simplification of the readability of information and further adaption to reliability criteria are required to support the education of patients with melanoma and laypersons, and to enhance transparency.

Clinical Validity, Understandability, and Actionability of Online Cardiovascular Disease Risk Calculators: Systematic Review

BACKGROUND

Online health information is particularly important for cardiovascular disease (CVD) prevention, where lifestyle changes are recommended until risk becomes high enough to warrant pharmacological intervention. Online information is abundant, but the quality is often poor and many people do not have adequate health literacy to access, understand, and use it effectively.

OBJECTIVE

This project aimed to review and evaluate the suitability of online CVD risk calculators for use by low health literate consumers in terms of clinical validity, understandability, and actionability.

METHODS

This systematic review of public websites from August to November 2016 used evaluation of clinical validity based on a high-risk patient profile and assessment of understandability and actionability using Patient Education Material Evaluation Tool for Print Materials.

RESULTS

A total of 67 unique webpages and 73 unique CVD risk calculators were identified. The same high-risk patient profile produced widely variable CVD risk estimates, ranging from as little as 3% to as high as a 43% risk of a CVD event over the next 10 years. One-quarter (25%) of risk calculators did not specify what model these estimates were based on. The most common clinical model was Framingham (44%), and most calculators (77%) provided a 10-year CVD risk estimate. The calculators scored moderately on understandability (mean score 64%) and poorly on actionability (mean score 19%). The absolute percentage risk was stated in most (but not all) calculators (79%), and only 18% included graphical formats consistent with recommended risk communication guidelines.

CONCLUSIONS

There is a plethora of online CVD risk calculators available, but they are not readily understandable and their actionability is poor. Entering the same clinical information produces widely varying results with little explanation. Developers need to address actionability as well as clinical validity and understandability to improve usefulness to consumers with low health literacy.