Assessment of Online Sites Reliability, Accountability, Readability, Accessibility, and Translation for Intravitreal Injections

Evaluating the Quality of Cancer-Related WeChat Public Accounts: Cross-Sectional Study

BACKGROUND

WeChat (Tencent) is one of the most important information sources for Chinese people. Relevantly, various health-related data are constantly transmitted among WeChat users. WeChat public accounts (WPAs) for health are rapidly emerging. Health-related WeChat public accounts have a significant impact on public health. Because of the rise in web-based health-seeking behavior, the general public has grown accustomed to obtaining cancer information from WPAs. Although WPAs make it easy for people to obtain health information, the quality of the information is questionable.

OBJECTIVE

This study aims to assess the quality and suitability of cancer-related WeChat public accounts (CWPAs).

METHODS

The survey was conducted from February 1 to 28, 2023. Based on the WPA monthly list provided by Qingbo Big Data, 28 CWPAs in the WeChat communication index were selected as the survey sample. Quality assessment of the included CWPAs was performed using the HONcode instrument. Furthermore, suitability was measured by using the Suitability Assessment of Materials. A total of 2 researchers conducted the evaluations independently.

RESULTS

Of the 28 CWPAs, 12 (43%) were academic and 16 (57%) were commercial. No statistical difference was found regarding the HONcode scores between the 2 groups (P=.96). The quality of the academic and commercial CWPAs evaluated using the HONcode instrument demonstrated mean scores of 5.58 (SD 2.02) and 5.63 (SD 2.16), respectively, corresponding to a moderate class. All CWPAs' compliance with the HONcode principles was unsatisfactory. A statistically significant difference between the 2 groups was observed in the Suitability Assessment of Materials scores (P=.04). The commercial WPAs reached an overall 55.1% (SD 5.5%) score versus the 50.2% (SD 6.4%) score reached by academic WPAs. The suitability of academic and commercial CWPAs was considered adequate.

CONCLUSIONS

This study revealed that CWPAs are not sufficiently credible. WPA owners must endeavor to create reliable health websites using approved tools such as the HONcode criteria. However, it is necessary to educate the public about the evaluation tools of health websites to assess their credibility before using the provided content. In addition, improving readability will allow the public to read and understand the content.

Automated Credibility Assessment of Web-Based Health Information Considering Health on the Net Foundation Code of Conduct (HONcode): Model Development and Validation Study

BACKGROUND

An increasing number of users are turning to web-based sources as an important source of health care guidance information. Thus, trustworthy sources of information should be automatically identifiable using objective criteria.

OBJECTIVE

The purpose of this study was to automate the assessment of the Health on the Net Foundation Code of Conduct (HONcode) criteria, enhancing our ability to pinpoint trustworthy health information sources.

METHODS

A data set of 538 web pages displaying health content was collected from 43 health-related websites. HONcode criteria have been considered as web page and website levels. For the website-level criteria (confidentiality, transparency, financial disclosure, and advertising policy), a bag of keywords has been identified to assess the criteria using a rule-based model. For the web page-level criteria (authority, complementarity, justifiability, and attribution) several machine learning (ML) approaches were used. In total, 200 web pages were manually annotated until achieving a balanced representation in terms of frequency. In total, 3 ML models-random forest, support vector machines (SVM), and Bidirectional Encoder Representations from Transformers (BERT)-were trained on the initial annotated data. A second step of training was implemented for the complementarity criterion using the BERT model for multiclass classification of the complementarity sentences obtained by annotation and data augmentation (positive, negative, and noncommittal sentences). Finally, the remaining web pages were classified using the selected model and 100 sentences were randomly selected for manual review.

RESULTS

For web page-level criteria, the random forest model showed a good performance for the attribution criterion while displaying subpar performance in the others. BERT and SVM had a stable performance across all the criteria. BERT had a better area under the curve (AUC) of 0.96, 0.98, and 1.00 for neutral sentences, justifiability, and attribution, respectively. SVM had the overall better performance for the classification of complementarity with the AUC equal to 0.98. Finally, SVM and BERT had an equal AUC of 0.98 for the authority criterion. For the website level criteria, the rule-based model was able to retrieve web pages with an accuracy of 0.97 for confidentiality, 0.82 for transparency, and 0.51 for both financial disclosure and advertising policy. The final evaluation of the sentences determined 0.88 of precision and the agreement level of reviewers was computed at 0.82.

CONCLUSIONS

Our results showed the potential power of automating the HONcode criteria assessment using ML approaches. This approach could be used with different types of pretrained models to accelerate the text annotation, and classification and to improve the performance in low-resource cases. Further work needs to be conducted to determine how to assign different weights to the criteria, as well as to identify additional characteristics that should be considered for consolidating these criteria into a comprehensive reliability score.

A Comprehensive Evaluation of the Quality, Readability, and Technical Quality of Online Information on Glaucoma

PURPOSE

To evaluate the quality and reliability of medical information, the technical quality of the presentation of information, and the readability of informational websites that publish content on the definition, causes, symptoms, and treatment of glaucoma.

DESIGN

A cross-sectional study was conducted to assess information published on websites with regard to glaucoma.

SUBJECTS

The top 150 websites populated on a Google search using the keywords glaucoma, high intraocular pressure, and high eye pressure were chosen for evaluation.

METHODS

Two independent reviewers assessed quality and reliability of each website using the DISCERN, Health on the Net Code (HONcode), and Journal of the American Medical Association (JAMA) criteria. The reviewers also evaluated technical quality by determining each website's ability to satisfy 10 unique features. Readability was assessed using the Readability Studio software (Oleander Software).

MAIN OUTCOME MEASURES

Quality of information was analyzed using the DISCERN, HONcode, and JAMA criteria. To assess readability, the Bormuth Cloze Mean, Bormuth Grade Placement, Flesch Kincaid Reading Ease, Coleman-Liau Index, Gunning Fog Score, Simple Measure of Gobbledygook Index, Readability Score, Fry Estimate, Raygor Estimate, and the Overall Mean Readability metrics were used. A separate subanalysis categorized websites into institutional and private categories.

RESULTS

Readability was poor among all websites, with most websites requiring a reading level higher than the 11th grade. The overall mean DISCERN score ± standard deviation (SD) was 3.0 ± 0.4, the mean HONcode score ± SD was 9.6 ± 1.8, and the mean JAMA score ± SD was 2.1 ± 1.1. The reviewers had moderate to excellent interrater reliability. Institutional websites (n = 39) had a higher mean DISCERN score (3.18 ± 0.33 vs. 2.95 ± 0.39, P < 0.05) and mean HONcode score (10.18 ± 1.90 vs. 9.34 ± 1.71, P < 0.05) than those of private websites (n = 111). Technical quality was higher among institutional websites (P < 0.05).

CONCLUSIONS

An overwhelming majority of websites presented information of low quality, reliability, and readability. Institutional websites generally received higher scores than those received by private websites; however, overall scores were still substandard, which necessitates improvement of online information on glaucoma.

Quality, Readability, and Accessibility of Online Content From a Google Search of “Macular Degeneration”: Critical Analysis

Purpose:

This work aims to assess the quality, accountability, readability, accessibility, and presence of Spanish translation in online material through a Google search of “macular degeneration”.

Methods:

In this retrospective cross-sectional analysis of website results from a Google search of “macular degeneration”, the quality and accountability for each website were assessed using the DISCERN criteria and the Health on the Net Foundation Code of Conduct (HONcode) principles. All 31 sites were independently graded by 2 ophthalmologists. Readability was evaluated using an online tool. The presence of accessibility features on the website and Spanish translation was recorded. The primary outcome measure was the DISCERN and HONcode quality and accountability scores of each website. Secondary outcome measures included the readability, accessibility, and presence of Spanish translation.

Results:

The mean ± SD of each criterion across all 15 DISCERN questions was 2.761 ± 0.666 (out of 5). The mean HONcode score for all websites was 7.355 ± 3.123. The mean consensus reading grade level was 10.258 ± 2.49. There were no statistically significant differences in any score between the top 5 websites and the bottom 26 websites evaluated. Accessibility was available on 10 of 31 websites. Spanish translation was available on 10 of 31 websites.

Conclusions:

The top 5 websites that appeared on a Google search did not have better quality or readability of online content. Improving quality, accountability, and readability can help improve patients’ health literacy regarding macular degeneration.

Assessment of the readability and quality of online patient education materials for the medical treatment of open-angle glaucoma

Objective

Patient adherence to glaucoma medications is poor, and is linked to low literacy levels. Patients commonly use the internet to access health information, and it is recommended that patient information is written at an 11-year-old reading level. The aim of this study is to assess the readability and quality of online patient education materials for the medical management of open angle glaucoma.

Methods and analysis

The top 10 relevant Google searches for nine glaucoma medications (timolol, brimonidine, apraclonidine, dorzolamide, latanoprost, bimatoprost, travoprost, tafluprost and brinzolamide) and three generic searches were analysed for readability and accountability. Readability was assessed using Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFI) and Simple Measure of Gobbledygook Index (SMOG). Webpages were classified by source and assessed using Journal of the American Medical Association (JAMA) benchmarks of accountability.

Results

111 articles were included in the analysis. Mean readability scores were: FRES 55.5 (95% CI 53.4 to 57.5); FKGL 9.7 (95% CI 9.3 to 10.0); GFI 12 (95% CI 11.6 to 12.4) and SMOG 9.3 (95% CI 8.9 to 9.6). One-way analysis of variance demonstrated no significant difference in readability score between source type. 9% of the webpages satisfied all 4 JAMA benchmarks. Pearson’s correlation coefficient showed a correlation between the FRES and accountability score (r=0.19, p=0.045).

Conclusion

The majority of online patient education materials for the medical treatment of glaucoma are written at a level too difficult for the general population and fail to meet accountability standards.

[Adherence to anti-VEGF treatment-Considerations and practical recommendations]

BACKGROUND

Numerous studies have identified a lack of treatment adherence as an important factor that often conflicts with the necessary number of anti-VEGF injections and therefore a better functional result.

OBJECTIVE

This article discusses approaches and possible measures to reduce the risk of late and infrequent intravitreal injections leading to the major issue of undertreatment.

MATERIAL AND METHOD

In the course of an expert dialogue, relevant parameters of treatment adherence and variables were identified. Meaningful processes were structured and assigned to organizational areas.

RESULTS

The compilation of meaningful measures enables practitioners to optimize their own implementation in different areas. Regular monitoring measures can identify the extent of treatment interruption and discontinuation. For specific indicators (treatments per time interval, longest pause interval, minimum coverage per unit time, delay) an effect on the development of visual function was demonstrated. Organizational measures, training of teams and referring physicians, redundant and iterative information transfer to patients have been proven in the experience of experts. The firm integration of these processes into the structures is facilitated by working with checklists.

CONCLUSION

An optimization of the processes is possible to improve the adherence and the functional results; however, interventional studies showing how adherence and persistence can be increased in the German treatment setting are still lacking.

Questionnaire for the assessment of adherence barriers of intravitreal therapy: the ABQ-IVT

Objective

To develop and validate a questionnaire for the investigation of non-adherence (NA) barriers in patients receiving intravitreal injection (IVT).

Design

Questionnaire development and cross-sectional patient survey combined with a retrospective medical chart review.

Participants

German patients with neovascular age-related macular degeneration (nAMD) or diabetic macular edema (DME) receiving anti-vascular endothelial growth factor (anti-VEGF) treatment via IVT.

Methods

The previously validated (indications: atrial fibrillation, human immunodeficiency virus, chronic inflammatory lung disease) Adherence Barriers Questionnaire (ABQ) was revised according to specifications of IVT, within the framework of an expert panel. The ABQ-IVT, which initially consisted of 24 items formulated as statements (4-point-Likert-scale ranging from “strongly agree” to “strongly disagree”), was applied in a cross-sectional survey. Evaluation of the questionnaire included an assessment of internal consistency and factor analysis. The occurrence of potential barriers in the patient sample was evaluated using descriptive statistics. To identify patient subpopulations, hierarchical cluster analysis was performed using ABQ-IVT answers as predictors. Due to difficulties in capturing NA as an external criterion, the evaluation of the questionnaire was limited to its internal validity and reliability.

Main outcome measures

Patients’ answers to the ABQ-IVT questionnaire and interviews.

Results

Of 253 patients, 234 (92%) were able to complete the ABQ-IVT questionnaire. Within the reliability analysis, the ABQ-IVT was reduced to 17 items. The condensed questionnaire demonstrated good internal consistency (Cronbach’s alpha = 0.78), and factor analysis showed no evidence for subscales of the questionnaire. Nearly half of the patients (49%) reported being affected by at least three different barriers. On average, a patient was affected by 3.1 barriers. The most frequently reported barriers were “Challenge due to time commitment of physician visits” (45% of the patients), “Depression” (29%) and “Travel and opportunity costs” (27%). Cluster analysis identified six patient subpopulations, each affected by different sets of barriers and differed regarding their patient characteristics.

Conclusions

The ABQ-IVT is a practical and reliable instrument for identifying patient-specific barriers to IVT treatment adherence. In practice, the questionnaire may be useful in assessing whether individual patients are at higher risk of NA due to specific adherence barriers. Aside from better awareness, this allows earlier interventions, though these still need to be validated. Patient subpopulations face different barriers and may, therefore, need distinct preventative care.

Supplementary Information

The online version contains supplementary material available at 10.1186/s40942-021-00311-x.