Web-Based Health Information Following the Renewal of the Cervical Screening Program in Australia: Evaluation of Readability, Understandability, and Credibility

Using Large Language Models to Generate Educational Materials on Childhood Glaucoma

PURPOSE

To evaluate the quality, readability, and accuracy of large language model (LLM)-generated patient education materials (PEMs) on childhood glaucoma, and their ability to improve existing the readability of online information.

DESIGN

Cross-sectional comparative study.

METHODS

We evaluated responses of ChatGPT-3.5, ChatGPT-4, and Bard to 3 separate prompts requesting that they write PEMs on "childhood glaucoma." Prompt A required PEMs be "easily understandable by the average American." Prompt B required that PEMs be written "at a 6th-grade level using Simple Measure of Gobbledygook (SMOG) readability formula." We then compared responses' quality (DISCERN questionnaire, Patient Education Materials Assessment Tool [PEMAT]), readability (SMOG, Flesch-Kincaid Grade Level [FKGL]), and accuracy (Likert Misinformation scale). To assess the improvement of readability for existing online information, Prompt C requested that LLM rewrite 20 resources from a Google search of keyword "childhood glaucoma" to the American Medical Association-recommended "6th-grade level." Rewrites were compared on key metrics such as readability, complex words (≥3 syllables), and sentence count.

RESULTS

All 3 LLMs generated PEMs that were of high quality, understandability, and accuracy (DISCERN ≥4, ≥70% PEMAT understandability, Misinformation score = 1). Prompt B responses were more readable than Prompt A responses for all 3 LLM (P ≤ .001). ChatGPT-4 generated the most readable PEMs compared to ChatGPT-3.5 and Bard (P ≤ .001). Although Prompt C responses showed consistent reduction of mean SMOG and FKGL scores, only ChatGPT-4 achieved the specified 6th-grade reading level (4.8 ± 0.8 and 3.7 ± 1.9, respectively).

CONCLUSIONS

LLMs can serve as strong supplemental tools in generating high-quality, accurate, and novel PEMs, and improving the readability of existing PEMs on childhood glaucoma.

Evaluation of online text-based information resources of gynaecological cancer symptoms

BACKGROUND

Gynaecological cancer symptoms are often vague and non-specific. Quality health information is central to timely cancer diagnosis and treatment. The aim of this study was to identify and evaluate the quality of online text-based patient information resources regarding gynaecological cancer symptoms.

METHODS

A targeted website search and Google search were conducted to identify health information resources published by the Australian government and non-government health organisations. Resources were classified by topic (gynaecological health, gynaecological cancers, cancer, general health); assessed for reading level (Simple Measure of Gobbledygook, SMOG) and difficulty (Flesch Reading Ease, FRE); understandability and actionability (Patient Education Materials Assessment Tool, PEMAT, 0-100), whereby higher scores indicate better understandability/actionability. Seven criteria were used to assess cultural inclusivity specific for Aboriginal and Torres Strait Islander people; resources which met 3-5 items were deemed to be moderately inclusive and 6+ items as inclusive.

RESULTS

A total of 109 resources were identified and 76% provided information on symptoms in the context of gynaecological cancers. The average readability was equivalent to a grade 10 reading level on the SMOG and classified as 'difficult to read' on the FRE. The mean PEMAT scores were 95% (range 58-100) for understandability and 13% (range 0-80) for actionability. Five resources were evaluated as being moderately culturally inclusive. No resource met all the benchmarks.

CONCLUSIONS

This study highlights the inadequate quality of online resources available on pre-diagnosis gynaecological cancer symptom information. Resources should be revised in line with the recommended standards for readability, understandability and actionability and to meet the needs of a culturally diverse population.

Investigation of the quality and health literacy demand of online information on pelvic floor exercises to reduce stress urinary incontinence

BACKGROUND

Many women are embarrassed to discuss pelvic floor exercises with health professionals during pregnancy; instead they search the internet for information on pelvic floor exercises.

AIMS

This project investigated the quality and health literacy demand in terms of readability, understandability and actionability of information available on the internet relating to pelvic floor exercises.

MATERIALS AND METHODS

An analysis was conducted on 150 websites and 43 videos obtained from three consecutive Google searches relating to pelvic floor exercises. Websites were assessed for Health on the Net certification. Readability was assessed using a readability calculator, and understandability/actionability of website and video information was assessed using the Patient Education Materials Assessment Tool (PEMAT) for Print or Audiovisual Materials.

RESULTS

Less than one-third (24%) of the websites were certified for quality. The median readability score for the websites was grade 9 and the websites and videos both rated highly for understandability and actionability. The median understandability score for the website information was 87%, and the median actionability score was 71%. The median understandability score of the video information was 95.5% and the median actionability score was 100%. The understandability and actionability of video information was also significantly greater than that for website information (P < 0.01).

CONCLUSION

Clinicians could consider directing patients to video-based resources when advising online pelvic floor exercise resources for women during pregnancy, especially those with lower health literacy.

The Readability and Quality of Web-Based Patient Information on Nasopharyngeal Carcinoma: Quantitative Content Analysis

BACKGROUND

Nasopharyngeal carcinoma (NPC) is a rare disease that is strongly associated with exposure to the Epstein-Barr virus and is characterized by the formation of malignant cells in nasopharynx tissues. Early diagnosis of NPC is often difficult owing to the location of initial tumor sites and the nonspecificity of initial symptoms, resulting in a higher frequency of advanced-stage diagnoses and a poorer prognosis. Access to high-quality, readable information could improve the early detection of the disease and provide support to patients during disease management.

OBJECTIVE

This study aims to assess the quality and readability of publicly available web-based information in the English language about NPC, using the most popular search engines.

METHODS

Key terms relevant to NPC were searched across 3 of the most popular internet search engines: Google, Yahoo, and Bing. The top 25 results from each search engine were included in the analysis. Websites that contained text written in languages other than English, required paywall access, targeted medical professionals, or included nontext content were excluded. Readability for each website was assessed using the Flesch Reading Ease score and the Flesch-Kincaid grade level. Website quality was assessed using the Journal of the American Medical Association (JAMA) and DISCERN tools as well as the presence of a Health on the Net Foundation seal.

RESULTS

Overall, 57 suitable websites were included in this study; 26% (15/57) of the websites were academic. The mean JAMA and DISCERN scores of all websites were 2.80 (IQR 3) and 57.60 (IQR 19), respectively, with a median of 3 (IQR 2-4) and 61 (IQR 49-68), respectively. Health care industry websites (n=3) had the highest mean JAMA score of 4 (SD 0). Academic websites (15/57, 26%) had the highest mean DISCERN score of 77.5. The Health on the Net Foundation seal was present on only 1 website, which also achieved a JAMA score of 3 and a DISCERN score of 50. Significant differences were observed between the JAMA score of hospital websites and the scores of industry websites (P=.04), news service websites (P<.048), charity and nongovernmental organization websites (P=.03). Despite being a vital source for patients, general practitioner websites were found to have significantly lower JAMA scores compared with charity websites (P=.05). The overall mean readability scores reflected an average reading age of 14.3 (SD 1.1) years.

CONCLUSIONS

The results of this study suggest an inconsistent and suboptimal quality of information related to NPC on the internet. On average, websites presented readability challenges, as written information about NPC was above the recommended reading level of sixth grade. As such, web-based information requires improvement in both quality and accessibility, and healthcare providers should be selective about information recommended to patients, ensuring they are reliable and readable.

Gynaecological cancer resources for Aboriginal and Torres Strait Islander women: A resource audit

ISSUE ADDRESSED

Aboriginal and Torres Strait Islander women experience considerable disparities in gynaecological cancer outcomes. Accessible and culturally appropriate health promotion resources about gynaecological cancers may support health literacy in this area. This study aimed to determine the understandability, actionability, readability, and cultural relevance of gynaecological cancer health literacy resources for Aboriginal and Torres Strait Islander consumers, families, and caregivers.

METHODS

We conducted a general Google search and targeted searches of Australian gynaecological cancer organisation websites in January and February 2022. Resources were assessed for understandability, actionability and cultural relevance to Aboriginal and Torres Strait Islander audiences.

RESULTS

We found 16 resources. The resources were generally understandable, actionable, readable, and culturally relevant, however, most resources were focused on cervical cancer prevention through vaccination and screening. Few resources focused other gynaecological cancer types or aspects of the cancer care continuum. While many resources contained elements that made them culturally relevant, areas for improvement were identified. These included: greater transparency relating to the Aboriginal and Torres Strait Islander leadership, governance, and involvement in the development of the resources as well as availability of different resource formats with an emphasis on visual aids.

CONCLUSIONS

This study highlighted a need for the development of resources relating to a wider range of gynaecological cancer types and different stages of the cancer care continuum for Aboriginal and Torres Strait Islander women. SO WHAT?: The development of a broader range of culturally appropriate gynaecological cancer health literacy resources, ideally developed through co-design with Aboriginal and Torres Strait Islander peoples, may contribute to addressing the disparities in gynaecological cancer outcomes for Aboriginal and Torres Strait Islander women.

Online information on chronic pain in 3 countries: an assessment of readability, credibility, and accuracy

Objectives

To assess the readability, credibility, and accuracy of online information on chronic pain in Australia, Mexico, and Nepal.

Methods

We assessed Google-based websites and government health websites about chronic pain for readability (using the Flesch Kincaid Readability Ease tool), credibility (using the Journal of American Medical Association [JAMA] benchmark criteria and Health on the Net Code [HONcode]), and accuracy (using 3 core concepts of pain science education: (1) pain does not mean my body is damaged; (2) thoughts, emotions, and experiences affect pain; and (3) I can retrain my overactive pain system).

Results

We assessed 71 Google-based websites and 15 government websites. There were no significant between-country differences in chronic pain information retrieved through Google for readability, credibility, or accuracy. Based on readability scores, the websites were "fairly difficult to read," suitable for ages 15 to 17 years or grades 10 to 12 years. For credibility, less than 30% of all websites met the full JAMA criteria, and more than 60% were not HONcode certified. For accuracy, all 3 core concepts were present in less than 30% of websites. Moreover, we found that the Australian government websites have low readability but are credible, and the majority provided all 3 core concepts in pain science education. A single Mexican government website had low readability without any core concepts but was credible.

Conclusion

The readability, credibility, and accuracy of online information on chronic pain should be improved internationally to support facilitating better management of chronic pain.

Promoting patient engagement in cancer genomics research programs: An environmental scan

Background: A national priority in the United States is to promote patient engagement in cancer genomics research, especially among diverse and understudied populations. Several cancer genomics research programs have emerged to accomplish this priority, yet questions remain about the meaning and methods of patient engagement. This study explored how cancer genomics research programs define engagement and what strategies they use to engage patients across stages in the conduct of research. Methods: An environmental scan was conducted of cancer genomics research programs focused on patient engagement. Research programs were identified and characterized using materials identified from publicly available sources (e.g., websites), a targeted literature review, and interviews with key informants. Descriptive information about the programs and their definitions of engagement, were synthesized using thematic analysis. The engagement strategies were synthesized and mapped to different stages in the conduct of research, including recruitment, consent, data collection, sharing results, and retention. Results: Ten research programs were identified, examples of which include the Cancer Moonshot Biobank, the MyPART Network, NCI-CONNECT, and the Participant Engagement and Cancer Genome Sequencing (PE-CGS) Network. All programs aimed to include understudied or underrepresented populations. Based on publicly available information, four programs explicitly defined engagement. These definitions similarly characterized engagement as being interpersonal, reciprocal, and continuous. Five general strategies of engagement were identified across the programs: 1) digital (such as websites) and 2) non-digital communications (such as radio broadcasts, or printed brochures); 3) partnering with community organizations; 4) providing incentives; and 5) affiliating with non-academic medical centers. Digital communications were the only strategy used across all stages of the conduct of research. Programs tailored these strategies to their study goals, including overcoming barriers to research participation among diverse populations. Conclusion: Programs studying cancer genomics are deeply committed to increasing research participation among diverse populations through patient engagement. Yet, the field needs to reach a consensus on the meaning of patient engagement, develop a taxonomy of patient engagement measures in cancer genomics research, and identify optimal strategies to engage patients in cancer genomics. Addressing these needs could enable patient engagement to fulfill its potential and accelerate the pace of cancer genomic discoveries.

Patient education information material assessment criteria: A scoping review

BACKGROUND

Patient education information material (PEIM) is an essential component of patient education programs in increasing patients' ability to cope with their diseases. Therefore, it is essential to consider the criteria that will be used to prepare and evaluate these resources.

OBJECTIVE

This paper aims to identify these criteria and recognize the tools or methods used to evaluate them.

METHODS

National and international databases and indexing banks, including PubMed, Scopus, Web of Science, ProQuest, the Cochrane Library, Magiran, SID and ISC, were searched for this review. Original or review articles, theses, short surveys, and conference papers published between January 1990 and June 2022 were included.

RESULTS

Overall, 4688 documents were retrieved, of which 298 documents met the inclusion criteria. The criteria were grouped into 24 overarching criteria. The most frequently used criteria were readability, quality, suitability, comprehensibility and understandability.

CONCLUSION

This review has provided empirical evidence to identify criteria, tools, techniques or methods for developing or evaluating a PEIM. The authors suggest that developing a comprehensive tool based on these findings is critical for evaluating the overall efficiency of PEIM using effective criteria.

An infodemiology study on exploring the quality and reliability of colorectal cancer immunotherapy information

Background

Immunotherapy is a new treatment modality which promises hope for advanced colorectal cancer patients. To our knowledge, no previous studies have evaluated the quality of patient information available on this topic online.

Objective

This study will explore the quality and reliability of colorectal cancer immunotherapy information using the Journal of the American Medical Association (JAMA) and DISCERN tools.

Methods

Design thinking methodology was integrated with systematic scoping reviews framework to inform our descriptive observational media analysis study. Google Chrome was used to run four searches using prespecified search terms selected according to the top patient concerns about immunotherapy. The first 20 relevant results were identified (n = 80) and then duplicates were removed. Descriptive narrative univariate and bivariate analysis was done for the relevant variables.

Results

The total of included websites was 17. Most websites score <3 points on JAMA and fair/poor on DISCERN. Most of the websites that score ≥3 points on JAMA and excellent/good on DISCERN have a charity affiliation. A total of 58.8% of the websites present the date, 41.2% demonstrate authorship, and sources are mentioned in 29.4% of the websites. Lack of content was noticed in providing the prognosis of patients if no treatment is given, clear aim and the effect of treatments on patient's quality of life.

Conclusion

Assessing the reliability of information about cancer treatments online remains a challenge. Further research is required to understand the patient perceptions and use of online information and whether it has an impact on their behavioural health outcomes.

Designing, Developing, Evaluating, and Implementing a Smartphone-Delivered, Rule-Based Conversational Agent (DISCOVER): Development of a Conceptual Framework

BACKGROUND

Conversational agents (CAs), also known as chatbots, are computer programs that simulate human conversations by using predetermined rule-based responses or artificial intelligence algorithms. They are increasingly used in health care, particularly via smartphones. There is, at present, no conceptual framework guiding the development of smartphone-based, rule-based CAs in health care. To fill this gap, we propose structured and tailored guidance for their design, development, evaluation, and implementation.

OBJECTIVE

The aim of this study was to develop a conceptual framework for the design, evaluation, and implementation of smartphone-delivered, rule-based, goal-oriented, and text-based CAs for health care.

METHODS

We followed the approach by Jabareen, which was based on the grounded theory method, to develop this conceptual framework. We performed 2 literature reviews focusing on health care CAs and conceptual frameworks for the development of mobile health interventions. We identified, named, categorized, integrated, and synthesized the information retrieved from the literature reviews to develop the conceptual framework. We then applied this framework by developing a CA and testing it in a feasibility study.

RESULTS

The Designing, Developing, Evaluating, and Implementing a Smartphone-Delivered, Rule-Based Conversational Agent (DISCOVER) conceptual framework includes 8 iterative steps grouped into 3 stages, as follows: design, comprising defining the goal, creating an identity, assembling the team, and selecting the delivery interface; development, including developing the content and building the conversation flow; and the evaluation and implementation of the CA. They were complemented by 2 cross-cutting considerations-user-centered design and privacy and security-that were relevant at all stages. This conceptual framework was successfully applied in the development of a CA to support lifestyle changes and prevent type 2 diabetes.

CONCLUSIONS

Drawing on published evidence, the DISCOVER conceptual framework provides a step-by-step guide for developing rule-based, smartphone-delivered CAs. Further evaluation of this framework in diverse health care areas and settings and for a variety of users is needed to demonstrate its validity. Future research should aim to explore the use of CAs to deliver health care interventions, including behavior change and potential privacy and safety concerns.

An evaluation of the quality of online perinatal depression information

BACKGROUND

During the perinatal period (including pregnancy and up to 12 months after childbirth), expectant and new mothers are at an elevated risk of developing depression. Inadequate knowledge about perinatal depression and treatment options may contribute to the low help-seeking rates exhibited by perinatal people. The Internet can be an accessible source of information about perinatal depression; however, the quality of this information remains to be evaluated. The purpose of this study was to assess the quality of perinatal depression information websites.

METHODS

After review, 37 websites were included in our sample. To assess overall website quality, we rated websites based on their reading level (Simple Measure of Gobbledegook; SMOG), information quality (DISCERN), usability (Patient Education Materials Assessment Tool; PEMAT), and visual design (Visual Aesthetics of Website Inventory; VisAWI).

RESULTS

Websites often exceeded the National Institute of Health's recommended reading level of grades 6-8, with scores ranging from 6.8 to 13.5. Website information quality ratings ranged from 1.8 to 4.3 out of 5, with websites often containing insufficient information about treatment choices. Website usability ratings were negatively impacted by the lack of information summaries, visual aids, and tangible tools. Visual design ratings ranged from 3.2 to 6.6 out of 7, with a need for more creative design elements to enhance user engagement.

CONCLUSIONS

This study outlines the characteristics of high-quality perinatal depression information websites. Our findings illustrate that perinatal depression websites are not meeting the needs of users in terms of reading level, information quality, usability, and visual design. Our results may be helpful in guiding healthcare providers to reliable, evidence-based online resources for their perinatal patients.

Automatic Diagnosis of Mental Healthcare Information Actionability: Developing Binary Classifiers

We aimed to develop a quantitative instrument to assist with the automatic evaluation of the actionability of mental healthcare information. We collected and classified two large sets of mental health information from certified mental health websites: generic and patient-specific mental healthcare information. We compared the performance of the optimised classifier with popular readability tools and non-optimised classifiers in predicting mental health information of high actionability for people with mental disorders. sensitivity of the classifier using both semantic and structural features as variables achieved statistically higher than that of the binary classifier using either semantic (p < 0.001) or structural features (p = 0.0010). The specificity of the optimized classifier was statistically higher than that of the classifier using structural variables (p = 0.002) and the classifier using semantic variables (p = 0.001). Differences in specificity between the full-variable classifier and the optimised classifier were statistically insignificant (p = 0.687). These findings suggest the optimised classifier using as few as 19 semantic-structural variables was the best-performing classifier. By combining insights of linguistics and statistical analyses, we effectively increased the interpretability and the diagnostic utility of the binary classifiers to guide the development, evaluation of the actionability and usability of mental healthcare information.

Assessing Communicative Effectiveness of Public Health Information in Chinese: Developing Automatic Decision Aids for International Health Professionals

Effective multilingual communication of authoritative health information plays an important role in helping to reduce health disparities and inequalities in developed and developing countries. Health information communication from the World Health Organization is governed by key principles including health information relevance, credibility, understandability, actionability, accessibility. Multilingual health information developed under these principles provide valuable benchmarks to assess the quality of health resources developed by local health authorities. In this paper, we developed machine learning classifiers for health professionals with or without Chinese proficiency to assess public-oriented health information in Chinese based on the definition of effective health communication by the WHO. We compared our optimized classifier (SVM_F5) with the state-of-art Chinese readability classifier (Chinese Readability Index Explorer CRIE 3.0), and classifiers adapted from established English readability formula, Gunning Fog Index, Automated Readability Index. Our optimized classifier achieved statistically significant higher area under the receiver operator curve (AUC of ROC), accuracy, sensitivity, and specificity than those of SVM using CRIE 3.0 features and SVM using linguistic features of Gunning Fog Index and Automated Readability Index (ARI). The statistically improved performance of our optimized classifier compared to that of SVM classifiers adapted from popular readability formula suggests that evaluation of health communication effectiveness as defined by the principles of the WHO is more complex than information readability assessment. Our SVM classifier validated on health information covering diverse topics (environmental health, infectious diseases, pregnancy, maternity care, non-communicable diseases, tobacco control) can aid effectively in the automatic assessment of original, translated Chinese public health information of whether they satisfy or not the current international standard of effective health communication as set by the WHO.

Evaluation of a Digital Health Initiative in Illicit Substance Use: Cross-sectional Survey Study

Background

The Cracks in the Ice (CITI) community toolkit was developed to provide evidence-based, up-to-date information and resources about crystal methamphetamine to Australians. Given the high rates of internet use in the community and the potential for misinformation, CITI has the potential to play an important role in improving knowledge and challenging misconceptions surrounding crystal methamphetamine.

Objective

This study aims to determine (1) whether the CITI toolkit is achieving its aim of disseminating evidence-based information and resources to people who use crystal methamphetamine, their family and friends, health professionals, and the general community and (2) examine the association between the use of CITI and the knowledge and attitudes about crystal methamphetamine.

Methods

A cross-sectional web-based survey, open to Australian residents (aged ≥18 years), was conducted from November 2018 to March 2019. People who had previously visited the website (referred to as “website visitors” in this study) and those who had not (“naïve”) were recruited. At baseline, knowledge, attitudes, and demographics were assessed. CITI website visitors then completed a series of site evaluation questions, including the System Usability Scale (SUS), and naïve participants were asked to undertake a guided site tour of a replicated version of the site before completing the evaluation questions and repeating knowledge and attitude scales.

Results

Of a total 2108 participants, 564 (26.7%) reported lifetime use of crystal methamphetamine, 434 (20.6%) were family/friends, 288 (13.7%) were health professionals, and 822 (38.9%) were community members. The average SUS score was 73.49 (SD 13.30), indicating good site usability. Health professionals reported significantly higher SUS scores than community members (P=.02) and people who used crystal methamphetamine (P<.001). Website visitors had significantly higher baseline knowledge than naïve participants (P<.001). Among naïve participants, knowledge scores increased following exposure to the website (mean 15.2, SE 0.05) compared to baseline (mean 14.4, SE 0.05; P<.001). The largest shifts in knowledge were observed for items related to prevalence, legal issues, and the effects of the drug. Stigmatizing attitude scores among the naïve group were significantly lower following exposure to CITI (mean 41.97, SE 0.21) compared to baseline (mean 44.3, SE 0.21; P<.001).

Conclusions

This study provides an innovative evaluation of a national eHealth resource. CITI is achieving its aim of disseminating evidence-based, nonstigmatizing, and useful information and resources about crystal methamphetamine to key end user groups and has received good usability scores across its target groups. Interaction with CITI led to immediate improvements in knowledge about crystal methamphetamine and a decrease in stigmatizing attitudes. CITI demonstrates the important role of digital information and support platforms for translating evidence into practice and improving knowledge and reducing stigma.

Predicting Writing Styles of Web-Based Materials for Children's Health Education Using the Selection of Semantic Features: Machine Learning Approach

BACKGROUND

Medical writing styles can have an impact on the understandability of health educational resources. Amid current web-based health information research, there is a dearth of research-based evidence that demonstrates what constitutes the best practice of the development of web-based health resources on children's health promotion and education.

OBJECTIVE

Using authoritative and highly influential web-based children's health educational resources from the Nemours Foundation, the largest not-for-profit organization promoting children's health and well-being, we aimed to develop machine learning algorithms to discriminate and predict the writing styles of health educational resources on children versus adult health promotion using a variety of health educational resources aimed at the general public.

METHODS

The selection of natural language features as predicator variables of algorithms went through initial automatic feature selection using ridge classifier, support vector machine, extreme gradient boost tree, and recursive feature elimination followed by revision by education experts. We compared algorithms using the automatically selected (n=19) and linguistically enhanced (n=20) feature sets, using the initial feature set (n=115) as the baseline.

RESULTS

Using five-fold cross-validation, compared with the baseline (115 features), the Gaussian Naive Bayes model (20 features) achieved statistically higher mean sensitivity (P=.02; 95% CI -0.016 to 0.1929), mean specificity (P=.02; 95% CI -0.016 to 0.199), mean area under the receiver operating characteristic curve (P=.02; 95% CI -0.007 to 0.140), and mean macro F1 (P=.006; 95% CI 0.016-0.167). The statistically improved performance of the final model (20 features) is in contrast to the statistically insignificant changes between the original feature set (n=115) and the automatically selected features (n=19): mean sensitivity (P=.13; 95% CI -0.1699 to 0.0681), mean specificity (P=.10; 95% CI -0.1389 to 0.4017), mean area under the receiver operating characteristic curve (P=.008; 95% CI 0.0059-0.1126), and mean macro F1 (P=.98; 95% CI -0.0555 to 0.0548). This demonstrates the importance and effectiveness of combining automatic feature selection and expert-based linguistic revision to develop the most effective machine learning algorithms from high-dimensional data sets.

CONCLUSIONS

We developed new evaluation tools for the discrimination and prediction of writing styles of web-based health resources for children's health education and promotion among parents and caregivers of children. User-adaptive automatic assessment of web-based health content holds great promise for distant and remote health education among young readers. Our study leveraged the precision and adaptability of machine learning algorithms and insights from health linguistics to help advance this significant yet understudied area of research.

Getting the timing right: Women's views on the best time to announce changes to cancer screening policy recommendations

•

Women suggest communicating changes to screening programs 6–12 months before implementation.

•

Opportunities to be involved in consultation about changes are important.

•

Preference for information using evidence, with an option of more information.

•

Recognition that the mode of delivery should differ by age.

In December 2017, the Australian National Cervical Screening Program (NCSP) changed from 2-yearly cervical cytology to 5-yearly primary human papillomavirus (HPV) testing, starting at age 25 and with an exit test when aged 70–74. Women showed limited awareness of these changes prior to their implementation. We explored women’s preferences for how similar cancer screening changes could be communicated to the public in the future, including when, how, and using what methods. Six focus groups including 49 women aged 18–74 were conducted in November 2017. Focus groups were guided by information available on the NCSP website and information developed by the researchers. Generally, women suggested that communication of changes to cancer screening programs would ideally occur between 6 and 12 months ahead of their implementation and that they would like the opportunity to be involved in consultation about the changes. The NCSP website was described as answering basic questions, but also raising further questions for which there were no answers provided. Most groups preferred information which included evidence behind the changes and wanted an option of more information. Similar suggestions were made across all focus groups about how communications could be delivered, with recognition that the mode of delivery should differ by age. Women were still seeking information about the test itself and a symptom list, in order to be aware of these over the five-year period. These findings make an important and timely contribution which could help inform other countries considering making changes to their cancer screening programs in the future.