1
|
Liu D, Hu X, Xiao C, Bai J, Barandouzi ZA, Lee S, Webster C, Brock LU, Lee L, Bold D, Lin Y. Evaluation of Large Language Models in Tailoring Educational Content for Cancer Survivors and Their Caregivers: Quality Analysis. JMIR Cancer 2025; 11:e67914. [PMID: 40192716 PMCID: PMC11995809 DOI: 10.2196/67914] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2024] [Revised: 02/27/2025] [Accepted: 02/28/2025] [Indexed: 04/16/2025] Open
Abstract
Background Cancer survivors and their caregivers, particularly those from disadvantaged backgrounds with limited health literacy or racial and ethnic minorities facing language barriers, are at a disproportionately higher risk of experiencing symptom burdens from cancer and its treatments. Large language models (LLMs) offer a promising avenue for generating concise, linguistically appropriate, and accessible educational materials tailored to these populations. However, there is limited research evaluating how effectively LLMs perform in creating targeted content for individuals with diverse literacy and language needs. Objective This study aimed to evaluate the overall performance of LLMs in generating tailored educational content for cancer survivors and their caregivers with limited health literacy or language barriers, compare the performances of 3 Generative Pretrained Transformer (GPT) models (ie, GPT-3.5 Turbo, GPT-4, and GPT-4 Turbo; OpenAI), and examine how different prompting approaches influence the quality of the generated content. Methods We selected 30 topics from national guidelines on cancer care and education. GPT-3.5 Turbo, GPT-4, and GPT-4 Turbo were used to generate tailored content of up to 250 words at a 6th-grade reading level, with translations into Spanish and Chinese for each topic. Two distinct prompting approaches (textual and bulleted) were applied and evaluated. Nine oncology experts evaluated 360 generated responses based on predetermined criteria: word limit, reading level, and quality assessment (ie, clarity, accuracy, relevance, completeness, and comprehensibility). ANOVA (analysis of variance) or chi-square analyses were used to compare differences among the various GPT models and prompts. Results Overall, LLMs showed excellent performance in tailoring educational content, with 74.2% (267/360) adhering to the specified word limit and achieving an average quality assessment score of 8.933 out of 10. However, LLMs showed moderate performance in reading level, with 41.1% (148/360) of content failing to meet the sixth-grade reading level. LLMs demonstrated strong translation capabilities, achieving an accuracy of 96.7% (87/90) for Spanish and 81.1% (73/90) for Chinese translations. Common errors included imprecise scopes, inaccuracies in definitions, and content that lacked actionable recommendations. The more advanced GPT-4 family models showed better overall performance compared to GPT-3.5 Turbo. Prompting GPTs to produce bulleted-format content was likely to result in better educational content compared with textual-format content. Conclusions All 3 LLMs demonstrated high potential for delivering multilingual, concise, and low health literacy educational content for cancer survivors and caregivers who face limited literacy or language barriers. GPT-4 family models were notably more robust. While further refinement is required to ensure simpler reading levels and fully comprehensive information, these findings highlight LLMs as an emerging tool for bridging gaps in cancer education and advancing health equity. Future research should integrate expert feedback, additional prompt engineering strategies, and specialized training data to optimize content accuracy and accessibility.
Collapse
Affiliation(s)
- Darren Liu
- Nell Hodgson Woodruff School of Nursing, Emory University, 1520 Clifton Rd NE, Atlanta, GA, 30322, United States, 1 4042514072
- Center for Data Science, Emory University, Atlanta, GA, United States
| | - Xiao Hu
- Nell Hodgson Woodruff School of Nursing, Emory University, 1520 Clifton Rd NE, Atlanta, GA, 30322, United States, 1 4042514072
- Center for Data Science, Emory University, Atlanta, GA, United States
| | - Canhua Xiao
- Nell Hodgson Woodruff School of Nursing, Emory University, 1520 Clifton Rd NE, Atlanta, GA, 30322, United States, 1 4042514072
- Winship Cancer Institute, Emory University, Atlanta, GA, United States
| | - Jinbing Bai
- Nell Hodgson Woodruff School of Nursing, Emory University, 1520 Clifton Rd NE, Atlanta, GA, 30322, United States, 1 4042514072
- Winship Cancer Institute, Emory University, Atlanta, GA, United States
| | - Zahra A Barandouzi
- Nell Hodgson Woodruff School of Nursing, Emory University, 1520 Clifton Rd NE, Atlanta, GA, 30322, United States, 1 4042514072
- Winship Cancer Institute, Emory University, Atlanta, GA, United States
| | - Stephanie Lee
- Nell Hodgson Woodruff School of Nursing, Emory University, 1520 Clifton Rd NE, Atlanta, GA, 30322, United States, 1 4042514072
| | - Caitlin Webster
- Nell Hodgson Woodruff School of Nursing, Emory University, 1520 Clifton Rd NE, Atlanta, GA, 30322, United States, 1 4042514072
| | - La-Urshalar Brock
- Nell Hodgson Woodruff School of Nursing, Emory University, 1520 Clifton Rd NE, Atlanta, GA, 30322, United States, 1 4042514072
- Winship Cancer Institute, Emory University, Atlanta, GA, United States
| | - Lindsay Lee
- Department of Medicine, University of Florida, Gainesville, FL, United States
| | - Delgersuren Bold
- Nell Hodgson Woodruff School of Nursing, Emory University, 1520 Clifton Rd NE, Atlanta, GA, 30322, United States, 1 4042514072
- Center for Data Science, Emory University, Atlanta, GA, United States
| | - Yufen Lin
- Nell Hodgson Woodruff School of Nursing, Emory University, 1520 Clifton Rd NE, Atlanta, GA, 30322, United States, 1 4042514072
- Winship Cancer Institute, Emory University, Atlanta, GA, United States
| |
Collapse
|
2
|
Han W, Wang T, He Z, Wang C, Hui Z, Lei S, Hao N, Li N, Wang X. Global research trends on gastrointestinal cancer and mental health (2004-2024): a bibliographic study. Front Med (Lausanne) 2025; 12:1515853. [PMID: 39935799 PMCID: PMC11811116 DOI: 10.3389/fmed.2025.1515853] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/23/2024] [Accepted: 01/08/2025] [Indexed: 02/13/2025] Open
Abstract
Background Gastrointestinal (GI) cancers impose a significant burden on global public health. Patients often experience mental health challenges due to physical changes and treatment-related symptoms, which can worsen their condition or delay recovery. Although research is mounting in this field, visual bibliometric analysis has not yet been conducted. This study aims to reveal the research hotspots and frontiers in this field using bibliometrics to guide future research. Methods The publications on GI cancer and mental health were retrieved in the Web of Science Core Collection from 2004 to 2024. VOS Viewer and CiteSpace, as commonly used bibliometric analysis tools, were employed to visualize the network structure of bibliometric data and uncover the evolving trends in scientific research fields. VOS Viewer was used to identify keyword co-occurrences, while CiteSpace was utilized to generate network visualizations, produce dual-map overlays of journals, and perform burst keyword analysis. Results A total of 1,118 publications were included for analysis. China had the highest number of publications in this field (341, 30.5%), while the United States held a central position (centrality = 0.48). The most productive author and institution were Floortje Mols and Tilburg University, respectively. Keyword analysis highlighted that "quality of life" (QoL) is a prominent research topic in the field, while "complications," "cancer-related fatigue," (CRF) "chronic stress," and "epidemiology" have been identified as key areas for future research. Conclusion Research interest in this field continues to grow. The research direction is mainly focused on personalized mental health interventions to improve QoL, as well as preoperative mental healthcare and ongoing care through internet-based multidisciplinary collaboration to reduce postoperative complications. More detailed clinical symptom assessment is needed to distinguish between CRF and mental health issues and to provide targeted intervention measures in the future. The mechanism of mental health effects on the occurrence and development of GI cancer will be a frontier.
Collapse
Affiliation(s)
- Wenjin Han
- School of Nursing, Xi’an Jiaotong University Health Science Center, Xi’an, China
| | - Tianmeng Wang
- School of Nursing, Xi’an Jiaotong University Health Science Center, Xi’an, China
| | - Zhiqiang He
- School of Nursing, Xi’an Jiaotong University Health Science Center, Xi’an, China
| | - Caihua Wang
- Medical School, Xi’an Peihua University, Xi’an, China
| | - Zhaozhao Hui
- School of Public Health, Xi’an Jiaotong University Health Science Center, Xi’an, China
| | - Shuangyan Lei
- Department of Radiotherapy, Shaanxi Provincial Cancer Hospital, Xi’an, China
| | - Nan Hao
- The First Affiliated Hospital of Xi’an Jiaotong University, Xi’an, China
| | - Ning Li
- School of Nursing, Xi’an Jiaotong University Health Science Center, Xi’an, China
| | - Xiaoqin Wang
- School of Nursing, Xi’an Jiaotong University Health Science Center, Xi’an, China
- The First Affiliated Hospital of Xi’an Jiaotong University, Xi’an, China
| |
Collapse
|
3
|
Lerdal A, Gay C, Bonsaksen T, Ekeberg Ø, Grimholt T, Heir T, Kottorp A, Lee KA, Skogstad L, Schou-Bredal I. Validation of a short version of the Lee fatigue scale in adults living in Norway: a cross-sectional population survey. BMC Public Health 2023; 23:2132. [PMID: 37904144 PMCID: PMC10617107 DOI: 10.1186/s12889-023-17036-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2022] [Accepted: 10/20/2023] [Indexed: 11/01/2023] Open
Abstract
BACKGROUND Due to the nature of fatigue, a brief reliable measure of fatigue severity is needed. Thus, the aim of our study was to evaluate a short version of the Lee Fatigue Scale (LFS) in the Norwegian general population. METHODS This cross-sectional survey consists of a representative sample from the Norwegian population drawn by The National Population Register in Norway. The study is part of a larger study (NORPOP) aimed at collecting normative data from several questionnaires focused on health in adults living in Norway. Registered citizens between 18 and 94 years of age were randomly selected stratified by age, sex and geographic region. Of the 4971 respondents eligible for the study, 1792 (36%) responded to the survey. In addition to age and sex, we collected responses on a 5-item version of the LFS measuring current fatige severity. The psychometric properties focusing on internal structure and precision of the LFS items were analyzed by a Rasch rating scale model. RESULTS Complete LFS scores for analyses were available for 1767 adults. Women had higher LFS-scores than men, and adults < 55 years old had higher scores than older respondents. Our analysis of the LFS showed that the average category on each item advanced monotonically. Two of the five items demonstrated misfit, while the three other items demonstrated goodness-of-fit to the model and uni-dimensionality. Items #1 and #4 (tired and fatigue respectively) showed differential item functioning (DIF) by sex, but no items showed DIFs in relation to age. The separation index of the LFS 3-item scale showed that the sample could be separated into three different groups according to the respondents' fatigue levels. The LFS-3 raw scores correlated strongly with the Rasch measure from the three items. The core dimensions in these individual items were very similarly expressed in the Norwegian language version and this may be a threat to the cultural-related or language validity of a short version of the LFS using these particular items. CONCLUSIONS The study provides validation of a short LFS 3-item version for estimating fatigue in the general population.
Collapse
Affiliation(s)
- Anners Lerdal
- Research Department, Lovisenberg Diaconal Hospital, Oslo, Norway.
- Department of Interdisciplinary Health Sciences, Faculty of Medicine, Institute of Health and Society, University of Oslo, Oslo, Norway.
| | - Caryl Gay
- Department of Interdisciplinary Health Sciences, Faculty of Medicine, Institute of Health and Society, University of Oslo, Oslo, Norway
- Department of Family Health Care Nursing, University of California, San Francisco, USA
| | - Tore Bonsaksen
- Department of Health and Nursing, Faculty of Social and Health Sciences, Inland Norway University of Applied Sciences, Elverum, Norway
- Department of Health, Faculty of Health Studies, VID Specialized University, Stavanger, Norway
| | - Øivind Ekeberg
- Psychosomatic and CL Psychiatry, Division of Mental Health and Addiction, Oslo University Hospital, Oslo, Norway
| | - Trine Grimholt
- Department of Health, Faculty of Health Studies, VID Specialized University, Oslo, Norway
- Department of Acute Medicine, Oslo University Hospital, Oslo, Norway
| | - Trond Heir
- Norwegian Center for Violence and Traumatic Stress Studies, Oslo, Norway
- Institute of Clinical Medicine, University of Oslo, Oslo, Norway
| | - Anders Kottorp
- Faculty of Health and Society, Malmö University, Malmö, Sweden
| | - Kathryn A Lee
- Department of Family Health Care Nursing, University of California, San Francisco, USA
| | - Laila Skogstad
- Faculty of Health Sciences, Department of Health and Care Sciences, UiT The Arctic University of Norway, Tromsø, Norway
| | - Inger Schou-Bredal
- Department of Public Health Science, Faculty of Medicine, Institute of Health and Society, University of Oslo, Oslo, Norway
| |
Collapse
|
4
|
Lin Y, Porter LS, Chee W, Alese OB, Curseen KA, Higgins MK, Northouse L, Xiao C. A Web-Based Dyadic Intervention to Manage Psychoneurological Symptoms for Patients With Colorectal Cancer and Their Caregivers: Protocol for a Mixed Methods Study. JMIR Res Protoc 2023; 12:e48499. [PMID: 37379055 PMCID: PMC10365620 DOI: 10.2196/48499] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Revised: 05/25/2023] [Accepted: 05/26/2023] [Indexed: 06/29/2023] Open
Abstract
BACKGROUND Patients with colorectal cancer (CRC) receiving chemotherapy often experience psychoneurological symptoms (PNS; ie, fatigue, depression, anxiety, sleep disturbance, pain, and cognitive dysfunction) that negatively impact both patients' and their caregivers' health outcomes. Limited information is available on PNS management for CRC patient and caregiver dyads. OBJECTIVE The purposes of this study are to (1) develop a web-based dyadic intervention for patients with CRC receiving chemotherapy and their caregivers (CRCweb) and (2) evaluate the feasibility, acceptability, and preliminary effects of CRCweb among patient-caregiver dyads in a cancer clinic. METHODS A mixed methods approach will be used. Semistructured interviews among 8 dyads will be conducted to develop CRCweb. A single-group pre- and posttest clinical trial will be used to examine the feasibility, acceptability, and preliminary effects of the intervention (CRCweb) among 20 dyads. Study assessments will be conducted before (T1) and after intervention (T2). Content analysis will be performed for semistructured interviews. Descriptive statistics will be calculated separately for patients and caregivers, and pre-post paired t tests will be used to evaluate treatment effects. RESULTS This study was funded in November 2022. As of April 2023, we have obtained institutional review board approval and completed clinical trial registration and are currently recruiting patient-caregiver dyads in a cancer clinic. The study is expected to be completed in October 2024. CONCLUSIONS Developing a web-based dyadic intervention holds great promise to reduce the PNS burden in patients with CRC receiving chemotherapy and their caregivers. The findings from this study will advance intervention development and implementation of symptom management and palliative care for patients with cancer and their caregivers. TRIAL REGISTRATION ClinicalTrials.gov NCT05663203; https://clinicaltrials.gov/ct2/show/NCT05663203. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID) PRR1-10.2196/48499.
Collapse
Affiliation(s)
- Yufen Lin
- Nell Hodgson Woodruff School of Nursing, Emory University, Atlanta, GA, United States
| | - Laura S Porter
- Department of Psychiatry and Behavioral Sciences, School of Medicine, Duke University, Durham, NC, United States
| | - Wonshik Chee
- Nell Hodgson Woodruff School of Nursing, Emory University, Atlanta, GA, United States
| | - Olatunji B Alese
- Winship Cancer Institute, Emory University, Atlanta, GA, United States
| | | | - Melinda K Higgins
- Nell Hodgson Woodruff School of Nursing, Emory University, Atlanta, GA, United States
| | - Laurel Northouse
- School of Nursing, University of Michigan, Ann Arbor, MI, United States
| | - Canhua Xiao
- Nell Hodgson Woodruff School of Nursing, Emory University, Atlanta, GA, United States
| |
Collapse
|