Beyond Dr. Google: Health information seeking from an intelligent online symptom checker: Cross-Sectional Questionnaire Study (Preprint)

Accuracy of online symptom assessment applications, large language models, and laypeople for self-triage decisions

Symptom-Assessment Application (SAAs, e.g., NHS 111 online) that assist laypeople in deciding if and where to seek care (self-triage) are gaining popularity and Large Language Models (LLMs) are increasingly used too. However, there is no evidence synthesis on the accuracy of LLMs, and no review has contextualized the accuracy of SAAs and LLMs. This systematic review evaluates the self-triage accuracy of both SAAs and LLMs and compares them to the accuracy of laypeople. A total of 1549 studies were screened and 19 included. The self-triage accuracy of SAAs was moderate but highly variable (11.5-90.0%), while the accuracy of LLMs (57.8-76.0%) and laypeople (47.3-62.4%) was moderate with low variability. Based on the available evidence, the use of SAAs or LLMs should neither be universally recommended nor discouraged; rather, we suggest that their utility should be assessed based on the specific use case and user group under consideration.

Exploring Laypersons' Experiences With a Mobile Symptom Checker App as an Interface Between eHealth Literacy, Health Literacy, and Health-Related Behavior: Qualitative Interview Study

BACKGROUND

Symptom checkers aim to help users recognize medical symptoms and recommend actions. However, they are not yet reliable for self-triage or diagnostics. Health literacy plays a role in their use, but the process from symptom recognition to health care consultation remains unclear.

OBJECTIVE

This qualitative observatory study explored how laypersons use symptom checkers, focusing on the process of use, entry points and outcomes, and the role of health literacy. Laypersons are defined as individuals who are neither medical professionals nor developers of such apps. Three research questions were addressed: (1) How do such users describe the process of using symptom checkers? (2) What are entry points and possible outcomes of symptom checker app use? (3) How are health literacy and eHealth literacy expressed during the use of symptom checker apps?

METHODS

As part of the Ethical, Legal, and Social Implications of Symptom Checker Apps in Primary Health Care project, 15 laypersons (n=9, 60% female and n=6, 40% male; mean age 30.7, SD 13.6 years) were interviewed about their experiences with the symptom checker Ada. The interviews were analyzed using an integrative approach combining social positioning, agency, and the Rubicon model as a heuristic framework.

RESULTS

App use follows a cyclic process comprising 4 steps: motivation (influenced by biography and context), intention formation (assigning a purpose), intention implementation (recruiting resources), and evaluation (transforming interactions into health-related insights). Biographical, social, and contextual factors shape process initiation. Users use symptom checkers for 3 main purposes: understanding their condition, receiving recommendations for action, and documenting or communicating health-related information. Each purpose requires specific planning and integration into health-related behaviors drawing on personal, social, and technological resources. Evaluation depends on contextual factors, app outputs, and the outcomes of users' health-related actions. Users assess whether the app aligns with their expectations, condition severity, and previous experiences, with health literacy playing a critical role in validation processes.

CONCLUSIONS

Symptom checker use is a complex, cyclic process shaped by context, biography, and health literacy. Users are motivated by health concerns influenced by personal, social, and contextual factors, with trust and attitudes impacting initial engagement. Intention formation reflects a balance between user skills and context, where app outputs inform decisions but may not always lead to action, especially in ambiguous situations. Users rely on personal resources and social networks to integrate app use into health-related behaviors, highlighting the limitations of symptom checkers in providing social or empathetic support. Symptom checkers have the potential to serve as an interface between users and health care, but future development must address the complexity of their use to unlock this potential.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.2196/34026.

Can Large Language Models Aid Caregivers of Pediatric Cancer Patients in Information Seeking? A Cross-Sectional Investigation

PURPOSE

Caregivers in pediatric oncology need accurate and understandable information about their child's condition, treatment, and side effects. This study assesses the performance of publicly accessible large language model (LLM)-supported tools in providing valuable and reliable information to caregivers of children with cancer.

METHODS

In this cross-sectional study, we evaluated the performance of the four LLM-supported tools-ChatGPT (GPT-4), Google Bard (Gemini Pro), Microsoft Bing Chat, and Google SGE-against a set of frequently asked questions (FAQs) derived from the Children's Oncology Group Family Handbook and expert input (In total, 26 FAQs and 104 generated responses). Five pediatric oncology experts assessed the generated LLM responses using measures including accuracy, clarity, inclusivity, completeness, clinical utility, and overall rating. Additionally, the content quality was evaluated including readability, AI disclosure, source credibility, resource matching, and content originality. We used descriptive analysis and statistical tests including Shapiro-Wilk, Levene's, Kruskal-Wallis H-tests, and Dunn's post hoc tests for pairwise comparisons.

RESULTS

ChatGPT shows high overall performance when evaluated by the experts. Bard also performed well, especially in accuracy and clarity of the responses, whereas Bing Chat and Google SGE had lower overall scores. Regarding the disclosure of responses being generated by AI, it was observed less frequently in ChatGPT responses, which may have affected the clarity of responses, whereas Bard maintained a balance between AI disclosure and response clarity. Google SGE generated the most readable responses whereas ChatGPT answered with the most complexity. LLM tools varied significantly (p < 0.001) across all expert evaluations except inclusivity. Through our thematic analysis of expert free-text comments, emotional tone and empathy emerged as a unique theme with mixed feedback on expectations from AI to be empathetic.

CONCLUSION

LLM-supported tools can enhance caregivers' knowledge of pediatric oncology. Each model has unique strengths and areas for improvement, indicating the need for careful selection based on specific clinical contexts. Further research is required to explore their application in other medical specialties and patient demographics, assessing broader applicability and long-term impacts.

The RepVig framework for designing use-case specific representative vignettes and evaluating triage accuracy of laypeople and symptom assessment applications

Most studies evaluating symptom-assessment applications (SAAs) rely on a common set of case vignettes that are authored by clinicians and devoid of context, which may be representative of clinical settings but not of situations where patients use SAAs. Assuming the use case of self-triage, we used representative design principles to sample case vignettes from online platforms where patients describe their symptoms to obtain professional advice and compared triage performance of laypeople, SAAs (e.g., WebMD or NHS 111), and Large Language Models (LLMs, e.g., GPT-4 or Claude) on representative versus standard vignettes. We found performance differences in all three groups depending on vignette type: When using representative vignettes, accuracy was higher (OR = 1.52 to 2.00, p < .001 to .03 in binary decisions, i.e., correct or incorrect), safety was higher (OR = 1.81 to 3.41, p < .001 to .002 in binary decisions, i.e., safe or unsafe), and the inclination to overtriage was also higher (OR = 1.80 to 2.66, p < .001 to p = .035 in binary decisions, overtriage or undertriage error). Additionally, we found changed rankings of best-performing SAAs and LLMs. Based on these results, we argue that our representative vignette sampling approach (that we call the RepVig Framework) should replace the practice of using a fixed vignette set as standard for SAA evaluation studies.

Accuracy of symptom checker for the diagnosis of sexually transmitted infections using machine learning and Bayesian network algorithms

BACKGROUND

A significant proportion of individuals with symptoms of sexually transmitted infection (STI) delay or avoid seeking healthcare, and digital diagnostic tools may prompt them to seek healthcare earlier. Unfortunately, none of the currently available tools fully mimic clinical assessment or cover a wide range of STIs.

METHODS

We prospectively invited attendees presenting with STI-related symptoms at Melbourne Sexual Health Centre to answer gender-specific questionnaires covering the symptoms of 12 common STIs using a computer-assisted self-interviewing system between 2015 and 2018. Then, we developed an online symptom checker (iSpySTI.org) using Bayesian networks. In this study, various machine learning algorithms were trained and evaluated for their ability to predict these STI and anogenital conditions. We used the Z-test to compare their average area under the ROC curve (AUC) scores with the Bayesian networks for diagnostic accuracy.

RESULTS

The study population included 6,162 men (median age 30, IQR: 26-38; approximately 40% of whom had sex with men in the past 12 months) and 4,358 women (median age 27, IQR: 24-31). Non-gonococcal urethritis (NGU) (23.6%, 1447/6121), genital warts (11.7%, 718/6121) and balanitis (8.9%, 546/6121) were the most common conditions in men. Candidiasis (16.6%, 722/4538) and bacterial vaginosis (16.2%, 707/4538) were the most common conditions in women. During evaluation with unseen datasets, machine learning models performed well for most male conditions, with the AUC ranging from 0.81 to 0.95, except for urinary tract infections (UTI) (AUC 0.72). Similarly, the models achieved AUCs ranging from 0.75 to 0.95 for female conditions, except for cervicitis (AUC 0.58). Urethral discharge and other urinary symptoms were important features for predicting urethral gonorrhoea, NGU and UTIs. Similarly, participants selected skin images that were similar to their own lesions, and the location of the anogenital skin lesions were also strong predictors. The vaginal discharge (odour, colour) and itchiness were important predictors for bacterial vaginosis and candidiasis. The performance of the machine learning models was significantly better than Bayesian models for male balanitis, molluscum contagiosum and genital warts (P < 0.05) but was similar for the other conditions.

CONCLUSIONS

Both machine learning and Bayesian models could predict correct diagnoses with reasonable accuracy using prospectively collected data for 12 STIs and other common anogenital conditions. Further work should expand the number of anogenital conditions and seek ways to improve the accuracy, potentially using patient collected images to supplement questionnaire data.

Patients Facing Large Language Models in Oncology: A Narrative Review

The integration of large language models (LLMs) into oncology is transforming patients' journeys through education, diagnosis, treatment monitoring, and follow-up. This review examines the current landscape, potential benefits, and associated ethical and regulatory considerations of the application of LLMs for patients in the oncologic domain.

Statistical refinement of patient-centered case vignettes for digital health research

Digital health research often relies on case vignettes (descriptions of fictitious or real patients) to navigate ethical and practical challenges. Despite their utility, the quality and lack of standardization of these vignettes has often been criticized, especially in studies on symptom-assessment applications (SAAs) and self-triage decision-making. To address this, our paper introduces a method to refine an existing set of vignettes, drawing on principles from classical test theory. First, we removed any vignette with an item difficulty of zero and an item-total correlation below zero. Second, we stratified the remaining vignettes to reflect the natural base rates of symptoms that SAAs are typically approached with, selecting those vignettes with the highest item-total correlation in each quota. Although this two-step procedure reduced the size of the original vignette set by 40%, comparing self-triage performance on the reduced and the original vignette sets, we found a strong correlation (r = 0.747 to r = 0.997, p < .001). This indicates that using our refinement method helps identifying vignettes with high predictive power of an agent's self-triage performance while simultaneously increasing cost-efficiency of vignette-based evaluation studies. This might ultimately lead to higher research quality and more reliable results.

Exploring psychological variables in users' health information-seeking behavior: A systematic review

One of the important factors that play a fundamental role in people's information behavior is psychological factors. The aim of the current research is to identify the psychological factors that impact users' health information-seeking behavior through a systematic review. Innovation in this work emphasizes the use of a systematic approach to identify psychological factors that influence individuals' information behavior. By employing a systematic method, this research can have high scientific value and provides greater confidence in identifying and describing psychological factors related to information behavior. The research method of this study was carried out using a systematic review method. After searching in WoS, PubMed, and Scopus databases, 4162 articles were reviewed, after removing repetition and applying article selection criteria, 31 articles were selected for analysis. In this article, a systematic review of the Prisma flowchart tool has been utilized. The Prisma flowchart is a valuable instrument for ensuring methodological transparency and facilitating the reporting of systematic reviews and meta-analyses. It provides a structured framework for outlining the various stages of the review process, including study identification, screening, eligibility assessment, data extraction, and synthesis. By employing the Prisma flowchart, researchers can enhance the rigor and reproducibility of their systematic reviews, thereby promoting evidence-based decision making in various fields of study. The findings reveal that out of 31 articles, 28 were surveys, and 3 were descriptive studies. Furthermore, one article employed an intervention methodology, targeting community members, pregnant women, or patients as the statistical population. The research findings highlight anxiety, uncertainty, and avoidance of information as the most commonly identified psychological variables influencing Health information-seeking behavior. Psychological factors play an important role in the health information behavior of information users in different societies; however, in the published articles in the field of health information behavior, more attention has been paid to information carriers and less attention has been paid to the psychological characteristics of people, which originate from the human psyche and mind. The importance of dealing with non-communicable diseases has been emphasized in the "Research and Technology Policies and Priorities" documents. These documents highlight disease management, self-care, and the role of education and information in disease control and reducing the burden of non-communicable diseases. Therefore, it is essential that planners and policymakers can take important steps by focusing on these factors in order to improve the quality of information acquisition. Also, this work provides the possibility for researchers to study the information in future research with more knowledge by knowing the existing gaps in the field of psychologically effective factors on information behavior.

Triage Accuracy and the Safety of User-Initiated Symptom Assessment With an Electronic Symptom Checker in a Real-Life Setting: Instrument Validation Study

BACKGROUND

Previous studies have evaluated the accuracy of the diagnostics of electronic symptom checkers (ESCs) and triage using clinical case vignettes. National Omaolo digital services (Omaolo) in Finland consist of an ESC for various symptoms. Omaolo is a medical device with a Conformité Européenne marking (risk class: IIa), based on Duodecim Clinical Decision Support, EBMEDS.

OBJECTIVE

This study investigates how well triage performed by the ESC nurse triage within the chief symptom list available in Omaolo (anal region symptoms, cough, diarrhea, discharge from the eye or watery or reddish eye, headache, heartburn, knee symptom or injury, lower back pain or injury, oral health, painful or blocked ear, respiratory tract infection, sexually transmitted disease, shoulder pain or stiffness or injury, sore throat or throat symptom, and urinary tract infection). In addition, the accuracy, specificity, sensitivity, and safety of the Omaolo ESC were assessed.

METHODS

This is a clinical validation study in a real-life setting performed at multiple primary health care (PHC) centers across Finland. The included units were of the walk-in model of primary care, where no previous phone call or contact was required. Upon arriving at the PHC center, users (patients) answered the ESC questions and received a triage recommendation; a nurse then assessed their triage. Findings on 877 patients were analyzed by matching the ESC recommendations with triage by the triage nurse.

RESULTS

Safe assessments by the ESC accounted for 97.6% (856/877; 95% CI 95.6%-98.0%) of all assessments made. The mean of the exact match for all symptom assessments was 53.7% (471/877; 95% CI 49.2%-55.9%). The mean value of the exact match or overly conservative but suitable for all (ESC's assessment was 1 triage level higher than the nurse's triage) symptom assessments was 66.6% (584/877; 95% CI 63.4%-69.7%). When the nurse concluded that urgent treatment was needed, the ESC's exactly matched accuracy was 70.9% (244/344; 95% CI 65.8%-75.7%). Sensitivity for the Omaolo ESC was 62.6% and specificity of 69.2%. A total of 21 critical assessments were identified for further analysis: there was no indication of compromised patient safety.

CONCLUSIONS

The primary objectives of this study were to evaluate the safety and to explore the accuracy, specificity, and sensitivity of the Omaolo ESC. The results indicate that the ESC is safe in a real-life setting when appraised with assessments conducted by triage nurses. Furthermore, the Omaolo ESC exhibits the potential to guide patients to appropriate triage destinations effectively, helping them to receive timely and suitable care.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.2196/41423.

Facilitating public involvement in research about healthcare AI: A scoping review of empirical methods

OBJECTIVE

With the recent increase in research into public views on healthcare artificial intelligence (HCAI), the objective of this review is to examine the methods of empirical studies on public views on HCAI. We map how studies provided participants with information about HCAI, and we examine the extent to which studies framed publics as active contributors to HCAI governance.

MATERIALS AND METHODS

We searched 5 academic databases and Google Advanced for empirical studies investigating public views on HCAI. We extracted information including study aims, research instruments, and recommendations.

RESULTS

Sixty-two studies were included. Most were quantitative (N = 42). Most (N = 47) reported providing participants with background information about HCAI. Despite this, studies often reported participants' lack of prior knowledge about HCAI as a limitation. Over three quarters (N = 48) of the studies made recommendations that envisaged public views being used to guide governance of AI.

DISCUSSION

Provision of background information is an important component of facilitating research with publics on HCAI. The high proportion of studies reporting participants' lack of knowledge about HCAI as a limitation reflects the need for more guidance on how information should be presented. A minority of studies adopted technocratic positions that construed publics as passive beneficiaries of AI, rather than as active stakeholders in HCAI design and implementation.

CONCLUSION

This review draws attention to how public roles in HCAI governance are constructed in empirical studies. To facilitate active participation, we recommend that research with publics on HCAI consider methodological designs that expose participants to diverse information sources.

Only the anxious ones? Identifying characteristics of symptom checker app users: a cross-sectional survey

BACKGROUND

Symptom checker applications (SCAs) may help laypeople classify their symptoms and receive recommendations on medically appropriate actions. Further research is necessary to estimate the influence of user characteristics, attitudes and (e)health-related competencies.

OBJECTIVE

The objective of this study is to identify meaningful predictors for SCA use considering user characteristics.

METHODS

An explorative cross-sectional survey was conducted to investigate German citizens' demographics, eHealth literacy, hypochondria, self-efficacy, and affinity for technology using German language-validated questionnaires. A total of 869 participants were eligible for inclusion in the study. As n = 67 SCA users were assessed and matched 1:1 with non-users, a sample of n = 134 participants were assessed in the main analysis. A four-step analysis was conducted involving explorative predictor selection, model comparisons, and parameter estimates for selected predictors, including sensitivity and post hoc analyses.

RESULTS

Hypochondria and self-efficacy were identified as meaningful predictors of SCA use. Hypochondria showed a consistent and significant effect across all analyses OR: 1.24-1.26 (95% CI: 1.1-1.4). Self-efficacy OR: 0.64-0.93 (95% CI: 0.3-1.4) showed inconsistent and nonsignificant results, leaving its role in SCA use unclear. Over half of the SCA users in our sample met the classification for hypochondria (cut-off on the WI of 5).

CONCLUSIONS

Hypochondria has emerged as a significant predictor of SCA use with a consistently stable effect, yet according to the literature, individuals with this trait may be less likely to benefit from SCA despite their greater likelihood of using it. These users could be further unsettled by risk-averse triage and unlikely but serious diagnosis suggestions.

TRIAL REGISTRATION

The study was registered in the German Clinical Trials Register (DRKS) DRKS00022465, DERR1- https://doi.org/10.2196/34026 .

How Virtual Triage Can Improve Patient Experience and Satisfaction: A Narrative Review and Look Forward

Objective

To complete a review of the literature on patient experience and satisfaction as relates to the potential for virtual triage (VT) or symptom checkers to enhance and enable improvements in these important health care delivery objectives.

Methods

Review and synthesis of the literature on patient experience and satisfaction as informed by emerging evidence, indicating potential for VT to favorably impact these clinical care objectives and outcomes.

Results/Conclusions

VT enhances potential clinical effectiveness through early detection and referral, can reduce avoidable care delivery due to late clinical presentation, and can divert primary care needs to more clinically appropriate outpatient settings rather than high-acuity emergency departments. Delivery of earlier and faster, more acuity level-appropriate care, as well as patient avoidance of excess care acuity (and associated cost), offer promise as contributors to improved patient experience and satisfaction. The application of digital triage as a front door to health care delivery organizations offers care engagement that can help reduce patient need to visit a medical facility for low-acuity conditions more suitable for self-care, thus avoiding unpleasant queues and reducing microbiological and other patient risks associated with visits to medical facilities. VT also offers an opportunity for providers to make patient health care experiences more personalized.

Characteristics of Users and Nonusers of Symptom Checkers in Germany: Cross-Sectional Survey Study

BACKGROUND

Previous studies have revealed that users of symptom checkers (SCs, apps that support self-diagnosis and self-triage) are predominantly female, are younger than average, and have higher levels of formal education. Little data are available for Germany, and no study has so far compared usage patterns with people's awareness of SCs and the perception of usefulness.

OBJECTIVE

We explored the sociodemographic and individual characteristics that are associated with the awareness, usage, and perceived usefulness of SCs in the German population.

METHODS

We conducted a cross-sectional online survey among 1084 German residents in July 2022 regarding personal characteristics and people's awareness and usage of SCs. Using random sampling from a commercial panel, we collected participant responses stratified by gender, state of residence, income, and age to reflect the German population. We analyzed the collected data exploratively.

RESULTS

Of all respondents, 16.3% (177/1084) were aware of SCs and 6.5% (71/1084) had used them before. Those aware of SCs were younger (mean 38.8, SD 14.6 years, vs mean 48.3, SD 15.7 years), were more often female (107/177, 60.5%, vs 453/907, 49.9%), and had higher formal education levels (eg, 72/177, 40.7%, vs 238/907, 26.2%, with a university/college degree) than those unaware. The same observation applied to users compared to nonusers. It disappeared, however, when comparing users to nonusers who were aware of SCs. Among users, 40.8% (29/71) considered these tools useful. Those considering them useful reported higher self-efficacy (mean 4.21, SD 0.66, vs mean 3.63, SD 0.81, on a scale of 1-5) and a higher net household income (mean EUR 2591.63, SD EUR 1103.96 [mean US $2798.96, SD US $1192.28], vs mean EUR 1626.60, SD EUR 649.05 [mean US $1756.73, SD US $700.97]) than those who considered them not useful. More women considered SCs unhelpful (13/44, 29.5%) compared to men (4/26, 15.4%).

CONCLUSIONS

Concurring with studies from other countries, our findings show associations between sociodemographic characteristics and SC usage in a German sample: users were on average younger, of higher socioeconomic status, and more commonly female compared to nonusers. However, usage cannot be explained by sociodemographic differences alone. It rather seems that sociodemographics explain who is or is not aware of the technology, but those who are aware of SCs are equally likely to use them, independently of sociodemographic differences. Although in some groups (eg, people with anxiety disorder), more participants reported to know and use SCs, they tended to perceive them as less useful. In other groups (eg, male participants), fewer respondents were aware of SCs, but those who used them perceived them to be more useful. Thus, SCs should be designed to fit specific user needs, and strategies should be developed to help reach individuals who could benefit but are not aware of SCs yet.

Ethical, legal, and social aspects of symptom checker applications: a scoping review

Symptom Checker Applications (SCA) are mobile applications often designed for the end-user to assist with symptom assessment and self-triage. SCA are meant to provide the user with easily accessible information about their own health conditions. However, SCA raise questions regarding ethical, legal, and social aspects (ELSA), for example, regarding fair access to this new technology. The aim of this scoping review is to identify the ELSA of SCA in the scientific literature. A scoping review was conducted to identify the ELSA of SCA. Ten databases (e.g., Web of Science and PubMed) were used. Studies on SCA that address ELSA, written in English or German, were included in the review. The ELSA of SCA were extracted and synthesized using qualitative content analysis. A total of 25,061 references were identified, of which 39 were included in the analysis. The identified aspects were allotted to three main categories: (1) Technology; (2) Individual Level; and (3) Healthcare system. The results show that there are controversial debates in the literature on the ethical and social challenges of SCA usage. Furthermore, the debates are characterised by a lack of a specific legal perspective and empirical data. The review provides an overview on the spectrum of ELSA regarding SCA. It offers guidance to stakeholders in the healthcare system, for example, patients, healthcare professionals, and insurance providers and could be used in future empirical research to investigate the perspectives of those affected, such as users.