Digital and online symptom checkers and health assessment/triage services for urgent health problems: systematic review

Controlling Inputter Variability in Vignette Studies Assessing Web-Based Symptom Checkers: Evaluation of Current Practice and Recommendations for Isolated Accuracy Metrics

BACKGROUND

The rapid growth of web-based symptom checkers (SCs) is not matched by advances in quality assurance. Currently, there are no widely accepted criteria assessing SCs' performance. Vignette studies are widely used to evaluate SCs, measuring the accuracy of outcome. Accuracy behaves as a composite metric as it is affected by a number of individual SC- and tester-dependent factors. In contrast to clinical studies, vignette studies have a small number of testers. Hence, measuring accuracy alone in vignette studies may not provide a reliable assessment of performance due to tester variability.

OBJECTIVE

This study aims to investigate the impact of tester variability on the accuracy of outcome of SCs, using clinical vignettes. It further aims to investigate the feasibility of measuring isolated aspects of performance.

METHODS

Healthily's SC was assessed using 114 vignettes by 3 groups of 3 testers who processed vignettes with different instructions: free interpretation of vignettes (free testers), specified chief complaints (partially free testers), and specified chief complaints with strict instruction for answering additional symptoms (restricted testers). κ statistics were calculated to assess agreement of top outcome condition and recommended triage. Crude and adjusted accuracy was measured against a gold standard. Adjusted accuracy was calculated using only results of consultations identical to the vignette, following a review and selection process. A feasibility study for assessing symptom comprehension of SCs was performed using different variations of 51 chief complaints across 3 SCs.

RESULTS

Intertester agreement of most likely condition and triage was, respectively, 0.49 and 0.51 for the free tester group, 0.66 and 0.66 for the partially free group, and 0.72 and 0.71 for the restricted group. For the restricted group, accuracy ranged from 43.9% to 57% for individual testers, averaging 50.6% (SD 5.35%). Adjusted accuracy was 56.1%. Assessing symptom comprehension was feasible for all 3 SCs. Comprehension scores ranged from 52.9% and 68%.

CONCLUSIONS

We demonstrated that by improving standardization of the vignette testing process, there is a significant improvement in the agreement of outcome between testers. However, significant variability remained due to uncontrollable tester-dependent factors, reflected by varying outcome accuracy. Tester-dependent factors, combined with a small number of testers, limit the reliability and generalizability of outcome accuracy when used as a composite measure in vignette studies. Measuring and reporting different aspects of SC performance in isolation provides a more reliable assessment of SC performance. We developed an adjusted accuracy measure using a review and selection process to assess data algorithm quality. In addition, we demonstrated that symptom comprehension with different input methods can be feasibly compared. Future studies reporting accuracy need to apply vignette testing standardization and isolated metrics.

Longitudinal Changes in Diagnostic Accuracy of a Differential Diagnosis List Developed by an AI-Based Symptom Checker: Retrospective Observational Study

BACKGROUND

Artificial intelligence (AI) symptom checker models should be trained using real-world patient data to improve their diagnostic accuracy. Given that AI-based symptom checkers are currently used in clinical practice, their performance should improve over time. However, longitudinal evaluations of the diagnostic accuracy of these symptom checkers are limited.

OBJECTIVE

This study aimed to assess the longitudinal changes in the accuracy of differential diagnosis lists created by an AI-based symptom checker used in the real world.

METHODS

This was a single-center, retrospective, observational study. Patients who visited an outpatient clinic without an appointment between May 1, 2019, and April 30, 2022, and who were admitted to a community hospital in Japan within 30 days of their index visit were considered eligible. We only included patients who underwent an AI-based symptom checkup at the index visit, and the diagnosis was finally confirmed during follow-up. Final diagnoses were categorized as common or uncommon, and all cases were categorized as typical or atypical. The primary outcome measure was the accuracy of the differential diagnosis list created by the AI-based symptom checker, defined as the final diagnosis in a list of 10 differential diagnoses created by the symptom checker. To assess the change in the symptom checker's diagnostic accuracy over 3 years, we used a chi-square test to compare the primary outcome over 3 periods: from May 1, 2019, to April 30, 2020 (first year); from May 1, 2020, to April 30, 2021 (second year); and from May 1, 2021, to April 30, 2022 (third year).

RESULTS

A total of 381 patients were included. Common diseases comprised 257 (67.5%) cases, and typical presentations were observed in 298 (78.2%) cases. Overall, the accuracy of the differential diagnosis list created by the AI-based symptom checker was 172 (45.1%), which did not differ across the 3 years (first year: 97/219, 44.3%; second year: 32/72, 44.4%; and third year: 43/90, 47.7%; P=.85). The accuracy of the differential diagnosis list created by the symptom checker was low in those with uncommon diseases (30/124, 24.2%) and atypical presentations (12/83, 14.5%). In the multivariate logistic regression model, common disease (P<.001; odds ratio 4.13, 95% CI 2.50-6.98) and typical presentation (P<.001; odds ratio 6.92, 95% CI 3.62-14.2) were significantly associated with the accuracy of the differential diagnosis list created by the symptom checker.

CONCLUSIONS

A 3-year longitudinal survey of the diagnostic accuracy of differential diagnosis lists developed by an AI-based symptom checker, which has been implemented in real-world clinical practice settings, showed no improvement over time. Uncommon diseases and atypical presentations were independently associated with a lower diagnostic accuracy. In the future, symptom checkers should be trained to recognize uncommon conditions.

Recommendation endpoints and safety of an online self-triage for depression symptoms

INTRODUCTION

Online symptom checkers are a way to address patient concerns and potentially offload a burdened healthcare system. However, safety outcomes of self-triage are unknown, so we reviewed triage recommendations and outcomes of our institution's depression symptom checker.

METHODS

We examined endpoint recommendations and follow-up encounters seven days afterward during 2 December 2021 to 13 December 2022. Patients with an emergency department visit or hospitalization within seven days of self-triaging had a manual review of the electronic health record to determine if the visit was related to depression, suicidal ideation, or suicide attempt. Charts were reviewed for deaths within seven days of self-triage.

RESULTS

There were 287 unique encounters from 263 unique patients. In 86.1% (247/287), the endpoint was an instruction to call nurse triage; in 3.1% of encounters (9/287), instruction was to seek emergency care. Only 20.2% (58/287) followed the recommendations given. Of the 229 patients that did not follow the endpoint recommendations, 121 (52.8%) had some type of follow-up within seven days. Nearly 11% (31/287) were triaged to endpoints not requiring urgent contact and 9.1% (26/287) to an endpoint that would not need any healthcare team input. No patients died in the study period.

CONCLUSIONS

Most patients did not follow the recommendations for follow-up care although ultimately most patients did receive care within seven days. Self-triage appears to appropriately sort patients with depressed mood to emergency care. On-line self-triaging tools for depression have the potential to safely offload some work from clinic personnel.

Testing Two Online Symptom Checkers With Vulnerable Groups: Usability Study to Improve Cognitive Accessibility of eHealth Services

BACKGROUND

The popularity of eHealth services has surged significantly, underscoring the importance of ensuring their usability and accessibility for users with diverse needs, characteristics, and capabilities. These services can pose cognitive demands, especially for individuals who are unwell, fatigued, or experiencing distress. Additionally, numerous potentially vulnerable groups, including older adults, are susceptible to digital exclusion and may encounter cognitive limitations related to perception, attention, memory, and language comprehension. Regrettably, many studies overlook the preferences and needs of user groups likely to encounter challenges associated with these cognitive aspects.

OBJECTIVE

This study primarily aims to gain a deeper understanding of cognitive accessibility in the practical context of eHealth services. Additionally, we aimed to identify the specific challenges that vulnerable groups encounter when using eHealth services and determine key considerations for testing these services with such groups.

METHODS

As a case study of eHealth services, we conducted qualitative usability testing on 2 online symptom checkers used in Finnish public primary care. A total of 13 participants from 3 distinct groups participated in the study: older adults, individuals with mild intellectual disabilities, and nonnative Finnish speakers. The primary research methods used were the thinking-aloud method, questionnaires, and semistructured interviews.

RESULTS

We found that potentially vulnerable groups encountered numerous issues with the tested services, with similar problems observed across all 3 groups. Specifically, clarity and the use of terminology posed significant challenges. The services overwhelmed users with excessive information and choices, while the terminology consisted of numerous complex medical terms that were difficult to understand. When conducting tests with vulnerable groups, it is crucial to carefully plan the sessions to avoid being overly lengthy, as these users often require more time to complete tasks. Additionally, testing with vulnerable groups proved to be quite efficient, with results likely to benefit a wider audience as well.

CONCLUSIONS

Based on the findings of this study, it is evident that older adults, individuals with mild intellectual disability, and nonnative speakers may encounter cognitive challenges when using eHealth services, which can impede or slow down their use and make the services more difficult to navigate. In the worst-case scenario, these challenges may lead to errors in using the services. We recommend expanding the scope of testing to include a broader range of eHealth services with vulnerable groups, incorporating users with diverse characteristics and capabilities who are likely to encounter difficulties in cognitive accessibility.

'Better see a doctor?' Status quo of symptom checker apps in Germany: A cross-sectional survey with a mixed-methods design (CHECK.APP)

Background

Symptom checker apps (SCAs) offer symptom classification and low-threshold self-triage for laypeople. They are already in use despite their poor accuracy and concerns that they may negatively affect primary care. This study assesses the extent to which SCAs are used by medical laypeople in Germany and which software is most popular. We examined associations between satisfaction with the general practitioner (GP) and SCA use as well as the number of GP visits and SCA use. Furthermore, we assessed the reasons for intentional non-use.

Methods

We conducted a survey comprising standardised and open-ended questions. Quantitative data were weighted, and open-ended responses were examined using thematic analysis.

Results

This study included 850 participants. The SCA usage rate was 8%, and approximately 50% of SCA non-users were uninterested in trying SCAs. The most commonly used SCAs were NetDoktor and Ada. Surprisingly, SCAs were most frequently used in the age group of 51-55 years. No significant associations were found between SCA usage and satisfaction with the GP or the number of GP visits and SCA usage. Thematic analysis revealed skepticism regarding the results and recommendations of SCAs and discrepancies between users' requirements and the features of apps.

Conclusion

SCAs are still widely unknown in the German population and have been sparsely used so far. Many participants were not interested in trying SCAs, and we found no positive or negative associations of SCAs and primary care.

An introduction to digital determinants of health

In recent years, technology has been increasingly incorporated within healthcare for the provision of safe and efficient delivery of services. Although this can be attributed to the benefits that can be harnessed, digital technology has the potential to exacerbate and reinforce preexisting health disparities. Previous work has highlighted how sociodemographic, economic, and political factors affect individuals' interactions with digital health systems and are termed social determinants of health [SDOH]. But, there is a paucity of literature addressing how the intrinsic design, implementation, and use of technology interact with SDOH to influence health outcomes. Such interactions are termed digital determinants of health [DDOH]. This paper will, for the first time, propose a definition of DDOH and provide a conceptual model characterizing its influence on healthcare outcomes. Specifically, DDOH is implicit in the design of artificial intelligence systems, mobile phone applications, telemedicine, digital health literacy [DHL], and other forms of digital technology. A better appreciation of DDOH by the various stakeholders at the individual and societal levels can be channeled towards policies that are more digitally inclusive. In tandem with ongoing work to minimize the digital divide caused by existing SDOH, further work is necessary to recognize digital determinants as an important and distinct entity.

"You're only a receptionist, what do you want to know for?": Street-level bureaucracy on the front line of primary care in the United Kingdom

Introduction

In care settings across the globe non-clinical staff are involved in filtering patients to the most appropriate source of care. This includes primary care where general practice receptionists are key in facilitating access to individual surgeries and the wider National Health Service. Despite the complexity and significance of their role little is known of how the decision-making behaviors of receptionists impact policy implementation and service delivery. By combining the agent-based implementation theory of street-level bureaucracy with a tri-level analytical framework this work acknowledges the impact of the decisions made by receptionists as street-level bureaucrats and demonstrates the benefits of using the novel framework to provide practical insight of the factors influencing those decisions.

Methods

A secondary analysis of qualitative data gathered from a series of semi-structured interviews conducted with 19 receptionists in the United Kingdom in 2019 was used to populate a tri-level framework: the micro-level relates to influences on decision making acting at an individual level, the meso-level influences at group and organizational levels, and the macro-level influences at a societal or policy level.

Results

At the micro-level we determined how receptionists are influenced by the level of rapport developed with patients and would use common sense to interpret urgency. At the meso-level, influences included their position at the forefront of premises, the culture of the workplace, and the processes and protocols used by their practice. At the macro-level, participants described the impact of limited health service capacity, the lack of mandatory training, and the growth in the use of digital technologies.

Conclusions

Street-level bureaucracy, complemented with a tri-level contextual analysis, is a useful theoretical framework to understand how health workers, such as receptionists, attempt to provide universality without sufficient resource, and could potentially be applied to other kinds of public service workers in this way. This theoretical framework also benefits from being an accessible foundation on which to base practice and policy changes.

Redesigning Primary Care: The Emergence of Artificial-Intelligence-Driven Symptom Diagnostic Tools

Modern healthcare is facing a juxtaposition of increasing patient demands owing to an aging population and a decreasing general practitioner workforce, leading to strained access to primary care. The coronavirus disease 2019 pandemic has emphasized the potential for alternative consultation methods, highlighting opportunities to minimize unnecessary care. This article discusses the role of artificial-intelligence-driven symptom checkers, particularly their efficiency, utility, and challenges in primary care. Based on a study conducted in Italian general practices, insights from both physicians and patients were gathered regarding this emergent technology, highlighting differences in perceived utility, user satisfaction, and potential challenges. While symptom checkers are seen as potential tools for addressing healthcare challenges, concerns regarding their accuracy and the potential for misdiagnosis persist. Patients generally viewed them positively, valuing their ease of use and the empowerment they provide in managing health. However, some general practitioners perceive these tools as challenges to their expertise. This article proposes that artificial-intelligence-based symptom checkers can optimize medical-history taking for the benefit of both general practitioners and patients, with potential enhancements in complex diagnostic tasks rather than routine diagnoses. It underscores the importance of carefully integrating digital innovations while preserving the essential human touch in healthcare. Symptom checkers offer promising solutions; ensuring their accuracy, reliability, and effective integration into primary care requires rigorous research, clinical guidance, and an understanding of varied user perceptions. Collaboration among technologists, clinicians, and patients is paramount for the successful evolution of digital tools in healthcare.

Supporting primary care through symptom checking artificial intelligence: a study of patient and physician attitudes in Italian general practice

BACKGROUND

Rapid advancements in artificial intelligence (AI) have led to the adoption of AI-driven symptom checkers in primary care. This study aimed to evaluate both patients' and physicians' attitudes towards these tools in Italian general practice settings, focusing on their perceived utility, user satisfaction, and potential challenges.

METHODS

This feasibility study involved ten general practitioners (GPs) and patients visiting GP offices. The patients used a chatbot-based symptom checker before their medical visit and conducted anamnestic screening for COVID-19 and a medical history algorithm concerning the current medical problem. The entered data were forwarded to the GP as medical history aid. After the medical visit, both physicians and patients evaluated their respective symptoms. Additionally, physicians performed a final overall evaluation of the symptom checker after the conclusion of the practice phase.

RESULTS

Most patients did not use symptom checkers. Overall, 49% of patients and 27% of physicians reported being rather or very satisfied with the symptom checker. The most frequent patient-reported reasons for satisfaction were ease of use, precise and comprehensive questions, perceived time-saving potential, and encouragement of self-reflection. Every other patient would consider at-home use of the symptom checker for the first appraisal of health problems to save time, reduce unnecessary visits, and/or as an aid for the physician. Patients' attitudes towards the symptom checker were not significantly associated with age, sex, or level of education. Most patients (75%) and physicians (84%) indicated that the symptom checker had no effect on the duration of the medical visit. Only a few participants found the use of the symptom checker to be disruptive to the medical visit or its quality.

CONCLUSIONS

The findings suggest a positive reception of the symptom checker, albeit with differing focus between patients and physicians. With the potential to be integrated further into primary care, these tools require meticulous clinical guidance to maximize their benefits.

TRIAL REGISTRATION

The study was not registered, as it did not include direct medical intervention on human participants.

Can clinical decision support systems be an asset in medical education? An experimental approach

BACKGROUND

Diagnostic accuracy is one of the major cornerstones of appropriate and successful medical decision-making. Clinical decision support systems (CDSSs) have recently been used to facilitate physician's diagnostic considerations. However, to date, little is known about the potential assets of CDSS for medical students in an educational setting. The purpose of our study was to explore the usefulness of CDSSs for medical students assessing their diagnostic performances and the influence of such software on students' trust in their own diagnostic abilities.

METHODS

Based on paper cases students had to diagnose two different patients using a CDSS and conventional methods such as e.g. textbooks, respectively. Both patients had a common disease, in one setting the clinical presentation was a typical one (tonsillitis), in the other setting (pulmonary embolism), however, the patient presented atypically. We used a 2x2x2 between- and within-subjects cluster-randomised controlled trial to assess the diagnostic accuracy in medical students, also by changing the order of the used resources (CDSS first or second).

RESULTS

Medical students in their 4th and 5th year performed equally well using conventional methods or the CDSS across the two cases (t(164) = 1,30; p = 0.197). Diagnostic accuracy and trust in the correct diagnosis were higher in the typical presentation condition than in the atypical presentation condition (t(85) = 19.97; p < .0001 and t(150) = 7.67; p < .0001).These results refute our main hypothesis that students diagnose more accurately when using conventional methods compared to the CDSS.

CONCLUSIONS

Medical students in their 4th and 5th year performed equally well in diagnosing two cases of common diseases with typical or atypical clinical presentations using conventional methods or a CDSS. Students were proficient in diagnosing a common disease with a typical presentation but underestimated their own factual knowledge in this scenario. Also, students were aware of their own diagnostic limitations when presented with a challenging case with an atypical presentation for which the use of a CDSS seemingly provided no additional insights.

Effects of Combinational Use of Additional Differential Diagnostic Generators on the Diagnostic Accuracy of the Differential Diagnosis List Developed by an Artificial Intelligence-Driven Automated History-Taking System: Pilot Cross-Sectional Study

BACKGROUND

Low diagnostic accuracy is a major concern in automated medical history-taking systems with differential diagnosis (DDx) generators. Extending the concept of collective intelligence to the field of DDx generators such that the accuracy of judgment becomes higher when accepting an integrated diagnosis list from multiple people than when accepting a diagnosis list from a single person may be a possible solution.

OBJECTIVE

The purpose of this study is to assess whether the combined use of several DDx generators improves the diagnostic accuracy of DDx lists.

METHODS

We used medical history data and the top 10 DDx lists (index DDx lists) generated by an artificial intelligence (AI)-driven automated medical history-taking system from 103 patients with confirmed diagnoses. Two research physicians independently created the other top 10 DDx lists (second and third DDx lists) per case by imputing key information into the other 2 DDx generators based on the medical history generated by the automated medical history-taking system without reading the index lists generated by the automated medical history-taking system. We used the McNemar test to assess the improvement in diagnostic accuracy from the index DDx lists to the three types of combined DDx lists: (1) simply combining DDx lists from the index, second, and third lists; (2) creating a new top 10 DDx list using a 1/n weighting rule; and (3) creating new lists with only shared diagnoses among DDx lists from the index, second, and third lists. We treated the data generated by 2 research physicians from the same patient as independent cases. Therefore, the number of cases included in analyses in the case using 2 additional lists was 206 (103 cases × 2 physicians' input).

RESULTS

The diagnostic accuracy of the index lists was 46% (47/103). Diagnostic accuracy was improved by simply combining the other 2 DDx lists (133/206, 65%, P<.001), whereas the other 2 combined DDx lists did not improve the diagnostic accuracy of the DDx lists (106/206, 52%, P=.05 in the collective list with the 1/n weighting rule and 29/206, 14%, P<.001 in the only shared diagnoses among the 3 DDx lists).

CONCLUSIONS

Simply adding each of the top 10 DDx lists from additional DDx generators increased the diagnostic accuracy of the DDx list by approximately 20%, suggesting that the combinational use of DDx generators early in the diagnostic process is beneficial.

Advancing AI in healthcare: A comprehensive review of best practices

Artificial Intelligence (AI) and Machine Learning (ML) are powerful tools shaping the healthcare sector. This review considers twelve key aspects of AI in clinical practice: 1) Ethical AI; 2) Explainable AI; 3) Health Equity and Bias in AI; 4) Sponsorship Bias; 5) Data Privacy; 6) Genomics and Privacy; 7) Insufficient Sample Size and Self-Serving Bias; 8) Bridging the Gap Between Training Datasets and Real-World Scenarios; 9) Open Source and Collaborative Development; 10) Dataset Bias and Synthetic Data; 11) Measurement Bias; 12) Reproducibility in AI Research. These categories represent both the challenges and opportunities of AI implementation in healthcare. While AI holds significant potential for improving patient care, it also presents risks and challenges, such as ensuring privacy, combating bias, and maintaining transparency and ethics. The review underscores the necessity of developing comprehensive best practices for healthcare organizations and fostering a diverse dialogue involving data scientists, clinicians, patient advocates, ethicists, economists, and policymakers. We are at the precipice of significant transformation in healthcare powered by AI. By continuing to reassess and refine our approach, we can ensure that AI is implemented responsibly and ethically, maximizing its benefit to patient care and public health.

Characteristics of Users and Nonusers of Symptom Checkers in Germany: Cross-Sectional Survey Study

BACKGROUND

Previous studies have revealed that users of symptom checkers (SCs, apps that support self-diagnosis and self-triage) are predominantly female, are younger than average, and have higher levels of formal education. Little data are available for Germany, and no study has so far compared usage patterns with people's awareness of SCs and the perception of usefulness.

OBJECTIVE

We explored the sociodemographic and individual characteristics that are associated with the awareness, usage, and perceived usefulness of SCs in the German population.

METHODS

We conducted a cross-sectional online survey among 1084 German residents in July 2022 regarding personal characteristics and people's awareness and usage of SCs. Using random sampling from a commercial panel, we collected participant responses stratified by gender, state of residence, income, and age to reflect the German population. We analyzed the collected data exploratively.

RESULTS

Of all respondents, 16.3% (177/1084) were aware of SCs and 6.5% (71/1084) had used them before. Those aware of SCs were younger (mean 38.8, SD 14.6 years, vs mean 48.3, SD 15.7 years), were more often female (107/177, 60.5%, vs 453/907, 49.9%), and had higher formal education levels (eg, 72/177, 40.7%, vs 238/907, 26.2%, with a university/college degree) than those unaware. The same observation applied to users compared to nonusers. It disappeared, however, when comparing users to nonusers who were aware of SCs. Among users, 40.8% (29/71) considered these tools useful. Those considering them useful reported higher self-efficacy (mean 4.21, SD 0.66, vs mean 3.63, SD 0.81, on a scale of 1-5) and a higher net household income (mean EUR 2591.63, SD EUR 1103.96 [mean US $2798.96, SD US $1192.28], vs mean EUR 1626.60, SD EUR 649.05 [mean US $1756.73, SD US $700.97]) than those who considered them not useful. More women considered SCs unhelpful (13/44, 29.5%) compared to men (4/26, 15.4%).

CONCLUSIONS

Concurring with studies from other countries, our findings show associations between sociodemographic characteristics and SC usage in a German sample: users were on average younger, of higher socioeconomic status, and more commonly female compared to nonusers. However, usage cannot be explained by sociodemographic differences alone. It rather seems that sociodemographics explain who is or is not aware of the technology, but those who are aware of SCs are equally likely to use them, independently of sociodemographic differences. Although in some groups (eg, people with anxiety disorder), more participants reported to know and use SCs, they tended to perceive them as less useful. In other groups (eg, male participants), fewer respondents were aware of SCs, but those who used them perceived them to be more useful. Thus, SCs should be designed to fit specific user needs, and strategies should be developed to help reach individuals who could benefit but are not aware of SCs yet.

Triage and Diagnostic Accuracy of Online Symptom Checkers: Systematic Review

BACKGROUND

In the context of a deepening global shortage of health workers and, in particular, the COVID-19 pandemic, there is growing international interest in, and use of, online symptom checkers (OSCs). However, the evidence surrounding the triage and diagnostic accuracy of these tools remains inconclusive.

OBJECTIVE

This systematic review aimed to summarize the existing peer-reviewed literature evaluating the triage accuracy (directing users to appropriate services based on their presenting symptoms) and diagnostic accuracy of OSCs aimed at lay users for general health concerns.

METHODS

Searches were conducted in MEDLINE, Embase, CINAHL, Health Management Information Consortium (HMIC), and Web of Science, as well as the citations of the studies selected for full-text screening. We included peer-reviewed studies published in English between January 1, 2010, and February 16, 2022, with a controlled and quantitative assessment of either or both triage and diagnostic accuracy of OSCs directed at lay users. We excluded tools supporting health care professionals, as well as disease- or specialty-specific OSCs. Screening and data extraction were carried out independently by 2 reviewers for each study. We performed a descriptive narrative synthesis.

RESULTS

A total of 21,296 studies were identified, of which 14 (0.07%) were included. The included studies used clinical vignettes, medical records, or direct input by patients. Of the 14 studies, 6 (43%) reported on triage and diagnostic accuracy, 7 (50%) focused on triage accuracy, and 1 (7%) focused on diagnostic accuracy. These outcomes were assessed based on the diagnostic and triage recommendations attached to the vignette in the case of vignette studies or on those provided by nurses or general practitioners, including through face-to-face and telephone consultations. Both diagnostic accuracy and triage accuracy varied greatly among OSCs. Overall diagnostic accuracy was deemed to be low and was almost always lower than that of the comparator. Similarly, most of the studies (9/13, 69 %) showed suboptimal triage accuracy overall, with a few exceptions (4/13, 31%). The main variables affecting the levels of diagnostic and triage accuracy were the severity and urgency of the condition, the use of artificial intelligence algorithms, and demographic questions. However, the impact of each variable differed across tools and studies, making it difficult to draw any solid conclusions. All included studies had at least one area with unclear risk of bias according to the revised Quality Assessment of Diagnostic Accuracy Studies-2 tool.

CONCLUSIONS

Although OSCs have potential to provide accessible and accurate health advice and triage recommendations to users, more research is needed to validate their triage and diagnostic accuracy before widescale adoption in community and health care settings. Future studies should aim to use a common methodology and agreed standard for evaluation to facilitate objective benchmarking and validation.

TRIAL REGISTRATION

PROSPERO CRD42020215210; https://tinyurl.com/3949zw83.

Mapping Inequities in Digital Health Technology Within the World Health Organization's European Region Using PROGRESS PLUS: Scoping Review

BACKGROUND

The use of digital technologies within health care rapidly increased as services transferred to web-based platforms during the COVID-19 pandemic. Inequalities in digital health across the domains of equity are not routinely examined; yet, the long-term integration of digitally delivered services needs to consider such inequalities to ensure equitable benefits.

OBJECTIVE

This scoping review aimed to map inequities in access, use, and engagement with digital health technologies across equity domains.

METHODS

We searched 4 electronic databases (MEDLINE, ASSIA, PsycINFO, and Scopus) for quantitative and mixed methods reviews and meta-analyses published between January 2016 and May 2022. Reviews were limited to those that included studies from the World Health Organization's European region. Extracted data were mapped against Cochrane's PROGRESS PLUS (place of residence, race, ethnicity, culture, and language, occupation, gender and sex, religion, education, socioeconomic status, social capital, and other characteristics) dimensions of equity.

RESULTS

In total, 404 unique citations were identified from the searches, and 2 citations were identified from other sources. After eligibility assessment, 22 reviews were included. Consistent evidence was found showing higher access to digital health technologies among patients who were of White ethnicity, were English speaking, and had no disability. There were no reviews that explored differences in access to digital health care by age, gender and sex, occupation, education, or homeless or substance misuse. Higher use of digital health technologies was observed among populations that were White, English speaking, younger, with a higher level of education, of higher economic status, and residents in urban areas. No clear evidence of differences in the use of digital technologies by occupation, gender and sex, disability, or homeless or substance misuse was found, nor was clear evidence found in the included reviews on inequalities in the engagement with digital technologies. Finally, no reviews were identified that explored differences by place of residence.

CONCLUSIONS

Despite awareness of the potential impact of inequalities in digital health, there are important evidence gaps across multiple equity domains. The development of a common framework for evaluating digital health equity in new health initiatives and consistency in reporting findings is needed.

Engagement With the Centers for Disease Control and Prevention Coronavirus Self-Checker and Guidance Provided to Users in the United States From March 23, 2020, to April 19, 2021: Thematic and Trend Analysis

BACKGROUND

In 2020, at the onset of the COVID-19 pandemic, the United States experienced surges in healthcare needs, which challenged capacity throughout the healthcare system. Stay-at-home orders in many jurisdictions, cancellation of elective procedures, and closures of outpatient medical offices disrupted patient access to care. To inform symptomatic persons about when to seek care and potentially help alleviate the burden on the healthcare system, Centers for Disease Control and Prevention (CDC) and partners developed the CDC Coronavirus Self-Checker ("Self-Checker"). This interactive tool assists individuals seeking information about COVID-19 to determine the appropriate level of care by asking demographic, clinical, and nonclinical questions during an online "conversation."

OBJECTIVE

This paper describes user characteristics, trends in use, and recommendations delivered by the Self-Checker between March 23, 2020, and April 19, 2021, for pursuing appropriate levels of medical care depending on the severity of user symptoms.

METHODS

User characteristics and trends in completed conversations that resulted in a care message were analyzed. Care messages delivered by the Self-Checker were manually classified into three overarching conversation themes: (1) seek care immediately; (2) take no action, or stay home and self-monitor; and (3) conversation redirected. Trends in 7-day averages of conversations and COVID-19 cases were examined with development and marketing milestones that potentially impacted Self-Checker user engagement.

RESULTS

Among 16,718,667 completed conversations, the Self-Checker delivered recommendations for 69.27% (n=11,580,738) of all conversations to "take no action, or stay home and self-monitor"; 28.8% (n=4,822,138) of conversations to "seek care immediately"; and 1.89% (n=315,791) of conversations were redirected to other resources without providing any care advice. Among 6.8 million conversations initiated for self-reported sick individuals without life-threatening symptoms, 59.21% resulted in a recommendation to "take no action, or stay home and self-monitor." Nearly all individuals (99.8%) who were not sick were also advised to "take no action, or stay home and self-monitor."

CONCLUSIONS

The majority of Self-Checker conversations resulted in advice to take no action, or stay home and self-monitor. This guidance may have reduced patient volume on the medical system; however, future studies evaluating patients' satisfaction, intention to follow the care advice received, course of action, and care modality pursued could clarify the impact of the Self-Checker and similar tools during future public health emergencies.

Rapid response nursing triage outcomes for COVID-19: factors associated with patient's participation in triage recommendations

BACKGROUND

COVID-19 is an ongoing global health crisis with prevention and treatment recommendations rapidly changing. Rapid response telephone triage and advice services are critical in providing timely care during pandemics. Understanding patient participation with triage recommendations and factors associated with patient participation can assist in developing sensitive and timely interventions for receiving the treatment to prevent adverse health effects of COVID-19.

METHODS

This cohort study aimed to assess patient participation (percentage of patients who followed nursing triage suggestions from the COVID hotline) and identify factors associated with patient participation in four quarterly electronic health records from March 2020 to March 2021 (Phase 1: 14 March 2020-6 June 2020; Phase 2: 17 June 2020-16 September 2020; Phase 3: 17 September 2020-16 December 2020; Phase 4: 17 December 2020-16 March 2021). All callers who provided their symptoms (including asymptomatic with exposure to COVID) and received nursing triage were included in the study. Factors associated with patient participation were identified using multivariable logistic regression analyses, including demographic variables, comorbidity variables, health behaviors, and COVID-19-related symptoms.

RESULTS

The aggregated data included 9849 encounters/calls from 9021 unique participants. Results indicated: (1) 72.5% of patient participation rate; (2) participants advised to seek emergency department care had the lowest patient participation rate (43.4%); (3) patient participation was associated with older age, a lower comorbidity index, a lack of unexplained muscle aches, and respiratory symptoms. The absence of respiratory symptoms was the only factor significantly associated with patient participation in all four phases (OR = 0.75, 0.60, 0.64, 0.52, respectively). Older age was associated with higher patient participation in three out of four phases (OR = 1.01-1.02), and a lower Charlson comorbidity index was associated with higher patient participation in phase 3 and phase 4 (OR = 0.83, 0.88).

CONCLUSION

Public participation in nursing triage during the COVID pandemic requires attention. This study supports using a nurse-led telehealth intervention and reveals crucial factors associated with patient participation. It highlighted the importance of timely follow-up in high-risk groups and the benefit of a telehealth intervention led by nurses serving as healthcare navigators during the COVID-19 pandemic.

Exploratory study: Evaluation of a symptom checker effectiveness for providing a diagnosis and evaluating the situation emergency compared to emergency physicians using simulated and standardized patients

BACKGROUND

The overloading of health care systems is an international problem. In this context, new tools such as symptom checker (SC) are emerging to improve patient orientation and triage. This SC should be rigorously evaluated and we can take a cue from the way we evaluate medical students, using objective structured clinical examinations (OSCE) with simulated patients.

OBJECTIVE

The main objective of this study was to evaluate the efficiency of a symptom checker versus emergency physicians using OSCEs as an assessment method.

METHODS

We explored a method to evaluate the ability to set a diagnosis and evaluate the emergency of a situation with simulation. A panel of medical experts wrote 220 simulated patients cases. Each situation was played twice by an actor trained to the role: once for the SC, then for an emergency physician. Like a teleconsultation, only the patient's voice was accessible. We performed a prospective non-inferiority study. If primary analysis had failed to detect non-inferiority, we have planned a superiority analysis.

RESULTS

The SC established only 30% of the main diagnosis as the emergency physician found 81% of these. The emergency physician was also superior compared to the SC in the suggestion of secondary diagnosis (92% versus 52%). In the matter of patient triage (vital emergency or not), there is still a medical superiority (96% versus 71%). We prove a non-inferiority of the SC compared to the physician in terms of interviewing time.

CONCLUSIONS AND RELEVANCE

We should use simulated patients instead of clinical cases in order to evaluate the effectiveness of SCs.

A multinational survey of patient utilization of and value conveyed through virtual symptom triage and healthcare referral

Objective

To describe the use patterns, impact and derived patient-user value of a mobile web-based virtual triage/symptom checker.

Methods

Online survey of 2,113 web-based patient-users of a virtual triage/symptom checker was completed over an 8-week period. Questions focused on triage and care objectives, pre- and post-triage care intent, frequency of use, value derived and satisfaction with virtual triage. Responses were analyzed and stratified to characterize patient-user pre-triage and post-triage intent relative to triage engine output.

Results

Seventy-eight percent of virtual triage users were female, and 37% were 18-24 years old or younger, 28% were 25-44, 16% were 45-54, and 19% were 55 years or older; 41.2% completed the survey from the U.S., 12.5% from the U.K., 9.1% from Canada, 5.6% from India, 3.8% from South Africa. Motivations were to determine need to consult a physician (44.2%), to secure medical advice without visiting a physician (21.0%), and to confirm a diagnosis received (14.2%). Forty-three percent were first time users of virtual triage, 36.6% utilized a triage engine at least once every few months or more often. Pre-triage, 40.5% did not know what level of healthcare they were planning to utilize, 33.9% stated they intended to seek a physician consultation, 23.7% engage self-care and 1.8% seek emergency care. Virtual triage recommended 56.8% of patient-users consult a physician, 33.8% seek emergency care and 9.4% engage self-care. In three-fourths, virtual triage helped users decide level of care to pursue. Among 74.1%, triage recommended care different than pre-triage intentions. Post-triage, those who remained uncertain of their care path decreased by 25.4%. Patient-user experience and satisfaction with virtual triage was high, with 80.1% stating that they were highly likely or likely to use it again, and interest in and willingness to use telemedicine doubled.

Conclusion

Virtual triage successfully redirected patient-users who initially planned to seek an inappropriate level of care acuity, reduced patient uncertainty of care path, and doubled the percentage of patients amenable to telemedicine and virtual health engagement. Patient-users were highly satisfied with virtual triage and the virtual triage patient experience, and a large majority will use virtual triage recurrently in the future.

A scoping review on the use and usefulness of online symptom checkers and triage systems: How to proceed?

Background

Patients are increasingly turning to the Internet for health information. Numerous online symptom checkers and digital triage tools are currently available to the general public in an effort to meet this need, simultaneously acting as a demand management strategy to aid the overburdened health care system. The implementation of these services requires an evidence-based approach, warranting a review of the available literature on this rapidly evolving topic.

Objective

This scoping review aims to provide an overview of the current state of the art and identify research gaps through an analysis of the strengths and weaknesses of the presently available literature.

Methods

A systematic search strategy was formed and applied to six databases: Cochrane library, NICE, DARE, NIHR, Pubmed, and Web of Science. Data extraction was performed by two researchers according to a pre-established data charting methodology allowing for a thematic analysis of the results.

Results

A total of 10,250 articles were identified, and 28 publications were found eligible for inclusion. Users of these tools are often younger, female, more highly educated and technologically literate, potentially impacting digital divide and health equity. Triage algorithms remain risk-averse, which causes challenges for their accuracy. Recent evolutions in algorithms have varying degrees of success. Results on impact are highly variable, with potential effects on demand, accessibility of care, health literacy and syndromic surveillance. Both patients and healthcare providers are generally positive about the technology and seem amenable to the advice given, but there are still improvements to be made toward a more patient-centered approach. The significant heterogeneity across studies and triage systems remains the primary challenge for the field, limiting transferability of findings.

Conclusion

Current evidence included in this review is characterized by significant variability in study design and outcomes, highlighting the significant challenges for future research.An evolution toward more homogeneous methodologies, studies tailored to the intended setting, regulation and standardization of evaluations, and a patient-centered approach could benefit the field.

Dr. Google: Physicians-The Web-Patients Triangle: Digital Skills and Attitudes towards e-Health Solutions among Physicians in South Eastern Poland-A Cross-Sectional Study in a Pre-COVID-19 Era

The investment in digital e-health services is a priority direction in the development of global healthcare systems. While people are increasingly using the Web for health information, it is not entirely clear what physicians' attitudes are towards digital transformation, as well as the acceptance of new technologies in healthcare. The aim of this cross-sectional survey study was to investigate physicians' self-digital skills and their opinions on obtaining online health knowledge by patients, as well as the recognition of physicians' attitudes towards e-health solutions. Principal component analysis (PCA) was performed to emerge the variables from self-designed questionnaire and cross-sectional analysis, comparing descriptive statistics and correlations for dependent variables using the one-way ANOVA (F-test). A total of 307 physicians participated in the study, reported as using the internet mainly several times a day (66.8%). Most participants (70.4%) were familiar with new technologies and rated their e-health literacy high, although 84.0% reported the need for additional training in this field and reported a need to introduce a larger number of subjects shaping digital skills (75.9%). 53.4% of physicians perceived Internet-sourced information as sometimes reliable and, in general, assessed the effects of its use by their patients negatively (41.7%). Digital skills increased significantly with frequency of internet use (F = 13.167; p = 0.0001) and decreased with physicians' age and the need for training. Those who claimed that patients often experienced health benefits from online health showed higher digital skills (-1.06). Physicians most often recommended their patients to obtain laboratory test results online (32.2%) and to arrange medical appointments via the Internet (27.0%). Along with the deterioration of physicians' digital skills, the recommendation of e-health solutions decreased (r = 0.413) and lowered the assessment of e-health solutions for the patient (r = 0.449). Physicians perceive digitization as a sign of the times and frequently use its tools in daily practice. The evaluation of Dr. Google's phenomenon and online health is directly related to their own e-health literacy skills, but there is still a need for practical training to deal with the digital revolution.

Telehealth for COVID-19: A Conceptual Framework

The world has been going through the global crisis of the coronavirus (COVID-19). It is a challenging situation for every country to tackle its healthcare system. COVID-19 spreads through physical contact with COVID-positive patients and causes potential damage to the country's health and economy system. Therefore, to overcome the chance of spreading the disease, the only preventive measure is to maintain social distancing. In this vulnerable situation, virtual resources have been utilized in order to maintain social distance, i.e., the telehealth system has been proposed and developed to access healthcare services remotely and manage people's health conditions. The telehealth system could become a regular part of our healthcare system, and during any calamity or natural disaster, it could be used as an emergency response to deal with the catastrophe. For this purpose, we proposed a conceptual telehealth framework in response to COVID-19. We focused on identifying critical issues concerning the use of telehealth in healthcare setups. Furthermore, the factors influencing the implementation of the telehealth system have been explored in detail. The proposed telehealth system utilizes artificial intelligence and data science to regulate and maintain the system efficiently. Before implementing the telehealth system, it is required that prearrangements be made, such as appropriate funding measures, the skills to know technological usage, training sessions, and staff endorsement. The barriers and influencing factors provided in this article can be helpful for future developments in telehealth systems and for making fruitful progress in fighting pandemics like COVID-19. At the same time, the same approach can be used to save the lives of many frontline workers.

How suitable are clinical vignettes for the evaluation of symptom checker apps? A test theoretical perspective

Objective

To evaluate the ability of case vignettes to assess the performance of symptom checker applications and to suggest refinements to the methodology used in case vignette-based audit studies.

Methods

We re-analyzed the publicly available data of two prominent case vignette-based symptom checker audit studies by calculating common metrics of test theory. Furthermore, we developed a new metric, the Capability Comparison Score (CCS), which compares symptom checker capability while controlling for the difficulty of the set of cases each symptom checker evaluated. We then scrutinized whether applying test theory and the CCS altered the performance ranking of the investigated symptom checkers.

Results

In both studies, most symptom checkers changed their rank order when adjusting the triage capability for item difficulty (ID) with the CCS. The previously reported triage accuracies commonly overestimated the capability of symptom checkers because they did not account for the fact that symptom checkers tend to selectively appraise easier cases (i.e., with high ID values). Also, many case vignettes in both studies showed insufficient (very low and even negative) values of item-total correlation (ITC), suggesting that individual items or the composition of item sets are of low quality.

Conclusions

A test-theoretic perspective helps identify previously undetected threats to the validity of case vignette-based symptom checker assessments and provides guidance and specific metrics to improve the quality of case vignettes, in particular by controlling for the difficulty of the vignettes an app was (not) able to evaluate correctly. Such measures might prove more meaningful than accuracy alone for the competitive assessment of symptom checkers. Our approach helps elaborate and standardize the methodology used for appraising symptom checker capability, which, ultimately, may yield more reliable results.

The place of digital triage in a complex healthcare system: An interview study with key stakeholders in Australia's national provider

Background

Digital triage tools such as telephone advice and online symptom checkers are now commonplace in health systems internationally. Research has focused on consumers' adherence to advice, health outcomes, satisfaction, and the degree to which these services manage demand for general practice or emergency departments. Such studies have had mixed findings, leaving equivocal the role of these services in healthcare.

Objective

We examined stakeholders' perspectives on Healthdirect, Australia's national digital triage provider, focusing on its role in the health system, and barriers to operation, in the context of the COVID-19 pandemic.

Methods

Key stakeholders took part in semi-structured interviews conducted online in the third quarter of 2021. Transcripts were coded and thematically analysed.

Results

Participants (n  =  41) were Healthdirect staff (n  =  13), employees of Primary Health Networks (PHNs; n  =  12), clinicians (n  =  9), shareholder representatives (n  =  4), consumer representatives (n  =  2) and other policymakers (n  =  1). Eight themes emerged from the analysis: (1) information and guidance in navigating the system, (2) efficiency through appropriate care, (3) value for consumers? (4) the difficulties in triage at a distance, (5) competition and the unfulfilled promise of integration, (6) challenges in promoting Healthdirect, (7) monitoring and evaluating digital triage services and (8) rapid change, challenge and opportunity from COVID-19.

Conclusion

Stakeholders varied in their views of the purpose of Healthdirect's digital triage services. They identified challenges in lack of integration, competition, and the limited public profile of the services, issues largely reflective of the complexity of the policy and health system landscape. There was acknowledgement of the value of the services during the COVID-19 pandemic, and an expectation of them realising greater potential in the wake of the rapid uptake of telehealth.

Online Self-Triage of Ear or Hearing Concerns in a Patient Portal: Comparison of Subsequent Diagnoses and Hospitalizations to National Emergency Department and National Ambulatory Ear or Hearing Visits

Background

Although online self-triage is easily accessible, little is known about the patients who use self-triage or their subsequent diagnoses. We compared ear/hearing self-triage subsequent diagnoses to ear/hearing visit diagnoses in emergency departments (ED) and ambulatory clinics across the United States.

Methods

We compared International Classification of Diseases version 10 (ICD10) coded diagnoses following online self-triage for ear/hearing concerns with those from national ED and ambulatory clinic samples. We used data from the Centers for Disease Control (CDC) National Hospital Ambulatory Medical Care Survey (NHAMCS) and National Ambulatory Medical Care Survey (NAMCS) for comparison. Using matched ear/hearing diagnostic categories for those aged 1 and over, we compared self-triage diagnosis frequencies with national ED and ambulatory diagnosis frequencies.

Results

Following ear/hearing self-triage, there were 1092 subsequent office visits with a primary diagnosis code. For five frequently diagnosed ear/hearing conditions (i.e., suppurative and nonsuppurative otitis media [OM], otalgia, otitis externa, and cerumen impaction), there was a strong correlation between diagnosis counts made following self-triage and estimated counts of national ED visit diagnoses (r = 0.94; CI 95% [0.37 to 0.99]; p = .016, adjusted r2 = 0.85). Seven diagnoses were available to compare with the national ambulatory sample; correlation was r = 0.79; CI 95% [0.08 to 0.97]; p = .037, adjusted r2 = 0.54. For ages 1 and over, estimated hospital admissions from the national ED visits for ear/hearing were 0.76%, CI 95% [0.28-2.1%]; estimated total national ear/hearing ED visits were 7.5 million (for 4 years, 2016 through 2019).

Conclusion

The strong correlation of ear-related self-triage diagnoses with national ED diagnoses and the low hospitalization risk for these diagnoses suggests that there is an opportunity for self-triage of ear/hearing concerns to decrease ED visits for these symptoms.

Challenges in evaluating the accuracy of AI-containing digital triage systems: A systematic review

INTRODUCTION

Patient-operated digital triage systems with AI components are becoming increasingly common. However, previous reviews have found a limited amount of research on such systems' accuracy. This systematic review of the literature aimed to identify the main challenges in determining the accuracy of patient-operated digital AI-based triage systems.

METHODS

A systematic review was designed and conducted in accordance with PRISMA guidelines in October 2021 using PubMed, Scopus and Web of Science. Articles were included if they assessed the accuracy of a patient-operated digital triage system that had an AI-component and could triage a general primary care population. Limitations and other pertinent data were extracted, synthesized and analysed. Risk of bias was not analysed as this review studied the included articles' limitations (rather than results). Results were synthesized qualitatively using a thematic analysis.

RESULTS

The search generated 76 articles and following exclusion 8 articles (6 primary articles and 2 reviews) were included in the analysis. Articles' limitations were synthesized into three groups: epistemological, ontological and methodological limitations. Limitations varied with regards to intractability and the level to which they can be addressed through methodological choices. Certain methodological limitations related to testing triage systems using vignettes can be addressed through methodological adjustments, whereas epistemological and ontological limitations require that readers of such studies appraise the studies with limitations in mind.

DISCUSSION

The reviewed literature highlights recurring limitations and challenges in studying the accuracy of patient-operated digital triage systems with AI components. Some of these challenges can be addressed through methodology whereas others are intrinsic to the area of inquiry and involve unavoidable trade-offs. Future studies should take these limitations in consideration in order to better address the current knowledge gaps in the literature.

"Nothing works without the doctor:" Physicians' perception of clinical decision-making and artificial intelligence

Introduction

Artificial intelligence-driven decision support systems (AI-DSS) have the potential to help physicians analyze data and facilitate the search for a correct diagnosis or suitable intervention. The potential of such systems is often emphasized. However, implementation in clinical practice deserves continuous attention. This article aims to shed light on the needs and challenges arising from the use of AI-DSS from physicians' perspectives.

Methods

The basis for this study is a qualitative content analysis of expert interviews with experienced nephrologists after testing an AI-DSS in a straightforward usage scenario.

Results

The results provide insights on the basics of clinical decision-making, expected challenges when using AI-DSS as well as a reflection on the test run.

Discussion

While we can confirm the somewhat expectable demand for better explainability and control, other insights highlight the need to uphold classical strengths of the medical profession when using AI-DSS as well as the importance of broadening the view of AI-related challenges to the clinical environment, especially during treatment. Our results stress the necessity for adjusting AI-DSS to shared decision-making. We conclude that explainability must be context-specific while fostering meaningful interaction with the systems available.

Ethical, legal, and social aspects of symptom checker applications: a scoping review

Symptom Checker Applications (SCA) are mobile applications often designed for the end-user to assist with symptom assessment and self-triage. SCA are meant to provide the user with easily accessible information about their own health conditions. However, SCA raise questions regarding ethical, legal, and social aspects (ELSA), for example, regarding fair access to this new technology. The aim of this scoping review is to identify the ELSA of SCA in the scientific literature. A scoping review was conducted to identify the ELSA of SCA. Ten databases (e.g., Web of Science and PubMed) were used. Studies on SCA that address ELSA, written in English or German, were included in the review. The ELSA of SCA were extracted and synthesized using qualitative content analysis. A total of 25,061 references were identified, of which 39 were included in the analysis. The identified aspects were allotted to three main categories: (1) Technology; (2) Individual Level; and (3) Healthcare system. The results show that there are controversial debates in the literature on the ethical and social challenges of SCA usage. Furthermore, the debates are characterised by a lack of a specific legal perspective and empirical data. The review provides an overview on the spectrum of ELSA regarding SCA. It offers guidance to stakeholders in the healthcare system, for example, patients, healthcare professionals, and insurance providers and could be used in future empirical research to investigate the perspectives of those affected, such as users.

Assessing data gathering of chatbot based symptom checkers - a clinical vignettes study

BACKGROUND

The burden on healthcare systems is mounting continuously owing to population growth and aging, overuse of medical services, and the recent COVID-19 pandemic. This overload is also causing reduced healthcare quality and outcomes. One solution gaining momentum is the integration of intelligent self-assessment tools, known as symptom-checkers, into healthcare-providers' systems. To the best of our knowledge, no study so far has investigated the data-gathering capabilities of these tools, which represent a crucial resource for simulating doctors' skills in medical-interviews.

OBJECTIVES

The goal of this study was to evaluate the data-gathering function of currently available chatbot symptom-checkers.

METHODS

We evaluated 8 symptom-checkers using 28 clinical vignettes from the repository of MSD-Manual case studies. The mean number of predefined pertinent findings for each case was 31.8 ± 6.8. The vignettes were entered into the platforms by 3 medical students who simulated the role of the patient. For each conversation, we obtained the number of pertinent findings retrieved and the number of questions asked. We then calculated the recall-rates (pertinent-findings retrieved out of all predefined pertinent-findings), and efficiency-rates (pertinent-findings retrieved out of the number of questions asked) of data-gathering, and compared them between the platforms.

RESULTS

The overall recall rate for all symptom-checkers was 0.32(2,280/7,112;95 %CI 0.31-0.33) for all pertinent findings, 0.37(1,110/2,992;95 %CI 0.35-0.39) for present findings, and 0.28(1140/4120;95 %CI 0.26-0.29) for absent findings. Among the symptom-checkers, Kahun platform had the highest recall rate with 0.51(450/889;95 %CI 0.47-0.54). Out of 4,877 questions asked overall, 2,280 findings were gathered, yielding an efficiency rate of 0.46(95 %CI 0.45-0.48) across all platforms. Kahun was the most efficient tool 0.74 (95 %CI 0.70-0.77) without a statistically significant difference from Your.MD 0.69(95 %CI 0.65-0.73).

CONCLUSION

The data-gathering performance of currently available symptom checkers is questionable. From among the tools available, Kahun demonstrated the best overall performance.

Utilization patterns and efficiency gains from use of a fully EHR-integrated COVID-19 self-triage and self-scheduling tool: a retrospective analysis

OBJECTIVE

Symptom checkers can help address high demand for SARS-CoV2 (COVID-19) testing and care by providing patients with self-service access to triage recommendations. However, health systems may be hesitant to invest in these tools, as their associated efficiency gains have not been studied. We aimed to quantify the operational efficiency gains associated with use of an online COVID-19 symptom checker as an alternative to a telephone hotline.

METHODS

In our health system, ambulatory patients can either use an online symptom checker or a telephone hotline to be triaged and connected to COVID-19 care. We performed a retrospective analysis of adults who used either method between October 20, 2021 and January 10, 2022, using call logs, electronic health record data, and local wages to calculate labor costs.

RESULTS

Of the 15 549 total COVID-19 triage encounters, 1820 (11.7%) used only the telephone hotline and 13 729 (88.3%) used the symptom checker. Only 271 (2%) of the patients who used the symptom checker also called the hotline. Hotline encounters required more clinician time compared to those involving the symptom checker (17.8 vs 0.4 min/encounter), resulting in higher average labor costs ($24.21 vs $0.55 per encounter). The symptom checker resulted in over 4200 clinician labor hours saved.

CONCLUSION

When given the option, most patients completed COVID-19 triage and visit scheduling online, resulting in substantial efficiency gains. These benefits may encourage health system investment in such tools.

Online Symptom Checkers: Recommendations for a Vignette-Based Clinical Evaluation Standard

The use of patient-facing online symptom checkers (OSCs) has expanded in recent years, but their accuracy, safety, and impact on patient behaviors and health care systems remain unclear. The lack of a standardized process of clinical evaluation has resulted in significant variation in approaches to OSC validation and evaluation. The aim of this paper is to characterize a set of congruent requirements for a standardized vignette-based clinical evaluation process of OSCs. Discrepancies in the findings of comparative studies to date suggest that different steps in OSC evaluation methodology can significantly influence outcomes. A standardized process with a clear specification for vignette-based clinical evaluation is urgently needed to guide developers and facilitate the objective comparison of OSCs. We propose 15 recommendation requirements for an OSC evaluation standard. A third-party evaluation process and protocols for prospective real-world evidence studies should also be prioritized to quality assure OSC assessment.

WtsWrng Interim Comparative Effectiveness Evaluation and Description of the Challenges to Develop, Assess, and Introduce This Novel Digital Application in a Traditional Health System

Science and technology have evolved quickly during the two decades of the 21st century, but healthcare systems are grounded in last century's structure and processes. Changes in the way health care is provided are demanded; digital transformation is a key driver making healthcare systems more accessible, agile, efficient, and citizen-centered. Nevertheless, the way healthcare systems function challenges the development (Innovation + Development and regulatory requirements), assessment (methodological guidance weaknesses), and adoption of digital applications (DAs). WtsWrng (WW), an innovative DA which uses images to interact with citizens for symptom triage and monitoring, is used as an example to show the challenges faced in its development and clinical validation and how these are being overcome. To prove WW's value from inception, novel approaches for evidence generation that allows for an agile and patient-centered development have been applied. Early scientific advice from NICE (UK) was sought for study design, an iterative development and interim analysis was performed, and different statistical parameters (Kappa, B statistic) were explored to face development and assessment challenges. WW triage accuracy at cutoff time ranged from 0.62 to 0.94 for the most frequent symptoms attending the Emergency Department (ED), with the observed concordance for the 12 most frequent diagnostics at hospital discharge fluctuating between 0.4 to 0.97; 8 of the diagnostics had a concordance greater than 0.8. This experience should provoke reflective thinking for DA developers, digital health scientists, regulators, health technology assessors, and payers.

Evaluation of Diagnostic and Triage Accuracy and Usability of a Symptom Checker in an Emergency Department: Observational Study

Background

Symptom checkers are clinical decision support apps for patients, used by tens of millions of people annually. They are designed to provide diagnostic and triage advice and assist users in seeking the appropriate level of care. Little evidence is available regarding their diagnostic and triage accuracy with direct use by patients for urgent conditions.

Objective

The aim of this study is to determine the diagnostic and triage accuracy and usability of a symptom checker in use by patients presenting to an emergency department (ED).

Methods

We recruited a convenience sample of English-speaking patients presenting for care in an urban ED. Each consenting patient used a leading symptom checker from Ada Health before the ED evaluation. Diagnostic accuracy was evaluated by comparing the symptom checker’s diagnoses and those of 3 independent emergency physicians viewing the patient-entered symptom data, with the final diagnoses from the ED evaluation. The Ada diagnoses and triage were also critiqued by the independent physicians. The patients completed a usability survey based on the Technology Acceptance Model.

Results

A total of 40 (80%) of the 50 participants approached completed the symptom checker assessment and usability survey. Their mean age was 39.3 (SD 15.9; range 18-76) years, and they were 65% (26/40) female, 68% (27/40) White, 48% (19/40) Hispanic or Latino, and 13% (5/40) Black or African American. Some cases had missing data or a lack of a clear ED diagnosis; 75% (30/40) were included in the analysis of diagnosis, and 93% (37/40) for triage. The sensitivity for at least one of the final ED diagnoses by Ada (based on its top 5 diagnoses) was 70% (95% CI 54%-86%), close to the mean sensitivity for the 3 physicians (on their top 3 diagnoses) of 68.9%. The physicians rated the Ada triage decisions as 62% (23/37) fully agree and 24% (9/37) safe but too cautious. It was rated as unsafe and too risky in 22% (8/37) of cases by at least one physician, in 14% (5/37) of cases by at least two physicians, and in 5% (2/37) of cases by all 3 physicians. Usability was rated highly; participants agreed or strongly agreed with the 7 Technology Acceptance Model usability questions with a mean score of 84.6%, although “satisfaction” and “enjoyment” were rated low.

Conclusions

This study provides preliminary evidence that a symptom checker can provide acceptable usability and diagnostic accuracy for patients with various urgent conditions. A total of 14% (5/37) of symptom checker triage recommendations were deemed unsafe and too risky by at least two physicians based on the symptoms recorded, similar to the results of studies on telephone and nurse triage. Larger studies are needed of diagnosis and triage performance with direct patient use in different clinical environments.

Understanding the invisible workforce: lessons for general practice from a survey of receptionists

Introduction

The significance of the role of receptionists during the recent shift to remote triage has been widely recognised and they will have a significant role to play in UK general practice as it continues to cope with a huge increase in demand exacerbated by the COVID-19 pandemic. To maximise their contribution, it is important the social and occupational characteristics of the modern receptionist are understood, alongside their attitudes towards the role and their perceptions of the support and training they receive .

Methods

We used convenience and cross-sectional sampling to survey the demographic characteristics of receptionists and various aspects of their role and responsibilities. This included the training received, specific tasks performed, job satisfaction, the importance of the role, and their interaction with clinical and non-clinical colleagues. We also captured data on the characteristics of their practice including the number of GPs and location.

Results

A total of 70 participants completed the survey (16 postal and 54 online responses) of whom the majority were white (97.2%), female (98.6%), and aged 40 and over (56.7%). The majority of the training focussed on customer service (72.9%), telephone (64.3%), and medical administration skills (58.6%). Just over a quarter had received training in basic triage (25.7%). A standard multiple regression model revealed that the strongest predictor of satisfaction was support from practice GPs (β = .65, p <.001) there were also significant positive correlations between satisfaction and appreciation from GPs, r(68) = .609, p < .001.

Conclusion

This study has provided a much-needed update on the demographics, duties, and job satisfaction of GP receptionists. The need for diversification of the workforce to reflect the range of primary care patients warrants consideration in light of continuing variation in access along lines of gender andethnicity. Training continues to focus on administrative duties not on the clinically relevant aspects of their role such as triage.

The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review

Digital and online symptom checkers are an increasingly adopted class of health technologies that enable patients to input their symptoms and biodata to produce a set of likely diagnoses and associated triage advice. However, concerns regarding the accuracy and safety of these symptom checkers have been raised. This systematic review evaluates the accuracy of symptom checkers in providing diagnoses and appropriate triage advice. MEDLINE and Web of Science were searched for studies that used either real or simulated patients to evaluate online or digital symptom checkers. The primary outcomes were the diagnostic and triage accuracy of the symptom checkers. The QUADAS-2 tool was used to assess study quality. Of the 177 studies retrieved, 10 studies met the inclusion criteria. Researchers evaluated the accuracy of symptom checkers using a variety of medical conditions, including ophthalmological conditions, inflammatory arthritides and HIV. A total of 50% of the studies recruited real patients, while the remainder used simulated cases. The diagnostic accuracy of the primary diagnosis was low across included studies (range: 19–37.9%) and varied between individual symptom checkers, despite consistent symptom data input. Triage accuracy (range: 48.8–90.1%) was typically higher than diagnostic accuracy. Overall, the diagnostic and triage accuracy of symptom checkers are variable and of low accuracy. Given the increasing push towards adopting this class of technologies across numerous health systems, this study demonstrates that reliance upon symptom checkers could pose significant patient safety hazards. Large-scale primary studies, based upon real-world data, are warranted to demonstrate the adequate performance of these technologies in a manner that is non-inferior to current best practices. Moreover, an urgent assessment of how these systems are regulated and implemented is required.

Protocol: Remote care as the 'new normal'? Multi-site case study in UK general practice

Background

Following a pandemic-driven shift to remote service provision, UK general practices offer telephone, video or online consultation options alongside face-to-face. This study explores practices' varied experiences over time as they seek to establish remote forms of accessing and delivering care.

Methods

This protocol is for a mixed-methods multi-site case study with co-design and national stakeholder engagement. 11 general practices were selected for diversity in geographical location, size, demographics, ethos, and digital maturity. Each practice has a researcher-in-residence whose role is to become familiar with its context and activity, follow it longitudinally for two years using interviews, public-domain documents and ethnography, and support improvement efforts. Research team members meet regularly to compare and contrast across cases. Practice staff are invited to join online learning events. Patient representatives work locally within their practice patient involvement groups as well as joining an online patient learning set or linking via a non-digital buddy system. NHS Research Ethics Approval has been granted. Governance includes a diverse independent advisory group with lay chair. We also have policy in-reach (national stakeholders sit on our advisory group) and outreach (research team members sit on national policy working groups).

Results anticipated

We expect to produce rich narratives of contingent change over time, addressing cross-cutting themes including access, triage and capacity; digital and wider inequities; quality and safety of care (e.g. continuity, long-term condition management, timely diagnosis, complex needs); workforce and staff wellbeing (including non-clinical staff, students and trainees); technologies and digital infrastructure; patient perspectives; and sustainability (e.g. carbon footprint).

Conclusion

By using case study methods focusing on depth and detail, we hope to explain why digital solutions that work well in one practice do not work at all in another. We plan to inform policy and service development through inter-sectoral network-building, stakeholder workshops and topic-focused policy briefings.

Patient Experience and Feedback after Use of an EHR-integrated COVID-19 Symptom Checker

BACKGROUND

Symptom checkers have been widely used during the COVID-19 pandemic to alleviate strain on health systems and offer patients a 24/7 self-service triage option. Although studies suggest that users may positively perceive online symptom checkers, no studies have quantified user feedback after use of an electronic health record (EHR)-integrated COVID-19 symptom checker with self-scheduling functionality.

OBJECTIVE

We aimed to understand user experience, user satisfaction, and user-reported alternatives to use of a COVID-19 symptom checker with self-triage and self-scheduling functionality.

METHODS

We launched a patient-portal based self-triage and self-scheduling tool in March 2020 for patients with COVID-19 symptoms, exposures, or questions. We made an optional, anonymous Qualtrics survey available to patients immediately after they completed the symptom checker.

RESULTS

Between December 16th, 2021 and March 28th, 2022, there were 395 unique responses to the survey. Overall, respondents reported high satisfaction across all demographics, with a median rating of 8 out of 10, and 47.6% of respondents giving a rating of 9 or 10 out of 10. User satisfaction scores were not associated with any demographic factors. The most common user-reported alternatives had the online tool not been available were calling the COVID-19 telephone hotline and sending a patient-portal message to their physician for advice. The ability to schedule a test online was the most important symptom checker feature for respondents. The most common categories of user feedback were regarding other COVID-19 services (e.g. telephone hotline), policies or procedures, or requesting additional features or functionality.

CONCLUSIONS

This analysis suggests that COVID-19 symptom checkers with self-triage and self-scheduling functionality may have high overall user satisfaction, regardless of user demographics. By allowing users to self-triage and self-schedule tests and visits, tools like this may prevent unnecessary calls and messages to clinicians. Individual feedback suggested that the user experience for this type of tool is highly dependent on the organization's operational workflows for COVID-19 testing and care. The study provides insight for the implementation and improvement of COVID-19 symptom checkers to ensure high user satisfaction.  .

A Novel Bayesian General Medical Diagnostic Assistant Achieves Superior Accuracy With Sparse History

Online AI symptom checkers and diagnostic assistants (DAs) have tremendous potential to reduce misdiagnosis and cost, while increasing the quality, convenience, and availability of healthcare, but only if they can perform with high accuracy. We introduce a novel Bayesian DA designed to improve diagnostic accuracy by addressing key weaknesses of Bayesian Network implementations for clinical diagnosis. We compare the performance of our prototype DA (MidasMed) to that of physicians and six other publicly accessible DAs (Ada, Babylon, Buoy, Isabel, Symptomate, and WebMD) using a set of 30 publicly available case vignettes, and using only sparse history (no exam findings or tests). Our results demonstrate superior performance of the MidasMed DA, with the correct diagnosis being the top ranked disorder in 93% of cases, and in the top 3 in 96% of cases.

Effects and utility of an online forward triage tool during the SARS-CoV-2 pandemic: a mixed method study and patient perspectives, Switzerland

OBJECTIVE

To assess the effects (quantitatively) and the utility (qualitatively) of a COVID-19 online forward triage tool (OFTT) in a pandemic context.

DESIGN

A mixed method sequential explanatory study was employed. Quantitative data of all OFTT users, between 2 March 2020 and 12 May 2020, were collected. Second, qualitative data were collected through key informant interviews (n=19) to explain the quantitative findings, explore tool utility, user experience and elicit recommendations.

SETTING

The working group e-emergency medicine at the emergency department developed an OFTT, which was made available online.

PARTICIPANTS

Participants included all users above the age of 18 that used the OFTT between 2 March 2020 and 12 May 2020.

INTERVENTION

An OFTT that displayed the current test recommendations of the Federal Office of Public Health on whether someone needed testing for COVID-19 or not. No diagnosis was provided.

RESULTS

In the study period, 6272 users consulted our OFTT; 40.2% (1626/4049) would have contacted a healthcare provider had the tool not existed. 560 participants consented to a follow-up survey and provided a valid email address. 31.4% (176/560) participants returned a complete follow-up questionnaire. 84.7% (149/176) followed the recommendations given. 41.5% (73/176) reported that their fear was allayed after using the tool. Qualitatively, seven overarching themes emerged namely (1) accessibility of tool, (2) user-friendliness of tool, (3) utility of tool as an information source, (4) utility of tool in allaying fear and anxiety, (5) utility of tool in medical decision-making (6) utility of tool in reducing the potential for onward transmissions and (7) utility of tool in reducing health system burden.

CONCLUSION

Our findings demonstrated that a COVID-19 OFTT does not only reduce the health system burden but can also serve as an information source, reduce anxiety and fear, reduce potential for cross infections and facilitate medical decision-making.

The Utility of a Pediatric COVID-19 Online Forward Triage Tool in Switzerland

Background

To offset the burden on the health system, hospitals set up telehealth interventions, such as online forward triage tools (OFTT). The website www.coronabambini.ch was developed to specifically address the needs of children and their families in Switzerland and to facilitate the decision to test, isolate, attend school, or access the health care system.

Methods

Video interviews were held with key informants (n = 20) from a population of parents, teachers, guardians, as well as doctors who had used the child-specific COVID-19 OFTT and had consented to a further study. Convenience and quota sampling were done to include a variety of key informants. Interviews were recorded, transcribed verbatim, and analyzed for themes.

Results

Three main themes emerged: i) the usefulness of the OFTT to the users, ii) expectation management and importance of stakeholder involvement in OFTT development, and iii) OFTT limitations.

Conclusion

Our study highlights opportunities, limitations, and lessons to consider when developing a pediatric COVID-19 OFTT. The involvement of stakeholders, parents, teachers, and health care providers in the design, set up, implementation, and evaluation of telehealth interventions is critical as this can help with expectation management and enhance OFTT utility.

Triage Errors in Primary and Pre-Primary Care

Triage errors are a major concern in health care due to resulting harmful delays in treatments or inappropriate allocation of resources. With the increasing popularity of digital symptom checkers in pre–primary care settings, and amid claims that artificial intelligence outperforms doctors, the accuracy of triage by digital symptom checkers is ever more scrutinized. This paper examines the context and challenges of triage in primary care, pre–primary care, and emergency care, as well as reviews existing evidence on the prevalence of triage errors in all three settings. Implications for development, research, and practice are highlighted, and recommendations are made on how digital symptom checkers should be best positioned.

Digital teaching tools in sports medicine: A randomized control trial comparing the effectiveness of virtual seminar and virtual fishbowl teaching method in medical students

Background

Since the COVID-19 pandemic, the demand for online courses has increased enormously. Therefore, finding new methods to improve medical education is imperative.

Objective

The aim of this study was to compare the self-reports of the individual student-centered virtual teaching techniques (seminar versus fishbowl) in a group of medical students.

Methods

During the second semester of 2020, students in the clinical phase of the study (n = 144) participated in the optional subject of Sports Medicine. The students were divided into 2 groups. One group (n = 72) received the knowledge transfer in the form of a virtual seminar, the other group (n = 72) in the form of a virtual fishbowl.

Results

Virtual seminar and virtual fishbowl students gave insights into these teaching techniques. Most of the students from the virtual fishbowl group believed that the virtual fishbowl format allowed them to be more actively involved in learning. The mean quiz scores were statistically higher for students in the virtual fishbowl group than students in the virtual seminar group (p < 0. 001).

Conclusion

This study concluded that virtual seminars and virtual fishbowl formats could be served as structured learning and teaching formats. At the same time, the virtual fishbowl format can promote an active exchange of knowledge from students’ perspectives.

Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation

Background

Symptom checkers are digital tools assisting laypersons in self-assessing the urgency and potential causes of their medical complaints. They are widely used but face concerns from both patients and health care professionals, especially regarding their accuracy. A 2015 landmark study substantiated these concerns using case vignettes to demonstrate that symptom checkers commonly err in their triage assessment.

Objective

This study aims to revisit the landmark index study to investigate whether and how symptom checkers’ capabilities have evolved since 2015 and how they currently compare with laypersons’ stand-alone triage appraisal.

Methods

In early 2020, we searched for smartphone and web-based applications providing triage advice. We evaluated these apps on the same 45 case vignettes as the index study. Using descriptive statistics, we compared our findings with those of the index study and with publicly available data on laypersons’ triage capability.

Results

We retrieved 22 symptom checkers providing triage advice. The median triage accuracy in 2020 (55.8%, IQR 15.1%) was close to that in 2015 (59.1%, IQR 15.5%). The apps in 2020 were less risk averse (odds 1.11:1, the ratio of overtriage errors to undertriage errors) than those in 2015 (odds 2.82:1), missing >40% of emergencies. Few apps outperformed laypersons in either deciding whether emergency care was required or whether self-care was sufficient. No apps outperformed the laypersons on both decisions.

Conclusions

Triage performance of symptom checkers has, on average, not improved over the course of 5 years. It decreased in 2 use cases (advice on when emergency care is required and when no health care is needed for the moment). However, triage capability varies widely within the sample of symptom checkers. Whether it is beneficial to seek advice from symptom checkers depends on the app chosen and on the specific question to be answered. Future research should develop resources (eg, case vignette repositories) to audit the capabilities of symptom checkers continuously and independently and provide guidance on when and to whom they should be recommended.

Online Health Information Seeking for Self and Child: An Experimental Study of Parental Symptom Search

BACKGROUND

Parents often search the web for health-related information for themselves or on behalf of their children, which may impact their health-related decision-making and behaviors. In particular, searching for somatic symptoms such as headaches, fever, or fatigue is common. However, little is known about how psychological and relational factors relate to the characteristics of successful symptom-related internet searches. To date, few studies have used experimental designs that connect participant subjective search evaluation with objective search behavior metrics.

OBJECTIVE

This study aimed to examine the features of web-based health-related search behaviors based on video-coded observational data, to investigate which psychological and relational factors are related to successful symptom search appraisal, and to examine the differences in search-related outcomes among self-seekers and by-proxy seekers.

METHODS

In a laboratory setting, parents living in Austria (N=46) with a child aged between 0 and 6 years were randomized to search their own (n=23, 50%) or their child's (n=23, 50%) most recent somatic symptom on the web. Web-based activity was recorded and transcribed. Health anxiety, eHealth literacy, attitude toward web-based health information, relational variables, state of stress, participants' search appraisals, and quantitative properties of the search session were assessed. Differences in search appraisals and search characteristics among parents who searched for themselves or their children were examined.

RESULTS

Across both groups, searches were carried out for 17 different symptom clusters. Almost all parents started with Google (44/46, 96%), and a majority used initial elaborated key phrases with >1 search keyword (38/45, 84%) and performed on average 2.95 (SD 1.83) search queries per session. Search success was negatively associated with health anxiety (r_s=-0.39, P=.01), stress after the search (r_s=-0.33, P=.02), and the number of search queries (r_s=-0.29, P=.04) but was not significantly associated with eHealth literacy (r_s=0.22, P=.13). Of note, eHealth literacy was strongly and positively correlated with satisfaction during the search (r_s=0.50, P<.001) but did not significantly correlate with search characteristics as measured by search duration (r_s=0.08, P=.56), number of performed search queries (r_s=0.20, P=.17), or total clicks (r_s=0.14, P=.32). No differences were found between parents searching for their own symptoms and parents searching for their child's symptoms.

CONCLUSIONS

This study provides exploratory findings regarding relevant dimensions of appraisals for symptom-based information seeking on the web. Consistent with previous literature, health anxiety was found to be associated with poorer search evaluation. Contrary to expectations, eHealth literacy was related neither to search success nor to search characteristics. Interestingly, we did not find significant differences between self-seekers and by-proxy seekers, suggesting similar search and evaluation patterns in our sample. Further research with larger samples is needed to identify and evaluate guidelines for enhanced web-based health information seeking among parents and the general public.

Health Care Professionals' Experiences of Web-Based Symptom Checkers for Triage: Cross-sectional Survey Study

Background

Web-based symptom checkers are promising tools that provide help to patients seeking guidance on health problems. Many health organizations have started using them to enhance triage. Patients use the symptom checker to report their symptoms online and submit the report to the health care center through the system. Health care professionals (registered nurse, practical nurse, general physician, physiotherapist, etc) receive patient inquiries with urgency rating, decide on actions to be taken, and communicate these to the patients. The success of the adoption, however, depends on whether the tools can efficiently support health care professionals’ workflow and achieve their support.

Objective

This study explores the factors influencing health care professionals’ support for a web-based symptom checker for triage.

Methods

Data were collected through a web-based survey of 639 health care professionals using either of the two most used web-based symptom checkers in the Finnish public primary care. Linear regression models were fitted to study the associations between the study variables and health care professionals’ support for the symptom checkers. In addition, the health care professionals’ comments collected via survey were qualitatively analyzed to elicit additional insights about the benefits and challenges of the clinical use of symptom checkers.

Results

Results show that the perceived beneficial influence of the symptom checkers on health care professionals’ work and the perceived usability of the tools were positively associated with professionals’ support. The perceived benefits to patients and organizational support for use were positively associated, and threat to professionals’ autonomy was negatively associated with health care professionals’ support. These associations were, however, not independent of other factors included in the models. The influences on professionals’ work were both positive and negative; the tools streamlined work by providing preliminary information on patients and reduced the number of phone calls, but they also created extra work as the professionals needed to call patients and ask clarifying questions. Managing time between the use of symptom checkers and other tasks was also challenging. Meanwhile, according to health care professionals’ experience, the symptom checkers benefited patients as they received help quickly with a lower threshold for care.

Conclusions

The efficient use of symptom checkers for triage requires usable solutions that support health care professionals’ work. High-quality information about the patients’ conditions and an efficient way of communicating with patients are needed. Using a new eHealth tool also requires that health organizations and teams reorganize their workflows and work distributions to support clinical processes.

What is the suitability of clinical vignettes in benchmarking the performance of online symptom checkers? An audit study

OBJECTIVE

Assess the suitability of clinical vignettes in benchmarking the performance of online symptom checkers (OSCs).

DESIGN

Observational study using a publicly available free OSC.

PARTICIPANTS

Healthily OSC, which provided consultations in English, was used to record consultation outcomes from two lay and four expert inputters using 139 standardised patient vignettes. Each vignette included three diagnostic solutions and a triage recommendation in one of three categories of triage urgency. A panel of three independent general practitioners interpreted the vignettes to arrive at an alternative set of diagnostic and triage solutions. Both sets of diagnostic and triage solutions were consolidated to arrive at a final consolidated version for benchmarking.

MAIN OUTCOME MEASURES

Six inputters simulated 834 standardised patient evaluations using Healthily OSC and recorded outputs (triage solution, signposting, and whether the correct diagnostic solution appeared first or within the first three differentials). We estimated Cohen's kappa to assess how interpretations by different inputters could lead to divergent OSC output even when using the same vignette or when compared with a separate panel of physicians.

RESULTS

There was moderate agreement on triage recommendation (kappa=0.48), and substantial agreement on consultation outcomes between all inputters (kappa=0.73). OSC performance improved significantly from baseline when compared against the final consolidated diagnostic and triage solution (p<0.001).

CONCLUSIONS

Clinical vignettes are inherently limited in their utility to benchmark the diagnostic accuracy or triage safety of OSC. Real-world evidence studies involving real patients are recommended to benchmark the performance of OSC against a panel of physicians.

A Utility Framework for COVID-19 Online Forward Triage Tools: A Swiss Telehealth Case Study

The SARS-CoV-2 pandemic caused a surge in online tools commonly known as symptom checkers. The purpose of these symptom checkers was mostly to reduce the health system burden by providing worried people with testing criteria, where to test and how to self-care. Technical, usability and organizational challenges with regard to online forward triage tools have also been reported. Very few of these online forward triage tools have been evaluated. Evidence for decision frameworks may be of particular value in a pandemic setting where time frames are restricted, uncertainties are ubiquitous and the evidence base is changing rapidly. The objective was to develop a framework to evaluate the utility of COVID-19 online forward triage tools. The development of the online forward triage tool utility framework was conducted in three phases. The process was guided by the socio-ecological framework for adherence that states that patient (individual), societal and broader structural factors affect adherence to the tool. In a further step, pragmatic incorporation of themes on the utility of online forward triage tools that emerged from our study as well as from the literature was performed. Seven criteria emerged; tool accessibility, reliability as an information source, medical decision-making aid, allaying fear and anxiety, health system burden reduction, onward forward transmission reduction and systems thinking (usefulness in capacity building, planning and resource allocation, e.g., tests and personal protective equipment). This framework is intended to be a starting point and a generic tool that can be adapted to other online forward triage tools beyond COVID-19. A COVID-19 online forward triage tool meeting all seven criteria can be regarded as fit for purpose. How useful an OFTT is depends on its context and purpose.

Study protocol for a pilot prospective, observational study investigating the condition suggestion and urgency advice accuracy of a symptom assessment app in sub-Saharan Africa: the AFYA-'Health' Study

INTRODUCTION

Due to a global shortage of healthcare workers, there is a lack of basic healthcare for 4 billion people worldwide, particularly affecting low-income and middle-income countries. The utilisation of AI-based healthcare tools such as symptom assessment applications (SAAs) has the potential to reduce the burden on healthcare systems. The purpose of the AFYA Study (AI-based Assessment oF health sYmptoms in TAnzania) is to evaluate the accuracy of the condition suggestions and urgency advice provided by a user on a Swahili language Ada SAA.

METHODS AND ANALYSIS

This study is designed as an observational prospective clinical study. The setting is a waiting room of a Tanzanian district hospital. It will include patients entering the outpatient clinic with various conditions and age groups, including children and adolescents. Patients will be asked to use the SAA before proceeding to usual care. After usual care, they will have a consultation with a study-provided physician. Patients and healthcare practitioners will be blinded to the SAA's results. An expert panel will compare the Ada SAA's condition suggestions and urgency advice to usual care and study provided differential diagnoses and triage. The primary outcome measures are the accuracy and comprehensiveness of the Ada SAA evaluated against the gold standard differential diagnoses.

ETHICS AND DISSEMINATION

Ethical approval was received by the ethics committee (EC) of Muhimbili University of Health and Allied Sciences with an approval number MUHAS-REC-09-2019-044 and the National Institute for Medical Research, NIMR/HQ/R.8c/Vol. I/922. All amendments to the protocol are reported and adapted on the basis of the requirements of the EC. The results from this study will be submitted to peer-reviewed journals, local and international stakeholders, and will be communicated in editorials/articles by Ada Health.

TRIAL REGISTRATION NUMBER

NCT04958577.

Usability Testing of a Digital Assessment Routing Tool for Musculoskeletal Disorders: An Iterative Convergent Mixed Methods Study (Preprint)

Background

Musculoskeletal disorders negatively affect millions of patients worldwide, placing significant demand on health care systems. Digital technologies that improve clinical outcomes and efficiency across the care pathway are development priorities. We developed the musculoskeletal Digital Assessment Routing Tool (DART) to enable self-assessment and immediate direction to the right care.

Objective

We aimed to assess and resolve all serious DART usability issues to create a positive user experience and enhance system adoption before conducting randomized controlled trials for the integration of DART into musculoskeletal management pathways.

Methods

An iterative, convergent mixed methods design was used, with 22 adult participants assessing 50 different clinical presentations over 5 testing rounds across 4 DART iterations. Participants were recruited using purposive sampling, with quotas for age, habitual internet use, and English-language ability. Quantitative data collection was defined by the constructs within the International Organization for Standardization 9241-210-2019 standard, with user satisfaction measured by the System Usability Scale. Study end points were resolution of all grade 1 and 2 usability problems and a mean System Usability Scale score of ≥80 across a minimum of 3 user group sessions.

Results

All participants (mean age 48.6, SD 15.2; range 20-77 years) completed the study. Every assessment resulted in a recommendation with no DART system errors and a mean completion time of 5.2 (SD 4.44, range 1-18) minutes. Usability problems were reduced from 12 to 0, with trust and intention to act improving during the study. The relationship between eHealth literacy and age, as explored with a scatter plot and calculation of the Pearson correlation coefficient, was performed for all participants (r=–0.2; 20/22, 91%) and repeated with a potential outlier removed (r=–0.23), with no meaningful relationships observed or found for either. The mean satisfaction for daily internet users was highest (19/22, 86%; mean 86.5, SD 4.48; 90% confidence level [CL] 1.78 or –1.78), with nonnative English speakers (6/22, 27%; mean 78.1, SD 4.60; 90% CL 3.79 or –3.79) and infrequent internet users scoring the lowest (3/22, 14%; mean 70.8, SD 5.44; 90% CL 9.17 or –9.17), although the CIs overlap. The mean score across all groups was 84.3 (SD 4.67), corresponding to an excellent system, with qualitative data from all participants confirming that DART was simple to use.

Conclusions

All serious DART usability issues were resolved, and a good level of satisfaction, trust, and willingness to act on the DART recommendation was demonstrated, thus allowing progression to randomized controlled trials that assess safety and effectiveness against usual care comparators. The iterative, convergent mixed methods design proved highly effective in fully evaluating DART from a user perspective and could provide a blueprint for other researchers of mobile health systems.

International Registered Report Identifier (IRRID)

RR2-10.2196/27205

Safety of Triage Self-assessment Using a Symptom Assessment App for Walk-in Patients in the Emergency Care Setting: Observational Prospective Cross-sectional Study

Background

Increasing use of emergency departments (EDs) by patients with low urgency, combined with limited availability of medical staff, results in extended waiting times and delayed care. Technological approaches could possibly increase efficiency by providing urgency advice and symptom assessments.

Objective

The purpose of this study is to evaluate the safety of urgency advice provided by a symptom assessment app, Ada, in an ED.

Methods

The study was conducted at the interdisciplinary ED of Marburg University Hospital, with data collection performed between August 2019 and March 2020. This study had a single-center cross-sectional prospective observational design and included 378 patients. The app’s urgency recommendation was compared with an established triage concept (Manchester Triage System [MTS]), including patients from the lower 3 MTS categories only. For all patients who were undertriaged, an expert physician panel assessed the case to detect potential avoidable hazardous situations (AHSs).

Results

Of 378 participants, 344 (91%) were triaged the same or more conservatively and 34 (8.9%) were undertriaged by the app. Of the 378 patients, 14 (3.7%) had received safe advice determined by the expert panel and 20 (5.3%) were considered to be potential AHS. Therefore, the assessment could be considered safe in 94.7% (358/378) of the patients when compared with the MTS assessment. From the 3 lowest MTS categories, 43.4% (164/378) of patients were not considered as emergency cases by the app, but could have been safely treated by a general practitioner or would not have required a physician consultation at all.

Conclusions

The app provided urgency advice after patient self-triage that has a high rate of safety, a rate of undertriage, and a rate of triage with potential to be an AHS, equivalent to telephone triage by health care professionals while still being more conservative than direct ED triage. A large proportion of patients in the ED were not considered as emergency cases, which could possibly relieve ED burden if used at home. Further research should be conducted in the at-home setting to evaluate this hypothesis.

Trial Registration

German Clinical Trial Registration DRKS00024909; https://www.drks.de/drks_web/navigate.do? navigationId=trial.HTML&TRIAL_ID=DRKS00024909