Recommendation endpoints and safety of an online self-triage for depression symptoms

INTRODUCTION

Online symptom checkers are a way to address patient concerns and potentially offload a burdened healthcare system. However, safety outcomes of self-triage are unknown, so we reviewed triage recommendations and outcomes of our institution's depression symptom checker.

METHODS

We examined endpoint recommendations and follow-up encounters seven days afterward during 2 December 2021 to 13 December 2022. Patients with an emergency department visit or hospitalization within seven days of self-triaging had a manual review of the electronic health record to determine if the visit was related to depression, suicidal ideation, or suicide attempt. Charts were reviewed for deaths within seven days of self-triage.

RESULTS

There were 287 unique encounters from 263 unique patients. In 86.1% (247/287), the endpoint was an instruction to call nurse triage; in 3.1% of encounters (9/287), instruction was to seek emergency care. Only 20.2% (58/287) followed the recommendations given. Of the 229 patients that did not follow the endpoint recommendations, 121 (52.8%) had some type of follow-up within seven days. Nearly 11% (31/287) were triaged to endpoints not requiring urgent contact and 9.1% (26/287) to an endpoint that would not need any healthcare team input. No patients died in the study period.

CONCLUSIONS

Most patients did not follow the recommendations for follow-up care although ultimately most patients did receive care within seven days. Self-triage appears to appropriately sort patients with depressed mood to emergency care. On-line self-triaging tools for depression have the potential to safely offload some work from clinic personnel.

Performance of Artificial Intelligence (AI)-Powered Chatbots in the Assessment of Medical Case Reports: Qualitative Insights From Simulated Scenarios

Introduction With the expanding awareness and use of AI-powered chatbots, it seems possible that an increasing number of people could use them to assess and evaluate their medical symptoms. If chatbots are used for this purpose, that have not previously undergone a thorough medical evaluation for this specific use, various risks might arise. The aim of this study is to analyze and compare the performance of popular chatbots in differentiating between severe and less critical medical symptoms described from a patient's perspective and to examine the variations in substantive medical assessment accuracy and empathetic communication style among the chatbots' responses. Materials and methods Our study compared three different AI-supported chatbots - OpenAI's ChatGPT 3.5, Microsoft's Bing Chat, and Inflection's Pi AI. Three exemplary case reports for medical emergencies as well as three cases without an urgent reason for an emergency medical admission were constructed and analyzed. Each case report was accompanied by identical questions concerning the most likely suspected diagnosis and the urgency of an immediate medical evaluation. The respective answers of the chatbots were qualitatively compared with each other regarding the medical accuracy of the differential diagnoses mentioned and the conclusions drawn, as well as regarding patient-oriented and empathetic language. Results All examined chatbots were capable of providing medically plausible and probable diagnoses and classifying situations as acute or less critical. However, their responses varied slightly in the level of their urgency assessment. Clear differences could be seen in the level of detail of the differential diagnoses, the overall length of the answers, and how the chatbot dealt with the challenge of being confronted with medical issues. All given answers were comparable in terms of empathy level and comprehensibility. Conclusion Even AI chatbots that are not designed for medical applications already offer substantial guidance in assessing typical medical emergency indications but should always be provided with a disclaimer. In responding to medical queries, characteristic differences emerge among chatbots in the extent and style of their respective answers. Given the lack of medical supervision of many established chatbots, subsequent studies, and experiences are essential to clarify whether a more extensive use of these chatbots for medical concerns will have a positive impact on healthcare or rather pose major medical risks.

Assessing the Safety of a New Clinical Decision Support System for a National Helpline

We assessed the safety of a new clinical decision support system (CDSS) for nurses on Australia's national consumer helpline. Accuracy and safety of triage advice was assessed by testing the CDSS using 78 standardised patient vignettes (48 published and 30 proprietary). Testing was undertaken in two cycles using the CDSS vendor's online evaluation tool (Cycle 1: 47 vignettes; Cycle 2: 41 vignettes). Safety equivalence was examined by testing the existing CDSS with the 47 vignettes from Cycle 1. The new CDSS triaged 66% of vignettes correctly compared to 57% by the existing CDSS. 15% of vignettes were overtriaged by the new CDSS compared to 28% by the existing CDSS. 19% of vignettes were undertriaged by the new CDSS compared to 15% by the existing CDSS. Overall performance of the new CDSS appears consistent and comparable with current studies. The new CDSS is at least as safe as the old CDSS.

Assessment of a Digital Symptom Checker Tool's Accuracy in Suggesting Reproductive Health Conditions: Clinical Vignettes Study

BACKGROUND

Reproductive health conditions such as endometriosis, uterine fibroids, and polycystic ovary syndrome (PCOS) affect a large proportion of women and people who menstruate worldwide. Prevalence estimates for these conditions range from 5% to 40% of women of reproductive age. Long diagnostic delays, up to 12 years, are common and contribute to health complications and increased health care costs. Symptom checker apps provide users with information and tools to better understand their symptoms and thus have the potential to reduce the time to diagnosis for reproductive health conditions.

OBJECTIVE

This study aimed to evaluate the agreement between clinicians and 3 symptom checkers (developed by Flo Health UK Limited) in assessing symptoms of endometriosis, uterine fibroids, and PCOS using vignettes. We also aimed to present a robust example of vignette case creation, review, and classification in the context of predeployment testing and validation of digital health symptom checker tools.

METHODS

Independent general practitioners were recruited to create clinical case vignettes of simulated users for the purpose of testing each condition symptom checker; vignettes created for each condition contained a mixture of condition-positive and condition-negative outcomes. A second panel of general practitioners then reviewed, approved, and modified (if necessary) each vignette. A third group of general practitioners reviewed each vignette case and designated a final classification. Vignettes were then entered into the symptom checkers by a fourth, different group of general practitioners. The outcomes of each symptom checker were then compared with the final classification of each vignette to produce accuracy metrics including percent agreement, sensitivity, specificity, positive predictive value, and negative predictive value.

RESULTS

A total of 24 cases were created per condition. Overall, exact matches between the vignette general practitioner classification and the symptom checker outcome were 83% (n=20) for endometriosis, 83% (n=20) for uterine fibroids, and 88% (n=21) for PCOS. For each symptom checker, sensitivity was reported as 81.8% for endometriosis, 84.6% for uterine fibroids, and 100% for PCOS; specificity was reported as 84.6% for endometriosis, 81.8% for uterine fibroids, and 75% for PCOS; positive predictive value was reported as 81.8% for endometriosis, 84.6% for uterine fibroids, 80% for PCOS; and negative predictive value was reported as 84.6% for endometriosis, 81.8% for uterine fibroids, and 100% for PCOS.

CONCLUSIONS

The single-condition symptom checkers have high levels of agreement with general practitioner classification for endometriosis, uterine fibroids, and PCOS. Given long delays in diagnosis for many reproductive health conditions, which lead to increased medical costs and potential health complications for individuals and health care providers, innovative health apps and symptom checkers hold the potential to improve care pathways.

How Virtual Triage Can Improve Patient Experience and Satisfaction: A Narrative Review and Look Forward

Objective

To complete a review of the literature on patient experience and satisfaction as relates to the potential for virtual triage (VT) or symptom checkers to enhance and enable improvements in these important health care delivery objectives.

Methods

Review and synthesis of the literature on patient experience and satisfaction as informed by emerging evidence, indicating potential for VT to favorably impact these clinical care objectives and outcomes.

Results/Conclusions

VT enhances potential clinical effectiveness through early detection and referral, can reduce avoidable care delivery due to late clinical presentation, and can divert primary care needs to more clinically appropriate outpatient settings rather than high-acuity emergency departments. Delivery of earlier and faster, more acuity level-appropriate care, as well as patient avoidance of excess care acuity (and associated cost), offer promise as contributors to improved patient experience and satisfaction. The application of digital triage as a front door to health care delivery organizations offers care engagement that can help reduce patient need to visit a medical facility for low-acuity conditions more suitable for self-care, thus avoiding unpleasant queues and reducing microbiological and other patient risks associated with visits to medical facilities. VT also offers an opportunity for providers to make patient health care experiences more personalized.

Comparison of Diagnostic and Triage Accuracy of Ada Health and WebMD Symptom Checkers, ChatGPT, and Physicians for Patients in an Emergency Department: Clinical Data Analysis Study

BACKGROUND

Diagnosis is a core component of effective health care, but misdiagnosis is common and can put patients at risk. Diagnostic decision support systems can play a role in improving diagnosis by physicians and other health care workers. Symptom checkers (SCs) have been designed to improve diagnosis and triage (ie, which level of care to seek) by patients.

OBJECTIVE

The aim of this study was to evaluate the performance of the new large language model ChatGPT (versions 3.5 and 4.0), the widely used WebMD SC, and an SC developed by Ada Health in the diagnosis and triage of patients with urgent or emergent clinical problems compared with the final emergency department (ED) diagnoses and physician reviews.

METHODS

We used previously collected, deidentified, self-report data from 40 patients presenting to an ED for care who used the Ada SC to record their symptoms prior to seeing the ED physician. Deidentified data were entered into ChatGPT versions 3.5 and 4.0 and WebMD by a research assistant blinded to diagnoses and triage. Diagnoses from all 4 systems were compared with the previously abstracted final diagnoses in the ED as well as with diagnoses and triage recommendations from three independent board-certified ED physicians who had blindly reviewed the self-report clinical data from Ada. Diagnostic accuracy was calculated as the proportion of the diagnoses from ChatGPT, Ada SC, WebMD SC, and the independent physicians that matched at least one ED diagnosis (stratified as top 1 or top 3). Triage accuracy was calculated as the number of recommendations from ChatGPT, WebMD, or Ada that agreed with at least 2 of the independent physicians or were rated "unsafe" or "too cautious."

RESULTS

Overall, 30 and 37 cases had sufficient data for diagnostic and triage analysis, respectively. The rate of top-1 diagnosis matches for Ada, ChatGPT 3.5, ChatGPT 4.0, and WebMD was 9 (30%), 12 (40%), 10 (33%), and 12 (40%), respectively, with a mean rate of 47% for the physicians. The rate of top-3 diagnostic matches for Ada, ChatGPT 3.5, ChatGPT 4.0, and WebMD was 19 (63%), 19 (63%), 15 (50%), and 17 (57%), respectively, with a mean rate of 69% for physicians. The distribution of triage results for Ada was 62% (n=23) agree, 14% unsafe (n=5), and 24% (n=9) too cautious; that for ChatGPT 3.5 was 59% (n=22) agree, 41% (n=15) unsafe, and 0% (n=0) too cautious; that for ChatGPT 4.0 was 76% (n=28) agree, 22% (n=8) unsafe, and 3% (n=1) too cautious; and that for WebMD was 70% (n=26) agree, 19% (n=7) unsafe, and 11% (n=4) too cautious. The unsafe triage rate for ChatGPT 3.5 (41%) was significantly higher (P=.009) than that of Ada (14%).

CONCLUSIONS

ChatGPT 3.5 had high diagnostic accuracy but a high unsafe triage rate. ChatGPT 4.0 had the poorest diagnostic accuracy, but a lower unsafe triage rate and the highest triage agreement with the physicians. The Ada and WebMD SCs performed better overall than ChatGPT. Unsupervised patient use of ChatGPT for diagnosis and triage is not recommended without improvements to triage accuracy and extensive clinical evaluation.

The Role of Virtual Triage in Improving Clinician Experience and Satisfaction: A Narrative Review

Objective

This review examines the literature on improving clinician satisfaction with a focus on what has been most effective in improving experience from the perspective of clinicians, and the potential role that virtual triage (VT) technology can play in delivering positive clinician experiences that improve clinical care, and bring value to health care delivery organizations (HDOs).

Methods

Review and synthesis of evidence on clinician satisfaction indicating a potential for VT to favorably impact clinician experience, sense of effectiveness, efficiency, and reduction of administrative task burden. Analysis considers how to conceptualize and the value of improving clinician experience, leading clinician dissatisfiers, and the potential role of VT in improving clinician experience/satisfaction.

Results

Contributors to poor clinician experience/satisfaction where VT could have a beneficial impact include better managing resource limitations, administrative workload, lack of care coordination, information overload, and payer interactions. VT can improve clinician experience through the technology's ability to leverage real-time actionable data clinicians can use, streamlining patient-clinician communications, personalizing care delivery, optimizing care coordination, and better aligning digital/virtual services with clinical practice. From an organizational perspective, improvements in clinician experience and satisfaction derive from establishing an effective digital back door, increasing the clinical impact of and satisfaction derived from telemedicine and virtual care, and enhancing clinician centricity.

Conclusions

By embracing digital transformation and implementing solutions such as VT that focus on improving patient and clinician experience, HDOs can address barriers to delivery of high-quality, efficient, and cost-effective care. VT is a digital health tool that can create a more streamlined and satisfying experience for clinicians and the patients they care for. VT is a technology solution that can help clinicians make faster more informed decisions, reduces avoidable care, improves communication with patients and within care teams, and lowers their administrative burden so they have more quality time to care for patients.

Characteristics of Users and Nonusers of Symptom Checkers in Germany: Cross-Sectional Survey Study

BACKGROUND

Previous studies have revealed that users of symptom checkers (SCs, apps that support self-diagnosis and self-triage) are predominantly female, are younger than average, and have higher levels of formal education. Little data are available for Germany, and no study has so far compared usage patterns with people's awareness of SCs and the perception of usefulness.

OBJECTIVE

We explored the sociodemographic and individual characteristics that are associated with the awareness, usage, and perceived usefulness of SCs in the German population.

METHODS

We conducted a cross-sectional online survey among 1084 German residents in July 2022 regarding personal characteristics and people's awareness and usage of SCs. Using random sampling from a commercial panel, we collected participant responses stratified by gender, state of residence, income, and age to reflect the German population. We analyzed the collected data exploratively.

RESULTS

Of all respondents, 16.3% (177/1084) were aware of SCs and 6.5% (71/1084) had used them before. Those aware of SCs were younger (mean 38.8, SD 14.6 years, vs mean 48.3, SD 15.7 years), were more often female (107/177, 60.5%, vs 453/907, 49.9%), and had higher formal education levels (eg, 72/177, 40.7%, vs 238/907, 26.2%, with a university/college degree) than those unaware. The same observation applied to users compared to nonusers. It disappeared, however, when comparing users to nonusers who were aware of SCs. Among users, 40.8% (29/71) considered these tools useful. Those considering them useful reported higher self-efficacy (mean 4.21, SD 0.66, vs mean 3.63, SD 0.81, on a scale of 1-5) and a higher net household income (mean EUR 2591.63, SD EUR 1103.96 [mean US $2798.96, SD US $1192.28], vs mean EUR 1626.60, SD EUR 649.05 [mean US $1756.73, SD US $700.97]) than those who considered them not useful. More women considered SCs unhelpful (13/44, 29.5%) compared to men (4/26, 15.4%).

CONCLUSIONS

Concurring with studies from other countries, our findings show associations between sociodemographic characteristics and SC usage in a German sample: users were on average younger, of higher socioeconomic status, and more commonly female compared to nonusers. However, usage cannot be explained by sociodemographic differences alone. It rather seems that sociodemographics explain who is or is not aware of the technology, but those who are aware of SCs are equally likely to use them, independently of sociodemographic differences. Although in some groups (eg, people with anxiety disorder), more participants reported to know and use SCs, they tended to perceive them as less useful. In other groups (eg, male participants), fewer respondents were aware of SCs, but those who used them perceived them to be more useful. Thus, SCs should be designed to fit specific user needs, and strategies should be developed to help reach individuals who could benefit but are not aware of SCs yet.

A multinational survey of patient utilization of and value conveyed through virtual symptom triage and healthcare referral

Objective

To describe the use patterns, impact and derived patient-user value of a mobile web-based virtual triage/symptom checker.

Methods

Online survey of 2,113 web-based patient-users of a virtual triage/symptom checker was completed over an 8-week period. Questions focused on triage and care objectives, pre- and post-triage care intent, frequency of use, value derived and satisfaction with virtual triage. Responses were analyzed and stratified to characterize patient-user pre-triage and post-triage intent relative to triage engine output.

Results

Seventy-eight percent of virtual triage users were female, and 37% were 18-24 years old or younger, 28% were 25-44, 16% were 45-54, and 19% were 55 years or older; 41.2% completed the survey from the U.S., 12.5% from the U.K., 9.1% from Canada, 5.6% from India, 3.8% from South Africa. Motivations were to determine need to consult a physician (44.2%), to secure medical advice without visiting a physician (21.0%), and to confirm a diagnosis received (14.2%). Forty-three percent were first time users of virtual triage, 36.6% utilized a triage engine at least once every few months or more often. Pre-triage, 40.5% did not know what level of healthcare they were planning to utilize, 33.9% stated they intended to seek a physician consultation, 23.7% engage self-care and 1.8% seek emergency care. Virtual triage recommended 56.8% of patient-users consult a physician, 33.8% seek emergency care and 9.4% engage self-care. In three-fourths, virtual triage helped users decide level of care to pursue. Among 74.1%, triage recommended care different than pre-triage intentions. Post-triage, those who remained uncertain of their care path decreased by 25.4%. Patient-user experience and satisfaction with virtual triage was high, with 80.1% stating that they were highly likely or likely to use it again, and interest in and willingness to use telemedicine doubled.

Conclusion

Virtual triage successfully redirected patient-users who initially planned to seek an inappropriate level of care acuity, reduced patient uncertainty of care path, and doubled the percentage of patients amenable to telemedicine and virtual health engagement. Patient-users were highly satisfied with virtual triage and the virtual triage patient experience, and a large majority will use virtual triage recurrently in the future.

How suitable are clinical vignettes for the evaluation of symptom checker apps? A test theoretical perspective

Objective

To evaluate the ability of case vignettes to assess the performance of symptom checker applications and to suggest refinements to the methodology used in case vignette-based audit studies.

Methods

We re-analyzed the publicly available data of two prominent case vignette-based symptom checker audit studies by calculating common metrics of test theory. Furthermore, we developed a new metric, the Capability Comparison Score (CCS), which compares symptom checker capability while controlling for the difficulty of the set of cases each symptom checker evaluated. We then scrutinized whether applying test theory and the CCS altered the performance ranking of the investigated symptom checkers.

Results

In both studies, most symptom checkers changed their rank order when adjusting the triage capability for item difficulty (ID) with the CCS. The previously reported triage accuracies commonly overestimated the capability of symptom checkers because they did not account for the fact that symptom checkers tend to selectively appraise easier cases (i.e., with high ID values). Also, many case vignettes in both studies showed insufficient (very low and even negative) values of item-total correlation (ITC), suggesting that individual items or the composition of item sets are of low quality.

Conclusions

A test-theoretic perspective helps identify previously undetected threats to the validity of case vignette-based symptom checker assessments and provides guidance and specific metrics to improve the quality of case vignettes, in particular by controlling for the difficulty of the vignettes an app was (not) able to evaluate correctly. Such measures might prove more meaningful than accuracy alone for the competitive assessment of symptom checkers. Our approach helps elaborate and standardize the methodology used for appraising symptom checker capability, which, ultimately, may yield more reliable results.

Utilization patterns and efficiency gains from use of a fully EHR-integrated COVID-19 self-triage and self-scheduling tool: a retrospective analysis

OBJECTIVE

Symptom checkers can help address high demand for SARS-CoV2 (COVID-19) testing and care by providing patients with self-service access to triage recommendations. However, health systems may be hesitant to invest in these tools, as their associated efficiency gains have not been studied. We aimed to quantify the operational efficiency gains associated with use of an online COVID-19 symptom checker as an alternative to a telephone hotline.

METHODS

In our health system, ambulatory patients can either use an online symptom checker or a telephone hotline to be triaged and connected to COVID-19 care. We performed a retrospective analysis of adults who used either method between October 20, 2021 and January 10, 2022, using call logs, electronic health record data, and local wages to calculate labor costs.

RESULTS

Of the 15 549 total COVID-19 triage encounters, 1820 (11.7%) used only the telephone hotline and 13 729 (88.3%) used the symptom checker. Only 271 (2%) of the patients who used the symptom checker also called the hotline. Hotline encounters required more clinician time compared to those involving the symptom checker (17.8 vs 0.4 min/encounter), resulting in higher average labor costs ($24.21 vs $0.55 per encounter). The symptom checker resulted in over 4200 clinician labor hours saved.

CONCLUSION

When given the option, most patients completed COVID-19 triage and visit scheduling online, resulting in substantial efficiency gains. These benefits may encourage health system investment in such tools.

Study of the User Behaviour Caused by Automatic Recommendation Systems Call to Action

Diagnostics accuracy and usability of symptom checkers have been researched in several studies. Their ability to set a correct diagnosis especially in the urgent cases is questionable. There is one aspect of symptom checkers that has not been deeply studied yet. It is their ability to motivate patients to follow up after receiving a direct recommendation and to decrease a load on the health care professionals. The goal of this research is to study how patients behave after receiving a recommendation from a symptom checker and motivation of this behavior. We studied how patients react on the symptom checker recommendations and the motivation behind this behavior. In total we invited 3615 patients to have a symptom checker screening; 2374 of them agreed to run a symptom checker screening; 867 of them agreed to participate in the study. The proportion of the patients who agreed to have a symptom checker screening. So, we can clearly see that symptom checker screening doesn't result in a significant decrease of the load on healthcare professionals. This is supported by the quantitative study results. The patients emphasized the ease of use of the tool and clearness of the recommendations it gives. However, they perceived it as rather a second opinion tool or a tool that helps to prepare to the doctor's visit.

Triage Errors in Primary and Pre-Primary Care

Triage errors are a major concern in health care due to resulting harmful delays in treatments or inappropriate allocation of resources. With the increasing popularity of digital symptom checkers in pre–primary care settings, and amid claims that artificial intelligence outperforms doctors, the accuracy of triage by digital symptom checkers is ever more scrutinized. This paper examines the context and challenges of triage in primary care, pre–primary care, and emergency care, as well as reviews existing evidence on the prevalence of triage errors in all three settings. Implications for development, research, and practice are highlighted, and recommendations are made on how digital symptom checkers should be best positioned.

How Does Triage by an Electronic Symptom Checker Match with Triage by a Nurse?

Omaolo© electronic symptom checkers (ESCs) have been developed to make triage for primary health care patients in Finland. Based on the analysis of the patient's responses to a set of questions, the ESC classifies him/her as emergent, urgent, not urgent, or advices on self-care. In this study the user answered the questions posed by the electronic symptom checker, after which a nurse assessed the urgency of the same user's symptom. The triage nurse was not allowed to know the result of the electronic symptom assessment until he or she had assessed the patient's condition. The level of triage was compared between ESC and nurse in each individual case. Findings from 825 individual cases were analyzed. The mean "exactly matched" for all symptom estimates was 52.6%. The mean "exactly matched" or "overconservative but suitable" for all symptom assessments was 66.6%. Safe assessments of electronic symptom checkers accounted for 98.6% of all assessments. A case was defined as "safe" if the recommendation for action given by the symptom assessment was at most one level less urgent than the nurse's triage assessment of the same case. The findings show that electronic symptom assessments are safe compared to the assessment of an experienced nurse.

Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation

Background

Symptom checkers are digital tools assisting laypersons in self-assessing the urgency and potential causes of their medical complaints. They are widely used but face concerns from both patients and health care professionals, especially regarding their accuracy. A 2015 landmark study substantiated these concerns using case vignettes to demonstrate that symptom checkers commonly err in their triage assessment.

Objective

This study aims to revisit the landmark index study to investigate whether and how symptom checkers’ capabilities have evolved since 2015 and how they currently compare with laypersons’ stand-alone triage appraisal.

Methods

In early 2020, we searched for smartphone and web-based applications providing triage advice. We evaluated these apps on the same 45 case vignettes as the index study. Using descriptive statistics, we compared our findings with those of the index study and with publicly available data on laypersons’ triage capability.

Results

We retrieved 22 symptom checkers providing triage advice. The median triage accuracy in 2020 (55.8%, IQR 15.1%) was close to that in 2015 (59.1%, IQR 15.5%). The apps in 2020 were less risk averse (odds 1.11:1, the ratio of overtriage errors to undertriage errors) than those in 2015 (odds 2.82:1), missing >40% of emergencies. Few apps outperformed laypersons in either deciding whether emergency care was required or whether self-care was sufficient. No apps outperformed the laypersons on both decisions.

Conclusions

Triage performance of symptom checkers has, on average, not improved over the course of 5 years. It decreased in 2 use cases (advice on when emergency care is required and when no health care is needed for the moment). However, triage capability varies widely within the sample of symptom checkers. Whether it is beneficial to seek advice from symptom checkers depends on the app chosen and on the specific question to be answered. Future research should develop resources (eg, case vignette repositories) to audit the capabilities of symptom checkers continuously and independently and provide guidance on when and to whom they should be recommended.

Health Care Professionals' Experiences of Web-Based Symptom Checkers for Triage: Cross-sectional Survey Study

Background

Web-based symptom checkers are promising tools that provide help to patients seeking guidance on health problems. Many health organizations have started using them to enhance triage. Patients use the symptom checker to report their symptoms online and submit the report to the health care center through the system. Health care professionals (registered nurse, practical nurse, general physician, physiotherapist, etc) receive patient inquiries with urgency rating, decide on actions to be taken, and communicate these to the patients. The success of the adoption, however, depends on whether the tools can efficiently support health care professionals’ workflow and achieve their support.

Objective

This study explores the factors influencing health care professionals’ support for a web-based symptom checker for triage.

Methods

Data were collected through a web-based survey of 639 health care professionals using either of the two most used web-based symptom checkers in the Finnish public primary care. Linear regression models were fitted to study the associations between the study variables and health care professionals’ support for the symptom checkers. In addition, the health care professionals’ comments collected via survey were qualitatively analyzed to elicit additional insights about the benefits and challenges of the clinical use of symptom checkers.

Results

Results show that the perceived beneficial influence of the symptom checkers on health care professionals’ work and the perceived usability of the tools were positively associated with professionals’ support. The perceived benefits to patients and organizational support for use were positively associated, and threat to professionals’ autonomy was negatively associated with health care professionals’ support. These associations were, however, not independent of other factors included in the models. The influences on professionals’ work were both positive and negative; the tools streamlined work by providing preliminary information on patients and reduced the number of phone calls, but they also created extra work as the professionals needed to call patients and ask clarifying questions. Managing time between the use of symptom checkers and other tasks was also challenging. Meanwhile, according to health care professionals’ experience, the symptom checkers benefited patients as they received help quickly with a lower threshold for care.

Conclusions

The efficient use of symptom checkers for triage requires usable solutions that support health care professionals’ work. High-quality information about the patients’ conditions and an efficient way of communicating with patients are needed. Using a new eHealth tool also requires that health organizations and teams reorganize their workflows and work distributions to support clinical processes.

Safety of Triage Self-assessment Using a Symptom Assessment App for Walk-in Patients in the Emergency Care Setting: Observational Prospective Cross-sectional Study

Background

Increasing use of emergency departments (EDs) by patients with low urgency, combined with limited availability of medical staff, results in extended waiting times and delayed care. Technological approaches could possibly increase efficiency by providing urgency advice and symptom assessments.

Objective

The purpose of this study is to evaluate the safety of urgency advice provided by a symptom assessment app, Ada, in an ED.

Methods

The study was conducted at the interdisciplinary ED of Marburg University Hospital, with data collection performed between August 2019 and March 2020. This study had a single-center cross-sectional prospective observational design and included 378 patients. The app’s urgency recommendation was compared with an established triage concept (Manchester Triage System [MTS]), including patients from the lower 3 MTS categories only. For all patients who were undertriaged, an expert physician panel assessed the case to detect potential avoidable hazardous situations (AHSs).

Results

Of 378 participants, 344 (91%) were triaged the same or more conservatively and 34 (8.9%) were undertriaged by the app. Of the 378 patients, 14 (3.7%) had received safe advice determined by the expert panel and 20 (5.3%) were considered to be potential AHS. Therefore, the assessment could be considered safe in 94.7% (358/378) of the patients when compared with the MTS assessment. From the 3 lowest MTS categories, 43.4% (164/378) of patients were not considered as emergency cases by the app, but could have been safely treated by a general practitioner or would not have required a physician consultation at all.

Conclusions

The app provided urgency advice after patient self-triage that has a high rate of safety, a rate of undertriage, and a rate of triage with potential to be an AHS, equivalent to telephone triage by health care professionals while still being more conservative than direct ED triage. A large proportion of patients in the ED were not considered as emergency cases, which could possibly relieve ED burden if used at home. Further research should be conducted in the at-home setting to evaluate this hypothesis.

Trial Registration

German Clinical Trial Registration DRKS00024909; https://www.drks.de/drks_web/navigate.do? navigationId=trial.HTML&TRIAL_ID=DRKS00024909

A Novel Diagnostic Decision Support System for Medical Professionals: Prospective Feasibility Study

BACKGROUND

Continuously growing medical knowledge and the increasing amount of data make it difficult for medical professionals to keep track of all new information and to place it in the context of existing information. A variety of digital technologies and artificial intelligence-based methods are currently available as persuasive tools to empower physicians in clinical decision-making and improve health care quality. A novel diagnostic decision support system (DDSS) prototype developed by Ada Health GmbH with a focus on traceability, transparency, and usability will be examined more closely in this study.

OBJECTIVE

The aim of this study is to test the feasibility and functionality of a novel DDSS prototype, exploring its potential and performance in identifying the underlying cause of acute dyspnea in patients at the University Hospital Basel.

METHODS

A prospective, observational feasibility study was conducted at the emergency department (ED) and internal medicine ward of the University Hospital Basel, Switzerland. A convenience sample of 20 adult patients admitted to the ED with dyspnea as the chief complaint and a high probability of inpatient admission was selected. A study physician followed the patients admitted to the ED throughout the hospitalization without interfering with the routine clinical work. Routinely collected health-related personal data from these patients were entered into the DDSS prototype. The DDSS prototype's resulting disease probability list was compared with the gold-standard main diagnosis provided by the treating physician.

RESULTS

The DDSS presented information with high clarity and had a user-friendly, novel, and transparent interface. The DDSS prototype was not perfectly suited for the ED as case entry was time-consuming (1.5-2 hours per case). It provided accurate decision support in the clinical inpatient setting (average of cases in which the correct diagnosis was the first diagnosis listed: 6/20, 30%, SD 2.10%; average of cases in which the correct diagnosis was listed as one of the top 3: 11/20, 55%, SD 2.39%; average of cases in which the correct diagnosis was listed as one of the top 5: 14/20, 70%, SD 2.26%) in patients with dyspnea as the main presenting complaint.

CONCLUSIONS

The study of the feasibility and functionality of the tool was successful, with some limitations. Used in the right place, the DDSS has the potential to support physicians in their decision-making process by showing new pathways and unintentionally ignored diagnoses. The DDSS prototype had some limitations regarding the process of data input, diagnostic accuracy, and completeness of the integrated medical knowledge. The results of this study provide a basis for the tool's further development. In addition, future studies should be conducted with the aim to overcome the current limitations of the tool and study design.

TRIAL REGISTRATION

ClinicalTrials.gov NCT04827342; https://clinicaltrials.gov/ct2/show/NCT04827342.

Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients

BACKGROUND

Digital technologies have become a common starting point for health-related information-seeking. Web- or app-based symptom checkers aim to provide rapid and accurate condition suggestions and triage advice but have not yet been investigated for mental disorders in routine health care settings.

OBJECTIVE

This study aims to test the diagnostic performance of a widely available symptom checker in the context of formal diagnosis of mental disorders when compared with therapists' diagnoses based on structured clinical interviews.

METHODS

Adult patients from an outpatient psychotherapy clinic used the app-based symptom checker Ada-check your health (ADA; Ada Health GmbH) at intake. Accuracy was assessed as the agreement of the first and 1 of the first 5 condition suggestions of ADA with at least one of the interview-based therapist diagnoses. In addition, sensitivity, specificity, and interrater reliabilities (Gwet first-order agreement coefficient [AC1]) were calculated for the 3 most prevalent disorder categories. Self-reported usability (assessed using the System Usability Scale) and acceptance of ADA (assessed using an adapted feedback questionnaire) were evaluated.

RESULTS

A total of 49 patients (30/49, 61% women; mean age 33.41, SD 12.79 years) were included in this study. Across all patients, the interview-based diagnoses matched ADA's first condition suggestion in 51% (25/49; 95% CI 37.5-64.4) of cases and 1 of the first 5 condition suggestions in 69% (34/49; 95% CI 55.4-80.6) of cases. Within the main disorder categories, the accuracy of ADA's first condition suggestion was 0.82 for somatoform and associated disorders, 0.65 for affective disorders, and 0.53 for anxiety disorders. Interrater reliabilities ranged from low (AC1=0.15 for anxiety disorders) to good (AC1=0.76 for somatoform and associated disorders). The usability of ADA was rated as high in the System Usability Scale (mean 81.51, SD 11.82, score range 0-100). Approximately 71% (35/49) of participants would have preferred a face-to-face over an app-based diagnostic.

CONCLUSIONS

Overall, our findings suggest that a widely available symptom checker used in the formal diagnosis of mental disorders could provide clinicians with a list of condition suggestions with moderate-to-good accuracy. However, diagnostic performance was heterogeneous between disorder categories and included low interrater reliability. Although symptom checkers have some potential to complement the diagnostic process as a screening tool, the diagnostic performance should be tested in larger samples and in comparison with further diagnostic instruments.

The Impact of Explanations on Layperson Trust in Artificial Intelligence-Driven Symptom Checker Apps: Experimental Study

BACKGROUND

Artificial intelligence (AI)-driven symptom checkers are available to millions of users globally and are advocated as a tool to deliver health care more efficiently. To achieve the promoted benefits of a symptom checker, laypeople must trust and subsequently follow its instructions. In AI, explanations are seen as a tool to communicate the rationale behind black-box decisions to encourage trust and adoption. However, the effectiveness of the types of explanations used in AI-driven symptom checkers has not yet been studied. Explanations can follow many forms, including why-explanations and how-explanations. Social theories suggest that why-explanations are better at communicating knowledge and cultivating trust among laypeople.

OBJECTIVE

The aim of this study is to ascertain whether explanations provided by a symptom checker affect explanatory trust among laypeople and whether this trust is impacted by their existing knowledge of disease.

METHODS

A cross-sectional survey of 750 healthy participants was conducted. The participants were shown a video of a chatbot simulation that resulted in the diagnosis of either a migraine or temporal arteritis, chosen for their differing levels of epidemiological prevalence. These diagnoses were accompanied by one of four types of explanations. Each explanation type was selected either because of its current use in symptom checkers or because it was informed by theories of contrastive explanation. Exploratory factor analysis of participants' responses followed by comparison-of-means tests were used to evaluate group differences in trust.

RESULTS

Depending on the treatment group, two or three variables were generated, reflecting the prior knowledge and subsequent mental model that the participants held. When varying explanation type by disease, migraine was found to be nonsignificant (P=.65) and temporal arteritis, marginally significant (P=.09). Varying disease by explanation type resulted in statistical significance for input influence (P=.001), social proof (P=.049), and no explanation (P=.006), with counterfactual explanation (P=.053). The results suggest that trust in explanations is significantly affected by the disease being explained. When laypeople have existing knowledge of a disease, explanations have little impact on trust. Where the need for information is greater, different explanation types engender significantly different levels of trust. These results indicate that to be successful, symptom checkers need to tailor explanations to each user's specific question and discount the diseases that they may also be aware of.

CONCLUSIONS

System builders developing explanations for symptom-checking apps should consider the recipient's knowledge of a disease and tailor explanations to each user's specific need. Effort should be placed on generating explanations that are personalized to each user of a symptom checker to fully discount the diseases that they may be aware of and to close their information gap.

A Real-World Rheumatology Registry and Research Consortium: The German RheumaDatenRhePort (RHADAR) Registry

Real-world data are crucial to continuously improve the management of patients with rheumatic and musculoskeletal diseases (RMDs). The German RheumaDatenRhePort (RHADAR) registry encompasses a network of rheumatologists and researchers in Germany providing pseudonymized real-world patient data and allowing timely and continuous improvement in the care of RMD patients. The RHADAR modules allow automated anamnesis and adaptive coordination of appointments regarding individual urgency levels. Further modules focus on the collection and integration of electronic patient-reported outcomes in between consultations. The digital RHADAR modules ultimately allow a patient-centered adaptive approach to integrated medical care starting as early as possible in the disease course. Such a closed-loop system consisting of various modules along the whole patient pathway enables comprehensive and timely patient management in an unprecedented manner.

Participatory COVID-19 Surveillance Tool in Rural Appalachia : Real-Time Disease Monitoring and Regional Response

INTRODUCTION

Few US studies have examined the usefulness of participatory surveillance during the coronavirus disease 2019 (COVID-19) pandemic for enhancing local health response efforts, particularly in rural settings. We report on the development and implementation of an internet-based COVID-19 participatory surveillance tool in rural Appalachia.

METHODS

A regional collaboration among public health partners culminated in the design and implementation of the COVID-19 Self-Checker, a local online symptom tracker. The tool collected data on participant demographic characteristics and health history. County residents were then invited to take part in an automated daily electronic follow-up to monitor symptom progression, assess barriers to care and testing, and collect data on COVID-19 test results and symptom resolution.

RESULTS

Nearly 6500 county residents visited and 1755 residents completed the COVID-19 Self-Checker from April 30 through June 9, 2020. Of the 579 residents who reported severe or mild COVID-19 symptoms, COVID-19 symptoms were primarily reported among women (n = 408, 70.5%), adults with preexisting health conditions (n = 246, 70.5%), adults aged 18-44 (n = 301, 52.0%), and users who reported not having a health care provider (n = 131, 22.6%). Initial findings showed underrepresentation of some racial/ethnic and non-English-speaking groups.

PRACTICAL IMPLICATIONS

This low-cost internet-based platform provided a flexible means to collect participatory surveillance data on local changes in COVID-19 symptoms and adapt to guidance. Data from this tool can be used to monitor the efficacy of public health response measures at the local level in rural Appalachia.

Benchmarking Triage Capability of Symptom Checkers Against That of Medical Laypersons: Survey Study

Background

Symptom checkers (SCs) are tools developed to provide clinical decision support to laypersons. Apart from suggesting probable diagnoses, they commonly advise when users should seek care (triage advice). SCs have become increasingly popular despite prior studies rating their performance as mediocre. To date, it is unclear whether SCs can triage better than those who might choose to use them.

Objective

This study aims to compare triage accuracy between SCs and their potential users (ie, laypersons).

Methods

On Amazon Mechanical Turk, we recruited 91 adults from the United States who had no professional medical background. In a web-based survey, the participants evaluated 45 fictitious clinical case vignettes. Data for 15 SCs that had processed the same vignettes were obtained from a previous study. As main outcome measures, we assessed the accuracy of the triage assessments made by participants and SCs for each of the three triage levels (ie, emergency care, nonemergency care, self-care) and overall, the proportion of participants outperforming each SC in terms of accuracy, and the risk aversion of participants and SCs by comparing the proportion of cases that were overtriaged.

Results

The mean overall triage accuracy was similar for participants (60.9%, SD 6.8%; 95% CI 59.5%-62.3%) and SCs (58%, SD 12.8%). Most participants outperformed all but 5 SCs. On average, SCs more reliably detected emergencies (80.6%, SD 17.9%) than laypersons did (67.5%, SD 16.4%; 95% CI 64.1%-70.8%). Although both SCs and participants struggled with cases requiring self-care (the least urgent triage category), SCs more often wrongly classified these cases as emergencies (43/174, 24.7%) compared with laypersons (56/1365, 4.10%).

Conclusions

Most SCs had no greater triage capability than an average layperson, although the triage accuracy of the five best SCs was superior to the accuracy of most participants. SCs might improve early detection of emergencies but might also needlessly increase resource utilization in health care. Laypersons sometimes require support in deciding when to rely on self-care but it is in that very situation where SCs perform the worst. Further research is needed to determine how to best combine the strengths of humans and SCs.

'You should see a doctor', said the robot: Reflections on a digital diagnostic device in a pandemic age

AIMS

In three days at the beginning of the COVID-19 pandemic, the Copenhagen Emergency Medical Services developed a digital diagnostic device. The purpose was to assess and triage potential COVID-19 symptoms and to reduce the number of calls to public health-care helplines. The device was used almost 150,000 times in a few weeks and was described by politicians and administrators as a solution and success. However, high usage cannot serve as the sole criterion of success. What might be adequate criteria? And should digital triage for citizens by default be considered low risk?

METHODS

This paper reflects on the uncertain aspects of the performance, risks and issues of accountability pertaining to the digital diagnostic device in order to draw lessons for future improvements. The analysis is based on the principles of evidence-based medicine (EBM), the EU and US regulations of medical devices and the taxonomy of uncertainty in health care by Han et al.

RESULTS

Lessons for future digital devices are (a) the need for clear criteria of success, (b) the importance of awareness of other severe diseases when triaging, (c) the priority of designing the device to collect data for evaluation and (d) clear allocation of responsibilities.

CONCLUSIONS

A device meant to substitute triage for citizens according to its own criteria of success should not by default be considered as low risk. In a pandemic age dependent on digitalisation, it is therefore important not to abandon the ethos of EBM, but instead to prepare the ground for new ways of building evidence of effect.

Use Characteristics and Triage Acuity of a Digital Symptom Checker in a Large Integrated Health System: Population-Based Descriptive Study

Background

Pressure on the US health care system has been increasing due to a combination of aging populations, rising health care expenditures, and most recently, the COVID-19 pandemic. Responses to this pressure are hindered in part by reliance on a limited supply of highly trained health care professionals, creating a need for scalable technological solutions. Digital symptom checkers are artificial intelligence–supported software tools that use a conversational “chatbot” format to support rapid diagnosis and consistent triage. The COVID-19 pandemic has brought new attention to these tools due to the need to avoid face-to-face contact and preserve urgent care capacity. However, evidence-based deployment of these chatbots requires an understanding of user demographics and associated triage recommendations generated by a large general population.

Objective

In this study, we evaluate the user demographics and levels of triage acuity provided by a symptom checker chatbot deployed in partnership with a large integrated health system in the United States.

Methods

This population-based descriptive study included all web-based symptom assessments completed on the website and patient portal of the Sutter Health system (24 hospitals in Northern California) from April 24, 2019, to February 1, 2020. User demographics were compared to relevant US Census population data.

Results

A total of 26,646 symptom assessments were completed during the study period. Most assessments (17,816/26,646, 66.9%) were completed by female users. The mean user age was 34.3 years (SD 14.4 years), compared to a median age of 37.3 years of the general population. The most common initial symptom was abdominal pain (2060/26,646, 7.7%). A substantial number of assessments (12,357/26,646, 46.4%) were completed outside of typical physician office hours. Most users were advised to seek medical care on the same day (7299/26,646, 27.4%) or within 2-3 days (6301/26,646, 23.6%). Over a quarter of the assessments indicated a high degree of urgency (7723/26,646, 29.0%).

Conclusions

Users of the symptom checker chatbot were broadly representative of our patient population, although they skewed toward younger and female users. The triage recommendations were comparable to those of nurse-staffed telephone triage lines. Although the emergence of COVID-19 has increased the interest in remote medical assessment tools, it is important to take an evidence-based approach to their deployment.

A Comparison of Artificial Intelligence and Human Doctors for the Purpose of Triage and Diagnosis

AI virtual assistants have significant potential to alleviate the pressure on overly burdened healthcare systems by enabling patients to self-assess their symptoms and to seek further care when appropriate. For these systems to make a meaningful contribution to healthcare globally, they must be trusted by patients and healthcare professionals alike, and service the needs of patients in diverse regions and segments of the population. We developed an AI virtual assistant which provides patients with triage and diagnostic information. Crucially, the system is based on a generative model, which allows for relatively straightforward re-parameterization to reflect local disease and risk factor burden in diverse regions and population segments. This is an appealing property, particularly when considering the potential of AI systems to improve the provision of healthcare on a global scale in many regions and for both developing and developed countries. We performed a prospective validation study of the accuracy and safety of the AI system and human doctors. Importantly, we assessed the accuracy and safety of both the AI and human doctors independently against identical clinical cases and, unlike previous studies, also accounted for the information gathering process of both agents. Overall, we found that the AI system is able to provide patients with triage and diagnostic information with a level of clinical accuracy and safety comparable to that of human doctors. Through this approach and study, we hope to start building trust in AI-powered systems by directly comparing their performance to human doctors, who do not always agree with each other on the cause of patients’ symptoms or the most appropriate triage recommendation.

Characteristics and Symptoms of App Users Seeking COVID-19-Related Digital Health Information and Remote Services: Retrospective Cohort Study

BACKGROUND

Patient-facing digital health tools have been promoted to help patients manage concerns related to COVID-19 and to enable remote care and self-care during the COVID-19 pandemic. It has also been suggested that these tools can help further our understanding of the clinical characteristics of this new disease. However, there is limited information on the characteristics and use patterns of these tools in practice.

OBJECTIVE

The aims of this study are to describe the characteristics of people who use digital health tools to address COVID-19-related concerns; explore their self-reported symptoms and characterize the association of these symptoms with COVID-19; and characterize the recommendations provided by digital health tools.

METHODS

This study used data from three digital health tools on the K Health app: a protocol-based COVID-19 self-assessment, an artificial intelligence (AI)-driven symptom checker, and communication with remote physicians. Deidentified data were extracted on the demographic and clinical characteristics of adults seeking COVID-19-related health information between April 8 and June 20, 2020. Analyses included exploring features associated with COVID-19 positivity and features associated with the choice to communicate with a remote physician.

RESULTS

During the period assessed, 71,619 individuals completed the COVID-19 self-assessment, 41,425 also used the AI-driven symptom checker, and 2523 consulted with remote physicians. Individuals who used the COVID-19 self-assessment were predominantly female (51,845/71,619, 72.4%), with a mean age of 34.5 years (SD 13.9). Testing for COVID-19 was reported by 2901 users, of whom 433 (14.9%) reported testing positive. Users who tested positive for COVID-19 were more likely to have reported loss of smell or taste (relative rate [RR] 6.66, 95% CI 5.53-7.94) and other established COVID-19 symptoms as well as ocular symptoms. Users communicating with a remote physician were more likely to have been recommended by the self-assessment to undergo immediate medical evaluation due to the presence of severe symptoms (RR 1.19, 95% CI 1.02-1.32). Most consultations with remote physicians (1940/2523, 76.9%) were resolved without need for referral to an in-person visit or to the emergency department.

CONCLUSIONS

Our results suggest that digital health tools can help support remote care and self-management of COVID-19 and that self-reported symptoms from digital interactions can extend our understanding of the symptoms associated with COVID-19.

Patients' Utilization and Perception of an Artificial Intelligence-Based Symptom Assessment and Advice Technology in a British Primary Care Waiting Room: Exploratory Pilot Study

BACKGROUND

When someone needs to know whether and when to seek medical attention, there are a range of options to consider. Each will have consequences for the individual (primarily considering trust, convenience, usefulness, and opportunity costs) and for the wider health system (affecting clinical throughput, cost, and system efficiency). Digital symptom assessment technologies that leverage artificial intelligence may help patients navigate to the right type of care with the correct degree of urgency. However, a recent review highlighted a gap in the literature on the real-world usability of these technologies.

OBJECTIVE

We sought to explore the usability, acceptability, and utility of one such symptom assessment technology, Ada, in a primary care setting.

METHODS

Patients with a new complaint attending a primary care clinic in South London were invited to use a custom version of the Ada symptom assessment mobile app. This exploratory pilot study was conducted between November 2017 and January 2018 in a practice with 20,000 registered patients. Participants were asked to complete an Ada self-assessment about their presenting complaint on a study smartphone, with assistance provided if required. Perceptions on the app and its utility were collected through a self-completed study questionnaire following completion of the Ada self-assessment.

RESULTS

Over a 3-month period, 523 patients participated. Most were female (n=325, 62.1%), mean age 39.79 years (SD 17.7 years), with a larger proportion (413/506, 81.6%) of working-age individuals (aged 15-64) than the general population (66.0%). Participants rated Ada's ease of use highly, with most (511/522, 97.8%) reporting it was very or quite easy. Most would use Ada again (443/503, 88.1%) and agreed they would recommend it to a friend or relative (444/520, 85.3%). We identified a number of age-related trends among respondents, with a directional trend for more young respondents to report Ada had provided helpful advice (50/54, 93%, 18-24-year olds reported helpful) than older respondents (19/32, 59%, adults aged 70+ reported helpful). We found no sex differences on any of the usability questions fielded. While most respondents reported that using the symptom checker would not have made a difference in their care-seeking behavior (425/494, 86.0%), a sizable minority (63/494, 12.8%) reported they would have used lower-intensity care such as self-care, pharmacy, or delaying their appointment. The proportion was higher for patients aged 18-24 (11/50, 22%) than aged 70+ (0/28, 0%).

CONCLUSIONS

In this exploratory pilot study, the digital symptom checker was rated as highly usable and acceptable by patients in a primary care setting. Further research is needed to confirm whether the app might appropriately direct patients to timely care, and understand how this might save resources for the health system. More work is also needed to ensure the benefits accrue equally to older age groups.

Rapid design and implementation of an integrated patient self-triage and self-scheduling tool for COVID-19

OBJECTIVE

To rapidly deploy a digital patient-facing self-triage and self-scheduling tool in a large academic health system to address the COVID-19 pandemic.

MATERIALS AND METHODS

We created a patient portal-based COVID-19 self-triage and self-scheduling tool and made it available to all primary care patients at the University of California, San Francisco Health, a large academic health system. Asymptomatic patients were asked about exposure history and were then provided relevant information. Symptomatic patients were triaged into 1 of 4 categories-emergent, urgent, nonurgent, or self-care-and then connected with the appropriate level of care via direct scheduling or telephone hotline.

RESULTS

This self-triage and self-scheduling tool was designed and implemented in under 2 weeks. During the first 16 days of use, it was completed 1129 times by 950 unique patients. Of completed sessions, 315 (28%) were by asymptomatic patients, and 814 (72%) were by symptomatic patients. Symptomatic patient triage dispositions were as follows: 193 emergent (24%), 193 urgent (24%), 99 nonurgent (12%), 329 self-care (40%). Sensitivity for detecting emergency-level care was 87.5% (95% CI 61.7-98.5%).

DISCUSSION

This self-triage and self-scheduling tool has been widely used by patients and is being rapidly expanded to other populations and health systems. The tool has recommended emergency-level care with high sensitivity, and decreased triage time for patients with less severe illness. The data suggests it also prevents unnecessary triage messages, phone calls, and in-person visits.

CONCLUSION

Patient self-triage tools integrated into electronic health record systems have the potential to greatly improve triage efficiency and prevent unnecessary visits during the COVID-19 pandemic.

Online symptom checker diagnostic and triage accuracy for HIV and hepatitis C

We sought to address the prior limitations of symptom checker accuracy by analysing the diagnostic and triage feasibility of online symptom checkers using a consecutive series of real-life emergency department (ED) patient encounters, and addressing a complex patient population – those with hepatitis C or HIV. We aimed to study the diagnostic and triage accuracy of these symptom checkers in relation to an emergency room physician-determined diagnosis. An ED retrospective analysis was performed on 8363 consecutive adult patients. Eligible patients included: 90 HIV, 67 hepatitis C, 11 both HIV and hepatitis C. Five online symptom checkers were utilised for diagnosis (Mayo Clinic, WebMD, Symptomate, Symcat, Isabel), three with triage capabilities. Symptom checker output was compared with ED physician-determined diagnosis data in regards to diagnostic accuracy and differential diagnosis listing, along with triage advice. All symptom checkers, whether for combined HIV and hepatitis C, HIV alone or hepatitis C alone had poor diagnostic accuracy in regards to Top1 (<20%), Top3 (<35%), Top10 (<40%), Listed at All (<45%). Significant variations existed for each individual symptom checker, as some appeared more accurate for listing the diagnosis in the top of the differential, vs. others more apt to list the diagnosis at all. In regards to ED triage data, a significantly higher percentage of hepatitis C patients (59.7%; 40/67) were found to have an initial diagnosis with emergent criteria than HIV patients (35.6%; 32/90). Symptom checker diagnostic capabilities are quite inferior to physician diagnostic capabilities. Complex patients such as those with HIV or hepatitis C may carry a more specific differential diagnosis, warranting symptom checkers to have diagnostic algorithms accounting for such complexity. Symptom checkers carry the potential for real-time epidemiologic monitoring of patient symptoms, as symptom entries and subsequent symptom checker diagnosis could allow health officials a means to track illnesses in specific patient populations and geographic regions. In order to do this, accurate and reliable symptom checkers are warranted.

Patient Perspectives on the Usefulness of an Artificial Intelligence-Assisted Symptom Checker: Cross-Sectional Survey Study

Background

Patients are increasingly seeking Web-based symptom checkers to obtain diagnoses. However, little is known about the characteristics of the patients who use these resources, their rationale for use, and whether they find them accurate and useful.

Objective

The study aimed to examine patients’ experiences using an artificial intelligence (AI)–assisted online symptom checker.

Methods

An online survey was administered between March 2, 2018, through March 15, 2018, to US users of the Isabel Symptom Checker within 6 months of their use. User characteristics, experiences of symptom checker use, experiences discussing results with physicians, and prior personal history of experiencing a diagnostic error were collected.

Results

A total of 329 usable responses was obtained. The mean respondent age was 48.0 (SD 16.7) years; most were women (230/304, 75.7%) and white (271/304, 89.1%). Patients most commonly used the symptom checker to better understand the causes of their symptoms (232/304, 76.3%), followed by for deciding whether to seek care (101/304, 33.2%) or where (eg, primary or urgent care: 63/304, 20.7%), obtaining medical advice without going to a doctor (48/304, 15.8%), and understanding their diagnoses better (39/304, 12.8%). Most patients reported receiving useful information for their health problems (274/304, 90.1%), with half reporting positive health effects (154/302, 51.0%). Most patients perceived it to be useful as a diagnostic tool (253/301, 84.1%), as a tool providing insights leading them closer to correct diagnoses (231/303, 76.2%), and reported they would use it again (278/304, 91.4%). Patients who discussed findings with their physicians (103/213, 48.4%) more often felt physicians were interested (42/103, 40.8%) than not interested in learning about the tool’s results (24/103, 23.3%) and more often felt physicians were open (62/103, 60.2%) than not open (21/103, 20.4%) to discussing the results. Compared with patients who had not previously experienced diagnostic errors (missed or delayed diagnoses: 123/304, 40.5%), patients who had previously experienced diagnostic errors (181/304, 59.5%) were more likely to use the symptom checker to determine where they should seek care (15/123, 12.2% vs 48/181, 26.5%; P=.002), but they less often felt that physicians were interested in discussing the tool’s results (20/34, 59% vs 22/69, 32%; P=.04).

Conclusions

Despite ongoing concerns about symptom checker accuracy, a large patient-user group perceived an AI-assisted symptom checker as useful for diagnosis. Formal validation studies evaluating symptom checker accuracy and effectiveness in real-world practice could provide additional useful information about their benefit.

Syndromic surveillance and its utilisation for mass gatherings

Tremendous advancements in syndromic surveillance strategies over the last two decades, and specifically from prior mass gatherings, have been incorporated into day-to-day healthcare analysis worldwide and have left a lasting indirect impact since their inception. Mass gatherings are a daily occurrence worldwide and provide a scenario ripe for public health aims and objectives utilising syndromic surveillance. Europe is less than a decade away from hosting a colossal worldwide gathering (2024 Summer Olympics) in likely a time when the global agreement is in flux. A call to arms is needed for additional surveillance strategies incorporating mobile application symptom checker data, telemedicine, social media and social data sensing. There remains a need for an optimal combination of real-time data sensing that captures the whole population, but to reach that goal we must incorporate new advancements into baseline epidemiologic data monitoring, otherwise we will be tracking real-time mass gathering events on top of inaccurate baseline epidemiologic data.

How Accurate Are Patients at Diagnosing the Cause of Their Knee Pain With the Help of a Web-based Symptom Checker?

Background:

Researching medical information is the third most popular activity online, and there are a variety of web-based symptom checker programs available.

Purpose:

This study evaluated a patient’s ability to self-diagnose their knee pain from a list of possible diagnoses supplied by an accurate symptom checker.

Study Design:

Cohort study (diagnosis); Level of evidence, 2.

Methods:

All patients older than 18 years who presented to the office of 7 different fellowship-trained sports medicine surgeons over an 8-month period with a complaint of knee pain were asked to participate. A web-based symptom checker for knee pain was used; the program has a reported accuracy of 89%. The symptom checker generates a list of potential diagnoses after patients enter symptoms and links each diagnosis to informative content. After exploring the informative content, patients selected all diagnoses they felt could explain their symptoms. Each patient was later examined by a physician who was blinded to the differential generated by the program as well as the patient-selected diagnoses. A blinded third party compared the diagnoses generated by the program with those selected by the patient as well as the diagnoses determined by the physician. The level of matching between the patient-selected diagnoses and the physician’s diagnoses determined the patient’s ability to correctly diagnose their knee pain.

Results:

There were 163 male and 165 female patients, with a mean age of 48 years (range, 18-76 years). The program generated a mean 6.6 diagnoses (range, 2-15) per patient. Each patient had a mean 1.7 physician diagnoses (range, 1-4). Patients selected a mean 2 diagnoses (range, 1-9). The patient-selected diagnosis matched the physician’s diagnosis 58% of the time.

Conclusion:

With the aid of an accurate symptom checker, patients were able to correctly identify the cause of their knee pain 58% of the time.

Accuracy of a computer-based diagnostic program for ambulatory patients with knee pain

BACKGROUND

Looking up information regarding a medical condition is the third most popular activity online, and there are a variety of web-based symptom-checking programs available to the patient. However, the authors are not aware of any that have been scientifically evaluated as an accurate measure for the cause of one's knee pain.

PURPOSE/HYPOTHESIS

The purpose of this study was to design and evaluate an Internet-based program that generates a differential diagnosis based on a history of knee pain entered by the patient. The hypothesis was that the program would accurately generate a differential diagnosis for patients presenting with knee pain.

STUDY DESIGN

Cohort study (diagnosis); Level of evidence, 2.

METHODS

A web-based program was created to collect knee pain history and generate a differential diagnosis for ambulatory patients with knee pain. The program selected from 26 common knee diagnoses. A total of 527 consecutive patients aged ≥18 years, who presented with a knee complaint to 7 different board-certified orthopaedic surgeons during a 3-month period, were asked to complete the questionnaire in the program. Upon completion, patients were examined by a board-certified orthopaedic surgeon. Both the patient and physician were blinded to the differential diagnosis generated by the program. A third party was responsible for comparing the diagnosis(es) generated by the program with that determined by the physician. The level of matching between diagnoses determined the accuracy of the program.

RESULTS

A total of 272 male and 255 female patients, with an average age of 47 years (range, 18-84 years), participated in the study. The median number of diagnoses generated by the program was 4.8 (range, 1-10), with this list containing the physician's diagnosis(es) 89% of the time. The specificity was 27%.

CONCLUSION

Despite a low specificity, the results of this study show the program to be an accurate method for generating a differential diagnosis for knee pain.

A perspective on healthcare related web interfaces

Consumer retrieval of health information through the internet has become prevalent. In the past, physicians provided filtered health information to the consumer. However, the availability of health related information including disease specific research trends over the World Wide Web is useful for clinicians and consumers. The use of internet based health care information by clinicians and consumers have increased in recent years. Nonetheless, consumers often have difficulties in evaluating such data in a comprehensive manner. Here, we describe the current status of health care related data over the World Wide Web.