Impact of NHS 111 Online on the NHS 111 telephone service and urgent care system: a mixed-methods study

Longitudinal Changes in Diagnostic Accuracy of a Differential Diagnosis List Developed by an AI-Based Symptom Checker: Retrospective Observational Study

BACKGROUND

Artificial intelligence (AI) symptom checker models should be trained using real-world patient data to improve their diagnostic accuracy. Given that AI-based symptom checkers are currently used in clinical practice, their performance should improve over time. However, longitudinal evaluations of the diagnostic accuracy of these symptom checkers are limited.

OBJECTIVE

This study aimed to assess the longitudinal changes in the accuracy of differential diagnosis lists created by an AI-based symptom checker used in the real world.

METHODS

This was a single-center, retrospective, observational study. Patients who visited an outpatient clinic without an appointment between May 1, 2019, and April 30, 2022, and who were admitted to a community hospital in Japan within 30 days of their index visit were considered eligible. We only included patients who underwent an AI-based symptom checkup at the index visit, and the diagnosis was finally confirmed during follow-up. Final diagnoses were categorized as common or uncommon, and all cases were categorized as typical or atypical. The primary outcome measure was the accuracy of the differential diagnosis list created by the AI-based symptom checker, defined as the final diagnosis in a list of 10 differential diagnoses created by the symptom checker. To assess the change in the symptom checker's diagnostic accuracy over 3 years, we used a chi-square test to compare the primary outcome over 3 periods: from May 1, 2019, to April 30, 2020 (first year); from May 1, 2020, to April 30, 2021 (second year); and from May 1, 2021, to April 30, 2022 (third year).

RESULTS

A total of 381 patients were included. Common diseases comprised 257 (67.5%) cases, and typical presentations were observed in 298 (78.2%) cases. Overall, the accuracy of the differential diagnosis list created by the AI-based symptom checker was 172 (45.1%), which did not differ across the 3 years (first year: 97/219, 44.3%; second year: 32/72, 44.4%; and third year: 43/90, 47.7%; P=.85). The accuracy of the differential diagnosis list created by the symptom checker was low in those with uncommon diseases (30/124, 24.2%) and atypical presentations (12/83, 14.5%). In the multivariate logistic regression model, common disease (P<.001; odds ratio 4.13, 95% CI 2.50-6.98) and typical presentation (P<.001; odds ratio 6.92, 95% CI 3.62-14.2) were significantly associated with the accuracy of the differential diagnosis list created by the symptom checker.

CONCLUSIONS

A 3-year longitudinal survey of the diagnostic accuracy of differential diagnosis lists developed by an AI-based symptom checker, which has been implemented in real-world clinical practice settings, showed no improvement over time. Uncommon diseases and atypical presentations were independently associated with a lower diagnostic accuracy. In the future, symptom checkers should be trained to recognize uncommon conditions.

Only the anxious ones? Identifying characteristics of symptom checker app users: a cross-sectional survey

BACKGROUND

Symptom checker applications (SCAs) may help laypeople classify their symptoms and receive recommendations on medically appropriate actions. Further research is necessary to estimate the influence of user characteristics, attitudes and (e)health-related competencies.

OBJECTIVE

The objective of this study is to identify meaningful predictors for SCA use considering user characteristics.

METHODS

An explorative cross-sectional survey was conducted to investigate German citizens' demographics, eHealth literacy, hypochondria, self-efficacy, and affinity for technology using German language-validated questionnaires. A total of 869 participants were eligible for inclusion in the study. As n = 67 SCA users were assessed and matched 1:1 with non-users, a sample of n = 134 participants were assessed in the main analysis. A four-step analysis was conducted involving explorative predictor selection, model comparisons, and parameter estimates for selected predictors, including sensitivity and post hoc analyses.

RESULTS

Hypochondria and self-efficacy were identified as meaningful predictors of SCA use. Hypochondria showed a consistent and significant effect across all analyses OR: 1.24-1.26 (95% CI: 1.1-1.4). Self-efficacy OR: 0.64-0.93 (95% CI: 0.3-1.4) showed inconsistent and nonsignificant results, leaving its role in SCA use unclear. Over half of the SCA users in our sample met the classification for hypochondria (cut-off on the WI of 5).

CONCLUSIONS

Hypochondria has emerged as a significant predictor of SCA use with a consistently stable effect, yet according to the literature, individuals with this trait may be less likely to benefit from SCA despite their greater likelihood of using it. These users could be further unsettled by risk-averse triage and unlikely but serious diagnosis suggestions.

TRIAL REGISTRATION

The study was registered in the German Clinical Trials Register (DRKS) DRKS00022465, DERR1- https://doi.org/10.2196/34026 .

How could online NHS 111 reduce demand for the telephone NHS 111 service? Qualitative study of user and staff views

BACKGROUND

Online NHS111 was introduced in 2018 in response to increasing and unsustainable demand for telephone NHS111. Despite high levels of use, there is little evidence of channel shift from the telephone to the online service. We explored user and staff perspectives of online NHS111 to understand how and why online NHS111 is used and whether there may be potential for shift from the telephone to online service.

METHODS

As part of a wider mixed-methods study, we used qualitative semistructured interviews to explore perspectives of recent users of online 111 who had responded to a user survey (n=32) and NHS 111 staff (n=16) between November 2019 and June 2020. Interviews were recorded and transcribed verbatim. The data sets were analysed separately using framework analysis (user interviews) and thematic analysis (staff interviews).

RESULTS

Telephone NHS111 health adviser skills in probing and obtaining 'soft information' were perceived as key to obtaining advice that was considered more appropriate and trusted than advice from online interactions, which relied on oversimplified or irrelevant questions.Online NHS111 was perceived to provide a useful and convenient adjunct to the telephone service and widened access to NHS111 services for some subgroups of users who would not otherwise access the telephone service (eg, communication barriers, social anxiety) or were concerned about 'bothering' a health professional. The nature of the online consultation meant that online NHS111 was perceived as more disposable and used more speculatively.

CONCLUSION

Online 111 was perceived as a useful adjunct but not a replacement for telephone NHS 111 with potential for channel shift hindered by reduced confidence in the online service due to the lack of human interaction. Further development of OL111 algorithms will be required if it is to meet the needs of people with more complex health needs.

"You're only a receptionist, what do you want to know for?": Street-level bureaucracy on the front line of primary care in the United Kingdom

Introduction

In care settings across the globe non-clinical staff are involved in filtering patients to the most appropriate source of care. This includes primary care where general practice receptionists are key in facilitating access to individual surgeries and the wider National Health Service. Despite the complexity and significance of their role little is known of how the decision-making behaviors of receptionists impact policy implementation and service delivery. By combining the agent-based implementation theory of street-level bureaucracy with a tri-level analytical framework this work acknowledges the impact of the decisions made by receptionists as street-level bureaucrats and demonstrates the benefits of using the novel framework to provide practical insight of the factors influencing those decisions.

Methods

A secondary analysis of qualitative data gathered from a series of semi-structured interviews conducted with 19 receptionists in the United Kingdom in 2019 was used to populate a tri-level framework: the micro-level relates to influences on decision making acting at an individual level, the meso-level influences at group and organizational levels, and the macro-level influences at a societal or policy level.

Results

At the micro-level we determined how receptionists are influenced by the level of rapport developed with patients and would use common sense to interpret urgency. At the meso-level, influences included their position at the forefront of premises, the culture of the workplace, and the processes and protocols used by their practice. At the macro-level, participants described the impact of limited health service capacity, the lack of mandatory training, and the growth in the use of digital technologies.

Conclusions

Street-level bureaucracy, complemented with a tri-level contextual analysis, is a useful theoretical framework to understand how health workers, such as receptionists, attempt to provide universality without sufficient resource, and could potentially be applied to other kinds of public service workers in this way. This theoretical framework also benefits from being an accessible foundation on which to base practice and policy changes.

Triage and Diagnostic Accuracy of Online Symptom Checkers: Systematic Review

BACKGROUND

In the context of a deepening global shortage of health workers and, in particular, the COVID-19 pandemic, there is growing international interest in, and use of, online symptom checkers (OSCs). However, the evidence surrounding the triage and diagnostic accuracy of these tools remains inconclusive.

OBJECTIVE

This systematic review aimed to summarize the existing peer-reviewed literature evaluating the triage accuracy (directing users to appropriate services based on their presenting symptoms) and diagnostic accuracy of OSCs aimed at lay users for general health concerns.

METHODS

Searches were conducted in MEDLINE, Embase, CINAHL, Health Management Information Consortium (HMIC), and Web of Science, as well as the citations of the studies selected for full-text screening. We included peer-reviewed studies published in English between January 1, 2010, and February 16, 2022, with a controlled and quantitative assessment of either or both triage and diagnostic accuracy of OSCs directed at lay users. We excluded tools supporting health care professionals, as well as disease- or specialty-specific OSCs. Screening and data extraction were carried out independently by 2 reviewers for each study. We performed a descriptive narrative synthesis.

RESULTS

A total of 21,296 studies were identified, of which 14 (0.07%) were included. The included studies used clinical vignettes, medical records, or direct input by patients. Of the 14 studies, 6 (43%) reported on triage and diagnostic accuracy, 7 (50%) focused on triage accuracy, and 1 (7%) focused on diagnostic accuracy. These outcomes were assessed based on the diagnostic and triage recommendations attached to the vignette in the case of vignette studies or on those provided by nurses or general practitioners, including through face-to-face and telephone consultations. Both diagnostic accuracy and triage accuracy varied greatly among OSCs. Overall diagnostic accuracy was deemed to be low and was almost always lower than that of the comparator. Similarly, most of the studies (9/13, 69 %) showed suboptimal triage accuracy overall, with a few exceptions (4/13, 31%). The main variables affecting the levels of diagnostic and triage accuracy were the severity and urgency of the condition, the use of artificial intelligence algorithms, and demographic questions. However, the impact of each variable differed across tools and studies, making it difficult to draw any solid conclusions. All included studies had at least one area with unclear risk of bias according to the revised Quality Assessment of Diagnostic Accuracy Studies-2 tool.

CONCLUSIONS

Although OSCs have potential to provide accessible and accurate health advice and triage recommendations to users, more research is needed to validate their triage and diagnostic accuracy before widescale adoption in community and health care settings. Future studies should aim to use a common methodology and agreed standard for evaluation to facilitate objective benchmarking and validation.

TRIAL REGISTRATION

PROSPERO CRD42020215210; https://tinyurl.com/3949zw83.

A multi-granular stacked regression for forecasting long-term demand in Emergency Departments

BACKGROUND

In the United Kingdom, Emergency Departments (EDs) are under significant pressure due to an ever-increasing number of attendances. Understanding how the capacity of other urgent care services and the health of a population may influence ED attendances is imperative for commissioners and policy makers to develop long-term strategies for reducing this pressure and improving quality and safety.

METHODS

We developed a novel multi-granular stacked regression (MGSR) model using publicly available data to predict future mean monthly ED attendances within Clinical Commissioning Group regions in England. The MGSR combines measures of population health and health service capacity in other related settings. We assessed model performance using the R-squared statistic, measuring variance explained, and the Mean Absolute Percentage Error (MAPE), measuring forecasting accuracy. We used the MGSR to forecast ED demand over a 4-year period under hypothetical scenarios where service capacity is increased, or population health is improved.

RESULTS

Measures of service capacity explain 41 ± 4% of the variance in monthly ED attendances and measures of population health explain 62 ± 22%. The MGSR leads to an overall improvement in performance, with an R-squared of 0.79 ± 0.02 and MAPE of 3% when forecasting mean monthly ED attendances per CCG. Using the MGSR to forecast long-term demand under different scenarios, we found improving population health would reduce peak ED attendances per CCG by approximately 1000 per month after 2 years.

CONCLUSION

Combining models of population health and wider urgent care service capacity for predicting monthly ED attendances leads to an improved performance compared to each model individually. Policies designed to improve population health will reduce ED attendances and enhance quality and safety in the long-term.

Self-Triage Use, Subsequent Healthcare Utilization, and Diagnoses: A Retrospective Study of Process and Clinical Outcomes Following Self-Triage and Self-Scheduling for Ear or Hearing Symptoms

Background

Self-triage is becoming more widespread, but little is known about the people who are using online self-triage tools and their outcomes. For self-triage researchers, there are significant barriers to capturing subsequent healthcare outcomes. Our integrated healthcare system was able to capture subsequent healthcare utilization of individuals who used self-triage integrated with self-scheduling of provider visits.

Methods

We retrospectively examined healthcare utilization and diagnoses after patients had used self-triage and self-scheduling for ear or hearing symptoms. Outcomes and counts of office visits, telemedicine interactions, emergency department visits, and hospitalizations were captured. Diagnosis codes associated with subsequent provider visits were dichotomously categorized as being associated with ear or hearing concerns or not. Nonvisit care encounters of patient-initiated messages, nurse triage calls, and clinical communications were also captured.

Results

For 2168 self-triage uses, we were able to capture subsequent healthcare encounters within 7 days of the self-triage for 80.5% (1745/2168). In subsequent 1092 office visits with diagnoses, 83.1% (891/1092) of the uses were associated with relevant ear, nose and throat diagnoses. Only 0.24% (4/1662) of patients with captured outcomes were associated with a hospitalization within 7 days. Self-triage resulted in a self-scheduled office visit in 7.2% (126/1745). Office visits resulting from a self-scheduled visit had significantly fewer combined non-visit care encounters per office visit (fewer combined nurse triage calls, patient messages, and clinical communication messages) than office visits that were not self-scheduled (-0.51; 95% CI, -0.72 to -0.29; P < .0001).

Conclusion

In an appropriate healthcare setting, self-triage outcomes can be captured in a high percentage of uses to examine for safety, patient adherence to recommendations, and efficiency of self-triage. With the ear or hearing self-triage, most uses had subsequent visit diagnoses relevant to ear or hearing, so most patients appeared to be selecting the appropriate self-triage pathway for their symptoms.

Measuring the impact introducing NHS 111 online had on the NHS 111 telephone service and the wider NHS urgent care system: an observational study

OBJECTIVES

To explore what impact introducing the National Health Service (NHS) 111 online service had on the number of phone calls to the NHS 111 telephone service and the NHS urgent care system.

DESIGN

Observational study using a dose-response interrupted time series model and random-effects meta- analysis to estimate the average effect.

SETTING AND PARTICIPANTS

NHS 111 telephone and online contacts for 18 NHS 111 area codes in England. NHS 111 telephone and online contacts data were collected between October 2010 to December 2019 and January 2018 to December 2019, respectively.

PRIMARY AND SECONDARY OUTCOME MEASURES

Primary outcome: the number of triaged calls to the NHS 111 telephone service following the introduction of NHS 111 online.

SECONDARY OUTCOMES

total calls to the NHS 111 telephone service, total number of emergency ambulance referrals or advice to contact 999, total number of advice to attend an emergency department or other urgent care treatment facility, and total number of advice to contact primary care.

RESULTS

For triaged calls, the overall incidence rate ratio (IRR) per 1000 online contacts was 1.013 (95% CI: 0.996 to 1.029, p=0.127). For total calls, the overall IRR per 1000 online contacts was 1.008 (95% CI: 0.992 to 1.025, p=0.313). For emergency ambulance referrals or advice to contact 999, the overall IRR per 1000 online contacts was 1.067 (95% CI: 1.035 to 1.100, p<0.001). For advice to attend an emergency department or other urgent care treatment facility, the overall IRR per 1000 online contacts is 1.050 (95% CI: 1.010 to 1.092, p=0.014). And finally, for those advised to contact primary care, the overall IRR per 1000 online contacts is 1.051 (95% CI: 1.027 to 1.076, p<0.001).

CONCLUSIONS

It was found that the NHS 111 online service has little impact on the number of triaged and total calls, suggesting that the workload for the NHS 111 telephone service has not increased or decreased as a result of introducing NHS 111 online. However, there was evidence to suggest an increase in the overall number of disposition recommendations (ambulance, emergency department and primary care) for NHS 111 telephone and online services combined following the introduction of the NHS 111 online service.

To what extent do callers follow the advice given by a non-emergency medical helpline (NHS 111): A retrospective cohort study

National Health Service (NHS) 111 helpline was set up to improve access to urgent care in England, efficiency and cost-effectiveness of first-contact health services. Following trusted, authoritative advice is crucial for improved clinical outcomes. We examine patient and call-related characteristics associated with compliance with advice given in NHS 111 calls. The importance of health interactions that are not face-to-face has recently been highlighted by COVID-19 pandemic. In this retrospective cohort study, NHS 111 call records were linked to urgent and emergency care services data. We analysed data of 3,864,362 calls made between October 2013 and September 2017 relating to 1,964,726 callers across London. A multiple logistic regression was used to investigate associations between compliance with advice given and patient and call characteristics. Caller’s action is ‘compliant with advice given if first subsequent service interaction following contact with NHS 111 is consistent with advice given. We found that most calls were made by women (58%), adults aged 30–59 years (33%) and people in the white ethnic category (36%). The most common advice was for caller to contact their General Practitioner (GP) or other local services (18.2%) with varying times scales. Overall, callers followed advice given in 49% of calls. Compliance with triage advice was more likely in calls for children aged <16 years, women, those from Asian/Asian British ethnicity, and calls made out of hours. The highest compliance was among callers advised to self-care without the need to contact any other healthcare service. This is one of the largest studies to describe pathway adherence following telephone advice and associated clinical and demographic features. These results could inform attempts to improve caller compliance with advice given by NHS 111, and as the NHS moves to more hybrid way of working, the lessons from this study are key to the development of remote healthcare services going forward.

Corrigendum: Impact of NHS 111 Online on the NHS 111 telephone service and urgent care system: a mixed-methods study

Correction to list of authors.