Sentiment analysis of user feedback on the HSE's Covid-19 contact tracing app

Effectiveness of digital contact tracing interventions for COVID-19: A systematic scoping review

OBJECTIVES

Digital contact tracing (DCT) interventions have been deployed at unprecedented scale during COVID-19. However, no comprehensive appraisal of the evidence exists to date regarding their effectiveness. We aimed to systematically review the global literature for a holistic understanding of DCT effectiveness during COVID-19, and to identify factors that enabled or hindered its effectiveness.

STUDY DESIGN

Systematic scoping review.

METHODS

We searched six databases for peer-reviewed literature relevant to the evaluation of DCT interventions during COVID-19 (January 2024) (CRD42021268586). We compiled implemented DCT interventions from grey literature. Effectiveness appraisals, different operationalizations, measurements, and definitions of DCT effectiveness, as well as associated factors were synthesized qualitatively. Study quality was assessed using the Mixed Methods Appraisal Tool. We followed Cochrane and PRISMA guidance.

RESULTS

We identified 133 studies evaluating 121 different DCT implementations. Seventy-three (60 %) studies found DCT to be effective, mostly when evaluating epidemiological impact metrics. Public trust emerged as crucial for DCT to be effective, which requires high and enforceable data safety and privacy standards, clear and transparent communication, high accuracy and reliability of the intervention, and an acceptance-enhancing implementation approach of other pandemic response measures by public health authorities more broadly. Most evaluations took place in high rather than low-resource settings.

CONCLUSION

While technical performance matters, DCT effectiveness primarily depends on a relatively small number of non-technical drivers centred around public trust. DCT should only be implemented as integrated part of a broader public health framework. Our findings hold important insights for the design, implementation, and evaluation of other digital technology for pandemic response.

Pre-trained language models in medicine: A survey

With the rapid progress in Natural Language Processing (NLP), Pre-trained Language Models (PLM) such as BERT, BioBERT, and ChatGPT have shown great potential in various medical NLP tasks. This paper surveys the cutting-edge achievements in applying PLMs to various medical NLP tasks. Specifically, we first brief PLMS and outline the research of PLMs in medicine. Next, we categorise and discuss the types of tasks in medical NLP, covering text summarisation, question-answering, machine translation, sentiment analysis, named entity recognition, information extraction, medical education, relation extraction, and text mining. For each type of task, we first provide an overview of the basic concepts, the main methodologies, the advantages of applying PLMs, the basic steps of applying PLMs application, the datasets for training and testing, and the metrics for task evaluation. Subsequently, a summary of recent important research findings is presented, analysing their motivations, strengths vs weaknesses, similarities vs differences, and discussing potential limitations. Also, we assess the quality and influence of the research reviewed in this paper by comparing the citation count of the papers reviewed and the reputation and impact of the conferences and journals where they are published. Through these indicators, we further identify the most concerned research topics currently. Finally, we look forward to future research directions, including enhancing models' reliability, explainability, and fairness, to promote the application of PLMs in clinical practice. In addition, this survey also collect some download links of some model codes and the relevant datasets, which are valuable references for researchers applying NLP techniques in medicine and medical professionals seeking to enhance their expertise and healthcare service through AI technology.

Self voting classification model for online meeting app review sentiment analysis and topic modeling

Online meeting applications (apps) have emerged as a potential solution for conferencing, education and meetings, etc. during the COVID-19 outbreak and are used by private companies and governments alike. A large number of such apps compete with each other by providing a different set of functions towards users' satisfaction. These apps take users' feedback in the form of opinions and reviews which are later used to improve the quality of services. Sentiment analysis serves as the key function to obtain and analyze users' sentiments from the posted feedback indicating the importance of efficient and accurate sentiment analysis. This study proposes the novel idea of self voting classification (SVC) where multiple variants of the same model are trained using different feature extraction approaches and the final prediction is based on the ensemble of these variants. For experiments, the data collected from the Google Play store for online meeting apps were used. Primarily, the focus of this study is to use a support vector machine (SVM) with the proposed SVC approach using both soft voting (SV) and hard voting (HV) criteria, however, decision tree, logistic regression, and k nearest neighbor have also been investigated for performance appraisal. Three variants of models are trained on a bag of words, term frequency-inverse document frequency, and hashing features to make the ensemble. Experimental results indicate that the proposed SVC approach can elevate the performance of traditional machine learning models substantially. The SVM obtains 1.00 and 0.98 accuracy scores, using HV and SV criteria, respectively when used with the proposed SVC approach. Topic-wise sentiment analysis using the latent Dirichlet allocation technique is performed as well for topic modeling.

Global User-Level Perception of COVID-19 Contact Tracing Applications: Data-Driven Approach Using Natural Language Processing

BACKGROUND

Contact tracing has been globally adopted in the fight to control the infection rate of COVID-19. To this aim, several mobile apps have been developed. However, there are ever-growing concerns over the working mechanism and performance of these applications. The literature already provides some interesting exploratory studies on the community's response to the applications by analyzing information from different sources, such as news and users' reviews of the applications. However, to the best of our knowledge, there is no existing solution that automatically analyzes users' reviews and extracts the evoked sentiments. We believe such solutions combined with a user-friendly interface can be used as a rapid surveillance tool to monitor how effective an application is and to make immediate changes without going through an intense participatory design method.

OBJECTIVE

In this paper, we aim to analyze the efficacy of AI and NLP techniques for automatically extracting and classifying the polarity of users' sentiments by proposing a sentiment analysis framework to automatically analyze users' reviews on COVID-19 contact tracing mobile apps. We also aim to provide a large-scale annotated benchmark data set to facilitate future research in the domain. As a proof of concept, we also developed a web application based on the proposed solutions, which is expected to help the community quickly analyze the potential of an application in the domain.

METHODS

We propose a pipeline starting from manual annotation via a crowd-sourcing study and concluding with the development and training of artificial intelligence (AI) models for automatic sentiment analysis of users' reviews. In detail, we collected and annotated a large-scale data set of user reviews on COVID-19 contact tracing applications. We used both classical and deep learning methods for classification experiments.

RESULTS

We used 8 different methods on 3 different tasks, achieving up to an average F1 score of 94.8%, indicating the feasibility of the proposed solution. The crowd-sourcing activity resulted in a large-scale benchmark data set composed of 34,534 manually annotated reviews.

CONCLUSIONS

The existing literature mostly relies on the manual or exploratory analysis of users' reviews on applications, which is tedious and time-consuming. In existing studies, generally, data from fewer applications are analyzed. In this work, we showed that AI and natural language processing techniques provide good results for analyzing and classifying users' sentiments' polarity and that automatic sentiment analysis can help to analyze users' responses more accurately and quickly. We also provided a large-scale benchmark data set. We believe the presented analysis, data set, and proposed solutions combined with a user-friendly interface can be used as a rapid surveillance tool to analyze and monitor mobile apps deployed in emergency situations leading to rapid changes in the applications without going through an intense participatory design method.

Digital Contact Tracing Apps for COVID-19: Development of a Citizen-Centered Evaluation Framework

BACKGROUND

The silent transmission of COVID-19 has led to an exponential growth of fatal infections. With over 4 million deaths worldwide, the need to control and stem transmission has never been more critical. New COVID-19 vaccines offer hope. However, administration timelines, long-term protection, and effectiveness against potential variants are still unknown. In this context, contact tracing and digital contact tracing apps (CTAs) continue to offer a mechanism to help contain transmission, keep people safe, and help kickstart economies. However, CTAs must address a wide range of often conflicting concerns, which make their development/evolution complex. For example, the app must preserve citizens' privacy while gleaning their close contacts and as much epidemiological information as possible.

OBJECTIVE

In this study, we derived a compare-and-contrast evaluative framework for CTAs that integrates and expands upon existing works in this domain, with a particular focus on citizen adoption; we call this framework the Citizen-Focused Compare-and-Contrast Evaluation Framework (C3EF) for CTAs.

METHODS

The framework was derived using an iterative approach. First, we reviewed the literature on CTAs and mobile health app evaluations, from which we derived a preliminary set of attributes and organizing pillars. These attributes and the probing questions that we formulated were iteratively validated, augmented, and refined by applying the provisional framework against a selection of CTAs. Each framework pillar was then subjected to internal cross-team scrutiny, where domain experts cross-checked sufficiency, relevancy, specificity, and nonredundancy of the attributes, and their organization in pillars. The consolidated framework was further validated on the selected CTAs to create a finalized version of C3EF for CTAs, which we offer in this paper.

RESULTS

The final framework presents seven pillars exploring issues related to CTA design, adoption, and use: (General) Characteristics, Usability, Data Protection, Effectiveness, Transparency, Technical Performance, and Citizen Autonomy. The pillars encompass attributes, subattributes, and a set of illustrative questions (with associated example answers) to support app design, evaluation, and evolution. An online version of the framework has been made available to developers, health authorities, and others interested in assessing CTAs.

CONCLUSIONS

Our CTA framework provides a holistic compare-and-contrast tool that supports the work of decision-makers in the development and evolution of CTAs for citizens. This framework supports reflection on design decisions to better understand and optimize the design compromises in play when evolving current CTAs for increased public adoption. We intend this framework to serve as a foundation for other researchers to build on and extend as the technology matures and new CTAs become available.

Mining user reviews of COVID contact-tracing apps: An exploratory analysis of nine European apps

CONTEXT

More than 78 countries have developed COVID contact-tracing apps to limit the spread of coronavirus. However, many experts and scientists cast doubt on the effectiveness of those apps. For each app, a large number of reviews have been entered by end-users in app stores.

OBJECTIVE

Our goal is to gain insights into the user reviews of those apps, and to find out the main problems that users have reported. Our focus is to assess the "software in society" aspects of the apps, based on user reviews.

METHOD

We selected nine European national apps for our analysis and used a commercial app-review analytics tool to extract and mine the user reviews. For all the apps combined, our dataset includes 39,425 user reviews.

RESULTS

Results show that users are generally dissatisfied with the nine apps under study, except the Scottish ("Protect Scotland") app. Some of the major issues that users have complained about are high battery drainage and doubts on whether apps are really working.

CONCLUSION

Our results show that more work is needed by the stakeholders behind the apps (e.g., app developers, decision-makers, public health experts) to improve the public adoption, software quality and public perception of these apps.

Public opinion of the Irish “COVID Tracker” digital contact tracing App: A national survey

Objective

This study aims to gather public opinion on the Irish “COVID Tracker” digital contact tracing (DCT) App, with particular focus on App usage, usability, usefulness, technological issues encountered, and potential changes to the App.

Methods

A 35-item online questionnaire was deployed for 10 days in October 2020, 3 months after the launch of the Irish DCT App.

Results

A total of 2889 completed responses were recorded, with 2553 (88%) respondents currently using the App. Although four in five users felt the App is easy to download, is easy to use and looks professional, 615 users (22%) felt it had slowed down their phone, and 757 (28%) felt it had a negative effect on battery life. Seventy-nine percent of respondents reported the App's main function is to aid contact tracing. Inclusion of national COVID-19 trends is a useful ancillary function according to 87% of respondents, and there was an appetite for more granular local data. Overall, 1265 (44%) respondents believed the App is helping the national effort, while 1089 (38%) were unsure.

Conclusions

DCT Apps may potentially augment traditional contact tracing methods. Despite some reports of negative effects on phone performance, just 7% of users who have tried the App have deleted it. Ancillary functionality, such as up-to-date regional COVID-19, may encourage DCT App use. This study describes general positivity toward the Irish COVID Tracker App among users but also highlights the need for transparency on effectiveness of App-enabled contact tracing and for study of non-users to better establish barriers to use.

Characteristics and determinants of population acceptance of COVID-19 digital contact tracing: a systematic review

BACKGROUND AND AIM

As recently outlined in the WHO-ECDC Indicator framework (1) to evaluate the public health effectiveness of digital proximity tracing solutions, one of the main barriers to digital contact tracing (DCT) is population acceptance, which, in turns, is influenced by digital literacy, attitudes and practice. DCT came to public prominence during the COVID-19 pandemic but evidence on its population acceptance have not been comprehensively analyzed.  Methods: We carried out a systematic review (PROSPERO: CRD42021253668) following the PRISMA guidelines to collect, systematize and critically appraise the available evidence on population DCT acceptance.  Original studies reporting on different measures of population DCT acceptance were included.

CONCLUSIONS

The systematic review was based on 41 articles meeting our a priori defined inclusion criteria, comprising aa total of 186144 surveyed subjects, 50000 tweets, 5025 Reddit posts and 714 written comments. Data extraction and synthesis required a qualitative outcome grouping, performed ex-post, in 14 different benchmarks components. They constitute a narrative analysis of actionable points for public health policy. Population acceptance is a key component of  DCT effective adoption and infection control during infectious diseases outbreaks. Assessing DCT acceptance's determinants in different settings, populations an cultural contexts it is of fundamental importance to inform the planning, implementation and monitoring of public health interventions. The results of our in-depth qualitative and quantitative analysis  will provide context for prospective improvements and actionable items and should guide future research aimed at exploring how digitalization can serve people-centred care.

Public perception on the use of digital contact tracing tools post COVID-19 lockdown: Sentiment analysis and opinion mining (Preprint)

Background

Singapore’s national digital contact-tracing (DCT) tool—TraceTogether—attained an above 70% uptake by December 2020 after a slew of measures. Sentiment analysis can help policymakers to assess public sentiments on the implementation of new policy measures in a short time, but there is a paucity of sentiment analysis studies on the usage of DCT tools.

Objective

We sought to understand the public’s knowledge of, concerns with, and sentiments on the use of TraceTogether over time and their preferences for the type of TraceTogether tool.

Methods

We conducted a cross-sectional survey at a large public hospital in Singapore after the COVID-19 lockdown, from July 2020 through February 2021. In total, 4097 respondents aged 21-80 years were sampled proportionately by sex and 4 age groups. The open-ended responses were processed and analyzed using natural language processing tools. We manually corrected the language and logic errors and replaced phrases with words available in the syuzhet sentiment library without altering the original meaning of the phrases. The sentiment scores were computed by summing the scores of all the tokens (phrases split into smaller units) in the phrase. Stopwords (prepositions and connectors) were removed, followed by implementing the bag-of-words model to calculate the bigram and trigram occurrence in the data set. Demographic and time filters were applied to segment the responses.

Results

Respondents’ knowledge of and concerns with TraceTogether changed from a focus on contact tracing and Bluetooth activation in July-August 2020 to QR code scanning and location check-ins in January-February 2021. Younger males had the highest TraceTogether uptake (24/40, 60%), while older females had the lowest uptake (8/34, 24%) in the first half of July 2020. This trend was reversed in mid-October after the announcement on mandatory TraceTogether check-ins at public venues. Although their TraceTogether uptake increased over time, older females continued to have lower sentiment scores. The mean sentiment scores were the lowest in January 2021 when the media reported that data collected by TraceTogether were used for criminal investigations. Smartphone apps were initially preferred over tokens, but the preference for the type of TraceTogether tool equalized over time as tokens became accessible to the whole population. The sentiments on token-related comments became more positive as the preference for tokens increased.

Conclusions

The public’s knowledge of and concerns with the use of a mandatory DCT tool varied with the national regulations and public communications over time with the evolution of the COVID-19 pandemic. Effective communications tailored to subpopulations and greater transparency in data handling will help allay public concerns with data misuse and improve trust in the authorities. Having alternative forms of the DCT tool can increase the uptake of and positive sentiments on DCT.

Best Practice Guidance for Digital Contact Tracing Apps: A Cross-disciplinary Review of the Literature

BACKGROUND

Digital contact tracing apps have the potential to augment contact tracing systems and disrupt COVID-19 transmission by rapidly identifying secondary cases prior to the onset of infectiousness and linking them into a system of quarantine, testing, and health care worker case management. The international experience of digital contact tracing apps during the COVID-19 pandemic demonstrates how challenging their design and deployment are.

OBJECTIVE

This study aims to derive and summarize best practice guidance for the design of the ideal digital contact tracing app.

METHODS

A collaborative cross-disciplinary approach was used to derive best practice guidance for designing the ideal digital contact tracing app. A search of the indexed and gray literature was conducted to identify articles describing or evaluating digital contact tracing apps. MEDLINE was searched using a combination of free-text terms and Medical Subject Headings search terms. Gray literature sources searched were the World Health Organization Institutional Repository for Information Sharing, the European Centre for Disease Prevention and Control publications library, and Google, including the websites of many health protection authorities. Articles that were acceptable for inclusion in this evidence synthesis were peer-reviewed publications, cohort studies, randomized trials, modeling studies, technical reports, white papers, and media reports related to digital contact tracing.

RESULTS

Ethical, user experience, privacy and data protection, technical, clinical and societal, and evaluation considerations were identified from the literature. The ideal digital contact tracing app should be voluntary and should be equitably available and accessible. User engagement could be enhanced by small financial incentives, enabling users to tailor aspects of the app to their particular needs and integrating digital contact tracing apps into the wider public health information campaign. Adherence to the principles of good data protection and privacy by design is important to convince target populations to download and use digital contact tracing apps. Bluetooth Low Energy is recommended for a digital contact tracing app's contact event detection, but combining it with ultrasound technology may improve a digital contact tracing app's accuracy. A decentralized privacy-preserving protocol should be followed to enable digital contact tracing app users to exchange and record temporary contact numbers during contact events. The ideal digital contact tracing app should define and risk-stratify contact events according to proximity, duration of contact, and the infectiousness of the case at the time of contact. Evaluating digital contact tracing apps requires data to quantify app downloads, use among COVID-19 cases, successful contact alert generation, contact alert receivers, contact alert receivers that adhere to quarantine and testing recommendations, and the number of contact alert receivers who subsequently are tested positive for COVID-19. The outcomes of digital contact tracing apps' evaluations should be openly reported to allow for the wider public to review the evaluation of the app.

CONCLUSIONS

In conclusion, key considerations and best practice guidance for the design of the ideal digital contact tracing app were derived from the literature.