Analysis of Errors in Dictated Clinical Documents Assisted by Speech Recognition Software and Professional Transcriptionists

Medicolegal aspects of documentation and the electronic health record

INTRODUCTION

The busiest times in the hospital are often met by the greatest challenges in complete and comprehensive documentation of the patient care event. The near complete transition to the Electronic Health Record (EHR) was to be the solution to a host of provider documentation concerns. It is clear the EHR provides reliability, reproducibility, integration, evidence based decision-making, multidisciplinary contribution across the entire healthcare spectrum.

METHODS

The use of a consensus of expert opinion supplemented by focused literature review allows a balanced evidence based presentation of data.

RESULTS

Documentation is not a perfect tool however, as issues with efficiency, reliability, use of shortcut maneuvers and potential for increased medico-legal risk have been raised. The solution is attention to documentation detail, and creation of systems that facilitate excellence. The focus on electronic documentation systems should include continual evaluation, ongoing improvement, involvement of a multidisciplinary patient care team and vendor receptiveness to in EHR development and operations.

CONCLUSION

The most effective use of the EHR as a risk management tool requires documentation knowledge, targeted analysis, product improvement and co-development of clinical-commercial resource.

From voice to ink (Vink): development and assessment of an automated, free-of-charge transcription tool

BACKGROUND

Verbatim transcription of qualitative audio data is a cornerstone of analytic quality and rigor, yet the time and energy required for such transcription can drain resources, delay analysis, and hinder the timely dissemination of qualitative insights. In recent years, software programs have presented a promising mechanism to accelerate transcription, but the broad application of such programs has been constrained due to expensive licensing or "per-minute" fees, data protection concerns, and limited availability of such programs in many languages. In this article, we outline our process of adapting a free, open-source, speech-to-text algorithm (Whisper by OpenAI) into a usable and accessible tool for qualitative transcription. Our program, which we have dubbed "Vink" for voice to ink, is available under a permissive open-source license (and thus free of cost).

RESULTS

We conducted a proof-of-principle assessment of Vink's performance in transcribing authentic interview audio data in 14 languages. A majority of pilot-testers evaluated the software performance positively and indicated that they were likely to use the tool in their future research. Our usability assessment indicates that Vink is easy-to-use, and we performed further refinements based on pilot-tester feedback to increase user-friendliness.

CONCLUSION

With Vink, we hope to contribute to facilitating rigorous qualitative research processes globally by reducing time and costs associated with transcription and by expanding free-of-cost transcription software availability to more languages. With Vink running on standalone computers, data privacy issues arising within many other solutions do not apply.

Using HIPAA (Health Insurance Portability and Accountability Act)-Compliant Transcription Services for Virtual Psychiatric Interviews: Pilot Comparison Study

BACKGROUND

Automatic speech recognition (ASR) technology is increasingly being used for transcription in clinical contexts. Although there are numerous transcription services using ASR, few studies have compared the word error rate (WER) between different transcription services among different diagnostic groups in a mental health setting. There has also been little research into the types of words ASR transcriptions mistakenly generate or omit.

OBJECTIVE

This study compared the WER of 3 ASR transcription services (Amazon Transcribe [Amazon.com, Inc], Zoom-Otter AI [Zoom Video Communications, Inc], and Whisper [OpenAI Inc]) in interviews across 2 different clinical categories (controls and participants experiencing a variety of mental health conditions). These ASR transcription services were also compared with a commercial human transcription service, Rev (Rev.Com, Inc). Words that were either included or excluded by the error in the transcripts were systematically analyzed by their Linguistic Inquiry and Word Count categories.

METHODS

Participants completed a 1-time research psychiatric interview, which was recorded on a secure server. Transcriptions created by the research team were used as the gold standard from which WER was calculated. The interviewees were categorized into either the control group (n=18) or the mental health condition group (n=47) using the Mini-International Neuropsychiatric Interview. The total sample included 65 participants. Brunner-Munzel tests were used for comparing independent sets, such as the diagnostic groupings, and Wilcoxon signed rank tests were used for correlated samples when comparing the total sample between different transcription services.

RESULTS

There were significant differences between each ASR transcription service's WER (P<.001). Amazon Transcribe's output exhibited significantly lower WERs compared with the Zoom-Otter AI's and Whisper's ASR. ASR performances did not significantly differ across the 2 different clinical categories within each service (P>.05). A comparison between the human transcription service output from Rev and the best-performing ASR (Amazon Transcribe) demonstrated a significant difference (P<.001), with Rev having a slightly lower median WER (7.6%, IQR 5.4%-11.35 vs 8.9%, IQR 6.9%-11.6%). Heat maps and spider plots were used to visualize the most common errors in Linguistic Inquiry and Word Count categories, which were found to be within 3 overarching categories: Conversation, Cognition, and Function.

CONCLUSIONS

Overall, consistent with previous literature, our results suggest that the WER between manual and automated transcription services may be narrowing as ASR services advance. These advances, coupled with decreased cost and time in receiving transcriptions, may make ASR transcriptions a more viable option within health care settings. However, more research is required to determine if errors in specific types of words impact the analysis and usability of these transcriptions, particularly for specific applications and in a variety of populations in terms of clinical diagnosis, literacy level, accent, and cultural origin.

Machine learning-based speech recognition system for nursing documentation - A pilot study

PURPOSE

Considering the significant workload of nursing tasks, enhancing the efficiency of nursing documentation is imperative. This study aimed to evaluate the effectiveness of a machine learning-based speech recognition (SR) system in reducing the clinical workload associated with typing nursing records, implemented in a psychiatry ward.

METHODS

The study was conducted between July 15, 2020, and June 30, 2021, at Cheng Hsin General Hospital in Taiwan. The language corpus was based on the existing records from the hospital nursing information system. The participating ward's nursing activities, clinical conversation, and accent data were also collected for deep learning-based SR-engine training. A total of 21 nurses participated in the evaluation of the SR system. Documentation time and recognition error rate were evaluated in parallel between SR-generated records and keyboard entry over 4 sessions. Any differences between SR and keyboard transcriptions were regarded as SR errors.

FINDINGS

A total of 200 data were obtained from four evaluation sessions, 10 participants were asked to use SR and keyboard entry in parallel at each session and 5 entries were collected from each participant. Overall, the SR system processed 30,112 words in 32,456 s (0.928 words per second). The mean accuracy of the SR system improved after each session, from 87.06% in 1st session to 95.07% in 4th session.

CONCLUSION

This pilot study demonstrated our machine learning-based SR system has an acceptable recognition accuracy and may reduce the burden of documentation for nurses. However, the potential error with the SR transcription should continually be recognized and improved. Further studies are needed to improve the integration of SR in digital documentation of nursing records, in terms of both productivity and accuracy across different clinical specialties.

Surgeon wellbeing in the 21st century

Physician time is under assault. Optimizing surgeons time and maximizing time spent on work that brings them the greatest professional fulfillment should be central tenants of these efforts.

Benefits, barriers, and facilitators of using speech recognition technology in nursing documentation and reporting: A cross-sectional study

Background and Aim

Nursing reports are necessary for clinical communication and provide an accurate reflection of nursing assessments, care provided, changes in clinical status, and patient-related information to support the multidisciplinary team to provide individualized care. Nurses always face challenges in recording and documenting nursing reports. Speech recognition systems (SRS), as one of the documentation technologies, can play a potential role in recording medical reports. Therefore, this study seeks to identify the barriers, benefits, and facilitators of utilizing speech recognition technology in nursing reports.

Materials and Methods

This cross-sectional was conducted through a researcher-made questionnaire in 2022. Invitations were sent to 200 ICU nurses working in the three educational hospitals of Imam Reza (AS), Qaem and Imam Zaman in Mashhad city (Iran), 125 of whom accepted our invitation. Finally, 73 nurses included the study based on inclusion and exclusion criteria. Data analysis was performed using SPSS 22.0.

Results

According to the nurses, "paperwork reduction" (3.96, ±1.96), "performance improvement" (3.96, ±0.93), and "cost reduction" (3.95, ±1.07) were the most common benefits of using the SRS. "Lack of specialized, technical, and experienced staff to teach nurses how to work with speech recognition systems" (3.59, ±1.18), "insufficient training of nurses" (3.59, ±1.11), and "need to edit and control quality and correct documents" (3.59, ±1.03) were the most common barriers to using SRS. As well as "ability to fully review documentation processes" (3.62, ±1.13), "creation of integrated data in record documentation" (3.58, ±1.15), "possibility of error correction for nurses" (3.51, ±1.16) were the most common facilitators. There was no significant relationship between nurses' demographic information and the benefits, barriers, and facilitators.

Conclusions

By providing information on the benefits, barriers, and facilitators of using this technology, hospital managers, nursing managers, and information technology managers of healthcare centers can make more informed decisions in selecting and implementing SRS for nursing report documentation. This will help to avoid potential challenges that may reduce the efficiency, effectiveness, and productivity of the systems.

"Mm-hm," "Uh-uh": are non-lexical conversational sounds deal breakers for the ambient clinical documentation technology?

OBJECTIVES

Ambient clinical documentation technology uses automatic speech recognition (ASR) and natural language processing (NLP) to turn patient-clinician conversations into clinical documentation. It is a promising approach to reducing clinician burden and improving documentation quality. However, the performance of current-generation ASR remains inadequately validated. In this study, we investigated the impact of non-lexical conversational sounds (NLCS) on ASR performance. NLCS, such as Mm-hm and Uh-uh, are commonly used to convey important information in clinical conversations, for example, Mm-hm as a "yes" response from the patient to the clinician question "are you allergic to antibiotics?"

MATERIALS AND METHODS

In this study, we evaluated 2 contemporary ASR engines, Google Speech-to-Text Clinical Conversation ("Google ASR"), and Amazon Transcribe Medical ("Amazon ASR"), both of which have their language models specifically tailored to clinical conversations. The empirical data used were from 36 primary care encounters. We conducted a series of quantitative and qualitative analyses to examine the word error rate (WER) and the potential impact of misrecognized NLCS on the quality of clinical documentation.

RESULTS

Out of a total of 135 647 spoken words contained in the evaluation data, 3284 (2.4%) were NLCS. Among these NLCS, 76 (0.06% of total words, 2.3% of all NLCS) were used to convey clinically relevant information. The overall WER, of all spoken words, was 11.8% for Google ASR and 12.8% for Amazon ASR. However, both ASR engines demonstrated poor performance in recognizing NLCS: the WERs across frequently used NLCS were 40.8% (Google) and 57.2% (Amazon), respectively; and among the NLCS that conveyed clinically relevant information, 94.7% and 98.7%, respectively.

DISCUSSION AND CONCLUSION

Current ASR solutions are not capable of properly recognizing NLCS, particularly those that convey clinically relevant information. Although the volume of NLCS in our evaluation data was very small (2.4% of the total corpus; and for NLCS that conveyed clinically relevant information: 0.06%), incorrect recognition of them could result in inaccuracies in clinical documentation and introduce new patient safety risks.

Evaluating the adoption of voice recognition technology for real-time dictation in a rural healthcare system: A retrospective analysis of dragon medical one

BACKGROUND

In 2013, Marshfield Clinic Health System (MCHS) implemented the Dragon Medical One (DMO) system provided by Nuance Management Center (NMC) for Real-Time Dictation (RTD), embracing the idea of streamlined clinic workflow, reduced dictation hours, and improved documentation legibility. Since then, MCHS has observed a trend of reduced time in documentation, however, the target goal of 100% adoption of voice recognition (VR)-based RTD has not been met.

OBJECTIVE

To evaluate the uptake/adoption of VR technology for RTD in MCHS, between 2018-2020.

METHODS

DMO data for 1,373 MCHS providers from 2018-2020 were analyzed. The study outcome was VR uptake, defined as the median number of hours each provider used VR technology to dictate patient information, and classified as no/yes. Covariates included sex, age, US-trained/international medical graduates, trend, specialty, and facility. Descriptive statistics and unadjusted and adjusted logistic regression analyses were performed. Stata/SE.version.17 was used for analyses. P-values less than/equal to 0.05 were considered statistically significant.

RESULTS

Of the 1,373 MCHS providers, the mean (SD) age was 48.3 (12.4) years. VR uptake was higher than no uptake (72.0% vs. 28.0%). In both unadjusted and adjusted analyses, VR uptake was 4.3 times and 7.7 times higher in 2019-2020 compared to 2018, respectively (OR:4.30,95%CI:2.44-7.46 and AOR:7.74,95%CI:2.51-23.86). VR uptake was 0.5 and 0.6 times lower among US-trained physicians compared to internationally-trained physicians (OR:0.53,95%CI:0.37-0.76 and AOR:0.58,95%CI:0.35-0.97). Uptake was 0.2 times lower among physicians aged 60/above than physicians aged 29/less (OR:0.20,95%CI:0.10-0.59, and AOR:0.17,95%CI:0.27-1.06).

CONCLUSION

Since 2018, VR adoption has increased significantly across MCHS. However, it was lower among US-trained physicians than among internationally-trained physicians (although internationally physicians were in minority) and lower among more senior physicians than among younger physicians. These findings provide critical information about VR trends, physician factors, and which providers could benefit from additional training to increase VR adoption in healthcare systems.

A Systematic Approach to Configuring MetaMap for Optimal Performance

BACKGROUND

 MetaMap is a valuable tool for processing biomedical texts to identify concepts. Although MetaMap is highly configurative, configuration decisions are not straightforward.

OBJECTIVE

 To develop a systematic, data-driven methodology for configuring MetaMap for optimal performance.

METHODS

 MetaMap, the word2vec model, and the phrase model were used to build a pipeline. For unsupervised training, the phrase and word2vec models used abstracts related to clinical decision support as input. During testing, MetaMap was configured with the default option, one behavior option, and two behavior options. For each configuration, cosine and soft cosine similarity scores between identified entities and gold-standard terms were computed for 40 annotated abstracts (422 sentences). The similarity scores were used to calculate and compare the overall percentages of exact matches, similar matches, and missing gold-standard terms among the abstracts for each configuration. The results were manually spot-checked. The precision, recall, and F-measure (β =1) were calculated.

RESULTS

 The percentages of exact matches and missing gold-standard terms were 0.6-0.79 and 0.09-0.3 for one behavior option, and 0.56-0.8 and 0.09-0.3 for two behavior options, respectively. The percentages of exact matches and missing terms for soft cosine similarity scores exceeded those for cosine similarity scores. The average precision, recall, and F-measure were 0.59, 0.82, and 0.68 for exact matches, and 1.00, 0.53, and 0.69 for missing terms, respectively.

CONCLUSION

 We demonstrated a systematic approach that provides objective and accurate evidence guiding MetaMap configurations for optimizing performance. Combining objective evidence and the current practice of using principles, experience, and intuitions outperforms a single strategy in MetaMap configurations. Our methodology, reference codes, measurements, results, and workflow are valuable references for optimizing and configuring MetaMap.

Applications of natural language processing in ophthalmology: present and future

Advances in technology, including novel ophthalmic imaging devices and adoption of the electronic health record (EHR), have resulted in significantly increased data available for both clinical use and research in ophthalmology. While artificial intelligence (AI) algorithms have the potential to utilize these data to transform clinical care, current applications of AI in ophthalmology have focused mostly on image-based deep learning. Unstructured free-text in the EHR represents a tremendous amount of underutilized data in big data analyses and predictive AI. Natural language processing (NLP) is a type of AI involved in processing human language that can be used to develop automated algorithms using these vast quantities of available text data. The purpose of this review was to introduce ophthalmologists to NLP by (1) reviewing current applications of NLP in ophthalmology and (2) exploring potential applications of NLP. We reviewed current literature published in Pubmed and Google Scholar for articles related to NLP and ophthalmology, and used ancestor search to expand our references. Overall, we found 19 published studies of NLP in ophthalmology. The majority of these publications (16) focused on extracting specific text such as visual acuity from free-text notes for the purposes of quantitative analysis. Other applications included: domain embedding, predictive modeling, and topic modeling. Future ophthalmic applications of NLP may also focus on developing search engines for data within free-text notes, cleaning notes, automated question-answering, and translating ophthalmology notes for other specialties or for patients, especially with a growing interest in open notes. As medicine becomes more data-oriented, NLP offers increasing opportunities to augment our ability to harness free-text data and drive innovations in healthcare delivery and treatment of ophthalmic conditions.

Speech recognition can help evaluate shared decision making and predict medication adherence in primary care setting

OBJECTIVE

Asthma is a common chronic illness affecting 19 million US adults. Inhaled corticosteroids are a safe and effective treatment for asthma, yet, medication adherence among patients remains poor. Shared decision-making, a patient activation strategy, can improve patient adherence to inhaled corticosteroids. This study aimed to explore whether audio-recorded patient-primary care provider encounters can be used to: 1. Evaluate the level of patient-perceived shared decision-making during the encounter, and 2. Predict levels of patient's inhaled corticosteroid adherence.

MATERIALS AND METHODS

Shared decision-making and inhaled corticosteroid adherence were assessed using the SDM Questionnaire-9 and the Medication Adherence Report Scale for Asthma (MARS-A). Speech-to-text algorithms were used to automatically transcribe 80 audio-recorded encounters between primary care providers and asthmatic patients. Machine learning algorithms (Naive Bayes, Support Vector Machines, Decision Tree) were applied to achieve the study's predictive goals.

RESULTS

The accuracy of automated speech-to-text transcription was relatively high (ROUGE F-score = .9). Machine learning algorithms achieved good predictive performance for shared decision-making (the highest F-score = .88 for the Naive Bayes) and inhaled corticosteroid adherence (the highest F-score = .87 for the Support Vector Machines).

DISCUSSION

This was the first study that trained machine learning algorithms on a dataset of audio-recorded patient-primary care provider encounters to successfully evaluate the quality of SDM and predict patient inhaled corticosteroid adherence.

CONCLUSION

Machine learning approaches can help primary care providers identify patients at risk for poor medication adherence and evaluate the quality of care by measuring levels of shared decision-making. Further work should explore the replicability of our results in larger samples and additional health domains.

Application of a Domain-specific BERT for Detection of Speech Recognition Errors in Radiology Reports

PURPOSE

To develop radiology domain-specific bidirectional encoder representations from transformers (BERT) models that can identify speech recognition (SR) errors and suggest corrections in radiology reports.

MATERIALS AND METHODS

A pretrained BERT model, Clinical BioBERT, was further pretrained on a corpus of 114 008 radiology reports between April 2016 and August 2019 that were retrospectively collected from two hospitals. Next, the model was fine-tuned on a training dataset of generated insertion, deletion, and substitution errors, creating Radiology BERT. This model was retrospectively evaluated on an independent dataset of radiology reports with generated errors (n = 18 885) and on unaltered report sentences (n = 2000) and prospectively evaluated on true clinical SR errors (n = 92). Correction Radiology BERT was separately trained to suggest corrections for detected deletion and substitution errors. Area under the receiver operating characteristic curve (AUC) and bootstrapped 95% CIs were calculated for each evaluation dataset.

RESULTS

Radiology-specific BERT had AUC values of >.99 (95% CI: >0.99, >0.99), 0.94 (95% CI: 0.93, 0.94), 0.98 (95% CI: 0.98, 0.98), and 0.97 (95% CI: 0.97, 0.97) for detecting insertion, deletion, substitution, and all errors, respectively, on the independently generated test set. Testing on unaltered report impressions revealed a sensitivity of 82% (28 of 34; 95% CI: 70%, 93%) and specificity of 88% (1521 of 1728; 95% CI: 87%, 90%). Testing on prospective SR errors showed an accuracy of 75% (69 of 92; 95% CI: 65%, 83%). Finally, the correct word was the top suggestion for 45.6% (475 of 1041; 95% CI: 42.5%, 49.3%) of errors.

CONCLUSION

Radiology-specific BERT models fine-tuned on generated errors were able to identify SR errors in radiology reports and suggest corrections.Keywords: Computer Applications, Technology Assessment Supplemental material is available for this article. © RSNA, 2022See also the commentary by Abajian and Cheung in this issue.

A dataset of simulated patient-physician medical interviews with a focus on respiratory cases

Artificial Intelligence (AI) is playing a major role in medical education, diagnosis, and outbreak detection through Natural Language Processing (NLP), machine learning models and deep learning tools. However, in order to train AI to facilitate these medical fields, well-documented and accurate medical conversations are needed. The dataset presented covers a series of medical conversations in the format of Objective Structured Clinical Examinations (OSCE), with a focus on respiratory cases in audio format and corresponding text documents. These cases were simulated, recorded, transcribed, and manually corrected with the underlying aim of providing a comprehensive set of medical conversation data to the academic and industry community. Potential applications include speech recognition detection for speech-to-text errors, training NLP models to extract symptoms, detecting diseases, or for educational purposes, including training an avatar to converse with healthcare professional students as a standardized patient during clinical examinations. The application opportunities for the presented dataset are vast, given that this calibre of data is difficult to access and costly to develop.

The Effects of Captioning Errors, Background Noise, and Hearing Loss on Memory for Text-Captioned Speech

PURPOSE

Previous studies have suggested that the negative effects of acoustic challenge on speech memory can be attenuated with assistive text captions, particularly among older adults with hearing impairment. However, no studies have systematically examined the effects of text-captioning errors, which are common in automated speech recognition (ASR) systems.

METHOD

In two experiments, we examined memory for text-captioned speech (with and without background noise) when captions had no errors (control) or had one of three common ASR errors: substitution, deletion, or insertion errors.

RESULTS

In both Experiment 1 (young adults with normal hearing) and Experiment 2 (older adults with varying hearing acuity), we observed similar additive effects of caption errors and background noise, such that increased background noise and the presence of captioning errors negatively impacted memory outcomes. Notably, the negative effects of captioning errors were largest among older adults with increased hearing thresholds, suggesting that older adults with hearing loss may show an increased reliance on text captions compared to adults with normal hearing.

CONCLUSION

Our findings show that even a single-word error can be deleterious to memory for text-captioned speech, especially in older adults with hearing loss. Therefore, to produce the greatest benefit to memory, it is crucial that text captions are accurate.

Accuracy of Cloud-Based Speech Recognition Open Application Programming Interface for Medical Terms of Korean

BACKGROUND

There are limited data on the accuracy of cloud-based speech recognition (SR) open application programming interfaces (APIs) for medical terminology. This study aimed to evaluate the medical term recognition accuracy of current available cloud-based SR open APIs in Korean.

METHODS

We analyzed the SR accuracy of currently available cloud-based SR open APIs using real doctor-patient conversation recordings collected from an outpatient clinic at a large tertiary medical center in Korea. For each original and SR transcription, we analyzed the accuracy rate of each cloud-based SR open API (i.e., the number of medical terms in the SR transcription per number of medical terms in the original transcription).

RESULTS

A total of 112 doctor-patient conversation recordings were converted with three cloud-based SR open APIs (Naver Clova SR from Naver Corporation; Google Speech-to-Text from Alphabet Inc.; and Amazon Transcribe from Amazon), and each transcription was compared. Naver Clova SR (75.1%) showed the highest accuracy with the recognition of medical terms compared to the other open APIs (Google Speech-to-Text, 50.9%, P < 0.001; Amazon Transcribe, 57.9%, P < 0.001), and Amazon Transcribe demonstrated higher recognition accuracy compared to Google Speech-to-Text (P < 0.001). In the sub-analysis, Naver Clova SR showed the highest accuracy in all areas according to word classes, but the accuracy of words longer than five characters showed no statistical differences (Naver Clova SR, 52.6%; Google Speech-to-Text, 56.3%; Amazon Transcribe, 36.6%).

CONCLUSION

Among three current cloud-based SR open APIs, Naver Clova SR which manufactured by Korean company showed highest accuracy of medical terms in Korean, compared to Google Speech-to-Text and Amazon Transcribe. Although limitations are existing in the recognition of medical terminology, there is a lot of rooms for improvement of this promising technology by combining strengths of each SR engines.

Evaluation and comparison of errors on nursing notes created by online and offline speech recognition technology and handwritten: an interventional study

BACKGROUND

Despite the rapid expansion of electronic health records, the use of computer mouse and keyboard, challenges the data entry into these systems. Speech recognition software is one of the substitutes for the mouse and keyboard. The objective of this study was to evaluate the use of online and offline speech recognition software on spelling errors in nursing reports and to compare them with errors in handwritten reports.

METHODS

For this study, online and offline speech recognition software were selected and customized based on unrecognized terms by these softwares. Two groups of 35 nurses provided the admission notes of hospitalized patients upon their arrival using three data entry methods (using the handwritten method or two types of speech recognition software). After at least a month, they created the same reports using the other methods. The number of spelling errors in each method was determined. These errors were compared between the paper method and the two electronic methods before and after the correction of errors.

RESULTS

The lowest accuracy was related to online software with 96.4% and accuracy. On the average per report, the online method 6.76, and the offline method 4.56 generated more errors than the paper method. After correcting the errors by the participants, the number of errors in the online reports decreased by 94.75% and the number of errors in the offline reports decreased by 97.20%. The highest number of reports with errors was related to reports created by online software.

CONCLUSION

Although two software had relatively high accuracy, they created more errors than the paper method that can be lowered by optimizing and upgrading these softwares. The results showed that error correction by users significantly reduced the documentation errors caused by the software.

Application of the i-PARIHS framework in the implementation of speech recognition technology as a way of addressing documentation burden within a mental health context

Documentation burden continues to be a critical issue in the adoption of comprehensive electronic health record systems. This case study demonstrates how the i-PARIHS framework can be applied to support the implementation of interventions in reducing documentation and EHR-related burden in a mental health context. As part of pre-adoption implementation activities for Speech Recognition Technology (SRT), a cross-sectional survey was conducted with physicians, residents, and fellows at an academic mental health hospital to explore their perceptions on SRT. Open-ended responses and follow-up interviews explored challenges and concerns on using SRT in practice. Through an analysis using the i-PARIHS framework, key considerations were mapped across the four components of the framework. This study demonstrates the value of applying well-established implementation frameworks, such as the i-PARIHS framework, in mitigating challenges related to documentation burden. Future studies should explore how implementation frameworks can be systematically embedded in addressing EHR-related burden.

Automated Misspelling Detection and Correction in Persian Clinical Text

Accurate electronic health records are important for clinical care, research, and patient safety assurance. Correction of misspelled words is required to ensure the correct interpretation of medical records. In the Persian language, the lack of automated misspelling detection and correction system is evident in the medicine and health care. In this article, we describe the development of an automated misspelling detection and correction system for radiology and ultrasound's free texts in the Persian language. To achieve our goal, we used n-gram language model and three different types of free texts related to abdominal and pelvic ultrasound, head and neck ultrasound, and breast ultrasound reports. Our system achieved the detection performance of up to 90.29% for radiology and ultrasound's free texts with the correction accuracy of 88.56%. Results indicated that high-quality spelling correction is possible in clinical reports. The system also achieved significant savings during the documentation process and final approval of the reports in the imaging department.

A Comparison of Voice Recognition Program and Traditional Keyboard Charting for Nurse Documentation

The purposes of this study are threefold: (1) compare the document times between a voice recognition system and keyboard charting, (2) compare the number of errors between the two methods, and (3) identify factors influencing documentation time. Voice recognition systems are considered a potential solution to decrease documentation time. However, little is known to what extent voice recognition systems can save nurses' documentation time. A pilot, simulation study was conducted using a voice recognition system and keyboard charting with 15 acute care nurses. A crossover method with repeated measures was utilized. Each nurse was given two simple and two complex assessment scenarios, assigned in random order, to document using both methods. Paired t-tests and multivariate linear regression models were used for data analysis. The voice recognition method saved the nurses 2.3 minutes (simple scenario) and 6.1 minutes (complex scenario) on average and was statistically significant (P < .001). There were no significant differences in errors or factors identified influencing documentation times. Eighty percent reported a preference of using voice recognition systems, and 87% agreed this method helped speed up charting. This study can show how a voice recognition system can improve documentation times compared with keyboard charting while still having thorough documentation.

REFINING AUTOMATIC SPEECH RECOGNITION SYSTEM FOR OLDER ADULTS

Building a high quality automatic speech recognition (ASR) system with limited training data has been a challenging task particularly for a narrow target population. Open-sourced ASR systems, trained on sufficient data from adults, are susceptible on seniors' speech due to acoustic mismatch between adults and seniors. With 12 hours of training data, we attempt to develop an ASR system for socially isolated seniors (80+ years old) with possible cognitive impairments. We experimentally identify that ASR for the adult population performs poorly on our target population and transfer learning (TL) can boost the system's performance. Standing on the fundamental idea of TL, tuning model parameters, we further improve the system by leveraging an attention mechanism to utilize the model's intermediate information. Our approach achieves 1.58% absolute improvements over the TL model.

The Computer Will See You Now: Overcoming Barriers to Adoption of Computer-Assisted History Taking (CAHT) in Primary Care

Patient health information is increasingly collected through multiple modalities, including electronic health records, wearables, and connected devices. Computer-assisted history taking could provide an additional channel to collect highly relevant, comprehensive, and accurate patient information while reducing the burden on clinicians and face-to-face consultation time. Considering restrictions to consultation time and the associated negative health outcomes, patient-provided health data outside of consultation can prove invaluable in health care delivery. Over the years, research has highlighted the numerous benefits of computer-assisted history taking; however, the limitations have proved an obstacle to adoption. In this viewpoint, we review these limitations under 4 main categories (accessibility, affordability, accuracy, and acceptability) and discuss how advances in technology, computing power, and ubiquity of personal devices offer solutions to overcoming these.

An extensive review of tools for manual annotation of documents

MOTIVATION

Annotation tools are applied to build training and test corpora, which are essential for the development and evaluation of new natural language processing algorithms. Further, annotation tools are also used to extract new information for a particular use case. However, owing to the high number of existing annotation tools, finding the one that best fits particular needs is a demanding task that requires searching the scientific literature followed by installing and trying various tools.

METHODS

We searched for annotation tools and selected a subset of them according to five requirements with which they should comply, such as being Web-based or supporting the definition of a schema. We installed the selected tools (when necessary), carried out hands-on experiments and evaluated them using 26 criteria that covered functional and technical aspects. We defined each criterion on three levels of matches and a score for the final evaluation of the tools.

RESULTS

We evaluated 78 tools and selected the following 15 for a detailed evaluation: BioQRator, brat, Catma, Djangology, ezTag, FLAT, LightTag, MAT, MyMiner, PDFAnno, prodigy, tagtog, TextAE, WAT-SL and WebAnno. Full compliance with our 26 criteria ranged from only 9 up to 20 criteria, which demonstrated that some tools are comprehensive and mature enough to be used on most annotation projects. The highest score of 0.81 was obtained by WebAnno (of a maximum value of 1.0).

Physician perceptions of documentation methods in electronic health records

This study sought to determine physician, specialty and practice factors influencing choice of method for electronic health record (EHR) documentation: direct typing (DT), electronic transcription (ET), human transcription (HT), and scribes. A survey assessing physician documentation practices was developed and distributed online. The primary outcome was the proportion of physicians using each method. Secondary outcomes were provider-rated accuracy, efficiency, and ease of navigation on a 1-5 Likert scale. Means were compared using linear mixed models with Bonferroni adjustment. The 818 respondents were mostly outpatient (46%) adult (79%) physicians, practiced for a mean 15.8 years, and used DT for EHR documentation (72%). Emergency physicians were more likely to use scribes (p < 0.0001). DT was rated less efficient than all other methods (p < 0.0001). ET was rated less accurate than DT (p < 0.001) and HT (p < 0.001). HT was rated less easy to navigate than DT (p = 0.002) and scribe (p < 0.001), and ET less than scribe (p = 0.002). Two hundred and forty-three respondents provided free-text comments that further described opinions. DT was the most commonly used EHR method but rated least efficient. Scribes were rated easy to navigate and efficient but infrequently used outside of emergency settings. Further innovation is needed to design systems responsive to all physician EHR needs.

Revealing the most common reporting errors through data mining of the report proofreading process

Objectives

To investigate the most common errors in residents’ preliminary reports, if structured reporting impacts error types and frequencies, and to identify possible implications for resident education and patient safety.

Material and methods

Changes in report content were tracked by a report comparison tool on a word level and extracted for 78,625 radiology reports dictated from September 2017 to December 2018 in our department. Following data aggregation according to word stems and stratification by subspecialty (e.g., neuroradiology) and imaging modality, frequencies of additions/deletions were analyzed for findings and impression report section separately and compared between subgroups.

Results

Overall modifications per report averaged 4.1 words, with demonstrably higher amounts of changes for cross-sectional imaging (CT: 6.4; MRI: 6.7) than non-cross-sectional imaging (radiographs: 0.2; ultrasound: 2.8). The four most frequently changed words (right, left, one, and none) remained almost similar among all subgroups (range: 0.072–0.117 per report; once every 9–14 reports). Albeit representing only 0.02% of analyzed words, they accounted for up to 9.7% of all observed changes. Subspecialties solely using structured reporting had substantially lower change ratios in the findings report section (mean: 0.2 per report) compared with prose-style reporting subspecialties (mean: 2.0). Relative frequencies of the most changed words remained unchanged.

Conclusion

Residents’ most common reporting errors in all subspecialties and modalities are laterality discriminator confusions (left/right) and unnoticed descriptor misregistration by speech recognition (one/none). Structured reporting reduces overall error rates, but does not affect occurrence of the most common errors. Increased error awareness and measures improving report correctness and ensuring patient safety are required.

Key Points

• The two most common reporting errors in residents’ preliminary reports are laterality discriminator confusions (left/right) and unnoticed descriptor misregistration by speech recognition (one/none).

• Structured reporting reduces the overall the error frequency in the findings report section by a factor of 10 (structured reporting: mean 0.2 per report; prose-style reporting: 2.0) but does not affect the occurrence of the two major errors.

• Staff radiologist review behavior noticeably differs between radiology subspecialties.

Electronic supplementary material

The online version of this article (10.1007/s00330-020-07306-6) contains supplementary material, which is available to authorized users.

Examination and diagnosis of electronic patient records and their associated ethics: a scoping literature review

Background

Electronic patient record (EPR) technology is a key enabler for improvements to healthcare service and management. To ensure these improvements and the means to achieve them are socially and ethically desirable, careful consideration of the ethical implications of EPRs is indicated. The purpose of this scoping review was to map the literature related to the ethics of EPR technology. The literature review was conducted to catalogue the prevalent ethical terms, to describe the associated ethical challenges and opportunities, and to identify the actors involved. By doing so, it aimed to support the future development of ethics guidance in the EPR domain.

Methods

To identify journal articles debating the ethics of EPRs, Scopus, Web of Science, and PubMed academic databases were queried and yielded 123 eligible articles. The following inclusion criteria were applied: articles need to be in the English language; present normative arguments and not solely empirical research; include an abstract for software analysis; and discuss EPR technology.

Results

The medical specialty, type of information captured and stored in EPRs, their use and functionality varied widely across the included articles. Ethical terms extracted were categorised into clusters ‘privacy’, ‘autonomy’, ‘risk/benefit’, ‘human relationships’, and ‘responsibility’. The literature shows that EPR-related ethical concerns can have both positive and negative implications, and that a wide variety of actors with rights and/or responsibilities regarding the safe and ethical adoption of the technology are involved.

Conclusions

While there is considerable consensus in the literature regarding EPR-related ethical principles, some of the associated challenges and opportunities remain underdiscussed. For example, much of the debate is presented in a manner more in keeping with a traditional model of healthcare and fails to take account of the multidimensional ensemble of factors at play in the EPR era and the consequent need to redefine/modify ethical norms to align with a digitally-enabled health service. Similarly, the academic discussion focuses predominantly on bioethical values. However, approaches from digital ethics may also be helpful to identify and deliberate about current and emerging EPR-related ethical concerns.

Role of Artificial Intelligence in Patient Safety Outcomes: Systematic Literature Review

BACKGROUND

Artificial intelligence (AI) provides opportunities to identify the health risks of patients and thus influence patient safety outcomes.

OBJECTIVE

The purpose of this systematic literature review was to identify and analyze quantitative studies utilizing or integrating AI to address and report clinical-level patient safety outcomes.

METHODS

We restricted our search to the PubMed, PubMed Central, and Web of Science databases to retrieve research articles published in English between January 2009 and August 2019. We focused on quantitative studies that reported positive, negative, or intermediate changes in patient safety outcomes using AI apps, specifically those based on machine-learning algorithms and natural language processing. Quantitative studies reporting only AI performance but not its influence on patient safety outcomes were excluded from further review.

RESULTS

We identified 53 eligible studies, which were summarized concerning their patient safety subcategories, the most frequently used AI, and reported performance metrics. Recognized safety subcategories were clinical alarms (n=9; mainly based on decision tree models), clinical reports (n=21; based on support vector machine models), and drug safety (n=23; mainly based on decision tree models). Analysis of these 53 studies also identified two essential findings: (1) the lack of a standardized benchmark and (2) heterogeneity in AI reporting.

CONCLUSIONS

This systematic review indicates that AI-enabled decision support systems, when implemented correctly, can aid in enhancing patient safety by improving error detection, patient stratification, and drug management. Future work is still needed for robust validation of these systems in prospective and real-world clinical environments to understand how well AI can predict safety outcomes in health care settings.

Assessing the accuracy of automatic speech recognition for psychotherapy

Accurate transcription of audio recordings in psychotherapy would improve therapy effectiveness, clinician training, and safety monitoring. Although automatic speech recognition software is commercially available, its accuracy in mental health settings has not been well described. It is unclear which metrics and thresholds are appropriate for different clinical use cases, which may range from population descriptions to individual safety monitoring. Here we show that automatic speech recognition is feasible in psychotherapy, but further improvements in accuracy are needed before widespread use. Our HIPAA-compliant automatic speech recognition system demonstrated a transcription word error rate of 25%. For depression-related utterances, sensitivity was 80% and positive predictive value was 83%. For clinician-identified harm-related sentences, the word error rate was 34%. These results suggest that automatic speech recognition may support understanding of language patterns and subgroup variation in existing treatments but may not be ready for individual-level safety surveillance.

Frequency and Types of Patient-Reported Errors in Electronic Health Record Ambulatory Care Notes

Importance

As health information transparency increases, patients more often seek their health data. More than 44 million patients in the US can now readily access their ambulatory visit notes online, and the practice is increasing abroad. Few studies have assessed documentation errors that patients identify in their notes and how these may inform patient engagement and safety strategies.

Objective

To assess the frequency and types of errors identified by patients who read open ambulatory visit notes.

Design, Setting, and Participants

In this survey study, a total of 136 815 patients at 3 US health care organizations with open notes, including 79 academic and community ambulatory care practices, received invitations to an online survey from June 5 to October 20, 2017. Patients who had at least 1 ambulatory note and had logged onto the portal at least once in the past 12 months were included. Data analysis was performed from July 3, 2018, to April 27, 2020.

Exposures

Access to ambulatory care open notes through patient portals for up to 7 years (2010-2017).

Main Outcomes and Measures

Proportion of patients reporting a mistake and how serious they perceived the mistake to be, factors associated with finding errors characterized by patients as serious, and categories of patient-reported errors.

Results

Of 136 815 patients who received survey invitations, 29 656 (21.7%) responded and 22 889 patients (mean [SD] age, 55.16 [15.96] years; 14 447 [63.1%] female; 18 301 [80.0%] white) read 1 or more notes in the past 12 months and completed error questions. Of these patients, 4830 (21.1%) reported a perceived mistake and 2043 (42.3%) reported that the mistake was serious (somewhat serious: 1563 [32.4%]; very serious: 480 [9.9%]). In multivariable analysis, female patients (relative risk [RR], 1.79; 95% CI, 1.72-1.85), more educated patients (RR, 1.38; 95% CI, 1.29-1.48), sicker patients (RR, 1.89; 95% CI, 1.84-1.94), those aged 45 to 64 years (RR, 2.23; 95% CI, 2.06-2.42), those 65 years or older (RR, 2.00; 95% CI, 1.73-2.32), and those who read more than 1 note (2-3 notes: RR, 1.82; 95% CI, 1.34-2.47; ≥4 notes: RR, 3.09; 95% CI, 2.02-4.73) were more likely to report a mistake that they found to be serious compared with their reference groups. After categorization of patient-reported very serious mistakes, those specifically mentioning the word diagnosis or describing a specific error in current or past diagnoses were most common (98 of 356 [27.5%]), followed by inaccurate medical history (85 of 356 [23.9%]), medications or allergies (50 of 356 [14.0%]), and tests, procedures, or results (30 of 356 [8.4%]). A total of 23 (6.5%) reflected notes reportedly written on the wrong patient. Of 433 very serious errors, 255 (58.9%) included at least 1 perceived error potentially associated with the diagnostic process (eg, history, physical examination, tests, referrals, and communication).

Conclusions and Relevance

In this study, patients who read ambulatory notes online perceived mistakes, a substantial proportion of which they found to be serious. Older and sicker patients were twice as likely to report a serious error compared with younger and healthier patients, indicating important safety and quality implications. Sharing notes with patients may help engage them to improve record accuracy and health care safety together with practitioners.

Pilot trial of semi-automated medical note writing using lexeme hypotheses

Clinicians write a billion free text notes per year. These notes are typically replete with errors of all types. No established automated method can extract data from this treasure trove. The practice of medicine therefore remains haphazard and chaotic, resulting in vast economic waste. The lexeme hypotheses are based on our analysis of how records are created. They enable a computer system to predict what issue a clinician will need to address next, based on the environment in which the clinician is working, and what responses the clinician has selected to date. The system uses a lexicon storing the issues (queries) and a range of responses to the issues. When the clinician selects a response, a text fragment is added to the output file. In the first phase of this work, the notes of 69 returning hemophilia patients were scrutinized, and the lexicon was expanded to 847 lexeme queries and 7995 responses to enable the construction of completed notes. The quality of lexeme-generated notes from 20 consecutive subjects was then compared to the clinicians' conventional clinic notes. The system generated grammatically correct notes. In comparison to the traditional clinic note, the lexeme-generated notes were more complete (88 % compared with 62 %), and had less typographical and grammatical errors (0.8 versus 3.5 errors per note). The system notes and traditional notes averaged about 800 words, but the traditional notes had a much wider distribution of lengths. The note-creation rate from marshalling the data to completion using the system averaged 80 wpm, twice as fast as the typical clinician can type. The lexeme method generates more complete, grammatical and organized notes faster than traditional methods. The notes are completely computerized at inception, and they incorporate prompts for clinicians to address otherwise overlooked items. This pilot justifies further exploration of this methodology.

Do you understand the words that are comin outta my mouth? Voice assistant comprehension of medication names

This study investigated the speech recognition abilities of popular voice assistants when being verbally asked about commonly dispensed medications by a variety of participants. Voice recordings of 46 participants (12 of which had a foreign accent in English) were played back to Amazon’s Alexa, Google Assistant, and Apple’s Siri for the brand- and generic names of the top 50 most dispensed medications in the United States. A repeated measures ANOVA indicated that Google Assistant achieved the highest comprehension accuracy for both brand medication names (M = 91.8%, SD = 4.2) and generic medication names (M = 84.3%, SD = 11.2), followed by Siri (brand names M = 58.5%, SD = 11.2; generic names M = 51.2%, SD = 16.0), and the lowest accuracy by Alexa (brand names M = 54.6%, SD = 10.8; generic names M = 45.5%, SD = 15.4). An interaction between voice assistant and participant accent was also found, demonstrating lower comprehension performance overall for those with a foreign accent using Siri (M = 48.8%, SD = 11.8) and Alexa (M = 41.7%, SD = 12.7), compared to participants without a foreign accent (Siri M = 57.0%, SD = 11.7; Alexa M = 53.0%, SD = 10.9). No significant difference between participant accents were found for Google Assistant. These findings show a substantial performance lead for Google Assistant compared to its voice assistant competitors when comprehending medication names, but there is still room for improvement.

Speech recognition for clinical documentation from 1990 to 2018: a systematic review

OBJECTIVE

The study sought to review recent literature regarding use of speech recognition (SR) technology for clinical documentation and to understand the impact of SR on document accuracy, provider efficiency, institutional cost, and more.

MATERIALS AND METHODS

We searched 10 scientific and medical literature databases to find articles about clinician use of SR for documentation published between January 1, 1990, and October 15, 2018. We annotated included articles with their research topic(s), medical domain(s), and SR system(s) evaluated and analyzed the results.

RESULTS

One hundred twenty-two articles were included. Forty-eight (39.3%) involved the radiology department exclusively and 10 (8.2%) involved emergency medicine; 10 (8.2%) mentioned multiple departments. Forty-eight (39.3%) articles studied productivity; 20 (16.4%) studied the effect of SR on documentation time, with mixed findings. Decreased turnaround time was reported in all 19 (15.6%) studies in which it was evaluated. Twenty-nine (23.8%) studies conducted error analyses, though various evaluation metrics were used. Reported percentage of documents with errors ranged from 4.8% to 71%; reported word error rates ranged from 7.4% to 38.7%. Seven (5.7%) studies assessed documentation-associated costs; 5 reported decreases and 2 reported increases. Many studies (44.3%) used products by Nuance Communications. Other vendors included IBM (9.0%) and Philips (6.6%); 7 (5.7%) used self-developed systems.

CONCLUSION

Despite widespread use of SR for clinical documentation, research on this topic remains largely heterogeneous, often using different evaluation metrics with mixed findings. Further, that SR-assisted documentation has become increasingly common in clinical settings beyond radiology warrants further investigation of its use and effectiveness in these settings.

Machine Learning in Relation to Emergency Medicine Clinical and Operational Scenarios: An Overview

Health informatics is a vital technology that holds great promise in the healthcare setting. We describe two prominent health informatics tools relevant to emergency care, as well as the historical background and the current state of informatics. We also identify recent research findings and practice changes. The recent advances in machine learning and natural language processing (NLP) are a prominent development in health informatics overall and relevant in emergency medicine (EM). A basic comprehension of machine-learning algorithms is the key to understand the recent usage of artificial intelligence in healthcare. We are using NLP more in clinical use for documentation. NLP has started to be used in research to identify clinically important diseases and conditions. Health informatics has the potential to benefit both healthcare providers and patients. We cover two powerful tools from health informatics for EM clinicians and researchers by describing the previous successes and challenges and conclude with their implications to emergency care.

The digital transformation of medicine can revitalize the patient-clinician relationship

Health professionals within the medical community feel that the principles of humanism in medicine have not been a point of emphasis for information and computer technology in healthcare. There is concern that the electronic health record is eroding the patient-clinician relationship and distancing clinicians from their patients. New analytic technologies, on the contrary, by taking over repetitive and mundane tasks, can provide an avenue to make medical care more patient-centered by freeing clinicians' time, and the time of the whole clinical care team, to engage with patients. Technology such as advanced speech recognition that optimizes clinicians' workflow could revitalize the patient-clinician relationship and perhaps also improve clinician well-being. Digital phenotyping can gain invaluable additional data from patients using technology that is already used for personal reasons by the majority of patients. The digital transformation of healthcare has the potential to make healthcare more humane and personalized, however, several important steps are needed to avoid the pitfalls that have come with prior iterations of information technology in medicine such as a heightened emphasis on data security and transparency. Both patients and clinicians should be involved from the early stages of development of medical technologies to ensure that they are person-centric. Technologists and engineers developing healthcare technologies should have experiences with the delivery of healthcare and the lives of patients and clinicians. These steps are necessary to develop a common commitment to the design concept that technology and humane care are not mutually exclusive, and in fact, can be symbiotic.