The IMPACT framework and implementation for accessible in silico clinical phenotyping in the digital era

Clinical phenotyping is often a foundational requirement for obtaining datasets necessary for the development of digital health applications. Traditionally done via manual abstraction, this task is often a bottleneck in development due to time and cost requirements, therefore raising significant interest in accomplishing this task via in-silico means. Nevertheless, current in-silico phenotyping development tends to be focused on a single phenotyping task resulting in a dearth of reusable tools supporting cross-task generalizable in-silico phenotyping. In addition, in-silico phenotyping remains largely inaccessible for a substantial portion of potentially interested users. Here, we highlight the barriers to the usage of in-silico phenotyping and potential solutions in the form of a framework of several desiderata as observed during our implementation of such tasks. In addition, we introduce an example implementation of said framework as a software application, with a focus on ease of adoption, cross-task reusability, and facilitating the clinical phenotyping algorithm development process.

Beyond mathematics, statistics, and programming: data science, machine learning, and artificial intelligence competencies and curricula for clinicians, informaticians, science journalists, and researchers

Data science, machine learning and artificial intelligence applications impact clinicians, informaticians, science journalists, and researchers. Most biomedical data science training focuses on learning a programming language in addition to higher mathematics and advanced statistics. This approach is appropriate for graduate students but greatly reduces the number of individuals in healthcare who can be involved in data science. To serve these four stakeholder audiences, we describe several curricular strategies focusing on solving real problems of interest to these audiences. Relevant competencies for these audiences include using intuitive programming tools that facilitate data exploration with minimal programming background, creating data models, evaluating results of data analyses, and assessing data science research reports, among others. Offering the curricula described here more broadly could broaden the stakeholder groups knowledgeable about and engaged in data science.

The International Academy of Health Sciences Informatics: 2021 Update

OBJECTIVES

To summarize the activities of the International Academy of Health Sciences Informatics (IAHSI) in 2021 and welcome its 2021 Class of Fellows.

METHODS

Report on governance, strategic directions, newly elected fellows, plenary meetings, and other activities of the Academy.

RESULTS

As in 2020, all of the Academy's activities were carried out virtually due to the COVID-19 pandemic. In 2021, new Board members were elected. Strategic activities in data standards and interoperability and in mentorship moved forward. A new class of 26 Fellows was elected, bringing the total membership of the Academy to 204 Fellows from all regions of the world. In addition, a virtual plenary meeting was held.

CONCLUSIONS

The Academy has continued to pursue its role as the honorific society globally for biomedical and health informatics. Expansion of strategic activities and membership will continue moving forward.

Clinical study applying machine learning to detect a rare disease: results and lessons learned

Machine learning has the potential to improve identification of patients for appropriate diagnostic testing and treatment, including those who have rare diseases for which effective treatments are available, such as acute hepatic porphyria (AHP). We trained a machine learning model on 205 571 complete electronic health records from a single medical center based on 30 known cases to identify 22 patients with classic symptoms of AHP that had neither been diagnosed nor tested for AHP. We offered urine porphobilinogen testing to these patients via their clinicians. Of the 7 who agreed to testing, none were positive for AHP. We explore the reasons for this and provide lessons learned for further work evaluating machine learning to detect AHP and other rare diseases.

The International Academy of Health Sciences Informatics (IAHSI): 2020 Report

OBJECTIVES

To summarize the major activities of the International Academy of Health Sciences Informatics (IAHSI) in the 2020 time period and to welcome its 2020 Class of Fellows.

METHOD

Report from the members of the Academy's Board.

RESULTS

Due to the SARS-CoV-2 pandemic, both Plenary meetings in 2020 had to be organized as virtual meetings. Scientific discussions, focusing on mobilizing computable biomedical knowledge and on data standards and interoperability formed major parts of these meetings. A statement on the use of informatics in pandemic situations was elaborated and sent to the World Health Organization. A panel on data standards and interoperability started its work. 34 Fellows were welcomed in the 2020 Class of Fellows so that the Academy now consists of 179 members.

CONCLUSIONS

There was a shift from supporting to strategic activities in the Academy's work. After having achieved organizational stability, the Academy can now focus on its strategic work and so on its main objective.

Searching for scientific evidence in a pandemic: An overview of TREC-COVID

We present an overview of the TREC-COVID Challenge, an information retrieval (IR) shared task to evaluate search on scientific literature related to COVID-19. The goals of TREC-COVID include the construction of a pandemic search test collection and the evaluation of IR methods for COVID-19. The challenge was conducted over five rounds from April to July 2020, with participation from 92 unique teams and 556 individual submissions. A total of 50 topics (sets of related queries) were used in the evaluation, starting at 30 topics for Round 1 and adding 5 new topics per round to target emerging topics at that state of the still-emerging pandemic. This paper provides a comprehensive overview of the structure and results of TREC-COVID. Specifically, the paper provides details on the background, task structure, topic structure, corpus, participation, pooling, assessment, judgments, results, top-performing systems, lessons learned, and benchmark datasets.

Digital Health during COVID-19: Informatics Dialogue with the World Health Organization

Background : On December 16, 2020 representatives of the International Medical Informatics Association (IMIA), a Non-Governmental Organization in official relations with the World Health Organization (WHO), along with its International Academy for Health Sciences Informatics (IAHSI), held an open dialogue with WHO Director General (WHO DG) Tedros Adhanom Ghebreyesus about the opportunities and challenges of digital health during the COVID-19 global pandemic.

Objectives : The aim of this paper is to report the outcomes of the dialogue and discussions with more than 200 participants representing different civil society organizations (CSOs).

Methods : The dialogue was held in form of a webinar. After an initial address of the WHO DG, short presentations by the panelists, and live discussions between panelists, the WHO DG and WHO representatives took place. The audience was able to post questions in written. These written discussions were saved with participants’ consent and summarized in this paper.

Results : The main themes that were brought up by the audience for discussion were: (a) opportunities and challenges in general; (b) ethics and artificial intelligence; (c) digital divide; (d) education. Proposed actions included the development of a roadmap based on the lessons learned from the COVID-19 pandemic.

Conclusions : Decision making by policy makers needs to be evidence-based and health informatics research should be used to support decisions surrounding digital health, and we further propose next steps in the collaboration between IMIA and WHO such as future engagement in the World Health Assembly.

A comparative analysis of system features used in the TREC-COVID information retrieval challenge

The COVID-19 pandemic has resulted in a rapidly growing quantity of scientific publications from journal articles, preprints, and other sources. The TREC-COVID Challenge was created to evaluate information retrieval (IR) methods and systems for this quickly expanding corpus. Using the COVID-19 Open Research Dataset (CORD-19), several dozen research teams participated in over 5 rounds of the TREC-COVID Challenge. While previous work has compared IR techniques used on other test collections, there are no studies that have analyzed the methods used by participants in the TREC-COVID Challenge. We manually reviewed team run reports from Rounds 2 and 5, extracted features from the documented methodologies, and used a univariate and multivariate regression-based analysis to identify features associated with higher retrieval performance. We observed that fine-tuning datasets with relevance judgments, MS-MARCO, and CORD-19 document vectors was associated with improved performance in Round 2 but not in Round 5. Though the relatively decreased heterogeneity of runs in Round 5 may explain the lack of significance in that round, fine-tuning has been found to improve search performance in previous challenge evaluations by improving a system’s ability to map relevant queries and phrases to documents. Furthermore, term expansion was associated with improvement in system performance, and the use of the narrative field in the TREC-COVID topics was associated with decreased system performance in both rounds. These findings emphasize the need for clear queries in search. While our study has some limitations in its generalizability and scope of techniques analyzed, we identified some IR techniques that may be useful in building search systems for COVID-19 using the TREC-COVID test collections.

Evaluation of patient-level retrieval from electronic health record data for a cohort discovery task

OBJECTIVE

Growing numbers of academic medical centers offer patient cohort discovery tools to their researchers, yet the performance of systems for this use case is not well understood. The objective of this research was to assess patient-level information retrieval methods using electronic health records for different types of cohort definition retrieval.

MATERIALS AND METHODS

We developed a test collection consisting of about 100 000 patient records and 56 test topics that characterized patient cohort requests for various clinical studies. Automated information retrieval tasks using word-based approaches were performed, varying 4 different parameters for a total of 48 permutations, with performance measured using B-Pref. We subsequently created structured Boolean queries for the 56 topics for performance comparisons. In addition, we performed a more detailed analysis of 10 topics.

RESULTS

The best-performing word-based automated query parameter settings achieved a mean B-Pref of 0.167 across all 56 topics. The way a topic was structured (topic representation) had the largest impact on performance. Performance not only varied widely across topics, but there was also a large variance in sensitivity to parameter settings across the topics. Structured queries generally performed better than automated queries on measures of recall and precision but were still not able to recall all relevant patients found by the automated queries.

CONCLUSION

While word-based automated methods of cohort retrieval offer an attractive solution to the labor-intensive nature of this task currently used at many medical centers, we generally found suboptimal performance in those approaches, with better performance obtained from structured Boolean queries. Future work will focus on using the test collection to develop and evaluate new approaches to query structure, weighting algorithms, and application of semantic methods.

TREC-COVID: rationale and structure of an information retrieval shared task for COVID-19

TREC-COVID is an information retrieval (IR) shared task initiated to support clinicians and clinical research during the COVID-19 pandemic. IR for pandemics breaks many normal assumptions, which can be seen by examining 9 important basic IR research questions related to pandemic situations. TREC-COVID differs from traditional IR shared task evaluations with special considerations for the expected users, IR modality considerations, topic development, participant requirements, assessment process, relevance criteria, evaluation metrics, iteration process, projected timeline, and the implications of data use as a post-task test collection. This article describes how all these were addressed for the particular requirements of developing IR systems under a pandemic situation. Finally, initial participation numbers are also provided, which demonstrate the tremendous interest the IR community has in this effort.

Addressing the Search Challenges of Precision Medicine with Information Retrieval Systems and Physician Readers

The Text REtrieval Conference (TREC), co-sponsored by the National Institute of Standards and Technology (NIST) in the US and US Department of Defense, was started in 1992. TREC's purpose is to support research within the information retrieval community by providing the infrastructure necessary for large-scale evaluation of text retrieval methodologies. In 2017, the TREC Precision Medicine (Roberts et al., 2017) track grew from the Clinical Decision Support track and focused on a narrower problem domain of precision oncology. After three years of computer runs being evaluated for relevance by physician readers, we provide a unique perspective of how to evaluate computer-generated articles and clinical trials pulled from PubMed and Clinicaltrials.gov to find relevant information on medical cases.

TREC-COVID

TREC-COVID is a community evaluation designed to build a test collection that captures the information needs of biomedical researchers using the scientific literature during a pandemic. One of the key characteristics of pandemic search is the accelerated rate of change: the topics of interest evolve as the pandemic progresses and the scientific literature in the area explodes. The COVID-19 pandemic provides an opportunity to capture this progression as it happens. TREC-COVID, in creating a test collection around COVID-19 literature, is building infrastructure to support new research and technologies in pandemic search.

Protecting healing relationships in the age of electronic health records: report from an international conference

We present findings of an international conference of diverse participants exploring the influence of electronic health records (EHRs) on the patient–practitioner relationship. Attendees united around a belief in the primacy of this relationship and the importance of undistracted attention. They explored administrative, regulatory, and financial requirements that have guided United States (US) EHR design and challenged patient-care documentation, usability, user satisfaction, interconnectivity, and data sharing. The United States experience was contrasted with those of other nations, many of which have prioritized patient-care documentation rather than billing requirements and experienced high user satisfaction. Conference participants examined educational methods to teach diverse learners effective patient-centered EHR use, including alternative models of care delivery and documentation, and explored novel ways to involve patients as healthcare partners like health-data uploading, chart co-creation, shared practitioner notes, applications, and telehealth. Future best practices must preserve human relationships, while building an effective patient–practitioner (or team)-EHR triad.

Automatic Indexing of Abstracts via Natural-language Processing Using a Simple Thesaurus

The authors describe CLARIT processing as an approach to automatic indexing. They also explore two elements of the indexing process, identifying concepts in text and selecting concepts to reflect the perspective of a domain.

Characteristics of the healthcare information technology workforce in the HITECH era: underestimated in size, still growing, and adapting to advanced uses

Objective

There is little readily available data about the size and characteristics of the healthcare information technology workforce. We sought to update a previous description of the size, growth, and characteristics of this workforce based on the Healthcare Information Management Systems Society (HIMSS) Analytics^® Database, a resource that includes hospital size, number of beds, amount of staffing, and an eight-stage model of electronic health record adoption (Electronic Medical Record Adoption Model, EMRAM℠).

Materials and Methods

We updated an analysis done using a 2007 snapshot of the HIMSS Analytics Database with a comparable snapshot from 2014 in order to estimate the size of the current workforce and project future needs. For the 2014 data, we applied the same weighted average analysis used in 2007 to obtain a ratio of information technology (IT) hospital full-time equivalent (FTE) to staffed beds, extrapolate the results to all US hospitals, and project the workforce needs as hospitals achieve higher EMRAM stages.

Results

Our estimated size of the healthcare information technology workforce in the US in 2014 was 161 160, which was 8.0% larger than the estimate based on the 2007 data. Based on the new data, we project a potential need for an additional 19 852 and 153 114 FTE, if all hospitals were to achieve EMRAM Stages 6 and 7, respectively. The distribution of FTE across job function category varies by EMRAM stage.

Discussion and Conclusions

Although these data are limited, especially for EMRAM Stage 7, there is likely need for substantial workforce growth as hospitals increase their adoption of advanced healthcare information technology. Further research with data better focused on workforce characteristics will provide a better picture of staffing requirements.

Evaluation of Meta-1 for a Concept-based Approach to the Automated Indexing and Retrieval of Bibliographic and Full-text Databases

SAPHIRE is a concept-based approach to information retrieval in the biomedical domain. Indexing and retrieval are based on a concept-matching algorithm that processes free text to identify concepts and map them to their canonical form. This process requires a large vocabulary containing a breadth of medical concepts and a diversity of synonym forms, which is provided by the Meta-1 vocabulary from the Unified Medical Language System Project of the National Library of Medicine. This paper describes the use of Meta-1 in SAPHIRE and an evaluation of both entities in the context of an information retrieval study.

Five Years of Clinical Informatics Board Certification for Physicians in the United States of America

Objectives:  To review the highlights of the new Clinical Informatics subspecialty including its history, certification requirements, development of and performance on the certification examination in the United States.

Methods:  We reviewed processes for the development of a subspecialty. Data from board certification examinations were collated and analyzed. We discussed eligibility requirements in the fellowship as well as practice pathways.

Results:  Lessons learned from the development of the Clinical Informatics subspecialty, opportunities, challenges, and future directions for the field are discussed.

Conclusions:  There remains a need for fellowship programs and creation and maintenance of a professional home for the subspecialty with the American Medical Informatics Association. Ongoing attention to the currency of the core content is required to maintain an examination designed to test the key concepts within the field of Clinical Informatics.

A publicly available benchmark for biomedical dataset retrieval: the reference standard for the 2016 bioCADDIE dataset retrieval challenge

Database URL

https://biocaddie.org/benchmark-data.

Early experiences of accredited clinical informatics fellowships

Since the launch of the clinical informatics subspecialty for physicians in 2013, over 1100 physicians have used the practice and education pathways to become board-certified in clinical informatics. Starting in 2018, only physicians who have completed a 2-year clinical informatics fellowship program accredited by the Accreditation Council on Graduate Medical Education will be eligible to take the board exam. The purpose of this viewpoint piece is to describe the collective experience of the first four programs accredited by the Accreditation Council on Graduate Medical Education and to share lessons learned in developing new fellowship programs in this novel medical subspecialty.

Barriers and facilitators to exchanging health information: a systematic review

OBJECTIVES

We conducted a systematic review of studies assessing facilitators and barriers to use of health information exchange (HIE).

METHODS

We searched MEDLINE, PsycINFO, CINAHL, and the Cochrane Library databases between January 1990 and February 2015 using terms related to HIE. English-language studies that identified barriers and facilitators of actual HIE were included. Data on study design, risk of bias, setting, geographic location, characteristics of the HIE, perceived barriers and facilitators to use were extracted and confirmed.

RESULTS

Ten cross-sectional, seven multiple-site case studies, and two before-after studies that included data from several sources (surveys, interviews, focus groups, and observations of users) evaluated perceived barriers and facilitators to HIE use. The most commonly cited barriers to HIE use were incomplete information, inefficient workflow, and reports that the exchanged information that did not meet the needs of users. The review identified several facilitators to use.

DISCUSSION

Incomplete patient information was consistently mentioned in the studies conducted in the US but not mentioned in the few studies conducted outside of the US that take a collective approach toward healthcare. Individual patients and practices in the US may exercise the right to participate (or not) in HIE which effects the completeness of patient information available to be exchanged. Workflow structure and user roles are key but understudied.

CONCLUSIONS

We identified several facilitators in the studies that showed promise in promoting electronic health data exchange: obtaining more complete patient information; thoughtful workflow that folds in HIE; and inclusion of users early in implementation.

Outcomes From Health Information Exchange: Systematic Review and Future Research Needs

BACKGROUND

Health information exchange (HIE), the electronic sharing of clinical information across the boundaries of health care organizations, has been promoted to improve the efficiency, cost-effectiveness, quality, and safety of health care delivery.

OBJECTIVE

To systematically review the available research on HIE outcomes and analyze future research needs.

METHODS

Data sources included citations from selected databases from January 1990 to February 2015. We included English-language studies of HIE in clinical or public health settings in any country. Data were extracted using dual review with adjudication of disagreements.

RESULTS

We identified 34 studies on outcomes of HIE. No studies reported on clinical outcomes (eg, mortality and morbidity) or identified harms. Low-quality evidence generally finds that HIE reduces duplicative laboratory and radiology testing, emergency department costs, hospital admissions (less so for readmissions), and improves public health reporting, ambulatory quality of care, and disability claims processing. Most clinicians attributed positive changes in care coordination, communication, and knowledge about patients to HIE.

CONCLUSIONS

Although the evidence supports benefits of HIE in reducing the use of specific resources and improving the quality of care, the full impact of HIE on clinical outcomes and potential harms are inadequately studied. Future studies must address comprehensive questions, use more rigorous designs, and employ a standard for describing types of HIE.

TRIAL REGISTRATION

PROSPERO Registry No CRD42014013285; http://www.crd.york.ac.uk/PROSPERO/ display_record.asp?ID=CRD42014013285 (Archived by WebCite at http://www.webcitation.org/6dZhqDM8t).

Participation in EHR based simulation improves recognition of patient safety issues

BACKGROUND

Electronic health records (EHR) are becoming increasingly integrated into the clinical environment. With the rapid proliferation of EHRs, a number of studies document an increase in adverse patient safety issues due to the EHR-user interface. Because of these issues, greater attention has been placed on novel educational activities which incorporate use of the EHR. The ICU environment presents many challenges to integrating an EHR given the vast amounts of data recorded each day, which must be interpreted to deliver safe and effective care. We have used a novel EHR based simulation exercise to demonstrate that everyday users fail to recognize a majority of patient safety issues in the ICU. We now sought to determine whether participation in the simulation improves recognition of said issues.

METHODS

Two ICU cases were created in our EHR simulation environment. Each case contained 14 safety issues, which differed in content but shared common themes. Residents were given 10 minutes to review a case followed by a presentation of management changes. Participants were given an immediate debriefing regarding missed issues and strategies for data gathering in the EHR. Repeated testing was performed in a cohort of subjects with the other case at least 1 week later.

RESULTS

116 subjects have been enrolled with 25 subjects undergoing repeat testing. There was no difference between cases in recognition of patient safety issues (39.5% vs. 39.4%). Baseline performance for subjects who participated in repeat testing was no different than the cohort as a whole. For both cases, recognition of safety issues was significantly higher among repeat participants compared to first time participants. Further, individual performance improved from 39.9% to 63.6% (p = 0.0002), a result independent of the order in which the cases were employed. The degree of improvement was inversely related to baseline performance. Further, repeat participants demonstrated a higher rate of recognition of changes in vitals, misdosing of antibiotics and oversedation compared to first time participants.

CONCLUSION

Participation in EHR simulation improves EHR use and identification of patient safety issues.

Beyond information retrieval and electronic health record use: competencies in clinical informatics for medical education

Physicians in the 21st century will increasingly interact in diverse ways with information systems, requiring competence in many aspects of clinical informatics. In recent years, many medical school curricula have added content in information retrieval (search) and basic use of the electronic health record. However, this omits the growing number of other ways that physicians are interacting with information that includes activities such as clinical decision support, quality measurement and improvement, personal health records, telemedicine, and personalized medicine. We describe a process whereby six faculty members representing different perspectives came together to define competencies in clinical informatics for a curriculum transformation process occurring at Oregon Health & Science University. From the broad competencies, we also developed specific learning objectives and milestones, an implementation schedule, and mapping to general competency domains. We present our work to encourage debate and refinement as well as facilitate evaluation in this area.

A national strategy to develop pragmatic clinical trials infrastructure

An important challenge in comparative effectiveness research is the lack of infrastructure to support pragmatic clinical trials, which compare interventions in usual practice settings and subjects. These trials present challenges that differ from those of classical efficacy trials, which are conducted under ideal circumstances, in patients selected for their suitability, and with highly controlled protocols. In 2012, we launched a 1-year learning network to identify high-priority pragmatic clinical trials and to deploy research infrastructure through the NIH Clinical and Translational Science Awards Consortium that could be used to launch and sustain them. The network and infrastructure were initiated as a learning ground and shared resource for investigators and communities interested in developing pragmatic clinical trials. We followed a three-stage process of developing the network, prioritizing proposed trials, and implementing learning exercises that culminated in a 1-day network meeting at the end of the year. The year-long project resulted in five recommendations related to developing the network, enhancing community engagement, addressing regulatory challenges, advancing information technology, and developing research methods. The recommendations can be implemented within 24 months and are designed to lead toward a sustained national infrastructure for pragmatic trials.

Recommendations for the use of operational electronic health record data in comparative effectiveness research

There is an increasing amount of clinical data in operational electronic health record (EHR) systems. Such data provide substantial opportunities for their re-use for many purposes, including comparative effectiveness research (CER). In a previous paper, we identified a number of caveats related to the use of such data, noting that they may be inaccurate, incomplete, transformed in ways that undermine their meaning, unrecoverable for research, of unknown provenance, of insufficient granularity, or incompatible with research protocols. In this paper, we provide recommendations for overcoming these caveats with the goal of leveraging such data to benefit CER and other health care activities. These recommendations include adaptation of "best evidence" approaches to use of data; processes to evaluate availability, completeness, quality, and transformability of data; creation of tools to manage data and their attributes; determination of metrics for assessing whether data are "research grade"; development of methods for comparative validation of data; construction of a methodology database for methods involving use of clinical data; standardized reporting methods for data and their attributes; appropriate use of informatics expertise; and a research agenda to determine biases inherent in operational data and to assess informatics approaches to their improvement.

Caveats for the use of operational electronic health record data in comparative effectiveness research

The growing amount of data in operational electronic health record systems provides unprecedented opportunity for its reuse for many tasks, including comparative effectiveness research. However, there are many caveats to the use of such data. Electronic health record data from clinical settings may be inaccurate, incomplete, transformed in ways that undermine their meaning, unrecoverable for research, of unknown provenance, of insufficient granularity, and incompatible with research protocols. However, the quantity and real-world nature of these data provide impetus for their use, and we develop a list of caveats to inform would-be users of such data as well as provide an informatics roadmap that aims to insure this opportunity to augment comparative effectiveness research can be best leveraged.

10x10 comes full circle: Spanish version back to United States in Puerto Rico

The adaptation of the 10x10 certificate program in health information systems for a Puerto Rican audience is described. The 10x10 program was initially developed in the USA by the Oregon Health Sciences University (OHSU), then adapted to Latin America by Hospital Italiano de Buenos Aires. Puerto Rico is in the intersection of the United States and Latin America, in terms of government, health care system, culture and language. Therefore, it seemed reasonable to re-adapt the program back to the USA, in Spanish, taking into account these facts and the experience of the team in delivering blended learning adapted to local needs. Forty professionals from Puerto Rico are currently taking the first version of the course, supported by the Regional Extension Center for Puerto Rico and the US Virgin Islands, and endorsed by the American Medical Informatics Association (AMIA).

Use of simulation to assess electronic health record safety in the intensive care unit: a pilot study

OBJECTIVE

To establish the role of high-fidelity simulation training to test the efficacy and safety of the electronic health record (EHR)-user interface within the intensive care unit (ICU) environment.

DESIGN

Prospective pilot study.

SETTING

Medical ICU in an academic medical centre.

PARTICIPANTS

Postgraduate medical trainees.

INTERVENTIONS

A 5-day-simulated ICU patient was developed in the EHR including labs, hourly vitals, medication administration, ventilator settings, nursing and notes. Fourteen medical issues requiring recognition and subsequent changes in management were included. Issues were chosen based on their frequency of occurrence within the ICU and their ability to test different aspects of the EHR-user interface. ICU residents, blinded to the presence of medical errors within the case, were provided a sign-out and given 10 min to review the case in the EHR. They then presented the case with their management suggestions to an attending physician. Participants were graded on the number of issues identified. All participants were provided with immediate feedback upon completion of the simulation.

PRIMARY AND SECONDARY OUTCOMES

To determine the frequency of error recognition in an EHR simulation. To determine factors associated with improved performance in the simulation.

RESULTS

38 participants including 9 interns, 10 residents and 19 fellows were tested. The average error recognition rate was 41% (range 6-73%), which increased slightly with the level of training (35%, 41% and 50% for interns, residents, and fellows, respectively). Over-sedation was the least-recognised error (16%); poor glycemic control was most often recognised (68%). Only 32% of the participants recognised inappropriate antibiotic dosing. Performance correlated with the total number of screens used (p=0.03).

CONCLUSIONS

Despite development of comprehensive EHRs, there remain significant gaps in identifying dangerous medical management issues. This gap remains despite high levels of medical training, suggesting that EHR-specific training may be beneficial. Simulation provides a novel tool in order to both identify these gaps as well as foster EHR-specific training.

Development and evaluation of an electronic health record configuration and customization laboratory course for clinical informatics students

OBJECTIVES

There is a need for informatics educational programs to develop laboratory courses that facilitate hands-on access to an EHR, and allow students to learn and evaluate functionality and configuration options. This is particularly relevant given the diversity of backgrounds of informatics students.

METHODS

We implemented an EHR laboratory course that allowed students to explore an EHR in both inpatient and outpatient clinical environments. The course focused on specific elements of the EHR including order set development, customization, clinical decision support, ancillary services, and billing and coding functionality. Students were surveyed at the end of the course for their satisfaction with the learning experience.

RESULTS AND CONCLUSIONS

We detailed challenges as well as lessons learned after analyzing student evaluations of this course. Features that promote the successful offering of an online EHR course, include (1) using more than one EHR to allow students to compare functionalities, (2) ensuring appropriate course calibration, (3) countering issues specific to EHR usability, and (4) fostering a fertile environment for rich online conversations are discussed.

A systematic literature review of automated clinical coding and classification systems

Clinical coding and classification processes transform natural language descriptions in clinical text into data that can subsequently be used for clinical care, research, and other purposes. This systematic literature review examined studies that evaluated all types of automated coding and classification systems to determine the performance of such systems. Studies indexed in Medline or other relevant databases prior to March 2009 were considered. The 113 studies included in this review show that automated tools exist for a variety of coding and classification purposes, focus on various healthcare specialties, and handle a wide variety of clinical document types. Automated coding and classification systems themselves are not generalizable, nor are the results of the studies evaluating them. Published research shows these systems hold promise, but these data must be considered in context, with performance relative to the complexity of the task and the desired outcome.

Synergies and distinctions between computational disciplines in biomedical research: perspective from the Clinical andTranslational Science Award programs

Clinical and translational research increasingly requires computation. Projects may involve multiple computationally oriented groups including information technology (IT) professionals, computer scientists, and biomedical informaticians. However, many biomedical researchers are not aware of the distinctions among these complementary groups, leading to confusion, delays, and suboptimal results. Although written from the perspective of Clinical and Translational Science Award (CTSA) programs within academic medical centers, this article addresses issues that extend beyond clinical and translational research. The authors describe the complementary but distinct roles of operational IT, research IT, computer science, and biomedical informatics using a clinical data warehouse as a running example. In general, IT professionals focus on technology. The authors distinguish between two types of IT groups within academic medical centers: central or administrative IT (supporting the administrative computing needs of large organizations) and research IT (supporting the computing needs of researchers). Computer scientists focus on general issues of computation such as designing faster computers or more efficient algorithms, rather than specific applications. In contrast, informaticians are concerned with data, information, and knowledge. Biomedical informaticians draw on a variety of tools, including but not limited to computers, to solve information problems in health care and biomedicine. The paper concludes with recommendations regarding administrative structures that can help to maximize the benefit of computation to biomedical research within academic health centers.

How well does a biomedical informatics curriculum map to health information management knowledge clusters? Analysis of a program

The disciplines of health information management (HIM) and biomedical informatics (BMI) have many historical differences from the content of their educational programs to the level offered (i.e., graduate vs. undergraduate). As the adoption of the electronic health record (EHR) grows, however, the two fields share increasingly similar interests, competencies, and educational programs. In our effort to establish an HIM track in our BMI graduate program, leading to Registered Health Information Administrator (RHIA) certification, we had to compare our BMI curriculum with the American Health Information Management Association (AHIMA) knowledge clusters. We present the results of our analysis, which provide insights into the similarities and differences between such curricula. These results show that existing BMI courses met several of the knowledge clusters, which means that only a few additional courses need to be developed.

The MERG Suite: Tools for discovering competencies and associated learning resources

BACKGROUND

As the demands for competency-based education grow, the need for standards-based tools to allow for publishing and discovery of competency-based learning content is more pressing. This project focused on developing federated discovery services for competency-based medical e-learning content.

METHODS

We built a tool suite for authoring and discovery of medical e-learning metadata. The end-user usability of the tool suite was evaluated through a web-based survey.

RESULTS

The suite, implemented as an open-source system, was evaluated to identify areas for improvement.

CONCLUSION

The MERG suite is a starting point for organizations implementing competency-based e-learning resources.

A comparative analysis of retrieval features used in the TREC 2006 Genomics Track passage retrieval task

OBJECTIVE

Identify the set of features that best explained the variation in the performance measure of TREC 2006 Genomics information extraction task, Mean Average Passage Precision (MAPP).

METHODS

A multivariate regression model was built using a backward-elimination approach as a function of certain generalized features that were common to all the algorithms used by TREC 2006 Genomics track participants.

RESULTS

Our regression analysis found that the following four factors were collectively associated with variation in MAPP: (1) Normalization of keywords in the query (2) Use of Entrez gene thesaurus for synonymous terms look-up (3) Unit of text retrieved using respective IR algorithms and (4) The way a passage was defined.

CONCLUSION

These reasonably likely hypotheses, generated by an exploratory data analysis, are informative in understanding results of the TREC 2006 Genomics passage extraction task. This approach has general value for analyzing the results of similar common challenge tasks.

Diagnosis, access and outcomes: Update of a systematic review of telemedicine services

Telemedicine services are being increasingly used. Although insurers and other payers are covering some services in the USA, the rationale for these coverage decisions is not always evidence-based. We reviewed the literature for telemedicine services that substitute for face-to-face medical diagnosis and treatment. We focused on three types of telemedicine services: store-and-forward, home-based and office/hospital-based services. Studies were included if they were relevant to at least one of the three study areas, addressed at least one key question and contained reported results. We excluded articles that did not study a service requiring face-to-face encounters (i.e. teleradiology was excluded). Our search initially identified 4083 citations. After review, 597 were judged to be potentially relevant at the title/abstract level. Following a full-text review, 106 studies were included. Store-and-forward services have been studied in many specialties, the most common being dermatology, wound care and ophthalmology. The evidence for their efficacy is mixed. Several limited studies showed the benefits of home-based telemedicine interventions in chronic diseases. Studies of office/hospital-based telemedicine suggest that telemedicine is most effective for verbal interactions, e.g. videoconferencing for diagnosis and treatment in specialties like neurology and psychiatry. There are still significant gaps in the evidence base between where telemedicine is used and where its use is supported by high-quality evidence. Further well-designed research is necessary to understand how best to deploy telemedicine services in health care.

The full spectrum of biomedical informatics education at Oregon Health & Science University

OBJECTIVES

The growing use of health information technology in operational settings, along with the maturation of the discipline of biomedical informatics, requires reorganization of educational programs in the field. The objective of this paper is to provide a context and description of the biomedical informatics education program at Oregon Health & Science University.

METHODS

The details of the program are provided.

RESULTS

The paper describes the overall program and its component curricula.

CONCLUSIONS

OHSU has developed a program that caters to the full spectrum of those who will work in the field, allowing education tailored to their career goals and needs. The maturation of Internet technologies also allow most aspects of the program to be delivered on-line. The informatics field must step up to the challenge of educating the best workforce to achieve our goals for the optimal use of HIT.

Advancing biomedical image retrieval: development and analysis of a test collection

OBJECTIVE

Develop and analyze results from an image retrieval test collection.

METHODS

After participating research groups obtained and assessed results from their systems in the image retrieval task of Cross-Language Evaluation Forum, we assessed the results for common themes and trends. In addition to overall performance, results were analyzed on the basis of topic categories (those most amenable to visual, textual, or mixed approaches) and run categories (those employing queries entered by automated or manual means as well as those using visual, textual, or mixed indexing and retrieval methods). We also assessed results on the different topics and compared the impact of duplicate relevance judgments.

RESULTS

A total of 13 research groups participated. Analysis was limited to the best run submitted by each group in each run category. The best results were obtained by systems that combined visual and textual methods. There was substantial variation in performance across topics. Systems employing textual methods were more resilient to visually oriented topics than those using visual methods were to textually oriented topics. The primary performance measure of mean average precision (MAP) was not necessarily associated with other measures, including those possibly more pertinent to real users, such as precision at 10 or 30 images.

CONCLUSIONS

We developed a test collection amenable to assessing visual and textual methods for image retrieval. Future work must focus on how varying topic and run types affect retrieval performance. Users' studies also are necessary to determine the best measures for evaluating the efficacy of image retrieval systems.

The TREC 2004 genomics track categorization task: classifying full text biomedical documents

BACKGROUND

The TREC 2004 Genomics Track focused on applying information retrieval and text mining techniques to improve the use of genomic information in biomedicine. The Genomics Track consisted of two main tasks, ad hoc retrieval and document categorization. In this paper, we describe the categorization task, which focused on the classification of full-text documents, simulating the task of curators of the Mouse Genome Informatics (MGI) system and consisting of three subtasks. One subtask of the categorization task required the triage of articles likely to have experimental evidence warranting the assignment of GO terms, while the other two subtasks were concerned with the assignment of the three top-level GO categories to each paper containing evidence for these categories.

RESULTS

The track had 33 participating groups. The mean and maximum utility measure for the triage subtask was 0.3303, with a top score of 0.6512. No system was able to substantially improve results over simply using the MeSH term Mice. Analysis of significant feature overlap between the training and test sets was found to be less than expected. Sample coverage of GO terms assigned to papers in the collection was very sparse. Determining papers containing GO term evidence will likely need to be treated as separate tasks for each concept represented in GO, and therefore require much denser sampling than was available in the data sets. The annotation subtask had a mean F-measure of 0.3824, with a top score of 0.5611. The mean F-measure for the annotation plus evidence codes subtask was 0.3676, with a top score of 0.4224. Gene name recognition was found to be of benefit for this task.

CONCLUSION

Automated classification of documents for GO annotation is a challenging task, as was the automated extraction of GO code hierarchies and evidence codes. However, automating these tasks would provide substantial benefit to biomedical curation, and therefore work in this area must continue. Additional experience will allow comparison and further analysis about which algorithmic features are most useful in biomedical document classification, and better understanding of the task characteristics that make automated classification feasible and useful for biomedical document curation. The TREC Genomics Track will be continuing in 2005 focusing on a wider range of triage tasks and improving results from 2004.

Enhancing access to the Bibliome: the TREC 2004 Genomics Track

BACKGROUND

The goal of the TREC Genomics Track is to improve information retrieval in the area of genomics by creating test collections that will allow researchers to improve and better understand failures of their systems. The 2004 track included an ad hoc retrieval task, simulating use of a search engine to obtain documents about biomedical topics. This paper describes the Genomics Track of the Text Retrieval Conference (TREC) 2004, a forum for evaluation of IR research systems, where retrieval in the genomics domain has recently begun to be assessed.

RESULTS

A total of 27 research groups submitted 47 different runs. The most effective runs, as measured by the primary evaluation measure of mean average precision (MAP), used a combination of domain-specific and general techniques. The best MAP obtained by any run was 0.4075. Techniques that expanded queries with gene name lists as well as words from related articles had the best efficacy. However, many runs performed more poorly than a simple baseline run, indicating that careful selection of system features is essential.

CONCLUSION

Various approaches to ad hoc retrieval provide a diversity of efficacy. The TREC Genomics Track and its test collection resources provide tools that allow improvement in information retrieval systems.

Reducing workload in systematic review preparation using automated citation classification

OBJECTIVE

To determine whether automated classification of document citations can be useful in reducing the time spent by experts reviewing journal articles for inclusion in updating systematic reviews of drug class efficacy for treatment of disease.

DESIGN

A test collection was built using the annotated reference files from 15 systematic drug class reviews. A voting perceptron-based automated citation classification system was constructed to classify each article as containing high-quality, drug class-specific evidence or not. Cross-validation experiments were performed to evaluate performance.

MEASUREMENTS

Precision, recall, and F-measure were evaluated at a range of sample weightings. Work saved over sampling at 95% recall was used as the measure of value to the review process.

RESULTS

A reduction in the number of articles needing manual review was found for 11 of the 15 drug review topics studied. For three of the topics, the reduction was 50% or greater.

CONCLUSION

Automated document citation classification could be a useful tool in maintaining systematic reviews of the efficacy of drug therapy. Further work is needed to refine the classification system and determine the best manner to integrate the system into the production of systematic reviews.

Telemedicine for the medicare population: update

CONTEXT

Telemedicine services are increasingly utilized by patients, clinicians, and institutions. Although private and Federal insurers are covering some telemedicine services, the rationale for these coverage decisions is not always evidence-based.

OBJECTIVES

The goal of this report was to assess the peer-reviewed literature for telemedicine services that substitute for face-to-face medical diagnosis and treatment that may apply to the Medicare population. We focused on three distinct areas: store-and-forward, home-based, and office/hospital-based services. We also sought to identify what progress had been made in expanding the evidence base since the publication of our initial report in 2001 (AHRQ Publication No. 01-E012).

DATA SOURCES

Ovid MEDLINE, reference lists of included studies, and non-indexed materials recommended by telemedicine experts.

STUDY SELECTION

Included studies had to be relevant to at least one of the three study areas, address at least one key question, and contain reported results. We excluded articles that did not study the Medicare population (e.g., children and pregnant adults) or used a service that does not require face-to-face encounters (e.g., radiology or pathology diagnosis).

DATA EXTRACTION

Our literature searches initially identified 4,083 citations. Using a dual-review process, 597 of these were judged to be potentially relevant to our study at the title/abstract level. Following a full-text review, 97 studies were identified that met our inclusion criteria and were subsequently included in the report's evidence tables.

DATA SYNTHESIS

Store-and-forward services have been studied in many specialties, the most prominent being dermatology, wound care, and ophthalmology. The evidence for their efficacy is mixed, and in most areas, there are not corresponding studies on outcomes or improved access to care. Several limited studies showed the benefits of home-based telemedicine interventions in chronic diseases. These interventions appear to enhance communication with health care providers and provide closer monitoring of general health, but the studies of these techniques were conducted in settings that required additional resources and dedicated staff. Studies of office/hospital-based telemedicine suggest that telemedicine is most effective for verbal interactions, e.g., videoconferencing for diagnosis and treatment in specialties like neurology and psychiatry.

CONCLUSIONS

There are still significant gaps in the evidence base between where telemedicine is used and where its use is supported by high-quality evidence. Further well-designed and targeted research that provides high-quality data will provide a strong contribution to understanding how best to deploy technological resources in health care.