Don Lindberg’s influence on future generations: The U.S. National Library of Medicine’s biomedical informatics research training programs

Through his visionary leadership as Director of the U.S. National Library of Medicine (NLM), Donald A. B. Lindberg M.D. influenced future generations of informatics professionals and the field of biomedical informatics itself. This chapter describes Dr. Lindberg’s role in sponsoring and shaping the NLM’s Institutional T15 training programs.

Don Lindberg's Influence on Future Generations: The U.S. National Library of Medicine's Biomedical Informatics Research Training Programs

Through his visionary leadership as Director of the U.S. National Library of Medicine (NLM), Donald A.B. Lindberg M.D. influenced future generations of informatics professionals and the field of biomedical informatics itself. This chapter describes Dr. Lindberg's role in sponsoring and shaping the NLM's Institutional T15 training programs.

Categorizing metadata to help mobilize computable biomedical knowledge

INTRODUCTION

Computable biomedical knowledge artifacts (CBKs) are digital objects conveying biomedical knowledge in machine-interpretable structures. As more CBKs are produced and their complexity increases, the value obtained from sharing CBKs grows. Mobilizing CBKs and sharing them widely can only be achieved if the CBKs are findable, accessible, interoperable, reusable, and trustable (FAIR+T). To help mobilize CBKs, we describe our efforts to outline metadata categories to make CBKs FAIR+T.

METHODS

We examined the literature regarding metadata with the potential to make digital artifacts FAIR+T. We also examined metadata available online today for actual CBKs of 12 different types. With iterative refinement, we came to a consensus on key categories of metadata that, when taken together, can make CBKs FAIR+T. We use subject-predicate-object triples to more clearly differentiate metadata categories.

RESULTS

We defined 13 categories of CBK metadata most relevant to making CBKs FAIR+T. Eleven of these categories (type, domain, purpose, identification, location, CBK-to-CBK relationships, technical, authorization and rights management, provenance, evidential basis, and evidence from use metadata) are evident today where CBKs are stored online. Two additional categories (preservation and integrity metadata) were not evident in our examples. We provide a research agenda to guide further study and development of these and other metadata categories.

CONCLUSION

A wide variety of metadata elements in various categories is needed to make CBKs FAIR+T. More work is needed to develop a common framework for CBK metadata that can make CBKs FAIR+T for all stakeholders.

Summary of fourth annual MCBK public meeting: Mobilizing computable biomedical knowledge-metadata and trust

The exponential growth of biomedical knowledge in computable formats challenges organizations to consider mobilizing artifacts in findable, accessible, interoperable, reusable, and trustable (FAIR+T) ways1. There is a growing need to apply biomedical knowledge artifacts to improve health in Learning Health Systems, health delivery organizations, and other settings. However, most organizations lack the infrastructure required to consume and apply computable knowledge, and national policies and standards adoption are insufficient to ensure that it is discoverable and used safely and fairly, nor is there widespread experience in the process of knowledge implementation as clinical decision support. The Mobilizing Computable Biomedical Knowledge (MCBK) community formed in 2016 to address these needs. This report summarizes the main outputs of the Fourth Annual MCBK public meeting, which was held virtually July 20 to July 21, 2021 and convened over 100 participants spanning diverse domains to frame and address important dimensions for mobilizing CBK.

Machine learning approaches for the prediction of bone mineral density by using genomic and phenotypic data of 5130 older men

The study aimed to utilize machine learning (ML) approaches and genomic data to develop a prediction model for bone mineral density (BMD) and identify the best modeling approach for BMD prediction. The genomic and phenotypic data of Osteoporotic Fractures in Men Study (n = 5130) was analyzed. Genetic risk score (GRS) was calculated from 1103 associated SNPs for each participant after a comprehensive genotype imputation. Data were normalized and divided into a training set (80%) and a validation set (20%) for analysis. Random forest, gradient boosting, neural network, and linear regression were used to develop BMD prediction models separately. Ten-fold cross-validation was used for hyper-parameters optimization. Mean square error and mean absolute error were used to assess model performance. When using GRS and phenotypic covariates as the predictors, all ML models' performance and linear regression in BMD prediction were similar. However, when replacing GRS with the 1103 individual SNPs in the model, ML models performed significantly better than linear regression (with lasso regularization), and the gradient boosting model performed the best. Our study suggested that ML models, especially gradient boosting, can improve BMD prediction in genomic data.

Summary of third annual MCBK public meeting: Mobilizing computable biomedical knowledge-Accelerating the second knowledge revolution

The volume of biomedical knowledge is growing exponentially and much of this knowledge is represented in computer executable formats, such as models, algorithms, and programmatic code. There is a growing need to apply this knowledge to improve health in Learning Health Systems, health delivery organizations, and other settings. However, most organizations do not yet have the infrastructure required to consume and apply computable knowledge, and national policies and standards adoption are not sufficient to ensure that it is discoverable and used safely and fairly, nor is there widespread experience in the process of knowledge implementation as clinical decision support. The Mobilizing Computable Biomedical Knowledge (MCBK) community was formed in 2016 to address these needs. This report summarizes the main outputs of the third annual MCBK public meeting, which was held virtually from June 30 to July 1, 2020 and brought together over 200 participants from various domains to frame and address important dimensions for mobilizing CBK.

Summary of second annual MCBK public meeting: Mobilizing Computable Biomedical Knowledge-A movement to accelerate translation of knowledge into action

The volume of biomedical knowledge is growing exponentially and much of this knowledge is represented in computer executable formats, such as models, algorithms and programmatic code. There is a growing need to apply this knowledge to improve health in Learning Health Systems, health delivery organizations, and other settings. However, most organizations do not yet have the infrastructure required to consume and apply computable knowledge, and national policies and standards adoption are not sufficient to ensure that it is discoverable and used safely and fairly, nor is there widespread experience in the process of knowledge implementation as clinical decision support. The Mobilizing Computable Biomedical Knowledge (MCBK) community formed in 2016 to address these needs. This report summarizes the main outputs of the Second Annual MCBK public meeting, which was held at the National Institutes of Health on July 18-19, 2019 and brought together over 150 participants from various domains to frame and address important dimensions for mobilizing CBK.

Preferences of older adults and their families for Meaningful Use clinical summaries

BACKGROUND

This study assessed the perceptions of older adults regarding the plan of care (POC) contained in the clinical summary mandated by the Electronic Health Records (EHR) Incentive Program.

METHODS

A qualitative descriptive design was selected for this study. Older adults (≥65) with chronic cardiac diagnoses were invited to participate. The investigator shadowed the physician during the patient encounter, interviewed the patients following their encounter, and asked patients to complete standard health literacy and cognitive screening tools and the Patient Activation Measure. Directed content analysis was used to analyze transcripts.

RESULTS

Patients (n=40) found the clinical summary useful for sharing information with family members and other physicians, reminding and informing, and for engaging in behavior change. Seventy-six percent reported that they would not go online to access the clinical summary for multiple reasons, including not being "computer savvy" and privacy concerns. Participant recommendations for a re-designed, improved clinical summary are included. The clinical summary helps patients and families communicate among health care professionals in a complex, disjointed health care system that often burdens patients with that responsibility. The majority of participants were happy with the paper version and offered multiple reasons for not wanting online access that may help us to focus on more compelling reasons for patient portal use.

CONCLUSIONS

Qualitatively, it appears that the clinical summary is a useful tool for engaging people with chronic disease in self-management. The participants in this study told us what many of us already know to be true; that the documentation we provide patients and families is less than ideal.

Decision Support at the Point of Care: Challenges in Knowledge Representation, Management, and Patient-specific Access

Many applications in a clinical information system can benefit from the incorporation of medical knowledge to provide patient-specific, point-of-care decision support. These include computer-based provider order entry, referral, clinical result interpretation, consultation, adverse event monitoring, scheduling, shared patient-doctor decision-making, and generation of alerts and reminders, among others. To be executable, knowledge must be represented in the form of rules, constraints, calculations, guidelines, and other logical/algorithmic formats. The main difficulty is that the integration of such knowledge into clinical applications, when it occurs, tends to be very system- and application-specific, often encoded in a programming language, or even in the formating specifications of a user interaction display. Also, the data references and services invoked are highly dependent on the system/platform and electronic medical record implementation. This makes it difficult and time-consuming to encode authoritative evidence-based knowledge, severely limits the ability to disseminate and share successes, and hampers efforts to review and update the logic as medical knowledge changes. Solutions to this problem involve the development of standards-based representations for medical knowledge, and tools for authoring/editing, dissemination, adaptation to local environments, and execution. Numerous approaches are being pursued, all of which will be described in this presentation.

Granular Quality Reporting for Cervical Cytology Testing

Quality reporting for cervical cancer prevention is focused on patients with normal cervical cytology, and excludes patients with cytological abnormalities that may be at higher risk. The major obstacles for granular reporting are the complexity of surveillance guidelines and free-text data. We performed automated chart review to compare the cytology testing rates for patients with 'atypical squamous cells of undetermined significance' (ASCUS) cytology, with the rates for patients with normal cytology. We modeled the surveillance guidelines, and extracted information from free-text cytology reports, to perform this study on 28101 female patients. Our results show that patients with ASCUS cytology had significantly higher adherence rates (94.9%) than those for patients with normal cytology (90.4%). Overall our study indicates that the quality of care varies significantly between the high and average risk patients. Our study demonstrates the use of health information technology for higher granularity of reporting for cervical cytology testing.

Automated recommendation for cervical cancer screening and surveillance

Because of the complexity of cervical cancer prevention guidelines, clinicians often fail to follow best-practice recommendations. Moreover, existing clinical decision support (CDS) systems generally recommend a cervical cytology every three years for all female patients, which is inappropriate for patients with abnormal findings that require surveillance at shorter intervals. To address this problem, we developed a decision tree-based CDS system that integrates national guidelines to provide comprehensive guidance to clinicians. Validation was performed in several iterations by comparing recommendations generated by the system with those of clinicians for 333 patients. The CDS system extracted relevant patient information from the electronic health record and applied the guideline model with an overall accuracy of 87%. Providers without CDS assistance needed an average of 1 minute 39 seconds to decide on recommendations for management of abnormal findings. Overall, our work demonstrates the feasibility and potential utility of automated recommendation system for cervical cancer screening and surveillance.

Evaluation of the effect of decision support on the efficiency of primary care providers in the outpatient practice

Background: Clinical decision support (CDS) for primary care has been shown to improve delivery of preventive services. However, there is little evidence for efficiency of physicians due to CDS assistance. In this article, we report a pilot study for measuring the impact of CDS on the time spent by physicians for deciding on preventive services and chronic disease management. Methods: We randomly selected 30 patients from a primary care practice, and assigned them to 10 physicians. The physicians were requested to perform chart review to decide on preventive services and chronic disease management for the assigned patients. The patients assignment was done in a randomized crossover design, such that each patient received 2 sets of recommendations—one from a physician with CDS assistance and the other from a different physician without CDS assistance. We compared the physician recommendations made using CDS assistance, with the recommendations made without CDS assistance. Results: The physicians required an average of 1 minute 44 seconds, when they were they had access to the decision support system and 5 minutes when they were unassisted. Hence the CDS assistance resulted in an estimated saving of 3 minutes 16 seconds (65%) of the physicians’ time, which was statistically significant (P < .0001). There was no statistically significant difference in the number of recommendations. Conclusion: Our findings suggest that CDS assistance significantly reduced the time spent by physicians for deciding on preventive services and chronic disease management. The result needs to be confirmed by performing similar studies at other institutions.

Formative evaluation of the accuracy of a clinical decision support system for cervical cancer screening

Objectives

We previously developed and reported on a prototype clinical decision support system (CDSS) for cervical cancer screening. However, the system is complex as it is based on multiple guidelines and free-text processing. Therefore, the system is susceptible to failures. This report describes a formative evaluation of the system, which is a necessary step to ensure deployment readiness of the system.

Materials and methods

Care providers who are potential end-users of the CDSS were invited to provide their recommendations for a random set of patients that represented diverse decision scenarios. The recommendations of the care providers and those generated by the CDSS were compared. Mismatched recommendations were reviewed by two independent experts.

Results

A total of 25 users participated in this study and provided recommendations for 175 cases. The CDSS had an accuracy of 87% and 12 types of CDSS errors were identified, which were mainly due to deficiencies in the system's guideline rules. When the deficiencies were rectified, the CDSS generated optimal recommendations for all failure cases, except one with incomplete documentation.

Discussion and conclusions

The crowd-sourcing approach for construction of the reference set, coupled with the expert review of mismatched recommendations, facilitated an effective evaluation and enhancement of the system, by identifying decision scenarios that were missed by the system's developers. The described methodology will be useful for other researchers who seek rapidly to evaluate and enhance the deployment readiness of complex decision support systems.

Workflow-based Data Reconciliation for Clinical Decision Support: Case of Colorectal Cancer Screening and Surveillance

A major barrier for computer-based clinical decision support (CDS), is the difficulty in obtaining the patient information required for decision making. The information gap is often due to deficiencies in the clinical documentation. One approach to address this gap is to gather and reconcile data from related documents or data sources. In this paper we consider the case of a CDS system for colorectal cancer screening and surveillance. We describe the use of workflow analysis to design data reconciliation processes. Further, we perform a quantitative analysis of the impact of these processes on system performance using a dataset of 106 patients. Results show that data reconciliation considerably improves the performance of the system. Our study demonstrates that, workflow-based data reconciliation can play a vital role in designing new-generation CDS systems that are based on complex guideline models and use natural language processing (NLP) to obtain patient data.

Healthcare information technology and economics

At the 2011 American College of Medical Informatics (ACMI) Winter Symposium we studied the overlap between health IT and economics and what leading healthcare delivery organizations are achieving today using IT that might offer paths for the nation to follow for using health IT in healthcare reform. We recognized that health IT by itself can improve health value, but its main contribution to health value may be that it can make possible new care delivery models to achieve much larger value. Health IT is a critically important enabler to fundamental healthcare system changes that may be a way out of our current, severe problem of rising costs and national deficit. We review the current state of healthcare costs, federal health IT stimulus programs, and experiences of several leading organizations, and offer a model for how health IT fits into our health economic future.

Clinical decision support with automated text processing for cervical cancer screening

Objective

To develop a computerized clinical decision support system (CDSS) for cervical cancer screening that can interpret free-text Papanicolaou (Pap) reports.

Materials and Methods

The CDSS was constituted by two rulebases: the free-text rulebase for interpreting Pap reports and a guideline rulebase. The free-text rulebase was developed by analyzing a corpus of 49 293 Pap reports. The guideline rulebase was constructed using national cervical cancer screening guidelines. The CDSS accesses the electronic medical record (EMR) system to generate patient-specific recommendations. For evaluation, the screening recommendations made by the CDSS for 74 patients were reviewed by a physician.

Results and Discussion

Evaluation revealed that the CDSS outputs the optimal screening recommendations for 73 out of 74 test patients and it identified two cases for gynecology referral that were missed by the physician. The CDSS aided the physician to amend recommendations in six cases. The failure case was because human papillomavirus (HPV) testing was sometimes performed separately from the Pap test and these results were reported by a laboratory system that was not queried by the CDSS. Subsequently, the CDSS was upgraded to look up the HPV results missed earlier and it generated the optimal recommendations for all 74 test cases.

Limitations

Single institution and single expert study.

Conclusion

An accurate CDSS system could be constructed for cervical cancer screening given the standardized reporting of Pap tests and the availability of explicit guidelines. Overall, the study demonstrates that free text in the EMR can be effectively utilized through natural language processing to develop clinical decision support tools.

Reducing diagnostic error with computer-based clinical decision support

Information technology approaches to delivering diagnostic clinical decision support (CDS) are the subject of the papers to follow in the proceedings. These will address the history of CDS and present day approaches (Miller), evaluation of diagnostic CDS methods (Friedman), and the role of clinical documentation in supporting diagnostic decision making (Schiff). In addition, several other considerations relating to this topic are interesting to ponder. We are moving toward increased understanding of gene regulation and gene expression, identification of biomarkers, and the ability to predict patient response to disease and to tailor treatments to these individual variations-referred to as "personalized" or, more recently, "predictive" medicine. Consequently, diagnostic decision making is more and more linked to management decision making, and generic diagnostic labels like "diabetes" or "colon cancer" will no longer be sufficient, because they don't tell us what to do. Ultimately, if we have more complete data including more structured capture of phenomic data as well as the characterization of the patient's genome, direct prediction from responses of highly refined subsets of similar patients in a database can be used to select appropriate management, the effectiveness of which was demonstrated in projects in selected limited domains as early as the 1970s. In general, there are six classes of methodologies, including the above, which can be applied to delivering CDS. In addition, patients are becoming more knowledgeable and should be regarded as active participants, not only in helping to obtain data but also in their own status assessment and as recipients of decision support. With the above advances, this is a very promising time to be engaged in pursuit of methods of CDS.

Commentary: Informatics in biomedicine and health care

During the last two decades, biomedical informatics (BMI) has become a critical component in biomedical research and health care delivery, as evidenced by two recent phenomena. One, as discussed in the article by Bernstam and colleagues in this issue, has been the introduction of Clinical and Translational Science Awards. Perhaps even more important has been the recent, arguably long overdue, emphasis on deployment of health information technology (IT) nationally. BMI utilizes IT and computer science as tools and methods for improving data acquisition, data management, data analysis, and knowledge generation, but it is driven by a focus on applications based in deep understanding of the science and practice, problems, interactions, culture, and milieu of biomedicine and health. Building from Bernstam and colleagues' distinction between BMI and other IT disciplines, the authors discuss the evolving role of BMI professionals as individuals uniquely positioned to work within the human and organizational context and culture in which the IT is being applied. The focus is not on the IT but on the combination--the interactions of IT systems, human beings, and organizations aimed at achieving a particular purpose. There has never been a time when the need for individuals well trained in BMI--those who understand the complexities of the human, social, and organizational milieu of biomedicine and health--has been more critical than it is now, as the nation seeks to develop a national infrastructure for biomedicine and health care, and as these fields seek to broadly deploy IT wisely and appropriately.

Informatics and a health care strategy for the future--general directions

Central problems in health care involve availability, access, quality, and cost. A major part of a health care strategy also involves disease prevention and promotion of healthy lifestyles, which go well beyond the purview of the health care system itself. Implementing any strategy involves health policy, finance, and management expertise. What then is the role of informatics? We take the position here that informatics is a key enabler both for addressing availability, access, quality, and cost, and also for supporting the work of health policy, finance, and management experts. Informatics provides the necessary information technology (IT) infrastructure, standards, tools, and data to be able to address these key topics and for carrying out the work of the experts. In that sense, we regard informatics as a means for social engineering--the availability of these capabilities brings stakeholders to the table who might otherwise not have reason to or be able to work together.

Testing the technology acceptance model for evaluating healthcare professionals' intention to use an adverse event reporting system

BACKGROUND

Many healthcare organizations have implemented adverse event reporting systems in the hope of learning from experience to prevent adverse events and medical errors. However, a number of these applications have failed or not been implemented as predicted.

OBJECTIVE

This study presents an extended technology acceptance model that integrates variables connoting trust and management support into the model to investigate what determines acceptance of adverse event reporting systems by healthcare professionals.

METHOD

The proposed model was empirically tested using data collected from a survey in the hospital environment. A confirmatory factor analysis was performed to examine the reliability and validity of the measurement model, and a structural equation modeling technique was used to evaluate the causal model.

RESULTS

The results indicated that perceived usefulness, perceived ease of use, subjective norm, and trust had a significant effect on a professional's intention to use an adverse event reporting system. Among them, subjective norm had the most contribution (total effect). Perceived ease of use and subjective norm also had a direct effect on perceived usefulness and trust, respectively. Management support had a direct effect on perceived usefulness, perceived ease of use, and subjective norm.

CONCLUSION

The proposed model provides a means to understand what factors determine the behavioral intention of healthcare professionals to use an adverse event reporting system and how this may affect future use. In addition, understanding the factors contributing to behavioral intent may potentially be used in advance of system development to predict reporting systems acceptance.

Biomedical informatics in the desert--a new and unique program at Arizona State University

OBJECTIVES

A new academic Biomedical Informatics (BMI) Program in Phoenix, Arizona, embodies a unique organizational structure to draw on the strengths of a computer science and informatics school and the biomedical and clinical strengths of a college of medicine, in an effort to infuse informatics approaches broadly.

METHODS

The program reflects a partnership of two state universities that situates the Arizona State University (ASU) Department of BMI on a new downtown Phoenix Biomedical Campus with the University of Arizona (UA) College of Medicine in partnership with ASU (COM-PHX). Plans call for development of faculty and expertise in the four major subdomains of BMI, as well as in various cross-cutting capabilities.

RESULTS

Coming into existence in a state that is investing significantly in biomedical science and technology, BMI has already developed Masters and PhD degree programs, is working with COM-PHX to integrate informatics intensively into the education of the medical students, and has been authorized to plan for an undergraduate program in BMI. Reflecting the statewide emphasis on the biomedical and health sector, the growing faculty are engaged in a number of research partnerships and collaborative centers.

CONCLUSIONS

As one of the newest academic BMI programs is taking shape in Arizona, it is embarking on a wide-ranging educational program and a broad research agenda that are now in their earliest stages.

SMART--an integrated wireless system for monitoring unattended patients

Monitoring vital signs and locations of certain classes of ambulatory patients can be useful in overcrowded emergency departments and at disaster scenes, both on-site and during transportation. To be useful, such monitoring needs to be portable and low cost, and have minimal adverse impact on emergency personnel, e.g., by not raising an excessive number of alarms. The SMART (Scalable Medical Alert Response Technology) system integrates wireless patient monitoring (ECG, SpO(2)), geo-positioning, signal processing, targeted alerting, and a wireless interface for caregivers. A prototype implementation of SMART was piloted in the waiting area of an emergency department and evaluated with 145 post-triage patients. System deployment aspects were also evaluated during a small-scale disaster-drill exercise.

Personal health records

Nationwide, momentum is growing to provide patients with computer tools called personal health records (PHRs). These allow patients to participate in their own healthcare management by viewing, editing, or discussing their own medical data. Historically, PHRs targeted consumers, but contemporary PHRs are increasingly aimed at providers and payers. This article reviews the types of PHRs that are currently available, discusses the PHR functionalities that offer the best value for a medical practice, and provides strategies for making purchasing decisions.

Design and implementation of the GLIF3 guideline execution engine

We have developed the GLIF3 Guideline Execution Engine (GLEE) as a tool for executing guidelines encoded in the GLIF3 format. In addition to serving as an interface to the GLIF3 guideline representation model to support the specified functions, GLEE provides defined interfaces to electronic medical records (EMRs) and other clinical applications to facilitate its integration with the clinical information system at a local institution. The execution model of GLEE takes the "system suggests, user controls" approach. A tracing system is used to record an individual patient's state when a guideline is applied to that patient. GLEE can also support an event-driven execution model once it is linked to the clinical event monitor in a local environment. Evaluation has shown that GLEE can be used effectively for proper execution of guidelines encoded in the GLIF3 format. When using it to execute each guideline in the evaluation, GLEE's performance duplicated that of the reference systems implementing the same guideline but taking different approaches. The execution flexibility and generality provided by GLEE, and its integration with a local environment, need to be further evaluated in clinical settings. Integration of GLEE with a specific event-monitoring and order-entry environment is the next step of our work to demonstrate its use for clinical decision support. Potential uses of GLEE also include quality assurance, guideline development, and medical education.

GLIF3: a representation format for sharable computer-interpretable clinical practice guidelines

The Guideline Interchange Format (GLIF) is a model for representation of sharable computer-interpretable guidelines. The current version of GLIF (GLIF3) is a substantial update and enhancement of the model since the previous version (GLIF2). GLIF3 enables encoding of a guideline at three levels: a conceptual flowchart, a computable specification that can be verified for logical consistency and completeness, and an implementable specification that is intended to be incorporated into particular institutional information systems. The representation has been tested on a wide variety of guidelines that are typical of the range of guidelines in clinical use. It builds upon GLIF2 by adding several constructs that enable interpretation of encoded guidelines in computer-based decision-support systems. GLIF3 leverages standards being developed in Health Level 7 in order to allow integration of guidelines with clinical information systems. The GLIF3 specification consists of an extensible object-oriented model and a structured syntax based on the resource description framework (RDF). Empirical validation of the ability to generate appropriate recommendations using GLIF3 has been tested by executing encoded guidelines against actual patient data. GLIF3 is accordingly ready for broader experimentation and prototype use by organizations that wish to evaluate its ability to capture the logic of clinical guidelines, to implement them in clinical systems, and thereby to provide integrated decision support to assist clinicians.

Positive attitudes and failed queries: an exploration of the conundrums of consumer health information retrieval

Several studies have found that consumers report a high level of satisfaction with the Internet as a health information resource. Belied by this positive attitude, however, are other studies reporting that consumers were often unsuccessful in searching for health information. In this paper, we present an interview and observation study in which we asked health consumers to search for health information on the Internet after first stating their search goals. Upon the conclusion of the session they were asked to evaluate their searches. We found that many consumers were unable to find satisfactory information when performing a specific query, while in general the group viewed health information retrieval (HIR) on the Internet in a positive light. We analyzed the observed search sessions to determine what factors accounted for the failure of specific searches and positive attitudes, and also discussed potential informatics solutions.

The InterMed approach to sharable computer-interpretable guidelines: a review

InterMed is a collaboration among research groups from Stanford, Harvard, and Columbia Universities. The primary goal of InterMed has been to develop a sharable language that could serve as a standard for modeling computer-interpretable guidelines (CIGs). This language, called GuideLine Interchange Format (GLIF), has been developed in a collaborative manner and in an open process that has welcomed input from the larger community. The goals and experiences of the InterMed project and lessons that the authors have learned may contribute to the work of other researchers who are developing medical knowledge-based tools. The lessons described include (1) a work process for multi-institutional research and development that considers different viewpoints, (2) an evolutionary lifecycle process for developing medical knowledge representation formats, (3) the role of cognitive methodology to evaluate and assist in the evolutionary development process, (4) development of an architecture and (5) design principles for sharable medical knowledge representation formats, and (6) a process for standardization of a CIG modeling language.

Description and status update on GELLO: a proposed standardized object-oriented expression language for clinical decision support

A major obstacle to sharing computable clinical knowledge is the lack of a common language for specifying expressions and criteria. Such a language could be used to specify decision criteria, formulae, and constraints on data and action. Al-though the Arden Syntax addresses this problem for clinical rules, its generalization to HL7's object-oriented data model is limited. The GELLO Expression language is an object-oriented language used for expressing logical conditions and computations in the GLIF3 (GuideLine Interchange Format, v. 3) guideline modeling language. It has been further developed under the auspices of the HL7 Clinical Decision Support Technical Committee, as a proposed HL7 standard., GELLO is based on the Object Constraint Language (OCL), because it is vendor-independent, object-oriented, and side-effect-free. GELLO expects an object-oriented data model. Although choice of model is arbitrary, standardization is facilitated by ensuring that the data model is compatible with the HL7 Reference Information Model (RIM).

Relationships among different subjective measurements of consumer health information retrieval performance

BACKGROUND

Millions of consumers perform health information retrieval (HIR) online. To better understand the consumers' perspective on HIR performance, we conducted an observation and interview study of 97 health information consumers.

METHODS

Consumers were asked to perform HIR tasks and we recorded their view regarding performance using several differ-ent subjective measurements: finding the desired information, usefulness of the information found, satisfaction with the information, and intention to continue searching. Statistical analysis was applied to verify if the multiple subjective measurements were redundant.

RESULT

The measurements ranged from slight agreement to no agreement among them. A number of reasons were identified for this lack of agreement.

CONCLUSION

Although related, the four subjective measurements of HIR performance are distinct from each other and carried different useful information

Design of a standards-based external rules engine for decision support in a variety of application contexts: report of a feasibility study at Partners HealthCare System

This project explored functional requirements for an institution-wide method, at Partners HealthCare, for interpreting clinical knowledge for decision support. Such knowledge is currently incorporated in a variety of clinical applications, yet the methods of representation and of execution vary and the ability to author/edit the rules by human experts is limited. We expanded on a 2002 "Knowledge Inventory" at Partners to evaluate feasibility of designing a single representation approach entailing: (a) exploration of specific needs of different applications, in terms of kinds of response required (synchronous/asynchronous, time criticality, etc.), context (e.g., implied patient, time frame, or episode), and kinds of actions to be triggered; (b) kind of representation of knowledge and feasibility of casting knowledge in the form of if em leader then statements; and (c) data and knowledge resources used (implied data model, and particular knowledge sources and terminology sources). The result of analysis was to design an architecture to accomplish this goal. We also did preliminary analysis of requirements for authoring for such a representation, and for implementation.

Characteristics of consumer terminology for health information retrieval

OBJECTIVES

As millions of consumers perform health information retrieval online, the mismatch between their terminology and the terminologies of the information sources could become a major barrier to successful retrievals. To address this problem, we studied the characteristics of consumer terminology for health information retrieval.

METHODS

Our study focused on consumer queries that were used on a consumer health service Web site and a consumer health information Web site. We analyzed data from the site-usage logs and conducted interviews with patients.

RESULTS

Our findings show that consumers' information retrieval performance is very poor. There are significant mismatches at all levels (lexical, semantic and mental models) between the consumer terminology and both the information source terminology and standard medical vocabularies.

CONCLUSIONS

Comprehensive terminology support on all levels is needed for consumer health information retrieval.

GELLO: an object-oriented query and expression language for clinical decision support

GELLO is a purpose-specific, object-oriented (OO) query and expression language. GELLO is the result of a concerted effort of the Decision Systems Group (DSG) working with the HL7 Clinical Decision Support Technical Committee (CDSTC) to provide the HL7 community with a common format for data encoding and manipulation. GELLO will soon be submitted for ballot to the HL7 CDSTC for consideration as a standard.

Applying axiomatic design methodology for guideline revision

We are investigating the use of axiomatic design (AD) as a principled approach to the revision of guidelines. AD models guidelines in a modular and hierarchical manner and captures interactions be-tween modules. To test this approach we applied AD to encode segments of three guidelines and their revised versions. Guideline encodings for the original versions were modified to incorporate changes made in the revised documents. The results indicate that AD is a promising approach for guideline modeling.

A method for subdividing clinical guidelines into process modules with associated triggers and objectives to facilitate implementation

Representation of multi-step clinical guidelines (CG) and their implementation in computerized decision support (DS) systems are complex and logistically challenging tasks. However, many simple rules based on CGs (e.g., medical logic modules), have been successfully implemented through a few popular DS models (e.g., prevention reminders, order entry systems). To facilitate mapping of CGs to practical DS models, we propose an empirical method for sub-dividing CGs into modules according to the locus in a clinical process flow model where implementation would be most effective (e.g., post-encounter provider order entry). We further propose a classification of triggers and objectives for CG modules that provides a framework for a DS system to implement the module Successful application of the method to ten diverse CGs in the outpatient setting is described.

A meta-data model for knowledge in decision support systems

Clinical decision support such as alerts, reminders and guidance are driven by rules often distributed among a variety of applications in a healthcare information system. Due to the increasing size of rule bases, there is a growing need to manage this dispersed knowledge in an integrated environment. A system for management of executable clinical knowledge such as rules should (1) assist in the development and maintenance of rules throughout the rules' life-cycles, (2) support search and retrieval of rules in the knowledge base (e.g., rules for diabetes, rules created by a particular individual), and (3) facilitate the analyses of rules in the knowledge base (e.g., identify rules not updated in the last year). In order to create such a clinical knowledge management system it is necessary to model the meta-data of rules. There have been efforts to document meta-data about rules within the Arden Syntax Medical Logical Modules' project. However, the maintenance and library categories in that project allow mainly free-text information about a rule. We have created a comprehensive meta-data structure and taxonomy for describing clinical rules that supports the features of a knowledge management system. We also tested this model using a representative set of rules.

Comparing computer-interpretable guideline models: a case-study approach

OBJECTIVES

Many groups are developing computer-interpretable clinical guidelines (CIGs) for use during clinical encounters. CIGs use "Task-Network Models" for representation but differ in their approaches to addressing particular modeling challenges. We have studied similarities and differences between CIGs in order to identify issues that must be resolved before a consensus on a set of common components can be developed.

DESIGN

We compared six models: Asbru, EON, GLIF, GUIDE, PRODIGY, and PROforma. Collaborators from groups that created these models represented, in their own formalisms, portions of two guidelines: American College of Chest Physicians cough guidelines [correction] and the Sixth Report of the Joint National Committee on Prevention, Detection, Evaluation, and Treatment of High Blood Pressure.

MEASUREMENTS

We compared the models according to eight components that capture the structure of CIGs. The components enable modelers to encode guidelines as plans that organize decision and action tasks in networks. They also enable the encoded guidelines to be linked with patient data-a key requirement for enabling patient-specific decision support.

RESULTS

We found consensus on many components, including plan organization, expression language, conceptual medical record model, medical concept model, and data abstractions. Differences were most apparent in underlying decision models, goal representation, use of scenarios, and structured medical actions.

CONCLUSION

We identified guideline components that the CIG community could adopt as standards. Some of the participants are pursuing standardization of these components under the auspices of HL7.

Developing a shared agenda for health care systems safety and quality

High quality, computer-interpretable, patient-specific knowledge at the point of need is essential, as we seek to incorporate decision support and other approaches in clinical information systems to achieve safety and increased quality of health care. This gives rise to the need for shared, standards-based approaches to representing the knowledge and tools for management of it.

Representation primitives, process models and patient data in computer-interpretable clinical practice guidelines: a literature review of guideline representation models

Representation of clinical practice guidelines in a computer-interpretable format is a critical issue for guideline development, implementation, and evaluation. We studied 11 types of guideline representation models that can be used to encode guidelines in computer-interpretable formats. We have consistently found in all reviewed models that primitives for representation of actions and decisions are necessary components of a guideline representation model. Patient states and execution states are important concepts that closely relate to each other. Scheduling constraints on representation primitives can be modeled as sequences, concurrences, alternatives, and loops in a guideline's application process. Nesting of guidelines provides multiple views to a guideline with different granularities. Integration of guidelines with electronic medical records can be facilitated by the introduction of a formal model for patient data. Data collection, decision, patient state, and intervention constitute four basic types of primitives in a guideline's logic flow. Decisions clarify our understanding on a patient's clinical state, while interventions lead to the change from one patient state to another.

Using the critical incident technique to define a minimal data set for requirements elicitation in public health

The introduction of computer-based information systems (ISs) in public health provides enhanced possibilities for service improvements and hence also for improvement of the population's health. Not least, new communication systems can help in the socialization and integration process needed between the different professions and geographical regions. Therefore, development of ISs that truly support public health practices require that technical, cognitive, and social issues be taken into consideration. A notable problem is to capture 'voices' of all potential users, i.e., the viewpoints of different public health practitioners. Failing to capture these voices will result in inefficient or even useless systems. The aim of this study is to develop a minimal data set for capturing users' voices on problems experienced by public health professionals in their daily work and opinions about how these problems can be solved. The issues of concern thus captured can be used both as the basis for formulating the requirements of ISs for public health professionals and to create an understanding of the use context. Further, the data can help in directing the design to the features most important for the users.

Future of medical knowledge management and decision support

Attempts to predict the future are typically off the mark. Beyond the challenges of forecasting the stock market or the weather, dramatic instances of notoriously inaccurate prognostications have been those by the US patent office in the late 1800s about the future of inventions, by Thomas Watson in the 1930s about the market for large computers, and by Bill Gates in the early 1990s about the significance of the Internet. When one seeks to make predictions about health care, one finds that, beyond the usual uncertainties regarding the future, additional impediments to forecasting are the discontinuities introduced by advances in biomedical science and technology, the impact of information technology, and the reorganizations and realignments attending various approaches to health care delivery and finance. Changes in all three contributing areas themselves can be measured in "PSPYs", or paradigm shifts per year. Despite these risks in forecasting, I believe that certain trends are sufficiently clear that I am willing to venture a few predictions. Further, the predictions I wish to make suggest a goal for the future that can be achieved, if we can align the prevailing political, financial, biomedical, and technical forces toward that end. Thus, in a sense this is a call to action, to shape the future rather than just let it happen. This chapter seeks to lay out the direction we are heading in knowledge management and decision support, and to delineate an information technology framework that appears desirable. I believe the framework to be discussed is of importance to the health care-related knowledge management and decision making activities of the consumer and patient, the health care provider, and health care delivery organizations and insurers. The approach is also relevant to the other dimensions of academic health care institution activities, notably the conduct of research and the processes of education and learning.

Training in medical informatics: combining onsite and online instruction

The Internet is promoting active exchange of teaching materials and discussion among geographically distant collaborators. We envision that training in medical informatics can be better achieved if both onsite and online instruction are combined, provided that cultural and technological barriers are anticipated and the training program is prepared accordingly. We describe our Brazil/USA program in medical informatics, which includes components of on-site and online education, and discuss lessons learned during its ongoing implementation. Three onsite courses and one workshop have been planned, and two online courses are being developed.

Adaptation of the critical incident technique to requirements engineering in public health

The introduction of modern information systems in public health provides new possibilities for improvements in public health services and hence also of population's health. However, development of information systems that truly supports public health practices requires that technical, cognitive, and social issues be taken into consideration. In requirements engineering for public health, a notable problem is that of capturing all aspects of the future user's voices, i.e., the viewpoints of different public health practitioners. Failing to capture these voices will result in inefficient or even useless systems. The aim of this paper is to report a requirements-engineering instrument to describe problems in the daily work of public health professionals. The issues of concern thus captured can be used as the basis for formulating the requirements of information systems for public health professionals.

Handling expressiveness and comprehensibility requirements in GLIF3

Clinical guidelines are aimed at standardizing patient care and improving its quality and cost effectiveness. Guidelines represented in a computer-interpretable (CI) format can be used to provide automatic decision support applied to individual patients during the clinical encounter. The process of creating computer-interpretable guidelines (CIG) re-moves ambiguities contained in paper-based guidelines, thus making the guideline more comprehensible. For these reasons, CIGs may have a larger impact on clinician behavior than paper-based guidelines. Since much effort goes into creating guidelines in a CI format, it is desirable that different institutions and software systems share them. In a guideline representation workshop hosted by the InterMed Collaboratory in March 2000, the need for a standard representation format for sharable CIGs was recognized. As a first step towards achieving this goal, we proposed a set of functional requirements for sharable CIGs. The requirements encompass the entire life cycle of a CIG: development, implementation, use and maintenance. In this paper we discuss requirements that are important during the development stage of a CIG. We have abstracted the requirements into two groups: expressiveness--the ability to ex-press the knowledge content of different types of guidelines--and comprehensibility--the ability to manage complexity, facilitate coherence, and visualize a guideline model to aid in human comprehension. The Guideline Interchange For-mat version 3 (GLIF3) is a language for structured representation of CIGs. It is under development to facilitate sharing CIGs among different institutions and systems. We illustrate how GLIF3 meets the specified development requirements.

Sharable computer-based clinical practice guidelines: rationale, obstacles, approaches, and prospects

Clinical practice guideline automation at the point of care is of growing interest, yet most guidelines are authored in unstructured narrative form. Computer-based execution depends on a formal structured representation, and also faces a number of other challenges at all stages of the guideline lifecycle: modeling, authoring, dissemination, implementation, and update. This is because of the multiplicity of conceptual models, authoring tools, authoring approaches, intended applications, implementation platforms, and local interface requirements and operational constraints. Complexity and time required for development and structure are also huge obstacles. These factors argue for convergence on a common shared model for representation that can be the basis of dissemination. A common model would facilitate direct interpretation or mapping to multiple implementation environments. GLIF (GuideLine Interchange Format) is a formal representation model for guidelines, created by the InterMed Collaboratory as a proposed basis for a shared representation. GLIF currently addresses the process of authoring and dissemination; the InterMed team's major focus now is on tools to facilitate these tasks and the mapping to clinical information system environments. Because of limitations in what can be done by a single team with finite resources, however, and the variety of additional perspectives that need to be accommodated, the InterMed team has determined that further development of a shared representation would be best served as an open process in which the world community is engaged. Under the auspices of the HL7 Decision Support Technical Committee, a GLIF Special Interest Group has been established, which is intended to be a forum for collaborative refinement and extension of a standard representation that can support the needs of the guideline lifecycle. Significant areas for future work will need to include demonstrations of effective means for incorporating guide-lines at point of care, reconciliation of functional requirements of different models and identification of those most important for supporting practical implementation, im-proved means for authoring and management of complexity, and methods for automatically analyzing and validating syntax, semantics, and logical consistency of guidelines.

Representation of clinical practice guidelines for computer-based implementations

Representation of clinical practice guidelines is a critical issue for computer-based guideline development, implementation and evaluation. We studied eight types of computer-based guideline representation models. Typical primitives for these models include decisions, actions, patient states and execution states. Temporal constraints and nesting are important aspects of guideline structure representation. Integration of guidelines with electronic medical records can be facilitated by the introduction of formal models of patient data. Patient states and execution states are closely related to one another. Data collection, decision, patient state and intervention are four basic steps in a guideline's logic flow.

Using a neural network with flow cytometry histograms to recognize cell surface protein binding patterns

Flow cytometric systems are being used increasingly in all branches of biological science including medicine. To develop analytic tools for identifying unknown molecules such as the antibodies that recognize different structure in the identical antigens, we explored use of a neural network in flow cytometry data comparison. Peak locations were extracted from flow cytometry histograms and we used the Marquardt backpropagation neural networks to recognize identical or similar binding patterns between antibodies and antigens based on the peak locations. The neural network showed 93.8% to 99.6% correct classification rates for identical or similar molecules. This suggests that the neural network technique can be useful in flow cytometry histogram data analysis.

Matching of flow-cytometry histograms using information theory in feature space

Flow cytometry is a widely available technique for analyzing cell-surface protein expression. Data obtained from flow cytometry is frequently used to produce fluorescence intensity histograms. Comparison of histograms can be useful in the identification of unknown molecules and in the analysis of protein expression. In this study, we examined the combination of a new smoothing technique called SiZer with information theory to measure the difference between cytometry histograms. SiZer provides cross-bandwidth smoothing and allowed analysis in feature space. The new methods were tested on a panel of monoclonal antibodies raised against proteins expressed on peripheral blood lymphocytes and compared with previous methods. The findings suggest that comparing information content of histograms in feature space is effective and efficient for identifying antibodies with similar cell-surface binding patterns.