Evaluating Natural Language Processors in the Clinical Domain

Evaluating natural language processing (NLP) systems in the clinical domain is a difficult task which is important for advancement of the field. A number of NLP systems have been reported that extract information from free-text clinical reports, but not many of the systems have been evaluated. Those that were evaluated noted good performance measures but the results were often weakened by ineffective evaluation methods. In this paper we describe a set of criteria aimed at improving the quality of NLP evaluation studies. We present an overview of NLP evaluations in the clinical domain and also discuss the Message Understanding Conferences (MUC) [1-41. Although these conferences constitute a series of NLP evaluation studies performed outside of the clinical domain, some of the results are relevant within medicine. In addition, we discuss a number of factors which contribute to the complexity that is inherent in the task of evaluating natural language systems.

Evaluating Consensus Among Physicians in Medical Knowledge Base Construction

This study evaluates inter-author variability in knowledge base construction. Seven board-certified internists independently profiled “acute perinephric abscess”, using as reference material a set of 109 peer-reviewed articles. Each participant created a list of findings associated with the disease, estimated the predictive value and sensitivity of each finding, and assessed the pertinence of each article for making each judgment. Agreement in finding selection was significantly different from chance: seven, six, and five participants selected the same finding 78.6, 9.8, and 1.6 times more often than predicted by chance. Findings with the highest sensitivity were most likely to be included by all participants. The selection of supporting evidence from the medical literature was significantly related to each physician’s agreement with the majority. The study shows that, with appropriate guidance, physicians can reproducibly extract information from the medical literature, and thus established a foundation for multi-author knowledge base construction.

Extracting Findings from Narrative Reports: Software Transferability and Sources of Physician Disagreement

While natural language processing systems are beginning to see clinical use, it remains unclear whether they can be disseminated effectively through the health care community. MedLEE, a general-purpose natural language processor developed for Columbia-Presbyterian Medical Center, was compared to physicians' ability to detect seven clinical conditions in 200 Brigham and Women's Hospital chest radiograph reports. Using the system on the new institution's reports resulted in a small but measurable drop in performance (it was distinguishable from physicians at p = 0.011). By making adjustments to the interpretation of the processor's coded output (without changing the processor itself), local behavior was better accommodated, and performance improved so that it was indistinguishable from the physicians. Pairs of physicians disagreed on at least one condition for 22% of reports; the source of disagreement appeared to be interpretation of findings, gauging likelihood and degree of disease, and coding errors.

Using Connectionist Modules for Decision Support

A connectionist model for decision support was constructed out of several back-propagation modules. Manifestations serve as input to the model; they may be real-valued, and the confidence in their measurement may be specified. The model produces as its output the posterior probability of disease. The model was trained on 1,000 cases taken from a simulated underlying population with three conditionally independent manifestations. The first manifestation had a linear relationship between value and posterior probability of disease, the second had a stepped relationship, and the third was normally distributed. An independent test set of 30,000 cases showed that the model was better able to estimate the posterior probability of disease (the standard deviation of residuals was 0.046, with a 95% confidence interval of 0.046-0.047) than a model constructed using logistic regression (with a standard deviation of residuals of 0.062, with a 95% confidence interval of 0.062-0.063). The model fitted the normal and stepped manifestations better than the linear one. It accommodated intermediate levels of confidence well.

Estimating summary statistics for electronic health record laboratory data for use in high-throughput phenotyping algorithms

We study the question of how to represent or summarize raw laboratory data taken from an electronic health record (EHR) using parametric model selection to reduce or cope with biases induced through clinical care. It has been previously demonstrated that the health care process (Hripcsak and Albers, 2012, 2013), as defined by measurement context (Hripcsak and Albers, 2013; Albers et al., 2012) and measurement patterns (Albers and Hripcsak, 2010, 2012), can influence how EHR data are distributed statistically (Kohane and Weber, 2013; Pivovarov et al., 2014). We construct an algorithm, PopKLD, which is based on information criterion model selection (Burnham and Anderson, 2002; Claeskens and Hjort, 2008), is intended to reduce and cope with health care process biases and to produce an intuitively understandable continuous summary. The PopKLD algorithm can be automated and is designed to be applicable in high-throughput settings; for example, the output of the PopKLD algorithm can be used as input for phenotyping algorithms. Moreover, we develop the PopKLD-CAT algorithm that transforms the continuous PopKLD summary into a categorical summary useful for applications that require categorical data such as topic modeling. We evaluate our methodology in two ways. First, we apply the method to laboratory data collected in two different health care contexts, primary versus intensive care. We show that the PopKLD preserves known physiologic features in the data that are lost when summarizing the data using more common laboratory data summaries such as mean and standard deviation. Second, for three disease-laboratory measurement pairs, we perform a phenotyping task: we use the PopKLD and PopKLD-CAT algorithms to define high and low values of the laboratory variable that are used for defining a disease state. We then compare the relationship between the PopKLD-CAT summary disease predictions and the same predictions using empirically estimated mean and standard deviation to a gold standard generated by clinical review of patient records. We find that the PopKLD laboratory data summary is substantially better at predicting disease state. The PopKLD or PopKLD-CAT algorithms are not meant to be used as phenotyping algorithms, but we use the phenotyping task to show what information can be gained when using a more informative laboratory data summary. In the process of evaluation our method we show that the different clinical contexts and laboratory measurements necessitate different statistical summaries. Similarly, leveraging the principle of maximum entropy we argue that while some laboratory data only have sufficient information to estimate a mean and standard deviation, other laboratory data captured in an EHR contain substantially more information than can be captured in higher-parameter models.

MAO inhibitory activity of bromo-2-phenylbenzofurans: synthesis, in vitro study, and docking calculations

Monoamine oxidase (MAO) is an enzyme responsible for metabolism of monoamine neurotransmitters which play an important role in brain development and function. This enzyme exists in two isoforms, and it has been demonstrated that MAO-B activity, but not MAO-A activity, increases with aging. MAO inhibitors show clinical value because besides the monoamine level regulation they reduce the formation of by-products of the MAO catalytic cycle, which are toxic to the brain. A series of 2-phenylbenzofuran derivatives was designed, synthesized and evaluated against hMAO-A and hMAO-B enzymes. A bromine substituent was introduced in the 2-phenyl ring, whereas position 5 or 7 of the benzofuran moiety was substituted with a methyl group. Most of the tested compounds inhibited preferentially MAO-B in a reversible manner, with IC50 values in the low micro or nanomolar range. The 2-(2'-bromophenyl)-5-methylbenzofuran (5) was the most active compound identified (IC50 = 0.20 μM). In addition, none of the studied compounds showed cytotoxic activity against the human neuroblastoma cell line SH-SY5Y. Molecular docking simulations were used to explain the observed hMAO-B structure-activity relationship for this type of compounds.

Progress in the development of small molecules as new human A3 adenosine receptor ligands based on the 3-thiophenylcoumarin core

3-Thiophenylcoumarins are described as adenosine receptor ligands. Synthesis, in vitro pharmacological assays and docking studies were performed.

Discovery of the first A1 adenosine receptor ligand based on the chromone scaffold

The first potent and selective hA₁AR ligand based on the chromone scaffold is reported in this work.

Navigating in chromone chemical space: discovery of novel and distinct A3 adenosine receptor ligands

One of the major hurdles in the development of effective drugs targeting GPCRs is finding ligands selective for a specific receptor subtype. Here we describe a potent and selective hormone-based hA₃ AR ligand (K_i of 167 nM) with a remarkable selectivity.

Design and discovery of tyrosinase inhibitors based on a coumarin scaffold

A novel series of 3-aryl and 3-heteroarylcoumarins displaying tyrosinase inhibitory activity.

Systems pharmacology augments drug safety surveillance

Small molecule drugs are the foundation of modern medical practice yet their use is limited by the onset of unexpected and severe adverse events (AEs). Regulatory agencies rely on post-marketing surveillance to monitor safety once drugs are approved for clinical use. Despite advances in pharmacovigilance methods that address issues of confounding bias, clinical data of AEs are inherently noisy. Systems pharmacology– the integration of systems biology and chemical genomics – can illuminate drug mechanisms of action. We hypothesize that these data can improve drug safety surveillance by highlighting drugs with a mechanistic connection to the target phenotype (enriching true positives) and filtering those that do not (depleting false positives). We present an algorithm, the modular assembly of drug safety subnetworks (MADSS), to combine systems pharmacology and pharmacovigilance data and significantly improve drug safety monitoring for four clinically relevant adverse drug reactions.

Similarity-based modeling applied to signal detection in pharmacovigilance

One of the main objectives in pharmacovigilance is the detection of adverse drug events (ADEs) through mining of healthcare databases, such as electronic health records or administrative claims data. Although different approaches have been shown to be of great value, research is still focusing on the enhancement of signal detection to gain efficiency in further assessment and follow-up. We applied similarity-based modeling techniques, using 2D and 3D molecular structure, ADE, target, and ATC (anatomical therapeutic chemical) similarity measures, to the candidate associations selected previously in a medication-wide association study for four ADE outcomes. Our results showed an improvement in the precision when we ranked the subset of ADE candidates using similarity scorings. This method is simple, useful to strengthen or prioritize signals generated from healthcare databases, and facilitates ADE detection through the identification of the most similar drugs for which ADE information is available.

A distribution-based method for assessing the differences between clinical trial target populations and patient populations in electronic health records

OBJECTIVE

To improve the transparency of clinical trial generalizability and to illustrate the method using Type 2 diabetes as an example.

METHODS

Our data included 1,761 diabetes clinical trials and the electronic health records (EHR) of 26,120 patients with Type 2 diabetes who visited Columbia University Medical Center of New-York Presbyterian Hospital. The two populations were compared using the Generalizability Index for Study Traits (GIST) on the earliest diagnosis age and the mean hemoglobin A1c (HbA1c) values.

RESULTS

Greater than 70% of Type 2 diabetes studies allow patients with HbA1c measures between 7 and 10.5, but less than 40% of studies allow HbA1c<7 and fewer than 45% of studies allow HbA1c>10.5. In the real-world population, only 38% of patients had HbA1c between 7 and 10.5, with 12% having values above the range and 52% having HbA1c<7. The GIST for HbA1c was 0.51. Most studies adopted broad age value ranges, with the most common restrictions excluding patients >80 or <18 years. Most of the real-world population fell within this range, but 2% of patients were <18 at time of first diagnosis and 8% were >80. The GIST for age was 0.75.

CONCLUSIONS

We contribute a scalable method to profile and compare aggregated clinical trial target populations with EHR patient populations. We demonstrate that Type 2 diabetes studies are more generalizable with regard to age than they are with regard to HbA1c. We found that the generalizability of age increased from Phase 1 to Phase 3 while the generalizability of HbA1c decreased during those same phases. This method can generalize to other medical conditions and other continuous or binary variables. We envision the potential use of EHR data for examining the generalizability of clinical trials and for defining population-representative clinical trial eligibility criteria.

Improving access to longitudinal patient health information within an emergency department

We designed and implemented an electronic patient tracking system with improved user authentication and patient selection. We then measured access to clinical information from previous clinical encounters before and after implementation of the system. Clinicians accessed longitudinal information for 16% of patient encounters before, and 40% of patient encounters after the intervention, indicating such a system can improve clinician access to information. We also attempted to evaluate the impact of providing this access on inpatient admissions from the emergency department, by comparing the odds of inpatient admission from an emergency department before and after the improved access was made available. Patients were 24% less likely to be admitted after the implementation of improved access. However, there were many potential confounders, based on the inherent pre-post design of the evaluation. Our experience has strong implications for current health information exchange initiatives.

Biclustering of adverse drug events in the FDA's spontaneous reporting system

In this article, we present a new pharmacovigilance data mining technique based on the biclustering paradigm, which is designed to identify drug groups that share a common set of adverse events (AEs) in the spontaneous reporting system (SRS) of the US Food and Drug Administration (FDA). A taxonomy of biclusters is developed, revealing that a significant number of bona fide adverse drug event (ADE) biclusters have been identified. Statistical tests indicate that it is extremely unlikely that the bicluster structures thus discovered, as well as their content, could have arisen by mere chance. Some of the biclusters classified as indeterminate provide support for previously unrecognized and potentially novel ADEs. In addition, we demonstrate the potential importance of the proposed methodology in several important aspects of pharmacovigilance such as providing insight into the etiology of ADEs, facilitating the identification of novel ADEs, suggesting methods and a rationale for aggregating terminologies, highlighting areas of focus, and providing an exploratory tool for data mining.

Making grandma's data secure: a security architecture for home telemedicine

Home telemedicine presents special challenges for data security and privacy. Experience in the Informatics for Diabetes Education And Telemedicine (IDEATel) project has demonstrated that data security is not a one-size-fits-all problem. The IDEATel users include elderly patients in their homes, nurse case managers, physicians, and researchers. The project supports multiple computer systems that require a variety of user interactions, including: data entry, data review, patient education, videoconferencing, and electronic monitoring. To meet these various needs, a number of different of security solutions were utilized, including: UserID/Password, PKI certificates, time-based tokens, IP filtering, VPNs, symmetric and asymmetric encryption schemes, firewalls and dedicated connections. These were combined in different ways to meet the needs of each user groups.

A knowledge model for the interpretation and visualization of NLP-parsed discharged summaries

At our institution, a Natural Language Processing (NLP) tool called MedLEE is used on a daily basis to parse medical texts including complete discharge summaries. MedLEE transforms written text into a generic structured format, which preserves the richness of the underlying natural language expressions by the use of concept modifiers (like change, certainty, degree and status). As a tradeoff, extraction of application-specific medical information is difficult without a clear understanding of how these modifiers combine. We report on a knowledge model for MedLEE modifiers that is helpful for a high level interpretation of NLP data and is used for the generation of two distinct views on NLP-parsed discharge summaries: A physician view offering a condensed overview of the severity of patient problems and a data mining view featuring binary problem states useful for machine learning.

Evaluating the UMLS as a source of lexical knowledge for medical language processing

Medical language processing (MLP) systems rely on specialized lexicons in order to recognize, classify, and normalize medical terminology, and the performance of an MLP system is dependent on the coverage and quality of such lexicons. However, the acquisition of lexical knowledge is expensive and time-consuming. The UMLS is a comprehensive resource that can be used to acquire lexical knowledge needed for medical language processing. This paper describes methods that use these resources to automatically create lexical entries and generate two lexicons. The first lexicon was created primarily using the UMLS, whereas the second was created by supplementing the lexicon of an existing MLP system called MedLEE with entries based on the UMLS. We subsequently carried out a study, which is the primary focus of this paper, using MedLEE with each of the two lexicons and also the current MedLEE lexicon to measure performance. Overall accuracy, sensitivity, and specificity using the lexicon primarily based on the UMLS were.86,.60, and.96 respectively. Those measures using the MedLEE lexicon alone were.93,.81, and.93, which was significantly better except for specificity; performance using the supplemental lexicon was exactly the same as performance using solely the MedLEE lexicon.

Coding neuroradiology reports for the Northern Manhattan Stroke Study: a comparison of natural language processing and manual review

Automated systems using natural language processing may greatly speed chart review tasks for clinical research, but their accuracy in this setting is unknown. The objective of this study was to compare the accuracy of automated and manual coding in the data acquisition tasks of an ongoing clinical research study, the Northern Manhattan Stroke Study(NOMASS). We identified 471 neuroradiology reports of brain images used in the NOMASS study. Using both automated and manual coding, we completed a standardized NOMASS imaging form with the information contained in these reports. We then generated ROC curves for both manual and automated coding by comparing our results to the original NOMASS data, where study in investigators directly coded their interpretations of brain images. The areas under the ROC curves for both manual and automated coding were the main outcome measure. The overall predictive value of the automated system (ROC area 0.85, 95% CI 0.84-0.87) was not statistically different from the predictive value of the manual coding (ROC area 0.87, 95% CI 0.83-0.91). Measured in terms of accuracy, the automated system performed slightly worse than manual coding. The overall accuracy of the automated system was 84% (CI 83-85%). The overall accuracy of manual coding was 86% (CI 84-88%). The difference in accuracy between the two methods was small but statistically significant (P = 0.026). Errors in manual coding appeared to be due to differences between neurologists' and nueroradiologists' interpretation, different use of detailed anatomic terms, and lack of clinical information. Automated systems can use natural language processing to rapidly perform complex data acquisition tasks. Although there is a small decrease in the accuracy of the data as compared to traditional methods, automated systems may greatly expand the power of chart review in clinical research design and implementation.

Medical text representations for inductive learning

Inductive learning algorithms have been proposed as methods for classifying medical text reports. Many of these proposed techniques differ in the way the text is represented for use by the learning algorithms. Slight differences can occur between representations that may be chosen arbitrarily, but such differences can significantly affect classification algorithm performance. We examined 8 different data representation techniques used for medical text, and evaluated their use with standard machine learning algorithms. We measured the loss of classification-relevant information due to each representation. Representations that captured status information explicitly resulted in significantly better performance. Algorithm performance was dependent on subtle differences in data representation.

Internet-based home asthma telemonitoring: can patients handle the technology?

STUDY OBJECTIVE

To evaluate the validity of spirometry self-testing during home telemonitoring and to assess the acceptance of an Internet-based home asthma telemonitoring system by asthma patients.

DESIGN

We studied an Internet-based telemonitoring system that collected spirometry data and symptom reports from asthma patients' homes for review by physicians in the medical center's clinical information system. After a 40-min training session, patients completed an electronic diary and performed spirometry testing twice daily on their own from their homes for 3 weeks. A medical professional visited each patient by the end of the third week of monitoring, 10 to 40 min after the patient had performed self-testing, and asked the patient to perform the spirometry test again under his supervision. We evaluated the validity of self-testing and surveyed the patients attitude toward the technology using a standardized questionnaire.

SETTING

Telemonitoring was conducted in patients' homes in a low-income inner city area.

PATIENTS

Thirty-one consecutive asthma patients without regard to computer experience.

MEASUREMENT AND RESULTS

Thirty-one asthma patients completed 3 weeks of monitoring. A paired t test showed no difference between unsupervised and supervised home spirometry self-testing. The variability of FVC (4.1%), FEV(1) (3. 7%), peak expiratory flow (7.9%), and other spirometric indexes in our study was similar to the within-subject variability reported by other researchers. Despite the fact that the majority of the patients (71%) had no computer experience, they indicated that the self-testing was "not complicated at all" or only "slightly complicated." The majority of patients (87.1%) were strongly interested in using home asthma telemonitoring in the future.

CONCLUSIONS

Spirometry self-testing by asthma patients during telemonitoring is valid and comparable to those tests collected under the supervision of a trained medical professional. Internet-based home asthma telemonitoring can be successfully implemented in a group of patients with no computer background.

Considering clustering: a methodological review of clinical decision support system studies

Computer-based clinical decision support systems (CDSSs) are often implemented at a cluster level, but standard statistical methods for sample estimation and analysis may not be appropriate for such studies. This review aims to determine whether the design and analysis methods of cluster-based studies were adequately addressed in reports of CDSS studies. We retrieved 61 reports of the CDSS controlled trials and identified 24 studies meeting our inclusion criteria. Of these, none included sample size calculations that allowed for clustering, while 14 (58%) took account of clustering in the analysis. Although there is increasing recognition of the methodological issues associated with cluster design in health care, many medical informaticians are still not aware of these issues. Investigators should publish estimates of the intracluster correlation coefficients and variance components in their reports to guide the planning of the future studies.

Generic data modeling for home telemonitoring of chronically ill patients

Management of many types of chronic diseases such as diabetes and asthma relies heavily on patients' self-monitoring of their disease conditions. In recent years, internet-based home telemonitoring systems that allow transmission of patient data to a central database and offer immediate access to the data by the care providers have become available. However, these systems often work with only one or a few types of medical devices and thus are limited in the types of diseases they can monitor. For example, a system designed to collect spirometry data from asthmatic patients cannot be easily adapted to collect blood glucose data from diabetic patients. This is because different medical devices produce different types of data and the existing telemonitoring systems are generally built around a proprietary data schema specific for the device used. In this paper, we describe a generic data schema for a telemonitoring system that is applicable to different types of medical devices and different diseases, and show an implementation of the schema in a relational database suitable for a variety of telemonitoring activities.

Natural language processing and its future in medicine

If accurate clinical information were available electronically, automated applications could be developed to use this information to improve patient care and lower costs. However, to be fully retrievable, clinical information must be structured or coded. Many online patient reports are not coded, but are recorded in natural-language text that cannot be reliably accessed. Natural language processing (NLP) can solve this problem by extracting and structuring text-based clinical information, making clinical data available for use. NLP systems are quite difficult to develop, as they require substantial amounts of knowledge, but progress has definitely been made. Some NLP systems have been developed and tested and have demonstrated promising performance in practical clinical applications; some of these systems have already been deployed. The authors provide background information about NLP, briefly describe some of the systems that have been recently developed, and discuss the future of NLP in medicine.

Telematic system for monitoring of asthma severity in patients' homes

Despite advances in the treatment of asthma the morbidity and mortality of this disease has increased significantly in the past several years. Recent studies have shown that monitoring of asthma severity in the patient home especially combined with patient education can reduce incidence of asthma exacerbation and subsequent hospitalization. The existing methods for home asthma monitoring are limited by four factors; they completely rely on a patient's ability to document and to evaluate test results; there is no easy way for a physician to review data in a timely manner; they use imprecise tools for evaluation of asthma severity and they don't provide clinical decision support tools. The goal of this study is to develop and to evaluate a telematic system for asthma severity monitoring which will minimize patients' efforts in performing self-testing at their homes and allow prompt reciprocal exchange of all relevant information between patients and health care providers. In our setting, patients use portable spirometer and pocket-sized palmtop computer for data exchange. Our system allows daily serial monitoring of asthma severity at patients' homes using Forced Vital Capacity test and symptom diary. The results of the tests become available for physicians review immediately after completion of self-testing procedures via Web browser. The results can be transmitted from patients' homes (or any other remote location) to the medical records database via landline or wireless networks in several minutes. Each time the remote server receives patient's results, it invokes the application which tests the validity of data, analyzes parameters trends and dispatches corresponding messages for the patient and, if necessary, for physicians. Such an approach provides constant feedback loop between asthma patient and physician. The system has been tested in 10 healthy volunteers and asthma patients. Patients participated in the study from two to 21 days. The test results showed that the system provides reliable reciprocal exchange of all relevant information between a physician and asthma patient in home settings. Average transmission time from the patient's palmtop to the remote central data repository was about 1 minute for 14.4 Kbps landline modem, 6 minutes for cellular network and 8 minutes for RAM Mobile network. After transmission, the test results were immediately available for review at our web site.

A health information network for managing innercity tuberculosis: bridging clinical care, public health, and home care

The purpose of this study was to use a health information network and innovative technology to coordinate tuberculosis care. An innercity medical center, a local health department, and a home care nurse service in northern Manhattan were used. The organizations were linked with computer networks. An automated decision support system with a natural language processor was used to detect tuberculosis cases and report them to the health department, and to select patients for respiratory isolation. Educational materials were placed on the World Wide Web and a Web-based kiosk. Home care nurses were outfitted with wireless pen-based computers, and data were relayed to the medical center. Automated tuberculosis case reporting resulted in time savings but not improved accuracy. Automated rules resulted in significant improvements in respiratory isolation. Kiosk educational materials were well-used. Wireless computing led to better access to information for both nurses and physicians, but not to reduction of workload. The key success element was recognition of critical priorities. It is concluded that innovative technology can facilitate the coordination of clinical care, public health, and home care.

Representing information in patient reports using natural language processing and the extensible markup language

OBJECTIVE

To design a document model that provides reliable and efficient access to clinical information in patient reports for a broad range of clinical applications, and to implement an automated method using natural language processing that maps textual reports to a form consistent with the model.

METHODS

A document model that encodes structured clinical information in patient reports while retaining the original contents was designed using the extensible markup language (XML), and a document type definition (DTD) was created. An existing natural language processor (NLP) was modified to generate output consistent with the model. Two hundred reports were processed using the modified NLP system, and the XML output that was generated was validated using an XML validating parser.

RESULTS

The modified NLP system successfully processed all 200 reports. The output of one report was invalid, and 199 reports were valid XML forms consistent with the DTD.

CONCLUSIONS

Natural language processing can be used to automatically create an enriched document that contains a structured component whose elements are linked to portions of the original textual report. This integrated document model provides a representation where documents containing specific information can be accurately and efficiently retrieved by querying the structured components. If manual review of the documents is desired, the salient information in the original reports can also be identified and highlighted. Using an XML model of tagging provides an additional benefit in that software tools that manipulate XML documents are readily available.

Classification algorithms applied to narrative reports

Narrative text reports represent a significant source of clinical data. However, the information stored in these reports is inaccessible to many automated decision support systems. Data mining techniques can assist in extracting information from narrative data. Multiple classification methods, such as rule generation, decision trees, Bayesian classifiers, and information retrieval were used to classify a set of 200 chest X-ray reports according to 6 clinical conditions indicated. A general-purpose natural language processor was used to convert the narrative text into a coded form that could be used by the classification algorithms. Significant differences in performance were found between algorithms. The best performing algorithm applied to the processor output was significantly better than information retrieval applied to raw text. Predictor variables from the coded processor output were limited to avoid overfitting. Methods that limited by domain knowledge performed significantly better than those that limited by conditional probabilities of the variables in the training set. Algorithms were also shown to be dependent on training set size.

WebCIS: large scale deployment of a Web-based clinical information system

WebCIS is a Web-based clinical information system. It sits atop the existing Columbia University clinical information system architecture, which includes a clinical repository, the Medical Entities Dictionary, an HL7 interface engine, and an Arden Syntax based clinical event monitor. WebCIS security features include authentication with secure tokens, authorization maintained in an LDAP server, SSL encryption, permanent audit logs, and application time outs. WebCIS is currently used by 810 physicians at the Columbia-Presbyterian center of New York Presbyterian Healthcare to review and enter data into the electronic medical record. Current deployment challenges include maintaining adequate database performance despite complex queries, replacing large numbers of computers that cannot run modern Web browsers, and training users that have never logged onto the Web. Although the raised expectations and higher goals have increased deployment costs, the end result is a far more functional, far more available system.

Automating a severity score guideline for community-acquired pneumonia employing medical language processing of discharge summaries

Obtaining encoded variables is often a key obstacle to automating clinical guidelines. Frequently the pertinent information occurs as text in patient reports, but text is inadequate for the task. This paper describes a retrospective study that automates determination of severity classes for patients with community-acquired pneumonia (i.e. classifies patients into risk classes 1-5), a common and costly clinical problem. Most of the variables for the automated application were obtained by writing queries based on output generated by MedLEE1, a natural language processor that encodes clinical information in text. Comorbidities, vital signs, and symptoms from discharge summaries as well as information from chest x-ray reports were used. The results were very good because when compared with a reference standard obtained manually by an independent expert, the automated application demonstrated an accuracy, sensitivity, and specificity of 93%, 92%, and 93% respectively for processing discharge summaries, and 96%, 87%, and 98% respectively for chest x-rays. The accuracy for vital sign values was 85%, and the accuracy for determining the exact risk class was 80%. The remaining 20% that did not match exactly differed by only one class.

A reliability study for evaluating information extraction from radiology reports

GOAL

To assess the reliability of a reference standard for an information extraction task.

SETTING

Twenty-four physician raters from two sites and two specialties judged whether clinical conditions were present based on reading chest radiograph reports.

METHODS

Variance components, generalizability (reliability) coefficients, and the number of expert raters needed to generate a reliable reference standard were estimated.

RESULTS

Per-rater reliability averaged across conditions was 0.80 (95% CI, 0.79-0.81). Reliability for the nine individual conditions varied from 0.67 to 0.97, with central line presence and pneumothorax the most reliable, and pleural effusion (excluding CHF) and pneumonia the least reliable. One to two raters were needed to achieve a reliability of 0.70, and six raters, on average, were required to achieve a reliability of 0.95. This was far more reliable than a previously published per-rater reliability of 0.19 for a more complex task. Differences between sites were attributable to changes to the condition definitions.

CONCLUSION

In these evaluations, physician raters were able to judge very reliably the presence of clinical conditions based on text reports. Once the reliability of a specific rater is confirmed, it would be possible for that rater to create a reference standard reliable enough to assess aggregate measures on a system. Six raters would be needed to create a reference standard sufficient to assess a system on a case-by-case basis. These results should help evaluators design future information extraction studies for natural language processors and other knowledge-based systems.

Evaluating natural language processors in the clinical domain

Evaluating natural language processing (NLP) systems in the clinical domain is a difficult task which is important for advancement of the field. A number of NLP systems have been reported that extract information from free-text clinical reports, but not many of the systems have been evaluated. Those that were evaluated noted good performance measures but the results were often weakened by ineffective evaluation methods. In this paper we describe a set of criteria aimed at improving the quality of NLP evaluation studies. We present an overview of NLP evaluations in the clinical domain and also discuss the Message Understanding Conferences (MUC) [1-4]. Although these conferences constitute a series of NLP evaluation studies performed outside of the clinical domain, some of the results are relevant within medicine. In addition, we discuss a number of factors which contribute to the complexity that is inherent in the task of evaluating natural language systems.

Developing online support for clinical information system developers: the FAQ approach

OBJECTIVE

We investigate a knowledge-based help system for developers of an integrated clinical information system (CIS). The first objective in the study was to determine the system's ability to answer users' questions effectively. User performance and behavior were studied. The second objective was to evaluate the effect of using questions and answers to augment or replace traditional program documentation.

DESIGN

A comparative study of user and system effectiveness using a collection of 47 veritable questions regarding the CIS, solicited from various CIS developers, is conducted. Most questions were concerning the clinical data model and acquiring the data.

MEASUREMENTS

Answers using current documentation known by users were compared to answers found using the help system. Answers existing within traditional documentation were compared to answers existing within question-answer exchanges (Q-A's).

RESULTS

The support system augmented 39% of users' answers to test questions. Though the Q-A's were less than 5% of the total documentation collected, these files contained answers to nearly 50% of the questions in the test group. The rest of the documentation contained about 75% of the answers.

CONCLUSIONS

A knowledge-based help system built by collecting questions and answers can be a viable alternative to large documentation files, providing the questions and answers can be collected effectively.

Respiratory isolation of tuberculosis patients using clinical guidelines and an automated clinical decision support system

OBJECTIVE

To evaluate a clinical guideline and an automated computer protocol for detection and respiratory isolation of tuberculosis (TB) patients.

DESIGN

An automated computer protocol was tested on a retrospective cohort of adult culture-positive TB patients admitted from 1992 to 1993 to Columbia-Presbyterian Medical Center and evaluated prospectively from July 1995 until July 1996.

SETTING

A large teaching hospital in New York City.

PATIENTS

171 adult patients admitted from 1992 to 1993 and 43 patients admitted between July 1995 and July 1996.

INTERVENTIONS

The 1990 Centers for Disease Control and Prevention guidelines for preventing transmission of TB were adapted to formulate clinical guidelines to ensure early isolation of TB patients at Columbia-Presbyterian Medical Center.

RESULTS

Implementation of a clinical respiratory isolation protocol resulted in a significant improvement in TB patient isolation rates, from 45 (51%) of 88 in 1992 to 62 (75%) of 83 in 1993 (P<.001). In testing automated protocols, the theoretical improvement would have identified an additional 27 patients not isolated by clinicians, making the overall isolation rate 134 (78%) of 171. For the prospective evaluation, 30 (70%) of 43 TB patients were isolated by clinicians adhering to the clinical protocol. Four additional patients were identified by the automated TB protocol, making the combined isolation rate 34 (79%) of 43.

CONCLUSIONS

A clinical policy to isolate TB patients and suspected human immunodeficiency virus-infected patients with cough, fever, or radiographic abnormalities improved isolation of culture-documented TB patients from 1992 to 1993. Automated computer protocols were successful in identifying additional potentially infectious patients that clinicians failed to place on respiratory isolation. Clinical and automated protocols combined resulted in better isolation rates than a clinical protocol alone.

An evaluation of natural language processing methodologies

Medical language processing (MLP) systems that codify information in textual patient reports have been developed to help solve the data entry problem. Some systems have been evaluated in order to assess performance, but there has been little evaluation of the underlying technology. Various methodologies are used by the different MLP systems but a comparison of the methods has not been performed although evaluations of MLP methodologies would be extremely beneficial to the field. This paper describes a study that evaluates different techniques. To accomplish this task an existing MLP system MedLEE was modified and results from a previous study were used. Based on confidence intervals and differences in sensitivity and specificity between each technique and all the others combined, the results showed that the two methods based on obtaining the largest well-formed segment within a sentence had significantly higher sensitivity than the others by 5% and 6%. The method based on recognizing a complete sentence had a significantly worse sensitivity than the others by 7% and a better specificity by .2%. None of the methods had significantly worse specificity.

Evolution of a knowledge base for a clinical decision support system encoded in the Arden Syntax

Clinical decision support systems (CDSS) are being used increasingly in medical practice. Thus, long-term maintenance of the knowledge bases (KB) of such systems becomes important. To quantify changes that occur as a KB evolves, we studied the KB at the Columbia-Presbyterian Medical Center. This KB has a total of 229 Medical Logic Modules (MLMs) encoded in the Arden Syntax. Eliminating those never used in practice, we retrospectively analyzed 156 MLMs developed over 78 months. We noted 2020 distinct versions of these MLMs that included 5528 changed statements over time. These changes occurred primarily in the logic slot (38.7% of all changes), the action slot (17.8%), in queries (15.0%) and in the data slot exclusive of queries (12.4%). We conclude that long-term maintenance of a KB for a CDSS requires significant changes over time. We discuss the implications of these results for the design of KB editors for the Arden Syntax.

Knowledge discovery and data mining to assist natural language understanding

As natural language processing systems become more frequent in clinical use, methods for interpreting the output of these programs become increasingly important. These methods require the effort of a domain expert, who must build specific queries and rules for interpreting the processor output. Knowledge discovery and data mining tools can be used instead of a domain expert to automatically generate these queries and rules. C5.0, a decision tree generator, was used to create a rule base for a natural language understanding system. A general-purpose natural language processor using this rule base was tested on a set of 200 chest radiograph reports. When a small set of reports, classified by physicians, was used as the training set, the generated rule base performed as well as lay persons, but worse than physicians. When a larger set of reports, using ICD9 coding to classify the set, was used for training the system, the rule base performed worse than the physicians and lay persons. It appears that a larger, more accurate training set is needed to increase performance of the method.

Patients' acceptance of Internet-based home asthma telemonitoring

We studied asthma patients from a low-income inner-city community without previous computer experience. The patients were given portable spirometers to perform spirometry tests and palmtop computers to enter symptoms in a diary, to exchange messages with physician and to review test results. The self-testing was performed at home on a daily basis. The results were transmitted to the hospital information system immediately after completion of each test. Physician could review results using an Internet Web browser from any location. A constantly active decision support server monitored all data traffic and dispatched alerts when certain clinical conditions were met. Seventeen patients, out of 19 invited, agreed to participate in the study and have been monitored for three weeks. They have been surveyed then using standardized questionnaire. Most of the patients (82.4%) characterized self-testing procedures as "not complicated at all." In 70.6% of cases self-testing did not interfere with usual activities, and 82.4% of patients felt the self-testing required a "very little" amount of their time. All patients stated that it is important for them to know that the results can be reviewed by professional staff in a timely manner. However, only 29.5% of patients reviewed their results at least once a week at home independently. The majority of the patients (94.1%) were strongly interested in using home asthma telemonitoring in the future. We concluded that Internet-based home asthma telemonitoring can be successfully implemented in the group of patients without previous computer background.

Extracting findings from narrative reports: software transferability and sources of physician disagreement

While natural language processing systems are beginning to see clinical use, it remains unclear whether they can be disseminated effectively through the health care community. MedLEE, a general-purpose natural language processor developed for Columbia-Presbyterian Medical Center, was compared to physicians' ability to detect seven clinical conditions in 200 Brigham and Women's Hospital chest radiograph reports. Using the system on the new institution's reports resulted in a small but measurable drop in performance (it was distinguishable from physicians at p = 0.011). By making adjustments to the interpretation of the processor's coded output (without changing the processor itself), local behavior was better accommodated, and performance improved so that it was indistinguishable from the physicians. Pairs of physicians disagreed on at least one condition for 22% of reports; the source of disagreement appeared to be interpretation of findings, gauging likelihood and degree of disease, and coding errors.

IAIMS architecture

An information system architecture defines the components of a system and the interfaces among the components. A good architecture is essential for creating an Integrated Advanced Information Management System (IAIMS) that works as an integrated whole yet is flexible enough to accommodate many users and roles, multiple applications, changing vendors, evolving user needs, and advancing technology. Modularity and layering promote flexibility by reducing the complexity of a system and by restricting the ways in which components may interact. Enterprise-wide mediation promotes integration by providing message routing, support for standards, dictionary-based code translation, a centralized conceptual data schema, business rule implementation, and consistent access to databases. Several IAIMS sites have adopted a client-server architecture, and some have adopted a three-tiered approach, separating user interface functions, application logic, and repositories.