Using a Natural Language Processing Approach to Support Rapid Knowledge Acquisition

Implementing artificial intelligence to extract insights from large, real-world clinical data sets can supplement and enhance knowledge management efforts for health sciences research and clinical care. At Vanderbilt University Medical Center (VUMC), the in-house developed Word Cloud natural language processing system extracts coded concepts from patient records in VUMC's electronic health record repository using the Unified Medical Language System terminology. Through this process, the Word Cloud extracts the most prominent concepts found in the clinical documentation of a specific patient or population. The Word Cloud provides added value for clinical care decision-making and research. This viewpoint paper describes a use case for how the VUMC Center for Knowledge Management leverages the condition-disease associations represented by the Word Cloud to aid in the knowledge generation needed to inform the interpretation of phenome-wide association studies.

Knowledgebase strategies to aid interpretation of clinical correlation research

OBJECTIVE

Knowledgebases are needed to clarify correlations observed in real-world electronic health record (EHR) data. We posit design principles, present a unifying framework, and report a test of concept.

MATERIALS AND METHODS

We structured a knowledge framework along 3 axes: condition of interest, knowledge source, and taxonomy. In our test of concept, we used hypertension as our condition of interest, literature and VanderbiltDDx knowledgebase as sources, and phecodes as our taxonomy. In a cohort of 832 566 deidentified EHRs, we modeled blood pressure and heart rate by sex and age, classified individuals by hypertensive status, and ran a Phenome-wide Association Study (PheWAS) for hypertension. We compared the correlations from PheWAS to the associations in our knowledgebase.

RESULTS

We produced PhecodeKbHtn: a knowledgebase comprising 167 hypertension-associated diseases, 15 of which were also negatively associated with blood pressure (pos+neg). Our hypertension PheWAS included 1914 phecodes, 129 of which were in the PhecodeKbHtn. Among the PheWAS association results, phecodes that were in PhecodeKbHtn had larger effect sizes compared with those phecodes not in the knowledgebase.

DISCUSSION

Each source contributed unique and additive associations. Models of blood pressure and heart rate by age and sex were consistent with prior cohort studies. All but 4 PheWAS positive and negative correlations for phecodes in PhecodeKbHtn may be explained by knowledgebase associations, hypertensive cardiac complications, or causes of hypertension independently associated with hypotension.

CONCLUSION

It is feasible to assemble a knowledgebase that is compatible with EHR data to aid interpretation of clinical correlation research.

Systematic review of international studies evaluating MDRD and CKD-EPI estimated glomerular filtration rate (eGFR) equations in Black adults

Use of race adjustment in estimating glomerular filtration rate (eGFR) has been challenged given concerns that it may negatively impact the clinical care of Black patients, as it results in Black patients being systematically assigned higher eGFR values than non-Black patients. We conducted a systematic review to assess how well eGFR, with and without race adjustment, estimates measured GFR (mGFR) in Black adults globally. A search across multiple databases for articles published from 1999 to May 2021 that compared eGFR to mGFR and reported outcomes by Black race was performed. We included studies that assessed eGFR using the Modification of Diet in Renal Disease (MDRD) and Chronic Kidney Disease Epidemiology Collaboration (CKD-EPICr) creatinine equations. Risk of study bias and applicability were assessed with the QUality Assessment of Diagnostic Accuracy Studies-2. Of 13,167 citations identified, 12 met the data synthesis criteria (unique patient cohorts in which eGFR was compared to mGFR with and without race adjustment). The studies included patients with and without kidney disease from Africa (n = 6), the United States (n = 3), Europe (n = 2), and Brazil (n = 1). Of 11 CKD-EPI equation studies, all assessed bias, 8 assessed accuracy, 6 assessed precision, and 5 assessed correlation/concordance. Of 7 MDRD equation studies, all assessed bias, 6 assessed accuracy, 5 assessed precision, and 3 assessed correlation/concordance. The majority of studies found that removal of race adjustment improved bias, accuracy, and precision of eGFR equations for Black adults. Risk of study bias was often unclear, but applicability concerns were low. Our systematic review supports the need for future studies to be conducted in diverse populations to assess the possibility of alternative approaches for estimating GFR. This study additionally provides systematic-level evidence for the American Society of Nephrology-National Kidney Foundation Task Force efforts to pursue other options for GFR estimation.

Design and implementation of a massive open online course on enhancing the recruitment of minorities in clinical trials - Faster Together

Background

Racial and ethnic minorities are often underrepresented in clinical trials, threatening the generalizability of trial results. Several factors may contribute to underrepresentation of minorities in clinical trials, including lack of training for researchers and staff on the importance of diversity in clinical trials and effective strategies for recruiting and retaining minority populations.

Methods

Applying community engaged research principles, we developed a massive open online course (MOOC) to help research team members develop knowledge and skills to enhance the recruitment of minorities in clinical trials. A transdisciplinary working group, consisting of clinical researchers, community engagement specialists, minority clinical trial recruitment and retention educators and specialists, and knowledge management information scientists, was formed to develop an evidence-based curriculum. Feedback from the Recruitment Innovation Center Community Advisory Board was incorporated to help finalize the curriculum. The course was implemented in Coursera, an online learning platform offering MOOCs. A bootstrap paired sample t-test was used to compare pre- and post-assessments of knowledge, attitudes, and intentions as it relates to minority recruitment.

Results

The final course, entitled Faster Together, was divided into eight 1-h modules. Each module included video presentations, reading assignments, and quizzes. After 10 months, 382 individuals enrolled in the course, 105 participants completed the pre-test, and 14 participants completed the post-test. Participants’ knowledge scores were higher with an increase in the mean number of correct answers from 15.4 (95% CI:12.1–18.7) on the pre-test to 18.7 (95% CI:17.42–20.2) on the post-test. All post-test respondents (n = 14) indicated that the course improved their professional knowledge, and 71.4% of respondents indicated that they were very likely to make changes to their recruitment practices.

Conclusions

Faster Together, a massive open online course, is an acceptable, accessible approach to educating research teams on minority recruitment in clinical trials. Preliminary evidence indicates the course increased knowledge on how to recruit minorities into clinical trials and could promote change in their recruitment practices.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01240-x.

Collection of social determinants of health in the community clinic setting: a cross-sectional study

BACKGROUND

Addressing social and behavioral determinants of health (SBDs) may help improve health outcomes of community clinic patients. This cross-sectional study explored how assessing SBDs can be used to complement health data collection strategies and provide clinicians with a more in-depth understanding of their patients.

METHODS

Adult patients, ages 18 and older, at an urban community health care clinic in Tennessee, U.S.A., were asked to complete a questionnaire regarding health status, health history and SBDs while waiting for their clinic appointment. The SBD component included items from the National Academy of Medicine, the Protocol for Responding to and Assessing Patient Assets, Risks, and Experiences instrument, and the Survey of Household Economics and Decisionmaking. Data collection and analysis occurred in 2017.

RESULTS

One hundred participants completed the study. The questionnaire took approximately 11 min to complete, and the response rate was 90% or higher for all items except annual household income (unanswered by 40 participants). The median number of negative SBDs was 4 (IQR 2.75-7.0), 96 participants had at least one unmet need, and the most common negative SBD was physical activity (75%; 75/100).

CONCLUSIONS

The hybrid questionnaire provided insight into a community clinic population's SBDs and allowed for a more complete understanding than a single questionnaire alone. The brief questionnaire administration time and low non-response rate support the questionnaire's feasibility in the community clinic setting, and results can be used by clinicians to further the personalization goals of precision medicine. Next steps include evaluating how to connect patients with appropriate resources for addressing their SBDs.

Internet-Based Assessment of Oncology Health Care Professional Learning Style and Optimization of Materials for Web-Based Learning: Controlled Trial With Concealed Allocation

Background

Precision medicine has resulted in increasing complexity in the treatment of cancer. Web-based educational materials can help address the needs of oncology health care professionals seeking to understand up-to-date treatment strategies.

Objective

This study aimed to assess learning styles of oncology health care professionals and to determine whether learning style-tailored educational materials lead to enhanced learning.

Methods

In all, 21,465 oncology health care professionals were invited by email to participate in the fully automated, parallel group study. Enrollment and follow-up occurred between July 13 and September 7, 2015. Self-enrolled participants took a learning style survey and were assigned to the intervention or control arm using concealed alternating allocation. Participants in the intervention group viewed educational materials consistent with their preferences for learning (reading, listening, and/or watching); participants in the control group viewed educational materials typical of the My Cancer Genome website. Educational materials covered the topic of treatment of metastatic estrogen receptor-positive (ER+) breast cancer using cyclin-dependent kinases 4/6 (CDK4/6) inhibitors. Participant knowledge was assessed immediately before (pretest), immediately after (posttest), and 2 weeks after (follow-up test) review of the educational materials. Study statisticians were blinded to group assignment.

Results

A total of 751 participants enrolled in the study. Of these, 367 (48.9%) were allocated to the intervention arm and 384 (51.1%) were allocated to the control arm. Of those allocated to the intervention arm, 256 (69.8%) completed all assessments. Of those allocated to the control arm, 296 (77.1%) completed all assessments. An additional 12 participants were deemed ineligible and one withdrew. Of the 552 participants, 438 (79.3%) self-identified as multimodal learners. The intervention arm showed greater improvement in posttest score compared to the control group (0.4 points or 4.0% more improvement on average; P=.004) and a higher follow-up test score than the control group (0.3 points or 3.3% more improvement on average; P=.02).

Conclusions

Although the study demonstrated more learning with learning style-tailored educational materials, the magnitude of increased learning and the largely multimodal learning styles preferred by the study participants lead us to conclude that future content-creation efforts should focus on multimodal educational materials rather than learning style-tailored content.

Institute of Medicine Measures of Social and Behavioral Determinants of Health: A Feasibility Study

INTRODUCTION

Social and behavioral factors are known to affect health but are not routinely assessed in medical practice. To date, no studies have assessed a parsimonious panel of measures of social and behavioral determinants of health (SBDs). This study evaluated the panel of SBD measures recommended by the Institute of Medicine and examined the effect of question order.

METHODS

Adults, aged ≥18 years, were recruited using ResearchMatch.org for this randomized, parallel design study conducted in 2015 (data analyzed in 2015-2016). Three versions of the SBD measures, sharing the same items but in different orders of presentation (Versions 1-3), were developed. Randomized to six groups, participants completed each version at least 1 week apart (Weeks 1-3). Version order was counterbalanced across each administration and randomization was stratified by gender, race, and age. Main outcomes were effect of question order, completion time, and non-response rates.

RESULTS

Of 781 participants, 624 (80%) completed the Week 1 questionnaire; median completion time for answering all SBD questions was 5 minutes, 583/624 participants answered all items, and no statistically significant differences associated with question order were observed when comparing responses across all versions. No significant differences in responses within assignment groups over time were found, with the exception of the stress measure for Group 5 (p=0.036).

CONCLUSION

Question order did not significantly impact participant responses. Time to complete the questionnaire was brief, and non-response rate was low. Findings support the feasibility of using the Institute of Medicine-recommended questionnaire to capture SBDs.

My Cancer Genome: Evaluating an Educational Model to Introduce Patients and Caregivers to Precision Medicine Information

This study tested an innovative model for creating consumer-level content about precision medicine based on health literacy and learning style principles. "Knowledge pearl" videos, incorporating multiple learning modalities, were created to explain genetic and cancer medicine concepts. Cancer patients and caregivers (n=117) were randomized to view professional-level content directly from the My Cancer Genome (MCG) website (Group A; control), content from MCG with knowledge pearls embedded (Group B), or a consumer translation, targeted at the sixth grade level, with knowledge pearls embedded (Group C). A multivariate analysis showed that Group C, but not Group B, showed greater knowledge gains immediately after viewing the educational material than Group A. Statistically significant group differences in test performance were no longer observed three weeks later. These findings suggest that adherence to health literacy and learning style principles facilitates comprehension of precision medicine concepts and that ongoing review of the educational information is necessary.

A personalized approach to deliver health care information to diabetic patients in community care clinics

OBJECTIVE

The study's purpose was to test the generalizability of an individualized information prescription model, which has been previously validated for educating patients about hypertension in emergency department and community health center settings. Study investigators assessed the effects of educational materials targeted to health literacy levels and learning styles on patients' diabetes knowledge in a community clinic setting.

METHODS

From May to August 2012, 160 patients were recruited and randomized into intervention (n = 81) and control (n = 79) groups. Inclusion criteria included 18 years or older, English or Spanish speaker, and a type 2 diabetes diagnosis. Measures included modified versions of the Diabetes Knowledge Test and Subjective Numeracy Scale, along with brief health literacy and learning style assessments. Study team members contacted both groups after 2 and 6 weeks to reassess diabetes knowledge.

RESULTS

The control group showed no significant change in diabetes knowledge at both follow-ups. In contrast, the mean number of diabetes knowledge questions answered correctly by the intervention group increased significantly after 2 weeks (Δ = 2.66, P = 0.000), which persisted at 6 weeks (Δ = 2.46, P = 0.00).

CONCLUSIONS

This study showed that patients' knowledge about diabetes increased significantly after exposure to educational materials targeted to their health literacy levels and learning style preferences and that the model is transferrable among health conditions.

Strategically aligning a mandala of competencies to advance a transformative vision

OBJECTIVE

This paper offers insight into the processes that have shaped the Eskind Biomedical Library's (EBL's) strategic direction and its alignment to the institution's transformative vision.

SETTING

The academic biomedical library has a notable track record for developing and pioneering roles for information professionals focused on a sophisticated level of information provision that draws from and fuels practice evolutions.

STRATEGY

The medical center's overall transformative vision informs the creation of a fully aligned library strategic plan designed to effectively contribute to the execution of key organizational goals. Annual goals reflect organizational priorities and contain quantifiable and measurable deliverables. Two strategic themes, facilitating genetic literacy and preserving community history, are described in detail to illustrate the concept of goal setting.

CONCLUSION

The strategic planning model reflects EBL's adaptation to the ever-changing needs of its organization. The paper provides a characterization of a workable model that can be replicated by other institutions.

Using health literacy and learning style preferences to optimize the delivery of health information

Limited patient understanding of hypertension contributes to poor health outcomes. In 2 sequential randomized studies, the authors determined the impact of administering information tailored to health literacy level alone or in combination with preferred learning style on patients' understanding of hypertension. Patients with high blood pressure were recruited in an academic emergency department. In Experiment 1 (N = 85), the control group received only the routine discharge instructions; the intervention group received discharge instructions combined with information consistent with their health literacy level as determined by the Short Test of Functional Health Literacy. In Experiment 2 (N = 87), the information provided to the intervention group was tailored to both health literacy and learning style, as indicated by the VARK™ Questionnaire. To measure learning, the authors compared scores on a hypertension assessment administered during the emergency department visit and 2 weeks after discharge. Participants who received materials tailored to both health literacy level and learning style preference showed greater gains in knowledge than did those receiving information customized for health literacy level only. This study demonstrates that personalizing health information to learning style preferences and literacy level improves patient understanding of hypertension.

A pilot study to evaluate learning style-tailored information prescriptions for hypertensive emergency department patients

OBJECTIVE

This pilot study explored whether learning style-tailored education materials, "information prescriptions," are effective in increasing hypertension knowledge in emergency room patients.

METHODS

In a randomized trial, hypertensive emergency medicine patients received either standard care discharge instructions or discharge instructions in combination with an information prescription individualized to each patient's learning-style preference. Two weeks post-visit, the study team assessed changes in hypertension knowledge via a survey.

RESULTS

No significant difference was observed for changes in quiz scores on the hypertension knowledge assessment, though patients receiving the tailored information prescriptions reported higher levels of satisfaction with intervention materials.

CONCLUSION

The study demonstrated the workflow feasibility of implementing a learning-style approach to patient education in the emergency department setting. Further research is needed to develop more robust measures of high blood pressure knowledge among the emergency department patient population. This work will contribute to establishing a framework for developing customized information prescriptions that can be broadly adapted for use in varied settings and with varied health care conditions.

Integrating best evidence into patient care: a process facilitated by a seamless integration with informatics tools

The Vanderbilt University paper discusses how the Eskind Biomedical Library at Vanderbilt University Medical Center transitioned from a simplistic approach that linked resources to the institutional electronic medical record system, StarPanel, to a value-added service that is designed to deliver highly relevant information. Clinical teams formulate complex patient-specific questions via an evidence-based medicine literature request basket linked to individual patient records. The paper transitions into discussing how the StarPanel approach acted as a springboard for two additional projects that use highly trained knowledge management librarians with informatics expertise to integrate evidence into both order sets and a patient portal, MyHealth@Vanderbilt.

A randomized effectiveness trial of a clinical informatics consult service: impact on evidence-based decision-making and knowledge implementation

OBJECTIVE

To determine the effectiveness of providing synthesized research evidence to inform patient care practices via an evidence based informatics program, the Clinical Informatics Consult Service (CICS).

DESIGN

Consults were randomly assigned to one of two conditions: CICS Provided, in which clinicians received synthesized information from the biomedical literature addressing the consult question or No CICS Provided, in which no information was provided.

MEASUREMENT

Outcomes were measured via online post-consult forms that assessed consult purpose, actual and potential impact, satisfaction, time spent searching, and other variables.

RESULTS

Two hundred twenty six consults were made during the 19-month study period. Clinicians primarily made requests in order to update themselves (65.0%, 147/226) and were satisfied with the service results (Mean 4.52 of possible 5.0, SD 0.94). Intention to treat (ITT) analyses showed that consults in the CICS Provided condition had a greater actual and potential impact on clinical actions and clinician satisfaction than No CICS consults. Evidence provided by the service primarily impacted the use of a new or different treatment (OR 8.19 95% CI 1.04-64.00). Reasons for no or little impact included a lack of evidence addressing the issue or that the clinician was already implementing the practices indicated by the evidence.

CONCLUSIONS

Clinical decision-making, particularly regarding treatment issues, was statistically significantly impacted by the service. Programs such as the CICS may provide an effective tool for facilitating the integration of research evidence into the management of complex patient care and may foster clinicians' engagement with the biomedical literature.

Evolution of a mature clinical informationist model

Achieving evidence-based practice will require new approaches to providing information during health care delivery and to integrating evidence and informatics at the point of care. To support evidence-based practice, Vanderbilt University Medical Center's Eskind Biomedical Library (EBL) introduced the role of clinical informationist, an information specialist with sufficient knowledge and insight to function as a true partner in the health care team. To further disseminate evidence-based knowledge, the Vanderbilt University Medical Center's (VUMC) electronic medical record system and pathway development processes integrate advanced information synthesis capabilities provided by clinical informationists. Combining clinical informationist expertise with informatics tools is an effective strategy for delivering the evidence needed to support patient care decisions.

Evolution of a mature clinical informationist model

Achieving evidence-based practice will require new approaches to providing information during health care delivery and to integrating evidence and informatics at the point of care. To support evidence-based practice, Vanderbilt University Medical Center's Eskind Biomedical Library (EBL) introduced the role of clinical informationist, an information specialist with sufficient knowledge and insight to function as a true partner in the health care team. To further disseminate evidence-based knowledge, the Vanderbilt University Medical Center's (VUMC) electronic medical record system and pathway development processes integrate advanced information synthesis capabilities provided by clinical informationists. Combining clinical informationist expertise with informatics tools is an effective strategy for delivering the evidence needed to support patient care decisions.

Benchmarking information needs and use in the Tennessee public health community

OBJECTIVE

The objective is to provide insight to understanding public health officials' needs and promote access to data repositories and communication tools.

METHODS

Survey questions were identified by a focus group with members drawn from the fields of librarianship, public health, and informatics. The resulting comprehensive information needs survey, organized in five distinct broad categories, was distributed to 775 Tennessee public health workers from ninety-five counties in 1999 as part of the National Library of Medicine-funded Partners in Information Access contract.

RESULTS

The assessment pooled responses from 571 public health workers (73% return rate) representing seventy-two of ninety-five counties (53.4% urban and 46.6% rural) about their information-seeking behaviors, frequency of resources used, computer skills, and level of Internet access. Sixty-four percent of urban and 43% of rural respondents had email access at work and more than 50% of both urban and rural respondents had email at home (N = 289). Approximately 70% of urban and 78% of rural public health officials never or seldom used or needed the Centers for Disease Control (CDC) Website. Frequency data pooled from eleven job categories representing a subgroup of 232 health care professionals showed 72% never or seldom used or needed MEDLINE. Electronic resources used daily or weekly were email, Internet search engines, internal databases and mailing lists, and the Tennessee Department of Health Website.

CONCLUSIONS

While, due to the small sample size, data cannot be generalized to the larger population, a clear trend of significant barriers to computer and Internet access can be identified across the public health community. This contributes to an overall limited use of existing electronic resources that inhibits evidence-based practice.

Print versus electronic journals: a preliminary investigation into the effect of journal format on research processes

PURPOSE

To begin investigating the impact of electronic journals on research processes such as information seeking, the authors conducted a pilot journal-use study to test the hypothesis that patrons use print and electronic journals differently.

METHODOLOGY

We placed fifteen high-use print titles also available in electronic format behind the circulation desk; patrons were asked to complete a survey upon requesting a journal. We also conducted a parallel survey of patrons using library computers. Both surveys asked patrons to identify themselves by user category and queried them about their journal use.

RESULTS

During the month-long study, patrons completed sixty-nine surveys of electronic and ninety surveys of print journal use. Results analysis indicated that fellows, students, and residents preferred electronic journals, and faculty preferred print journals. Patrons used print journals for reading articles and scanning contents; they employed electronic journals for printing articles and checking references. Users considered electronic journals easier to access and search than print journals; however, they reported that print journals had higher quality text and figures.

DISCUSSION/CONCLUSION

This study is an introductory step in examining how electronic journals affect research processes. Our data revealed that there were distinct preferences in format among categories. In addition to collection management implications for libraries, these data also have implications for publishers and educators; current electronic formats do not facilitate all types of uses and thus may be changing learning patterns as well.

Information needs of clinical teams: analysis of questions received by the Clinical Informatics Consult Service

OBJECTIVES

To examine the types of questions received by Clinical Informatics Consult Service (CICS) librarians from clinicians on rounds and to analyze the number of clearly differentiated viewpoints provided in response.

DESIGN

Questions were retrieved from an internal database, the CICS Knowledge Base, and analyzed for redundancy by subject analysis. The unique questions were classified into ten categories by subject. Treatment-related questions were analyzed for the number of viewpoints represented in the librarian's response.

RESULTS

The CICS Knowledge Base contained 476 unique questions and 71 redundant questions. Among the unique queries, the top two categories accounted for 67%: treatment (36%) and disease description (31%). Within the treatment-related subset, 138 questions (59%) required representation of more than one viewpoint in the librarian's response.

DISCUSSION

Questions generated by clinicians frequently require comprehensive, critical appraisal of the medical literature, a need that can be filled by librarians trained in such techniques. This study demonstrates that many questions require representation of more than one viewpoint to answer completely. Moreover, the redundancy rate underscores the need for resources like the CICS Knowledge Base. By critically analyzing the medical literature, CICS librarians are providing a time-saving and valuable service for clinicians and charting new territory for librarians.

The Patient Informatics Consult Service (PICS): an approach for a patient-centered service

The Patient Informatics Consult Service (PICS) at the Eskind Biomedical Library at Vanderbilt University Medical Center (VUMC) provides patients with consumer-friendly information by using an information prescription mechanism. Clinicians refer patients to the PICS by completing the prescription and noting the patient's condition and any relevant factors. In response, PICS librarians critically appraise and summarize consumer-friendly materials into a targeted information report. Copies of the report are given to both patient and clinician, thus facilitating doctor-patient communication and closing the clinician-librarian feedback loop. Moreover, the prescription form also circumvents many of the usual barriers for patients in locating information, namely, patients' unfamiliarity with medical terminology and lack of knowledge of authoritative sources. PICS librarians capture the time and expertise put into these reports by creating Web-based pathfinders on prescription topics. Pathfinders contain librarian-created disease overviews and links to authoritative resources and seek to minimize the consumer's exposure to unreliable information. Pathfinders also adhere to strict guidelines that act as a model for locating, appraising, and summarizing information for consumers. These mechanisms--the information prescription, research reports, and pathfinders--serve as steps toward the long-term goal of full integration of consumer health information into patient care at VUMC.

Developing a culture of lifelong learning in a library environment

Between 1995 and 1996, the Annette and Irwin Eskind Biomedical Library (EBL) at Vanderbilt University Medical Center (VUMC) radically revised the model of service it provides to the VUMC community. An in-depth training program was developed for librarians, who began to migrate to clinical settings and establish clinical librarianship and information brokerage services beyond the library's walls. To ensure that excellent service would continue within the library, EBL's training program was adapted for library assistants, providing them with access to information about a wide variety of work roles and processes over a four to eight-month training period. Concurrently, customer service areas were reorganized so that any question--whether reference or circulation--could be answered at any of four service points, eliminating the practice of passing customers from person to person between the reference and circulation desks. To provide an incentive for highly trained library assistants to remain at EBL, management and library assistants worked together to redesign the career pathway based on defined stages of achievement, self-directed participation in library-wide projects, and demonstrated commitment to lifelong learning. Education and training were the fundamental principles at the center of all this activity.

Diagnostic certainty and hospital resource use

OBJECTIVE

To determine whether the house staff's subjective probability estimates of their initial admitting diagnoses are independent predictors of in-hospital resource consumption.

DESIGN

Descriptive correlational study.

SETTING

Academic medical center inpatient setting.

PATIENTS

Patients admitted to general medicine wards.

MEASUREMENTS

A visual analog scale with hash marks at 0, 25, 50, 75 and 100 was used to obtain a subjective probability estimate that the house staff's initial admitting diagnosis was a correct diagnosis. This provided the measure of diagnostic "certainty" at the time of admission. Patient demographic data, prior hospital stays, distance from hospital, MedisGroups scores, outlier status and vital status at discharge were obtained from administrative systems. Length of stay, total charges, cost estimates (total departmental), and number of consultations were obtained from the hospital-based transaction master database.

RESULTS

House staff evaluated 1,778 admissions, 77.2% were assigned a diagnostic certainty rating of 75% or higher. In univariate analysis, the certainty rating did not vary with the MedisGroups score, outlier status or vital status at discharge. It varied with prior stays and measures of resource use. In multivariate analysis, the certainty rating was a significant factor accounting for variation in each of the measures of resource utilization with the exception of adjusted pharmacy charges.

CONCLUSIONS

Even in a teaching hospital only a small proportion of patients had an "uncertain" diagnosis (22.8%). Nonetheless, the certainty variables were significantly related to measures of resource consumption including length of stay, total costs and number of consults obtained.

Preparing librarians to meet the challenges of today's health care environment

OBJECTIVE

Refine the understanding of the desirable skills for health sciences librarians as a basis for developing a training program model that reflects the fundamental changes in health care delivery and information technology.

DESIGN

A four-step needs assessment process: focus groups developed lists of desirable skills; the research team organized candidate skills into a taxonomy; a survey of a random sample of librarians and library users assessed perception of importance of individual skills; and the research team framed, as a unifying hypothesis, a training model.

SURVEY METHODS

The survey was distributed to random samples of 150 librarians, stratified by type of library, and 150 library users, stratified by type of use. A non-randomized sample was obtained by mounting the survey on a World Wide Web server. The survey instrument included 96 distinct skills organized into 13 categories. Respondents rated the importance of each skill on a Likert scale and provided a separate ranking by identifying the ten most important skills for the profession.

RESULTS

Among the participants, 51% of librarians and 36% of library users responded to the survey. All categories of skills were rated above the midpoint of priority on the Likert scale. All groups rated personality characteristics and skills as most important, with an understanding of the health sciences, education, and research being rated comparably to technical skills.

CONCLUSIONS

Health sciences librarians need a new educational model that provides them with broad-based tools to discover new roles and new resources for acquiring individual skills as the need arises. A unifying training model would involve trainees in developing their learning plan in a way that promotes proactive inquiry and self-directed learning, and it would rotate the trainees through projects to provide skills and an understanding of end-user work processes.

Integrating health sciences librarians into biomedicine

Vanderbilt University Medical Center (VUMC) developed a model training program to prepare current and future health sciences librarians for roles that are integrated into the diverse fabric of the health care professions. As a complement to the traditional and theoretical aspects of a librarian's education, this mixture of supplemental coursework and intensive practical training emphasizes active management of information, problem-solving skills, learning in context, and direct participation in research, while providing the opportunity for advanced academic pursuits. The practical training will take place under the auspices of an established Integrated Advanced Information Management Systems (IAIMS) library that is fully integrated with the Health Center Information Management Unit and Academic Biomedical Informatics Unit. During the planning phase, investigators are analyzing the model's aims and requirements, concentrating on (a) refining the current understanding of the roles health sciences librarians occupy; (b) developing educational strategies that prepare librarians to fulfill expanded roles; and (c) planning for an evaluation process that will support iterative revision and refinement of the model.

Evaluation of long-term maintenance of a large medical knowledge base

OBJECTIVE

Evaluate the effects of long-term maintenance activities on existing portions of a large internal medicine knowledge base.

DESIGN

Five physicians who were not among the original developers of the knowledge base independently updated a total of 15 QMR disease profiles; each updated submission was modified by a review of group serving as the "gold standard, " and the pre- and post-study versions of each updated disease profile were compared.

MEASUREMENTS

Numbers and types of changes, defined as any difference between the original version and the final version of a disease profile; reason for each change; and bibliographic references cited by the physicians as supporting evidence.

RESULTS

A total of 16% of all entries were modified by the updating process; up to 95% of the entries in a disease profile were affected. The two most common modifications were changes to the frequency of an entry, and creation of a new entry. Laboratory findings were affected much more often than were history, symptom, or physical exam findings. The dominant reason for changes was appearance of new evidence in the medical literature. The literature cited ranged from 1944 to the present.

CONCLUSIONS

This study provides an evaluation of the rate of change within the QMR medical knowledge base due to long-term maintenance. The results show that this is a demanding activity that may profoundly affect certain portions of a knowledge base, and that different types of knowledge (e.g., simple laboratory vs expensive or invasive laboratory findings) are affected by the process in different ways.

Cross-platform hypermedia examinations on the Web

The authors developed a multiple-choice medical testing system delivered using the World Wide Web. It evolved from an older, single-platform, locally-developed computer-based examination. The old system offered a number of advantages over traditional paper-based examinations, such as digital graphics and quicker, easier scoring. The new system builds on these advantages with its true cross-platform design and the addition of hypertext learning responses. The benefits of this system will increase as more medical educational resources migrate to the Web. Faculty and student feedback has been positive. The authors encourage other institutions to experiment with Web-based teaching materials, including examinations.

Educational software evaluation process

The Active Digital Library at the Vanderbilt University Medical Center has created and implemented an educational software evaluation process to facilitate the timely recommendation for product acquisition. Using this process, breadth and depth of subject coverage, clarity of presentation, quality of construction, and ease of use are being assessed by content and technical experts. The process uses a team approach, employing a bi-level evaluation instrument based on existing software evaluation forms and system bug reports.

Information needs of health care professionals in an AIDS outpatient clinic as determined by chart review

OBJECTIVE

To examine the information needs of health care professionals in HIV-related clinical encounters, and to determine the suitability of existing information sources to address those needs.

SETTING

HIV outpatient clinic.

PARTICIPANTS

Seven health care professionals with diverse training and patient care involvement.

METHODS

Based on patient charts describing 120 patient encounters, participants generated 266 clinical questions. Printed and on-line information sources were used to answer questions in two phases: using commonly available sources and using all available medical library sources.

MEASUREMENTS

The questions were divided into 16 categories by subject. The number of questions answered, their categories, the information source(s) providing answers, and the time required to answer questions were recorded for each phase.

RESULTS

Each participant generated an average of 3.8 clinical questions per chart. Five categories accounted for almost 75% of all questions; the treatment protocols/regimens category was most frequent (24%). A total of 245 questions (92%) were answered, requiring an average of 15 minutes per question. Most (87%) of the questions were answered via electronic sources, even though paper sources were consulted first.

CONCLUSIONS

The participating professionals showed considerable information needs. A combination of on-line and paper sources was necessary to provide the answers. The study suggests that present-day information sources are not entirely satisfactory for answering clinical questions generated by examining charts of HIV-infected patients.

Consistency enforcement in medical knowledge base construction

Some aspects of knowledge base creation can be partially or completely automated, resulting in higher quality and smaller effort. Computer assistance is particularly valuable in ensuring the internal consistency of a knowledge base. The article describes several techniques for consistency enforcement in QMR-KAT, an interactive knowledge base editor for the INTERNIST-I/QMR medical knowledge base. Two strategies that improve consistency are applicable to a wide range of situations. The first strategy prevents simple (but common) inconsistencies. The second strategy reveals facts that are potentially (but not necessarily) inconsistent with known data, and may require further evaluation. Both strategies use the contents of the existing knowledge base in the evaluation of new facts.

Evaluating consensus among physicians in medical knowledge base construction

This study evaluates inter-author variability in knowledge base construction. Seven board-certified internists independently profiled "acute perinephric abscess", using as reference material a set of 109 peer-reviewed articles. Each participant created a list of findings associated with the disease, estimated the predictive value and sensitivity of each finding, and assessed the pertinence of each article for making each judgment. Agreement in finding selection was significantly different from chance: seven, six, and five participants selected the same finding 78.6, 9.8, and 1.6 times more often than predicted by chance. Findings with the highest sensitivity were most likely to be included by all participants. The selection of supporting evidence from the medical literature was significantly related to each physician's agreement with the majority. The study shows that, with appropriate guidance, physicians can reproducibly extract information from the medical literature, and thus established a foundation for multi-author knowledge base construction.

Comparing contents of a knowledge base to traditional information sources

Physicians rely on the medical literature as a major source of medical knowledge and data. The medical literature, however, is continually evolving and represents different sources at different levels of coverage and detail. The recent development of computerized medical knowledge bases has added a new form of information that can potentially be used to address the practicing physician's information needs. To understand how the information from various sources differs, we compared the description of a disease found in the QMR knowledge base to those found in two general internal medicine textbooks and two specialized nephrology textbooks. The study shows both differences in coverage and differences in the level of detail. Textbooks contain information about pathophysiology and therapy that is not present in the diagnostic knowledge base. The knowledge base contains a more detailed description of the associated findings, more quantitative information, and a greater number of references to peer-reviewed medical articles. The study demonstrates that computerized knowledge bases, if properly constructed, may be able to provide clinicians with a useful new source of medical knowledge that is complementary to existing sources.

An interlingua for electronic interchange of medical information: using frames to map between clinical vocabularies

The proliferation of medical knowledge has led to the development of extensive dictionaries for electronically accessing information resources. The task of standardizing terminology used for electronic hospital records and for knowledge bases for medical expert systems and indexing the medical literature cannot easily be met by developing a single, monolithic "official" medical vocabulary. Developing a monolithic vocabulary would require a massive effort, and its existence would not guarantee its use by third-party payors, by practicing clinicians, or by developers of electronic medical information systems. Recognizing this, the National Library of Medicine (NLM) has begun to develop the Unified Medical Language System (UMLS) as a means of promoting electronic information exchange among systems with controlled vocabularies. The authors describe a frame-based system developed as an experimental approach to mapping between controlled clinical vocabularies.

Heuristic determination of quantitative data for knowledge acquisition in medicine

Knowledge acquisition for medical knowledge bases can be aided by programs that suggest possible values for portions of the data. The paper presents an experiment which was used in designing a heuristic to help the process of knowledge acquisition. The heuristic helps to determine numerical data from stylized literature excerpts in the context of knowledge acquisition for the QMR medical knowledge base. Quantitative suggestions from the heuristics are shown to agree substantially with the data incorporated in the final version of the knowledge base. The experiment shows the potential of knowledge base specific heuristics in simplifying the task of knowledge base creation.

A tool for the computer-assisted creation of QMR medical knowledge base disease profiles

QMR-KAT is a computer-based tool which assists physicians in the construction of the QMR medical knowledge base. Each QMR disease profile results from an in-depth analysis of the published medical literature, and from consultations with expert clinicians. QMR-KAT is an interactive knowledge acquisition program which facilitates the creation of new disease profiles, records the supporting evidence for each disease profile entry, and enforces consistency with the existing knowledge base. The program has been used in the creation of all new QMR disease profiles over the past two years. It has also been used to support a study on the reproducibility of knowledge base construction.