A framework for a distributed, hybrid, multiple-ontology clinical-guideline library, and automated guideline-support tools

The Application of Computer Technology to Clinical Practice Guideline Implementation: A Scoping Review

Implementation of clinical practice guidelines (CPG) is a complex and challenging task. Computer technology, including artificial intelligence (AI), has been explored to promote the CPG implementation. This study has reviewed the main domains where computer technology and AI has been applied to CPG implementation. PubMed, Embase, Web of science, the Cochrane Library, China National Knowledge Infrastructure database, WanFang DATA, VIP database, and China Biology Medicine disc database were searched from inception to December 2021. Studies involving the utilization of computer technology and AI to promote the implementation of CPGs were eligible for review. A total of 10429 published articles were identified, 117 met the inclusion criteria. 21 (17.9%) focused on the utilization of AI techniques to classify or extract the relative content of CPGs, such as recommendation sentence, condition-action sentences. 47 (40.2%) focused on the utilization of computer technology to represent guideline knowledge to make it understandable by computer. 15 (12.8%) focused on the utilization of AI techniques to verify the relative content of CPGs, such as conciliation of multiple single-disease guidelines for comorbid patients. 34 (29.1%) focused on the utilization of AI techniques to integrate guideline knowledge into different resources, such as clinical decision support systems. We conclude that the application of computer technology and AI to CPG implementation mainly concentrated on the guideline content classification and extraction, guideline knowledge representation, guideline knowledge verification, and guideline knowledge integration. The AI methods used for guideline content classification and extraction were pattern-based algorithm and machine learning. In guideline knowledge representation, guideline knowledge verification, and guideline knowledge integration, computer techniques of knowledge representation were the most used.

APLUS: A Python library for usefulness simulations of machine learning models in healthcare

Despite the creation of thousands of machine learning (ML) models, the promise of improving patient care with ML remains largely unrealized. Adoption into clinical practice is lagging, in large part due to disconnects between how ML practitioners evaluate models and what is required for their successful integration into care delivery. Models are just one component of care delivery workflows whose constraints determine clinicians' abilities to act on models' outputs. However, methods to evaluate the usefulness of models in the context of their corresponding workflows are currently limited. To bridge this gap we developed APLUS, a reusable framework for quantitatively assessing via simulation the utility gained from integrating a model into a clinical workflow. We describe the APLUS simulation engine and workflow specification language, and apply it to evaluate a novel ML-based screening pathway for detecting peripheral artery disease at Stanford Health Care.

System for Context-Specific Visualization of Clinical Practice Guidelines (GuLiNav): Concept and Software Implementation

Background

Clinical decision support systems often adopt and operationalize existing clinical practice guidelines leading to higher guideline availability, increased guideline adherence, and data integration. Most of these systems use an internal state-based model of a clinical practice guideline to derive recommendations but do not provide the user with comprehensive insight into the model.

Objective

Here we present a novel approach based on dynamic guideline visualization that incorporates the individual patient’s current treatment context.

Methods

We derived multiple requirements to be fulfilled by such an enhanced guideline visualization. Using business process and model notation as the representation format for computer-interpretable guidelines, a combination of graph-based representation and logical inferences is adopted for guideline processing. A context-specific guideline visualization is inferred using a business rules engine.

Results

We implemented and piloted an algorithmic approach for guideline interpretation and processing. As a result of this interpretation, a context-specific guideline is derived and visualized. Our implementation can be used as a software library but also provides a representational state transfer interface. Spring, Camunda, and Drools served as the main frameworks for implementation. A formative usability evaluation of a demonstrator tool that uses the visualization yielded high acceptance among clinicians.

Conclusions

The novel guideline processing and visualization concept proved to be technically feasible. The approach addresses known problems of guideline-based clinical decision support systems. Further research is necessary to evaluate the applicability of the approach in specific medical use cases.

Encoding and verification of a computer-interpretable guideline: a case study of pressure-ulcer management

This study examined ways to improve the accuracy of translating clinical practice guidelines (CPGs) into a computer-interpretable guideline (CIG) for pressure-ulcer management using the Shareable Active Guideline Environment (SAGE) guideline model, and aimed to verify the accuracy of the obtained CIG. The study was conducted using the following procedures: selecting CPGs, extracting rules from the selected CPGs, developing a CIG using the SAGE guideline model, and verifying the obtained CIG with test cases using an execution engine. The CIG for pressure-ulcer management was developed based on 38 rules and three algorithms at the semiformal representation level using MS Excel and MS Visio. The CIG was encoded by two Activity Graphs consisting of 115 instances representing algorithms and rules as knowledge elements in the SAGE guideline model. Two errors were found and corrected. Results of the study demonstrated that a CIG representing knowledge on pressure-ulcer management can be effectively developed using commonly available programs and the SAGE guideline model, and that the obtained CIG can be verified with a locally developed execution engine. The CIG developed in the study could contribute to health information management once it is implemented successfully in a clinical decision support system.

Clinical Decision Support: a 25 Year Retrospective and a 25 Year Vision

OBJECTIVE

The objective of this review is to summarize the state of the art of clinical decision support (CDS) circa 1990, review progress in the 25 year interval from that time, and provide a vision of what CDS might look like 25 years hence, or circa 2040.

METHOD

Informal review of the medical literature with iterative review and discussion among the authors to arrive at six axes (data, knowledge, inference, architecture and technology, implementation and integration, and users) to frame the review and discussion of selected barriers and facilitators to the effective use of CDS.

RESULT

In each of the six axes, significant progress has been made. Key advances in structuring and encoding standardized data with an increased availability of data, development of knowledge bases for CDS, and improvement of capabilities to share knowledge artifacts, explosion of methods analyzing and inferring from clinical data, evolution of information technologies and architectures to facilitate the broad application of CDS, improvement of methods to implement CDS and integrate CDS into the clinical workflow, and increasing sophistication of the end-user, all have played a role in improving the effective use of CDS in healthcare delivery.

CONCLUSION

CDS has evolved dramatically over the past 25 years and will likely evolve just as dramatically or more so over the next 25 years. Increasingly, the clinical encounter between a clinician and a patient will be supported by a wide variety of cognitive aides to support diagnosis, treatment, care-coordination, surveillance and prevention, and health maintenance or wellness.

Ontology based intercultural patient practitioner assistive communications from qualitative gap analysis

Purpose

– There is an increasing interest in using information and communication technologies to support health services. But the adoption and development of even basic ICT communications services in many health services is limited, leaving enormous gaps in the broad understanding of its role in health care delivery. The purpose of this paper is to address a specific (intercultural) area of healthcare communications consumer disadvantage; and it examines the potential for ICT exploitation through the lens of a conceptual framework. The opportunity to pursue a new solutions pathway has been amplified in recent times through the development of computer-based ontologies and the resultant knowledge from ontologist activity and consequential research publishing.

Design/methodology/approach

– A specific intercultural area of patient disadvantage arises from variations in meaning and understanding of patient and clinician words, phrases and non-verbal expression. Collection and localization of data concepts, their attributes and individual instances were gathered from an Aboriginal trainee nurse focus group and from a qualitative gap analysis (QGA) of 130 criteria-selected sources of literature. These concepts, their relationships and semantic interpretations populate the computer ontology. The ontology mapping involves two domains, namely, Aboriginal English (AE) and Type II diabetes care guidelines. This is preparatory to development of the Patient Practitioner Assistive Communications (PPAC) system for Aboriginal rural and remote patient primary care.

Findings

– The combined QGA and focus group output reported has served to illustrate the call for three important drivers of change. First, there is no evidence to contradict the hypothesis that patient-practitioner interview encounters for many Australian Aboriginal patients and wellbeing outcomes are unsatisfactory at best. Second, there is a potent need for cultural competence knowledge and practice uptake on the part of health care providers; and third, the key contributory component to determine success or failures within healthcare for ethnic minorities is communication. Communication, however, can only be of value in health care if in practice it supports shared cognition; and mutual cognition is rarely achievable when biopsychosocial and other cultural worldview differences go unchallenged.

Research limitations/implications

– There has been no direct engagement with remote Aboriginal communities in this work to date. The authors have initially been able to rely upon a cohort of both Indigenous and non-Indigenous people with relevant cultural expertise and extended family relationships. Among these advisers are health care practitioners, academics, trainers, Aboriginal education researchers and workshop attendees. It must therefore be acknowledged that as is the case with the QGA, the majority of the concept data is from third parties. The authors have also discovered that urban influences and cultural sensitivities tend to reduce the extent of, and opportunity to, witness AE usage, thereby limiting the ability to capture more examples of code-switching. Although the PPAC system concept is qualitatively well developed, pending future work planned for rural and remote community engagement the authors presently regard the work as mostly allied to a hypothesis on ontology-driven communications. The concept data population of the AE home talk/health talk ontology has not yet reached a quantitative critical mass to justify application design model engineering and real-world testing.

Originality/value

– Computer ontologies avail us of the opportunity to use assistive communications technology applications as a dynamic support system to elevate the pragmatic experience of health care consultations for both patients and practitioners. The human-machine interactive development and use of such applications is required just to keep pace with increasing demand for healthcare and the growing health knowledge transfer environment. In an age when the worldwide web, communications devices and social media avail us of opportunities to confront the barriers described the authors have begun the first construction of a merged schema for two domains that already have a seemingly intractable negative connection. Through the ontology discipline of building syntactically and semantically robust and accessible concepts; explicit conceptual relationships; and annotative context-oriented guidance; the authors are working towards addressing health literacy and wellbeing outcome deficiencies of benefit to the broader communities of disadvantage patients.

Leveraging workflow control patterns in the domain of clinical practice guidelines

BACKGROUND

Clinical practice guidelines (CPGs) include recommendations describing appropriate care for the management of patients with a specific clinical condition. A number of representation languages have been developed to support executable CPGs, with associated authoring/editing tools. Even with tool assistance, authoring of CPG models is a labor-intensive task. We aim at facilitating the early stages of CPG modeling task. In this context, we propose to support the authoring of CPG models based on a set of suitable procedural patterns described in an implementation-independent notation that can be then semi-automatically transformed into one of the alternative executable CPG languages.

METHODS

We have started with the workflow control patterns which have been identified in the fields of workflow systems and business process management. We have analyzed the suitability of these patterns by means of a qualitative analysis of CPG texts. Following our analysis we have implemented a selection of workflow patterns in the Asbru and PROforma CPG languages. As implementation-independent notation for the description of patterns we have chosen BPMN 2.0. Finally, we have developed XSLT transformations to convert the BPMN 2.0 version of the patterns into the Asbru and PROforma languages.

RESULTS

We showed that although a significant number of workflow control patterns are suitable to describe CPG procedural knowledge, not all of them are applicable in the context of CPGs due to their focus on single-patient care. Moreover, CPGs may require additional patterns not included in the set of workflow control patterns. We also showed that nearly all the CPG-suitable patterns can be conveniently implemented in the Asbru and PROforma languages. Finally, we demonstrated that individual patterns can be semi-automatically transformed from a process specification in BPMN 2.0 to executable implementations in these languages.

CONCLUSIONS

We propose a pattern and transformation-based approach for the development of CPG models. Such an approach can form the basis of a valid framework for the authoring of CPG models. The identification of adequate patterns and the implementation of transformations to convert patterns from a process specification into different executable implementations are the first necessary steps for our approach.

Critical factors influencing physicians' intention to use computerized clinical practice guidelines: an integrative model of activity theory and the technology acceptance model

Background

With the widespread use of information communication technologies, computerized clinical practice guidelines are developed and considered as effective decision supporting tools in assisting the processes of clinical activities. However, the development of computerized clinical practice guidelines in Taiwan is still at the early stage and acceptance level among major users (physicians) of computerized clinical practice guidelines is not satisfactory. This study aims to investigate critical factors influencing physicians’ intention to computerized clinical practice guideline use through an integrative model of activity theory and the technology acceptance model.

Methods

The survey methodology was employed to collect data from physicians of the investigated hospitals that have implemented computerized clinical practice guidelines. A total of 505 questionnaires were sent out, with 238 completed copies returned, indicating a valid response rate of 47.1 %. The collected data was then analyzed by structural equation modeling technique.

Results

The results showed that attitudes toward using computerized clinical practice guidelines (γ = 0.451, p < 0.001), organizational support (γ = 0.285, p < 0.001), perceived usefulness of computerized clinical practice guidelines (γ = 0.219, p < 0.05), and social influence (γ = 0.213, p < 0.05) were critical factors influencing physicians’ intention to use computerized clinical practice guidelines, and these factors can explain 68.6 % of the variance in intention to use computerized clinical practice guidelines.

Conclusions

This study confirmed that some subject (human) factors, environment (organization) factors, tool (technology) factors mentioned in the activity theory should be carefully considered when introducing computerized clinical practice guidelines. Managers should pay much attention on those identified factors and provide adequate resources and incentives to help the promotion and use of computerized clinical practice guidelines. Through the appropriate use of computerized clinical practice guidelines, the clinical benefits, particularly in improving quality of care and facilitating the clinical processes, will be realized.

Development of a smartphone application for clinical-guideline-based obesity management

Objectives

The purpose of the study was to develop and evaluate a clinical-guideline-based smartphone application ('app') for obesity management.

Methods

Obesity-related knowledge and functional requirements were extracted from clinical practice guidelines, a literature review, and consultations with experts. The extracted knowledge was used to design obesity-management algorithms, and the functions of the developed app are presented through a use case diagram and activity diagrams. The database and user interface were designed and then an app was developed. The proficiency and efficiency of the algorithm were evaluated using scenarios, while the user interface was assessed using a mobile heuristics evaluation tool, with its usability determined using the Post-Study System Usability Questionnaire.

Results

In total, 131 obesity-related knowledge statements and 11 functions for the app were extracted, and 5 algorithms (comprising 1 main algorithm and 4 subalgorithms) were developed. The database comprised 11 tables and 41 screens. The app was developed using the Android SDK platform 4.0.3, JDK 1.7.0, and Eclipse. The overall proficiency and efficiency scores of the algorithm were 88.0 and 69.1, respectively. In heuristics tests, 57 comments were made, and the mean usability score was 3.47 out of 5. Thirteen usability problems were identified by the heuristics and usability evaluations.

Conclusions

The app developed in this study might be helpful for weight management because it can provide high-quality health information and intervention without spatial or temporal constraints. However, the clinical effectiveness of this app still requires further investigation.

An Ontology for Telemedicine Systems Resiliency to Technological Context Variations in Pervasive Healthcare

Clinical data are crucial for any medical case to study and understand a patient’s condition and to give the patient the best possible treatment. Pervasive healthcare systems apply information and communication technology to enable the usage of ubiquitous clinical data by authorized medical persons. However, quality of clinical data in these applications is, to a large extent, determined by the technological context of the patient. A technological context is characterized by potential technological disruptions that affect optimal functioning of technological resources. The clinical data based on input from these technological resources can therefore have quality degradations. If these degradations are not noticed, the use of this clinical data can lead to wrong treatment decisions, which potentially puts the patient’s safety at risk. This paper presents an ontology that specifies the relation among technological context, quality of clinical data, and patient treatment. The presented ontology provides a formal way to represent the knowledge to specify the effect of technological context variations in the clinical data quality and the impact of the clinical data quality on a patient’s treatment. Accordingly, this ontology is the foundation for a quality of data framework that enables the development of telemedicine systems that are capable of adapting the treatment when the quality of the clinical data degrades, and thus guaranteeing patients’ safety even when technological context varies.

Implementation and evaluation of an Asbru-based decision support system for adjuvant treatment in breast cancer

The domain of cancer treatment is a promising field for the implementation and evaluation of a protocol-based clinical decision support system, because of the algorithmic nature of treatment recommendations. However, many factors can limit such systems' potential to support the decision of clinicians: technical challenges related to the interoperability with existing electronic patient records and clinical challenges related to the inherent complexity of the decisions, often collectively taken by panels of different specialists. In this paper, we evaluate the performances of an Asbru-based decision support system implementing treatment protocols for breast cancer, which accesses data from an oncological electronic patient record. Focusing on the decision on the adjuvant pharmaceutical treatment for patients affected by early invasive breast cancer, we evaluate the matching of the system's recommendations with those issued by the multidisciplinary panel held weekly in a hospital.

Computer-interpretable clinical guidelines: a methodological review

Clinical practice guidelines (CPGs) aim to improve the quality of care, reduce unjustified practice variations and reduce healthcare costs. In order for them to be effective, clinical guidelines need to be integrated with the care flow and provide patient-specific advice when and where needed. Hence, their formalization as computer-interpretable guidelines (CIGs) makes it possible to develop CIG-based decision-support systems (DSSs), which have a better chance of impacting clinician behavior than narrative guidelines. This paper reviews the literature on CIG-related methodologies since the inception of CIGs, while focusing and drawing themes for classifying CIG research from CIG-related publications in the Journal of Biomedical Informatics (JBI). The themes span the entire life-cycle of CIG development and include: knowledge acquisition and specification for improved CIG design, including (1) CIG modeling languages and (2) CIG acquisition and specification methodologies, (3) integration of CIGs with electronic health records (EHRs) and organizational workflow, (4) CIG validation and verification, (5) CIG execution engines and supportive tools, (6) exception handling in CIGs, (7) CIG maintenance, including analyzing clinician's compliance to CIG recommendations and CIG versioning and evolution, and finally (8) CIG sharing. I examine the temporal trends in CIG-related research and discuss additional themes that were not identified in JBI papers, including existing themes such as overcoming implementation barriers, modeling clinical goals, and temporal expressions, as well as futuristic themes, such as patient-centric CIGs and distributed CIGs.

A study of diverse clinical decision support rule authoring environments and requirements for integration

Background

Efficient rule authoring tools are critical to allow clinical Knowledge Engineers (KEs), Software Engineers (SEs), and Subject Matter Experts (SMEs) to convert medical knowledge into machine executable clinical decision support rules. The goal of this analysis was to identify the critical success factors and challenges of a fully functioning Rule Authoring Environment (RAE) in order to define requirements for a scalable, comprehensive tool to manage enterprise level rules.

Methods

The authors evaluated RAEs in active use across Partners Healthcare, including enterprise wide, ambulatory only, and system specific tools, with a focus on rule editors for reminder and medication rules. We conducted meetings with users of these RAEs to discuss their general experience and perceived advantages and limitations of these tools.

Results

While the overall rule authoring process is similar across the 10 separate RAEs, the system capabilities and architecture vary widely. Most current RAEs limit the ability of the clinical decision support (CDS) interventions to be standardized, sharable, interoperable, and extensible. No existing system meets all requirements defined by knowledge management users.

Conclusions

A successful, scalable, integrated rule authoring environment will need to support a number of key requirements and functions in the areas of knowledge representation, metadata, terminology, authoring collaboration, user interface, integration with electronic health record (EHR) systems, testing, and reporting.

Computerization of workflows, guidelines, and care pathways: a review of implementation challenges for process-oriented health information systems

OBJECTIVE

There is a need to integrate the various theoretical frameworks and formalisms for modeling clinical guidelines, workflows, and pathways, in order to move beyond providing support for individual clinical decisions and toward the provision of process-oriented, patient-centered, health information systems (HIS). In this review, we analyze the challenges in developing process-oriented HIS that formally model guidelines, workflows, and care pathways.

METHODS

A qualitative meta-synthesis was performed on studies published in English between 1995 and 2010 that addressed the modeling process and reported the exposition of a new methodology, model, system implementation, or system architecture. Thematic analysis, principal component analysis (PCA) and data visualisation techniques were used to identify and cluster the underlying implementation 'challenge' themes.

RESULTS

One hundred and eight relevant studies were selected for review. Twenty-five underlying 'challenge' themes were identified. These were clustered into 10 distinct groups, from which a conceptual model of the implementation process was developed.

DISCUSSION AND CONCLUSION

We found that the development of systems supporting individual clinical decisions is evolving toward the implementation of adaptable care pathways on the semantic web, incorporating formal, clinical, and organizational ontologies, and the use of workflow management systems. These architectures now need to be implemented and evaluated on a wider scale within clinical settings.

A scalable architecture for incremental specification and maintenance of procedural and declarative clinical decision-support knowledge

Clinical guidelines have been shown to improve the quality of medical care and to reduce its costs. However, most guidelines exist in a free-text representation and, without automation, are not sufficiently accessible to clinicians at the point of care. A prerequisite for automated guideline application is a machine-comprehensible representation of the guidelines. In this study, we designed and implemented a scalable architecture to support medical experts and knowledge engineers in specifying and maintaining the procedural and declarative aspects of clinical guideline knowledge, resulting in a machine comprehensible representation. The new framework significantly extends our previous work on the Digital electronic Guidelines Library (DeGeL) The current study designed and implemented a graphical framework for specification of declarative and procedural clinical knowledge, Gesher. We performed three different experiments to evaluate the functionality and usability of the major aspects of the new framework: Specification of procedural clinical knowledge, specification of declarative clinical knowledge, and exploration of a given clinical guideline. The subjects included clinicians and knowledge engineers (overall, 27 participants). The evaluations indicated high levels of completeness and correctness of the guideline specification process by both the clinicians and the knowledge engineers, although the best results, in the case of declarative-knowledge specification, were achieved by teams including a clinician and a knowledge engineer. The usability scores were high as well, although the clinicians' assessment was significantly lower than the assessment of the knowledge engineers.

Ability of expert physicians to structure clinical guidelines: reality versus perception

RATIONALE, AIMS AND OBJECTIVES

Structuring Textual Clinical Guidelines (GLs) into a formal representation is a necessary prerequisite for supporting their automated application. We had developed a collaborative guideline-structuring methodology that involves expert physicians, clinical editors and knowledge engineers, to produce a machine-comprehensible representation for automated support of evidence-based, guideline-based care. Our goals in the current study were: (1) to investigate the perceptions of the expert physicians and clinical editors as to the relative importance, for the structuring process, of different aspects of the methodology; (2) to assess, for the clinical editors, the inter-correlations among (i) the reported level of understanding of the guideline structuring ontology's (knowledge scheme's) features, (ii) the reported ease of structuring each feature and (iii) the actual objective quality of structuring.

METHODS

A clinical consensus regarding the contents of three guidelines was prepared by an expert in the domain of each guideline. For each guideline, two clinical editors independently structured the guideline into a semi-formal representation, using the Asbru guideline ontology's features. The quality of the resulting structuring was assessed quantitatively. Each expert physician was asked which aspects were most useful for formation of the consensus. Each clinical editor filled questionnaires relating to: (1) the level of understanding of the ontology's features (before the structuring process); (2) the usefulness of various aspects in the structuring process (after the structuring process); (3) the ease of structuring each ontological feature (after the structuring process). Subjective reports were compared with objective quantitative measures of structuring correctness.

RESULTS

Expert physicians considered having medical expertise and understanding the ontological features as the aspects most useful for creation of a consensus. Clinical editors considered understanding the ontological features and the use of the structuring tools as the aspects most useful for structuring guidelines. There was a positive correlation (R = 0.87, P < 0.001) between the reported ease of understanding ontological features and the reported ease of structuring those features. However, there was no significant correlation between the reported level of understanding the features - or the reported ease of structuring by using those features - and the objective quality of the structuring of these features in actual guidelines.

CONCLUSIONS

Aspects considered important for formation of a clinical consensus differ from those for structuring of guidelines. Understanding the features of a structuring ontology is positively correlated with the reported ease of using these features, but neither of these subjective reports correlated with the actual objective quality of the structuring using these features.

A semantically enriched clinical guideline model enabling deployment in heterogeneous healthcare environments

Clinical guidelines are developed to assist healthcare practitioners to make decisions on patient's medical problems, and as such they communicate with external applications to retrieve patient data to initiate medical actions through clinical workflows, and transmit information to alert/reminder systems. The interoperability problems in the healthcare information technology domain prevent wider deployment of clinical guidelines because each deployment requires a tedious custom adaptation phase. In this paper, we provide machine-processable mechanisms that express the semantics of clinical guideline interfaces so that automated processes can be used to access the clinical resources for guideline deployment and execution. To be able to deploy the semantically extended guidelines to healthcare settings semiautomatically, the underlying application's semantics must also be available. We describe how this can be achieved based on two prominent implementation technologies in use in the eHealth domain: integrating healthcare enterprise cross-enterprise document sharing integration profile for discovering and exchanging electronic healthcare records and Web service technology for interacting with the clinical workflows and wireless medical sensor devices. The system described in this paper is realized within the scope of the SAPHIRE Project.

Vaidurya: a multiple-ontology, concept-based, context-sensitive clinical-guideline search engine

We designed and implemented a generic search engine (Vaidurya), as part of our Digital clinical-Guideline Library (DeGeL) framework. Two search methods were implemented in addition to full-text search: (1) concept-based search, which relies on pre-indexing the guidelines in a clinically meaningful fashion, and (2) context-sensitive search, which relies on first semi-structuring the guidelines according to a given ontology, then searching for terms within specific labeled text segments. The Vaidurya engine is fully functional and is used within the DeGeL system. We describe the Vaidurya ontological and algorithmic framework; we also briefly summarize the results of a detailed evaluation in the clinical-guideline domain, demonstrating that both concept-based and context-sensitive ontology-independent search are highly feasible and significantly improve on free text search retrieval performance. We conclude by analyzing the limitations and advantages of the approach, and the steps that we have started to take to extend it based on user feedback.

Computer-based execution of clinical guidelines: a review

PURPOSE

Clinical guidelines are useful tools to standardize and improve health care. The automation of the guideline execution process is a basic step towards its widespread use in medical centres. This paper presents an analysis and a comparison of eight systems that allow the enactment of clinical guidelines in a (semi) automatic fashion.

METHODS

This paper presents a review of the literature (2000-2007) collected from medical databases as well as international conferences in the medical informatics area.

RESULTS

Eight systems containing a guideline execution engine were selected. The language used to represent the guidelines as well as the architecture of these systems were compared. Different aspects have been assessed for each system, such as the integration with external elements or the coordination mechanisms used in the execution of clinical guidelines. Security and terminology issues complement the above study.

CONCLUSIONS

Although these systems could be beneficial for clinicians and patients, it is an ongoing research area, and they are not yet fully implemented and integrated into existing careflow management systems and hence used in daily practice in health care institutions.

Evaluation of an architecture for intelligent query and exploration of time-oriented clinical data

OBJECTIVE

Evaluate KNAVE-II, a knowledge-based framework for visualization, interpretation, and exploration of longitudinal clinical data, clinical concepts and patterns. KNAVE-II mediates queries to a distributed temporal-abstraction architecture (IDAN), which uses a knowledge-based problem-solving method specializing in on-the-fly computation of clinical queries.

METHODS

A two-phase, balanced cross-over study to compare efficiency and satisfaction of a group of clinicians when answering queries of variable complexity about time-oriented clinical data, typical for oncology protocols, using KNAVE-II, versus standard methods: both paper charts and a popular electronic spreadsheet (ESS) in Phase I; an ESS in Phase II. The measurements included the time required to answer and the correctness of answer for each query and each complexity category, and for all queries, assessed versus a predetermined gold standard set by a domain expert. User satisfaction was assessed by the Standard Usability Score (SUS) tool-specific questionnaire and by a "Usability of Tool Comparison" comparative questionnaire developed for this study.

RESULTS

In both evaluations, subjects answered higher-complexity queries significantly faster using KNAVE-II than when using paper charts or an ESS up to a mean of 255 s difference per query versus the ESS for hard queries (p=0.0003) in the second evaluation. Average correctness scores when using KNAVE-II versus paper charts, in the first phase, and the ESS, in the second phase, were significantly higher over all queries. In the second evaluation, 91.6% (110/120) of all of the questions asked within queries of all levels produced correct answers using KNAVE-II, opposed to only 57.5% (69/120) using the ESS (p<0.0001). User satisfaction with KNAVE-II was significantly superior compared to using either a paper chart or the ESS (p=0.006). Clinicians ranked KNAVE-II superior to both paper and the ESS.

CONCLUSIONS

An evaluation of the functionality and usability of KNAVE-II and its supporting knowledge-based temporal-mediation architecture has produced highly encouraging results regarding saving of physician time, enhancement of accuracy of clinical assessment, and user satisfaction.

A comparative evaluation of full-text, concept-based, and context-sensitive search

OBJECTIVES

Study comparatively (1) concept-based search, using documents pre-indexed by a conceptual hierarchy; (2) context-sensitive search, using structured, labeled documents; and (3) traditional full-text search. Hypotheses were: (1) more contexts lead to better retrieval accuracy; and (2) adding concept-based search to the other searches would improve upon their baseline performances.

DESIGN

Use our Vaidurya architecture, for search and retrieval evaluation, of structured documents classified by a conceptual hierarchy, on a clinical guidelines test collection.

MEASUREMENTS

Precision computed at different levels of recall to assess the contribution of the retrieval methods. Comparisons of precisions done with recall set at 0.5, using t-tests.

RESULTS

Performance increased monotonically with the number of query context elements. Adding context-sensitive elements, mean improvement was 11.1% at recall 0.5. With three contexts, mean query precision was 42% +/- 17% (95% confidence interval [CI], 31% to 53%); with two contexts, 32% +/- 13% (95% CI, 27% to 38%); and one context, 20% +/- 9% (95% CI, 15% to 24%). Adding context-based queries to full-text queries monotonically improved precision beyond the 0.4 level of recall. Mean improvement was 4.5% at recall 0.5. Adding concept-based search to full-text search improved precision to 19.4% at recall 0.5.

CONCLUSIONS

The study demonstrated usefulness of concept-based and context-sensitive queries for enhancing the precision of retrieval from a digital library of semi-structured clinical guideline documents. Concept-based searches outperformed free-text queries, especially when baseline precision was low. In general, the more ontological elements used in the query, the greater the resulting precision.

Runtime application of Hybrid-Asbru clinical guidelines

Clinical guidelines are a major tool in improving the quality of medical care. However, to support the automation of guideline-based care, several requirements must be filled, such as specification of the guidelines in a machine-interpretable format and a connection to an Electronic Patient Record (EPR). For several different reasons, it is beneficial to convert free-text guidelines gradually, through several intermediate representations, to a machine-interpretable format. It is also realistic to consider the case when an EPR is unavailable. We propose an innovative approach to the runtime application of intermediate-represented Hybrid-Asbru guidelines, with or without an available EPR. The new approach capitalizes on our extensive work on developing the Digital electronic Guideline Library (DeGeL) framework. The new approach was implemented as the Spock system. For evaluation, three guidelines were specified in an intermediate format and were applied to a set of simulated patient records designed to cover prototypical cases. In all cases, the Spock system produced the expected output, and did not produce an unexpected one. Thus, we have demonstrated the capability of the Spock system to apply guidelines encoded in the Hybrid-Asbru intermediate representation, when an EPR is not available.

Multiple hierarchical classification of free-text clinical guidelines

OBJECTIVE

Manual classification of free-text documents within a predefined hierarchy, commonly required in the medical domain, is highly time consuming task. We present an approach based on supervised learning to automate the classification of clinical guidelines into predefined hierarchical conceptual categories.

METHODS AND MATERIAL

Given a set of hierarchically categorized documents in the training stage the learning algorithm exploits the hierarchical structure of the concepts in order to overcome the low number of training examples. The classification task is thus decomposed into a continuous decision process, unlike searching within a decision tree, which follows the concept hierarchy and makes a single decision at each node on the path, multiple paths can be chosen. Classification is based on applying a similarity function at each concept. Several evaluation measures were used, based on the intended use of the hierarchy. In addition, conservative and aggressive stop-criterion strategies for stopping the search through the concept hierarchy were formulated. An evaluation of the approach, including several training methods and multiple evaluation measures, has been performed using a training set of 1136 guidelines from the National Guideline Clearing House set.

RESULTS

Based on a test collection consisting of 1038 clinical practice guidelines (CPGs) classified along two hierarchies, of roughly 5000 concepts, in which each CPG was classified by a mean of 10 concepts, a variable precision was observed from 44% to 60% depending on the settings of the training methods.

CONCLUSION

These results demonstrate the feasibility of the approach, especially when considering the low ratio of guidelines to classification indices (concepts) in the evaluation data set used here.