Crossing the Quality Chasm: The Role of Information Technology Departments

Validation of an Intensive Care Unit Data Mart for Research and Quality Improvement

Data derived from the electronic health record (EHR) is frequently extracted using undefined approaches that may affect the accuracy of collected variables. Further, efforts to assess data accuracy often suffer from limited collaboration between clinicians and data analysts who perform the extraction. In this manuscript, we describe the methodology behind creation of a structured, rigorously derived intensive care unit (ICU) data mart based on data automatically and routinely derived from the EHR. This ICU data mart includes high-quality data elements commonly used for quality improvement and research purposes. These data elements were identified by physicians working closely with data analysts to iteratively develop and refine algorithmic definitions for complex outcomes and risk factors. We contend that this methodology can be reproduced and applied across other institution or to other clinical domains to create high quality data marts, inclusive of complex outcomes data.

Content and Face Validation of a Novel, Interactive Nutrition Specific Physical Exam Competency Tool (INSPECT) to Evaluate Registered Dietitians' Competence: A Delphi Consensus from the United States

The nutrition-focused physical examination (NFPE) is an integral component of nutrition assessment performed by registered dietitian nutritionists (RDNs) to determine signs of malnutrition and other nutrition-related complications. Increased use of this essential skill among RDNs and the transformation of dietetics education to a competency-based model in the near future calls for appropriately validated tools to measure RDNs' NFPE competence. To fill the need for a validated competency tool, this study developed an Interactive Nutrition-Specific Physical Exam Competency Tool (INSPECT) utilizing the initial 70 items identified in the first phase of the study. The second phase of this study aimed to test the preliminary version of the INSPECT for content and face validity. An expert panel of 17 members provided consensus recommendations through the Delphi process. Internal consistency of the consensus was measured with Cronbach's alpha (α) and α of ≥0.70 was defined as acceptable a priori. Inter-rater agreement among the expert panel was determined using the intraclass correlation coefficient (ICC) and an a priori ICC of 0.75 to 0.9 was established as good and >0.9 as excellent agreement. The results showed acceptable face validity (α = 0.71) and excellent content validity for the INSPECT, with an internal consistency of α = 0.97 in the first round and α = 0.96 in the second round. The inter-rater agreement was also excellent with ICC = 0.95 for each of the Delphi rounds. A total of 52 items were retained from the preliminary version of the INSPECT. Open feedback from the experts allowed for the consolidation of 11 similar items for better scoring and evaluation and thus, a total of 41 items were included in the final version of the INSPECT. The final version of the INSPECT is currently being studied in real-life, multi-site clinical settings among practicing RDNs to examine construct validity, reliability, and item-level psychometric properties. Ultimately, the validated INSPECT will be available for the competency evaluation of RDNs practicing in clinical settings.

Expert appraisal of criteria for assessing gaming disorder: an international Delphi study

BACKGROUND AND AIMS

Following the recognition of 'internet gaming disorder' (IGD) as a condition requiring further study by the DSM-5, 'gaming disorder' (GD) was officially included as a diagnostic entity by the World Health Organization (WHO) in the 11th revision of the International Classification of Diseases (ICD-11). However, the proposed diagnostic criteria for gaming disorder remain the subject of debate, and there has been no systematic attempt to integrate the views of different groups of experts. To achieve a more systematic agreement on this new disorder, this study employed the Delphi expert consensus method to obtain expert agreement on the diagnostic validity, clinical utility and prognostic value of the DSM-5 criteria and ICD-11 clinical guidelines for GD.

METHODS

A total of 29 international experts with clinical and/or research experience in GD completed three iterative rounds of a Delphi survey. Experts rated proposed criteria in progressive rounds until a pre-determined level of agreement was achieved.

RESULTS

For DSM-5 IGD criteria, there was an agreement both that a subset had high diagnostic validity, clinical utility and prognostic value and that some (e.g. tolerance, deception) had low diagnostic validity, clinical utility and prognostic value. Crucially, some DSM-5 criteria (e.g. escapism/mood regulation, tolerance) were regarded as incapable of distinguishing between problematic and non-problematic gaming. In contrast, ICD-11 diagnostic guidelines for GD (except for the criterion relating to diminished non-gaming interests) were judged as presenting high diagnostic validity, clinical utility and prognostic value.

CONCLUSIONS

This Delphi survey provides a foundation for identifying the most diagnostically valid and clinically useful criteria for GD. There was expert agreement that some DSM-5 criteria were not clinically relevant and may pathologize non-problematic patterns of gaming, whereas ICD-11 diagnostic guidelines are likely to diagnose GD adequately and avoid pathologizing.

Towards reporting guidelines of research using whole-body vibration as training or treatment regimen in human subjects-A Delphi consensus study

BACKGROUND

Whole-body vibration (WBV) is a method utilizing vibrating platforms to expose individuals to mechanical vibration. In its various applications, it has been linked to improved muscular, skeletal, metabolic, or cognitive functioning, quality of life, and physiological parameters such as blood pressure. Most evidence concerning WBV is inconclusive and meta-analytical reviews may not readily produce insights since the research has a risk of misunderstandings of vibration parameters and incomplete reporting occurs. This study aims at laying an empirical foundation for reporting guidelines for human WBV studies to improve the quality of reporting and the currently limited comparability between studies.

METHOD

The Delphi methodology is employed to exploit the integrated knowledge of WBV experts to distil the specific aspects of WBV methodology that should be included in such guidelines. Over three rounds of completing online questionnaires, the expert panel (round 1/2/3: 51/40/37 experts respectively from 17 countries with an average of 19.4 years of WBV research experience) rated candidate items.

RESULTS

A 40-item list was established based on the ratings of the individual items from the expert panel with a large final consensus (94.6%).

CONCLUSION

The final consensus indicates comprehensiveness and valuableness of the list. The results are in line with previous guidelines but expand these extensively. The present results may therefore serve as a foundation for updated guidelines for reporting human WBV studies in order to improve the quality of reporting of WBV studies, improve comparability of studies and facilitate the development of WBV study designs.

Identifying current and emerging resources and tools utilized for detection, prediction, and visualization of viral zoonotic disease clusters: a Delphi study

Zoonotic disease surveillance presents a substantial problem in the management of public health. Globally, zoonoses have the potential to spread and negatively impact population health economic growth, and security. This research was conducted to investigate the current data sources, analytical methods, and limitations for cluster detection and prediction with particular interest in emerging bioinformatics tools and resources to inform the development of zoonotic surveillance spatial decision support systems. We recruited 10 local health personnel to participate in a Delphi study. Participants agreed cluster detection is a priority, though mathematical modeling methods and bioinformatics resources are not commonly used toward this endeavor. However, participants indicated a desire to utilize preventative measures. We identified many limitations for identifying clusters including software availability, appropriateness, training, and usage of emerging genetic data. Future decision support system development should focus on state health personnel priorities and tasks to better utilize emerging developments and available data.

Innovating From Within: A Process Model for User-Centered Digital Development in Academic Medical Centers

BACKGROUND

Design thinking and human-centered design approaches have become increasingly common in health care literature, particularly in relation to health information technology (HIT), as a pathway toward the development of usable, diffusible tools and processes. There is a need in academic medical centers tasked with digital innovation for a comprehensive process model to guide development that incorporates current industry trends, including design thinking and lean and agile approaches to digital development.

OBJECTIVE

This study aims to describe the foundations and phases of our model for user-centered HIT development.

METHODS

Based on our experience, we established an integrated approach and rigorous process for HIT development that leverages design thinking and lean and agile strategies in a pragmatic way while preserving methodological integrity in support of academic research goals.

RESULTS

A four-phased pragmatic process model was developed for user-centered digital development in HIT.

CONCLUSIONS

The model for user-centered HIT development that we developed is the culmination of diverse innovation projects and represents a multiphased, high-fidelity process for making more creative, flexible, efficient, and effective tools. This model is a critical step in building a rigorous approach to HIT design that incorporates a multidisciplinary, pragmatic perspective combined with academic research practices and state-of-the-art approaches to digital product development to meet the unique needs of health care.

Finding consensus on frailty assessment in acute care through Delphi method

OBJECTIVE

We seek to address gaps in knowledge and agreement around optimal frailty assessment in the acute medical care setting. Frailty is a common term describing older persons who are at increased risk of developing multimorbidity, disability, institutionalisation and death. Consensus has not been reached on the practical implementation of this concept to assess clinically and manage older persons in the acute care setting.

DESIGN

Modified Delphi, via electronic questionnaire. Questions included ranking items that best recognise frailty, optimal timing, location and contextual elements of a successful tool. Intraclass correlation coefficients for overall levels of agreement, with consensus and stability tested by 2-way ANOVA with absolute agreement and Fisher's exact test.

PARTICIPANTS

A panel of national experts (academics, front-line clinicians and specialist charities) were invited to electronic correspondence.

RESULTS

Variables reflecting accumulated deficit and high resource usage were perceived by participants as the most useful indicators of frailty in the acute care setting. The Acute Medical Unit and Care of the older Persons Ward were perceived as optimum settings for frailty assessment. 'Clinically meaningful and relevant', 'simple (easy to use)' and 'accessible by multidisciplinary team' were perceived as characteristics of a successful frailty assessment tool in the acute care setting. No agreement was reached on optimal timing, number of variables and organisational structures.

CONCLUSIONS

This study is a first step in developing consensus for a clinically relevant frailty assessment model for the acute care setting, providing content validation and illuminating contextual requirements. Testing on clinical data sets is a research priority.

Delphi survey to identify topics to be addressed at the initial follow-up consultation after oesophageal cancer surgery

Background

There is no consensus among patients and healthcare professionals (HCPs) on the topics that need to be addressed after oesophageal cancer surgery. The aim of this study was to identify these topics, using a two-round Delphi survey.

Methods

In round 1, patients and HCPs (surgeons, dieticians, nurses) were invited to rate the importance of 49 topics. The proportion of panellists that considered a topic to be of low, moderate or high importance was then calculated for each of these two groups. Based on these proportions and the i.q.r., topics were categorized as: ‘consensus to be included’, ‘consensus to be excluded’ and ‘no consensus’. Only topics in the first category were included in the second round. In round 2, panellists were provided with individual and group feedback. To be included in the final list, topics had to meet criteria for consensus and stability.

Results

There were 108 patients and 77 HCPs in the round 2 analyses. In general, patients and HCPs considered the same topics important. The final list included 23 topics and revealed that it was most important to address: cancer removed/lymph nodes, the new oesophagus, eating and drinking, surgery, alarming new complaints and the recovery period.

Conclusion

The study provides surgeons with a list of topics selected by patients and HCPs that may be addressed systematically at the initial follow-up consultation after oesophageal cancer surgery.

Building on the EGIPPS performance assessment: the multipolar framework as a heuristic to tackle the complexity of performance of public service oriented health care organisations

Background

Performance of health care systems is a key concern of policy makers and health service managers all over the world. It is also a major challenge, given its multidimensional nature that easily leads to conceptual and methodological confusion. This is reflected by a scarcity of models that comprehensively analyse health system performance.

Discussion

In health, one of the most comprehensive performance frameworks was developed by the team of Leggat and Sicotte. Their framework integrates 4 key organisational functions (goal attainment, production, adaptation to the environment, and values and culture) and the tensions between these functions.

We modified this framework to better fit the assessment of the performance of health organisations in the public service domain and propose an analytical strategy that takes it into the social complexity of health organisations. The resulting multipolar performance framework (MPF) is a meta-framework that facilitates the analysis of the relations and interactions between the multiple actors that influence the performance of health organisations.

Summary

Using the MPF in a dynamic reiterative mode not only helps managers to identify the bottlenecks that hamper performance, but also the unintended effects and feedback loops that emerge. Similarly, it helps policymakers and programme managers at central level to better anticipate the potential results and side effects of and required conditions for health policies and programmes and to steer their implementation accordingly.

Development of a clinical data warehouse from an intensive care clinical information system

There are relatively few institutions that have developed clinical data warehouses, containing patient data from the point of care. Because of the various care practices, data types and definitions, and the perceived incompleteness of clinical information systems, the development of a clinical data warehouse is a challenge. In order to deal with managerial and clinical information needs, as well as educational and research aims that are important in the setting of a university hospital, Erasmus Medical Center Rotterdam, The Netherlands, developed a data warehouse incrementally. In this paper we report on the in-house development of an integral part of the data warehouse specifically for the intensive care units (ICU-DWH). It was modeled using Atos Origin Metadata Frame method. The paper describes the methodology, the development process and the content of the ICU-DWH, and discusses the need for (clinical) data warehouses in intensive care.

The role of information technology in translating educational interventions into practice: an analysis using the PRECEDE/PROCEED model

OBJECTIVE

The evidence base for information technology (IT) has been criticized, especially with the current emphasis on translational science. The purpose of this paper is to present an analysis of the role of IT in the implementation of a geriatric education and quality improvement (QI) intervention.

DESIGN

A mixed-method three-group comparative design was used. The PRECEDE/PROCEED implementation model was used to qualitatively identify key factors in the implementation process. These results were further explored in a quantitative analysis.

METHOD

Thirty-three primary care clinics at three institutions (Intermountain Healthcare, VA Salt Lake City Health Care System, and University of Utah) participated. The program consisted of an onsite, didactic session, QI planning and 6 months of intense implementation support.

RESULTS

Completion rate was 82% with an average improvement rate of 21%. Important predisposing factors for success included an established electronic record and a culture of quality. The reinforcing and enabling factors included free continuing medical education credits, feedback, IT access, and flexible support. The relationship between IT and QI emerged as a central factor. Quantitative analysis found significant differences between institutions for pre-post changes even after the number and category of implementation strategies had been controlled for.

CONCLUSIONS

The analysis illustrates the complex dependence between IT interventions, institutional characteristics, and implementation practices. Access to IT tools and data by individual clinicians may be a key factor for the success of QI projects. Institutions vary widely in the degree of access to IT tools and support. This article suggests that more attention be paid to the QI and IT department relationship.

Quality outcomes management: Veterans Affairs case study

During the last decade, the Department of Veterans Affairs (VA) has made major strides in enhancing quality of medical, surgical, and mental health care for veterans. These improvements have been achieved through the will and commitment of VA leadership and by changes in the administrative structure, such as through the creation of Veteran Integrated Service Networks and patient care service lines, the use of state-of-the-art technology for electronic health records, implementation of high-value preventative and chronic disease management performance measures, and the ability to track their effectiveness. Parallel with these changes, the quality of mental heath care in the VA has also improved, as have mental health education and research.

A cognitive task analysis of information management strategies in a computerized provider order entry environment

OBJECTIVE

Computerized Provider Order Entry (CPOE) with electronic documentation, and computerized decision support dramatically changes the information environment of the practicing clinician. Prior work patterns based on paper, verbal exchange, and manual methods are replaced with automated, computerized, and potentially less flexible systems. The objective of this study is to explore the information management strategies that clinicians use in the process of adapting to a CPOE system using cognitive task analysis techniques.

DESIGN

Observation and semi-structured interviews were conducted with 88 primary-care clinicians at 10 Veterans Administration Medical Centers.

MEASUREMENTS

Interviews were taped, transcribed, and extensively analyzed to identify key information management goals, strategies, and tasks. Tasks were aggregated into groups, common components across tasks were clarified, and underlying goals and strategies identified.

RESULTS

Nearly half of the identified tasks were not fully supported by the available technology. Six core components of tasks were identified. Four meta-cognitive information management goals emerged: 1) Relevance Screening; 2) Ensuring Accuracy; 3) Minimizing memory load; and 4) Negotiating Responsibility. Strategies used to support these goals are presented.

CONCLUSION

Users develop a wide array of information management strategies that allow them to successfully adapt to new technology. Supporting the ability of users to develop adaptive strategies to support meta-cognitive goals is a key component of a successful system.