Logic Models for Evaluation of Complex Health Information Systems

BACKGROUND

Complex health IT needs to be planned and evaluated.

OBJECTIVES

To propose logic models for the evaluation of complex health IT.

METHODS

Logic models describe input, activities, output, outcome, and impact.

RESULTS AND CONCLUSION

This first example of a logic model for patient portals shows how health IT planning and evaluation may benefit from logic models.

Should App Self-Management Mean Self-Control? A Quantiative Study on App Supported Diabetes Self-Management

Apps have been attested to empower patients regarding disease self-management through numerous studies. However, it is still unclear what factors determine the perception of patients whether an app is a useful tool for this purpose. A multiple regression model that was informed by the Technology Acceptance Model (TAM 2) was tested based on the answers of 235 app users with Diabetes type 1 or 2. The model accounted for 59.2% of the variance of the perceived degree of self-management. Factors belonging to the relevance-usefulness-quality complex as well as factors reflecting the patient's self-control were found to be significant in the model. Patient demographics, i.e. age, gender, app experience and type of Diabetes did not play any significant role. In conclusion, this study raises the question whether apps should be designed to strengthen self-management in the sense of self-control (e.g. own measurements, diary) as opposed to guiding and advice giving.

The Effect of Innovation Capabilities of Health Care Organizations on the Quality of Health Information Technology: Model Development With Cross-sectional Data

Background

Large health organizations often struggle to build complex health information technology (HIT) solutions and are faced with ever-growing pressure to continuously innovate their information systems. Limited research has been conducted that explores the relationship between organizations’ innovative capabilities and HIT quality in the sense of achieving high-quality support for patient care processes.

Objective

The aim of this study is to explain how core constructs of organizational innovation capabilities are linked to HIT quality based on a conceptual sociotechnical model on innovation and quality of HIT, called the IQ_HIT model, to help determine how better information provision in health organizations can be achieved.

Methods

We designed a survey to assess various domains of HIT quality, innovation capabilities of health organizations, and context variables and administered it to hospital chief information officers across Austria, Germany, and Switzerland. Data from 232 hospitals were used to empirically fit the model using partial least squares structural equation modeling to reveal associations and mediating and moderating effects.

Results

The resulting empirical IQ_HIT model reveals several associations between the analyzed constructs, which can be summarized in 2 main insights. First, it illustrates the linkage between the constructs measuring HIT quality by showing that the professionalism of information management explains the degree of HIT workflow support (R²=0.56), which in turn explains the perceived HIT quality (R²=0.53). Second, the model shows that HIT quality was positively influenced by innovation capabilities related to the top management team, the information technology department, and the organization at large. The assessment of the model’s statistical quality criteria indicated valid model specifications, including sufficient convergent and discriminant validity for measuring the latent constructs that underlie the measures of HIT quality and innovation capabilities.

Conclusions

The proposed sociotechnical IQ_HIT model points to the key role of professional information management for HIT workflow support in patient care and perceived HIT quality from the viewpoint of hospital chief information officers. Furthermore, it highlights that organizational innovation capabilities, particularly with respect to the top management team, facilitate HIT quality and suggests that health organizations establish this link by applying professional information management practices. The model may serve to stimulate further scientific work in the field of HIT adoption and diffusion and to provide practical guidance to managers, policy makers, and educators on how to achieve better patient care using HIT.

Professionalism of Information Management in Health Care: Development and Validation of the Construct and Its Measurement

BACKGROUND

Against the background of a steadily increasing degree of digitalization in health care, a professional information management (IM) is required to successfully plan, implement, and evaluate information technology (IT). At its core, IM has to ensure a high quality of health data and health information systems to support patient care.

OBJECTIVES

The goal of the present study was to define what constitutes professional IM as a construct as well as to propose a reliable and valid measurement instrument.

METHODS

To develop and validate the construct of professionalism of information management (PIM) and its measurement, a stepwise approach followed an established procedure from information systems and behavioral research. The procedure included an analysis of the pertaining literature and expert rounds on the construct and the instrument, two consecutive and comprehensive surveys at the national and international level, exploratory and confirmatory factor analyses as well as reliability and validity testing.

RESULTS

Professionalism of information management was developed as a construct consisting of the three dimensions of strategic, tactical, and operational IM as well as of the regularity and cyclical phases of IM procedures as the two elements of professionalism. The PIM instrument operationalized the construct providing items that incorporated IM procedures along the three dimensions and cyclical phases. These procedures had to be evaluated against their degree of regularity in the instrument. The instrument proved to be reliable and valid in two consecutive measurement phases and across three countries.

CONCLUSION

It can be concluded that professionalism of information management is a meaningful construct that can be operationalized in a scientifically rigorous manner. Both science and practice can benefit from these developments in terms of improved self-assessment, benchmarking capabilities, and eventually, obtaining a better understanding of health IT maturity.

It Needs More Than Just User Participation: Combining Perspectives of Clinical Leaders and Chief Information Officers on Determinants of Hospitals' IT Innovativeness

Although user participation may facilitate the realisation of IT innovations, various literature analyses show only minimal to moderate evidence for such effects possibly due to disregard of mediating factors. Against this background, this study examines the extent to which joint intrapreneurship of clinical leaders and IT leaders as well as a distinct innovation culture mediate the effect of user participation on hospitals' IT innovativeness. IT innovativeness was measured by the availability and usability of IT functions and by the perceived 'innovative power' of a hospital. An empirical model was developed and tested with data from 168 clinical leaders and IT leaders who participated pairwise in a survey representing 84 German hospitals. Three parallel mediation analyses indicated that the participation of users could only lead to IT innovativeness if they were accompanied by intrapreneurial leadership on the part of clinical directors and IT leaders and if a pronounced innovation culture prevailed.

Ready for HIT Innovations? Developing a Tool to Assess the Professionalism of Information Management in Hospitals

Current frameworks postulate the success of health IT innovations to be determined by the professionalism of the information management (PIM). Still, empirical knowledge about PIM is scarce up until today. This study seeks to answer three research questions: (1.) How can PIM be measured in a reliable and valid way, (2.) how pronounced is PIM in German hospitals and (3.) do hospital characteristics have an impact on the degree of PIM? Based on the results of an expert workshop and frameworks for information management (IM) items for a PIM inventory were developed and the inventory sent to 1349 chief information officers of German hospitals. A principle component analysis based on the responses of 196 hospitals confirmed the three components that had been proposed by the frameworks: the strategic, the tactical and the operational level. The full inventory implied satisfying reliability and allowed a PIM composite-score to be calculated. The PIM scores for strategic and tactical IM were found to be far lower than for operational IM which hints at strong deficits in these areas. A stepwise regression model indicated that the degree of PIM significantly increased with the size of the hospital, which had been expected and hints the validity of the PIM inventory. This tool offers potentials for hospitals to classify and improve their IM.

Hospital CEOs Need Health IT Knowledge and Trust in CIOs: Insights from a Qualitative Study

BACKGROUND

IT is getting an increasing importance in hospitals. In this context, major IT decisions are often made by CEOs who are not necessarily IT experts.

OBJECTIVES

Therefore, this study aimed at a) exploring different types of IT decision makers at CEO level, b) identifying hypotheses if trust exists between these different types of CEOs and their CIOs and c) building hypotheses on potential consequences regarding risk taking and innovation.

METHODS

To this end, 14 qualitative interviews with German hospital CEOs were conducted to explore the research questions.

RESULTS

The study revealed three major types: IT savvy CEOs, IT enthusiastic CEOs and IT indifferent CEOs. Depending on these types, their relationship with the CIO varied in terms of trust and common language. In case of IT indifferent CEOs, a potential vicious circle of lack of IT knowledge, missing trust, low willingness to take risks and low innovation power could be identified.

CONCLUSION

In order to break of this circle, CEOs seem to need more IT knowledge and/or greater trust in their CIO.

Measuring the Availability of Electronic Patient Data Across the Hospital and Throughout Selected Clinical Workflows

The workflow-oriented dissemination of electronic patient data is a central goal of IT deployment in hospitals. Against this background, the present study examines two research questions: (1.) Are there differences in the availability of electronic patient data (AEPD) between different clinical workflows and data types and (2.) which structural and organizational factors determine AEPD? Based on a Germany wide hospital survey, AEPD was assessed along six clinical workflows. While AEPD was lowest for ward rounds, discharge showed the highest AEPD with pre- and post-surgery processes ranging in between. With regard to the data types analyzed, patient demographics and observation findings obtained the highest AEPD scores. Electrophysiological results, checklists and warnings were less common electronically and received lower AEPD scores. Multiple linear regression analysis resulted in a significant model that explained 34.4% of the variance of AEPD. Large hospitals and those with a professional information management, a high health IT related innovation culture and a nursing informatics officer possess higher AEPD scores and thus have better clinical information logistics mechanisms at their command.

Determinants of Clinical Information Logistics: Tracing Socio-Organisational Factors and Country Differences from the Perspective of Clinical Directors

The establishment of successful clinical information logistics (CIL) within the care processes is one of the main objectives of strategic health IT management in hospitals. While technical realisations in terms of useful, usable and interoperable IT solutions are essential precursors of CIL, there is limited empirical research on what socio-organisational factors underlie an innovation-friendly culture and how they can affect successful information provision. We applied factor analysis on survey data from 403 clinical directors from Germany, Austria and Switzerland and used the dimensions identified to explain the level of CIL with ordered logistic regression analysis. The intensity of collaboration and exchange with the IT department as well as the degree of executive IT leadership showed to be strongly associated with better CIL while personal views and attitudes of clinical directors were not. Analysing country differences revealed the degree of the exchange with the IT department to be significantly lower in German hospitals. This points at a potential strategic lever for German hospital executives to focus on.

Going Mobile: An Empirical Model for Explaining Successful Information Logistics in Ward Rounds

BACKGROUND

Medical ward rounds are critical focal points of inpatient care that call for uniquely flexible solutions to provide clinical information at the bedside. While this fact is undoubted, adoption rates of mobile IT solutions remain rather low.

OBJECTIVES

Our goal was to investigate if and how mobile IT solutions influence successful information provision at the bedside, i.e. clinical information logistics, as well as to shed light at socio-organizational factors that facilitate adoption rates from a user-centered perspective.

METHODS

Survey data were collected from 373 medical and nursing directors of German, Austrian and Swiss hospitals and analyzed using variance-based Structural Equation Modelling (SEM).

RESULTS

The adoption of mobile IT solutions explains large portions of clinical information logistics and is in itself associated with an organizational culture of innovation and end user participation.

CONCLUSION

Results should encourage decision makers to understand mobility as a core constituent of information logistics and thus to promote close end-user participation as well as to work towards building a culture of innovation.

Exploring Innovation Capabilities of Hospital CIOs: An Empirical Assessment

Hospital CIOs play a central role in the adoption of innovative health IT. Until now, it remained unclear which particular conditions constitute their capability to innovate in terms of intrapersonal as well as organisational factors. An inventory of 20 items was developed to capture these conditions and examined by analysing data obtained from 164 German hospital CIOs. Principal component analysis resulted in three internally consistent components that constitute large portions of the CIOs innovation capability: organisational innovation culture, entrepreneurship personality and openness towards users. Results were used to build composite indicators that allow further evaluations.

Implementing a Data Management Platform for Longitudinal Health Research

Health IT adoption research is rooted in Rogers' Diffusion of Innovation theory, which is based on longitudinal analyses. However, many studies in this field use cross-sectional designs. The aim of this study therefore was to design and implement a system to (i) consolidate survey data sets originating from different years (ii) integrate additional secondary data and (iii) query and statistically analyse these longitudinal data. Our system design comprises a 5-tier-architecture that embraces tiers for data capture, data representation, logics, presentation and integration. In order to historicize data properly and to separate data storage from data analytics a data vault schema was implemented. This approach allows the flexible integration of heterogeneous data sets and the selection of comparable items. Data analysis is prepared by compiling data in data marts and performed by R and related tools. IT Report Healthcare data from 2011, 2013 and 2017 could be loaded, analysed and combined with secondary longitudinal data.

Antecedents of CIOs' Innovation Capability in Hospitals: Results of an Empirical Study

CIOs' innovation capability is regarded as a precondition of successful HIT adoption in hospitals. Based on the data of 142 CIOs, this study aimed at identifying antecedents of perceived innovation capability. Eight features describing the status quo of the hospital IT management (e.g. use of IT governance frameworks), four features of the hospital structure (e.g. functional diversification) and four CIO characteristics (e.g. duration of employment) were tested as potential antecedents in an exploratory stepwise regression approach. Perceived innovation capability in its entirety and its three sub-dimensions served as criterion. The results show that CIOs' perceived innovation capability could be explained significantly (R²=0.34) and exclusively by facts that described the degree of formalism and structure of IT management in a hospital, e.g. intensive and formalised strategic communication, the existence of an IT strategy and the use of IT governance frameworks. Breaking down innovation capability into its constituents revealed that "innovative organisational culture" contributed to a large extent (R²=0.26) to the overall result sharing several predictors. In contrast, "intrapreneurial personality" (R²=0.11) and "openness towards users" (R²=0.18) could be predicted less well. These results hint at the relationship between working in a well-structured, formalised and strategy oriented environment and the overall feeling of being capable to promote IT innovation.

Understanding latent structures of clinical information logistics: A bottom-up approach for model building and validating the workflow composite score

BACKGROUND AND PURPOSE

Clinical information logistics is a construct that aims to describe and explain various phenomena of information provision to drive clinical processes. It can be measured by the workflow composite score, an aggregated indicator of the degree of IT support in clinical processes. This study primarily aimed to investigate the yet unknown empirical patterns constituting this construct. The second goal was to derive a data-driven weighting scheme for the constituents of the workflow composite score and to contrast this scheme with a literature based, top-down procedure. This approach should finally test the validity and robustness of the workflow composite score.

METHODS

Based on secondary data from 183 German hospitals, a tiered factor analytic approach (confirmatory and subsequent exploratory factor analysis) was pursued. A weighting scheme, which was based on factor loadings obtained in the analyses, was put into practice.

RESULTS

We were able to identify five statistically significant factors of clinical information logistics that accounted for 63% of the overall variance. These factors were "flow of data and information", "mobility", "clinical decision support and patient safety", "electronic patient record" and "integration and distribution". The system of weights derived from the factor loadings resulted in values for the workflow composite score that differed only slightly from the score values that had been previously published based on a top-down approach.

CONCLUSION

Our findings give insight into the internal composition of clinical information logistics both in terms of factors and weights. They also allowed us to propose a coherent model of clinical information logistics from a technical perspective that joins empirical findings with theoretical knowledge. Despite the new scheme of weights applied to the calculation of the workflow composite score, the score behaved robustly, which is yet another hint of its validity and therefore its usefulness.

Investigating the roots of successful IT adoption processes - an empirical study exploring the shared awareness-knowledge of Directors of Nursing and Chief Information Officers

Background

The majority of health IT adoption research focuses on the later stages of the IT adoption process: namely on the implementation phase. The first stage, however, which is defined as the knowledge-stage, remains widely unobserved. Following Rogers’ Diffusion of Innovation Theory (DOI) this paper presents a research framework to examine the possible lack of shared IT awareness-knowledge, i.e. an information gradient, of two crucial stakeholders, the Chief Information Officer (CIO) and the Director of Nursing (DoN). This study shall answer the following research questions: (1.) Does this gradient exist? (2.) Which direction does it have? (3.) Are certain health IT (HIT) attributes associated with a potential gradient? (4.) Which determinants of diffusion go along with this gradient?

Method

Results of two surveys that focused on the topic “IT support of clinical workflows” from the viewpoint of CIOs and DoNs with corresponding datasets from 75 hospitals were used in a secondary data analysis. The gradient was operationalised by measuring the disagreement of CIOs and DoNs on the availability and implementation status of 29 IT functions. HIT attributes tested were relevance and market penetration of the IT functions, determinants of diffusion were inter-professional leadership and IT service density.

Results

The analysis revealed a significant disagreement on the availability of 9 out of 29 HIT functions. In 23 HIT functions, the CIOs reported a higher implementation status than the DoNs, which pointed to a trend for a unidirectional gradient. The disagreement was significantly lower when the relevance of the IT function was high. Both determinants of diffusion correlated significantly negative with the degree of disagreement.

Conclusion

This is the first study to empirically examine shared awareness-knowledge of two IT-stakeholders that are crucial for triggering IT adoption on the frontline level in hospitals. It could be shown that a gradient and thus a lack of shared awareness-knowledge existed and was associated with certain factors. In conclusion, hospitals should implement improved cooperation between IT staff and clinicians and IT service density when establishing the prerequisites for successful IT adoption processes.

An Iterative Methodology for Developing National Recommendations for Nursing Informatics Curricula

The increasing importance of IT in nursing requires educational measures to support its meaningful application. However, many countries do not yet have national recommendations for nursing informatics competencies. We thus developed an iterative triple methodology to yield validated and country specific recommendations for informatics core competencies in nursing. We identified relevant competencies from national sources (step 1), matched and enriched these with input from the international literature (step 2) and fed the resulting 24 core competencies into a survey (120 invited experts from which 87 responded) and two focus group sessions with a total of 48 experts (steps 3a/3b). The subsequent focus group sessions confirmed and expanded the findings. As a result, we were able to define role specific informatics core competencies for three countries.

Developing a Workflow Composite Score to Measure Clinical Information Logistics. A Top-down Approach

BACKGROUND

Availability and usage of individual IT applications have been studied intensively in the past years. Recently, IT support of clinical processes is attaining increasing attention. The underlying construct that describes the IT support of clinical workflows is clinical information logistics. This construct needs to be better understood, operationalised and measured.

OBJECTIVES

It is therefore the aim of this study to propose and develop a workflow composite score (WCS) for measuring clinical information logistics and to examine its quality based on reliability and validity analyses.

METHODS

We largely followed the procedural model of MacKenzie and colleagues (2011) for defining and conceptualising the construct domain, for developing the measurement instrument, assessing the content validity, pretesting the instrument, specifying the model, capturing the data and computing the WCS and testing the reliability and validity.

RESULTS

Clinical information logistics was decomposed into the descriptors data and information, function, integration and distribution, which embraced the framework validated by an analysis of the international literature. This framework was refined selecting representative clinical processes. We chose ward rounds, pre- and post-surgery processes and discharge as sample processes that served as concrete instances for the measurements. They are sufficiently complex, represent core clinical processes and involve different professions, departments and settings. The score was computed on the basis of data from 183 hospitals of different size, ownership, location and teaching status. Testing the reliability and validity yielded encouraging results: the reliability was high with r(split-half) = 0.89, the WCS discriminated between groups; the WCS correlated significantly and moderately with two EHR models and the WCS received good evaluation results by a sample of chief information officers (n = 67). These findings suggest the further utilisation of the WCS.

CONCLUSION

As the WCS does not assume ideal workflows as a gold standard but measures IT support of clinical workflows according to validated descriptors a high portability of the WCS to other hospitals in other countries is very likely. The WCS will contribute to a better understanding of the construct clinical information logistics.

IT-benchmarking of clinical workflows: concept, implementation, and evaluation

Due to the emerging evidence of health IT as opportunity and risk for clinical workflows, health IT must undergo a continuous measurement of its efficacy and efficiency. IT-benchmarks are a proven means for providing this information. The aim of this study was to enhance the methodology of an existing benchmarking procedure by including, in particular, new indicators of clinical workflows and by proposing new types of visualisation. Drawing on the concept of information logistics, we propose four workflow descriptors that were applied to four clinical processes. General and specific indicators were derived from these descriptors and processes. 199 chief information officers (CIOs) took part in the benchmarking. These hospitals were assigned to reference groups of a similar size and ownership from a total of 259 hospitals. Stepwise and comprehensive feedback was given to the CIOs. Most participants who evaluated the benchmark rated the procedure as very good, good, or rather good (98.4%). Benchmark information was used by CIOs for getting a general overview, advancing IT, preparing negotiations with board members, and arguing for a new IT project.

Characteristics of German hospitals adopting health IT systems - results from an empirical study

Hospital characteristics that facilitate IT adoption have been described by the literature extensively, however with controversial results. The aim of this study therefore is to draw a set of the most important variables from previous studies and include them in a combined analysis for testing their contribution as single factors and their interactions. Total number of IT systems installed and number of clinical IT systems in the hospital were used as criterion variables. Data from a national survey of German hospitals served as basis. Based on a stepwise multiple regression analysis four variables were identified to significantly explain the degree of IT adoption (60% explained variance): 1) hospital size, 2) IT department, 3) reference customer and 4) ownership (private vs. public). Our results replicate previous findings with regard to hospital size and ownership. In addition our study emphasizes the importance of a reliable internal structure for IT projects (existence of an IT department) and the culture of testing and installing most recent IT products (being a reference customer). None of the interactions between factors was significant.