Psychometric examination of instruments using aggregated data

The Impact of Selecting Specific Cohorts for Benchmarking and Interpretation of Emergency Department Patient Satisfaction Scores

OBJECTIVES

Emergency departments (EDs) patient satisfaction metrics are highly valued by hospitals, health systems, and payers, yet these metrics are challenging to analyze and interpret. Accurate interpretation involves selection of the most appropriate peer group for benchmark comparisons. We hypothesized that the selection of different benchmark peer groups would yield different interpretations of Press Ganey (PG) patient satisfaction scores.

METHODS

Emergency department PG summary ratings of "doctors section" and "likelihood-to-recommend" raw scores and corresponding percentiles were derived for three benchmark peer groups from three academic years (2016, 2017, and 2018). The three benchmarks are: 1) the PG Large database; 2) the PG University HealthSystem Consortium (UHC) database; and 3) the Academy of Administrators in Academic Emergency Medicine (AAAEM) database, which is composed only of EDs from academic health centers with emergency medicine residency training programs. Raw scores were converted to percentile ranks for each distribution and then compared using Welch's ANOVA and Games-Howell pairwise comparisons.

RESULTS

For both patient satisfaction raw scores evaluated, the AAAEM database was noted to have significantly higher percentile ranks when compared to the PG Large and PG UHC databases. These results were consistent for all three time frames assessed.

CONCLUSIONS

Benchmarking with different peer groups provides different results, with similar patient satisfaction raw scores resulting in higher percentile ranks using the AAAEM database compared to the two PG databases. The AAAEM database should be considered the most appropriate peer group for benchmarking academic EDs.

Collaboration processes and perceived effectiveness of integrated care projects in primary care: a longitudinal mixed-methods study

Background

Collaborative partnerships are considered an essential strategy for integrating local disjointed health and social services. Currently, little evidence is available on how integrated care arrangements between professionals and organisations are achieved through the evolution of collaboration processes over time. The first aim was to develop a typology of integrated care projects (ICPs) based on the final degree of integration as perceived by multiple stakeholders. The second aim was to study how types of integration differ in changes of collaboration processes over time and final perceived effectiveness.

Methods

A longitudinal mixed-methods study design based on two data sources (surveys and interviews) was used to identify the perceived degree of integration and patterns in collaboration among 42 ICPs in primary care in The Netherlands. We used cluster analysis to identify distinct subgroups of ICPs based on the final perceived degree of integration from a professional, organisational and system perspective. With the use of ANOVAs, the subgroups were contrasted based on: 1) changes in collaboration processes over time (shared ambition, interests and mutual gains, relationship dynamics, organisational dynamics and process management) and 2) final perceived effectiveness (i.e. rated success) at the professional, organisational and system levels.

Results

The ICPs were classified into three subgroups with: ‘United Integration Perspectives (UIP)’, ‘Disunited Integration Perspectives (DIP)’ and ‘Professional-oriented Integration Perspectives (PIP)’. ICPs within the UIP subgroup made the strongest increase in trust-based (mutual gains and relationship dynamics) as well as control-based (organisational dynamics and process management) collaboration processes and had the highest overall effectiveness rates. On the other hand, ICPs with the DIP subgroup decreased on collaboration processes and had the lowest overall effectiveness rates. ICPs within the PIP subgroup increased in control-based collaboration processes (organisational dynamics and process management) and had the highest effectiveness rates at the professional level.

Conclusions

The differences across the three subgroups in terms of the development of collaboration processes and the final perceived effectiveness provide evidence that united stakeholders’ perspectives are achieved through a constructive collaboration process over time. Disunited perspectives at the professional, organisation and system levels can be aligned by both trust-based and control-based collaboration processes.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-015-1125-4) contains supplementary material, which is available to authorized users.

Exploring the success of an integrated primary care partnership: a longitudinal study of collaboration processes

BACKGROUND

Forming partnerships is a prominent strategy used to promote integrated service delivery across health and social service systems. Evidence about the collaboration process upon which partnerships evolve has rarely been addressed in an integrated-care setting. This study explores the longitudinal relationship of the collaboration process and the influence on the final perceived success of a partnership in such a setting. The collaboration process through which partnerships evolve is based on a conceptual framework which identifies five themes: shared ambition, interests and mutual gains, relationship dynamics, organisational dynamics and process management.

METHODS

Fifty-nine out of 69 partnerships from a national programme in the Netherlands participated in this survey study. At baseline, 338 steering committee members responded, and they returned 320 questionnaires at follow-up. Multiple-regression-analyses were conducted to explore the relationship between the baseline as well as the change in the collaboration process and the final success of the partnerships.

RESULTS

Mutual gains and process management were the most significant baseline predictors for the final success of the partnership. A positive change in the relationship dynamics had a significant effect on the final success of a partnership.

CONCLUSIONS

Insight into the collaboration process of integrated primary care partnerships offers a potentially powerful way of predicting their success. Our findings underscore the importance of monitoring the collaboration process during the development of the partnerships in order to achieve their full collaborative advantage.

Exploring the relationship between synergy and partnership functioning factors in health promotion partnerships

Intersectoral partnerships have been identified as a useful mechanism for addressing the health challenges that face society. In theory, partnerships achieve synergistic outcomes that amount to more than can be achieved by individual partners working on their own. This study aimed to identify key factors that influence health promotion partnership synergy. Data were collected from 337 partners in 40 health promotion partnerships using a postal survey. The questionnaire incorporated a number of multidimensional scales designed to assess the contribution of factors that influence partnership synergy. New validated scales were developed for synergy, trust, mistrust and power. Pearson's correlations and multiple regression analysis were used to identify the significance of each factor to partnership synergy. Trust, leadership and efficiency were shown to be the most important predictors of partnership synergy. Synergy is predicated on trust and leadership. Trust-building mechanisms need to be built into the partnership forming stage and this trust needs to be sustained throughout the collaborative process. We need to develop systems where the best leaders are put forward for intersectoral partnerships. This should be consistent across all sectors and organizations.

A multilevel confirmatory factor analysis of the Practice Environment Scale: a case study

BACKGROUND

Practice Environment Scale (PES) data are collected from RNs in nursing units in hospitals that are members of the National Database of Nursing Quality Indicators (NDNQI). Patient and RN information are collected to aid in quality improvement and research at the nursing unit level. The data were collected from the individual RN, but items are worded so that analyses can be conducted at the individual, unit, or hospital level. There is a need to examine the validity of the PES at both the individual and the unit level.

OBJECTIVE

To describe multilevel confirmatory factor analysis via a case study for investigating the validity of the PES, a measure of the nursing practice environment.

APPROACH

The PES was administered to 72,889 RNs from 4,783 nursing units (16 unit types; e.g., critical care and obstetric) in 2007. The PES has 31 items in five different domains. A multilevel confirmatory factor analytic model was fit with a structure on the basis of the five domains. From this model, an estimate was sought between unit loadings and within unit loadings to investigate factorial, convergent, and discriminant validity at both the unit and the RN levels. To investigate criterion-related validity, the five PES domains were correlated with the seven job enjoyment items adapted from the National Database of Nursing Quality Indicators at the unit and RN levels (also using a multilevel model).

RESULTS

The multilevel factor analysis provides evidence of factorial, convergent, discriminant, and criterion-related validity at both the unit and the RN levels.

DISCUSSION

The PES is a valid instrument for use in quality improvement and research both at the unit and individual RN levels.

Adding a Nursing Information Technology Subscale to the Practice Environment Scale of the Nursing Work Index

In the past decade, the use of information technology (IT) to support healthcare delivery has greatly expanded. Introducing new clinical information systems and updating existing systems require continued learning and training among registered nurses (RNs) and other clinicians. Consequently, a 5-item measure was designed as a subscale to the Practice Environment Scale of the Nursing Work Index. This new subscale, the Nursing Information Technology Subscale (NITS) measures RNs' perception of the extent to which IT in their practice environment supports patient care delivery. A psychometric evaluation of the NITS was conducted at 8 Veterans Affairs hospitals. The findings demonstrated that this brief subscale has acceptable reliability as well as convergent and discriminant validity.

Mapping the organizational culture research in nursing: a literature review

AIM

This paper reports a critical review of nursing organizational culture research studies with the objectives of: (1) reviewing theoretical and methodological characteristics of the studies and (2) drawing inferences specific to the state of knowledge in this field.

BACKGROUND

Organizational culture is regarded as significant in influencing research use in clinical practice yet it is not understood how culture shapes practitioners' behaviours. Only one review of this empirical literature in nursing has been completed.

METHOD

Using selected computerized databases, published nursing research studies in English that examine organizational culture were accessed. Organizational culture studies were categorized using Hatch's three perspectives on organizational culture: (1) modern, (2) symbolic-interpretive and (3) postmodern. The review was conducted in 2005.

RESULTS

Twenty-nine studies were in the final data set. Results pointed to variations in cultural definitions and incorporation of organizational sciences theory. In classifying the studies, modern perspectives dominated (n = 22), symbolic-interpretive approaches were an emerging group (n = 6) and one study was unclassifiable. Our results expand current cultural instrument reviews by pinpointing tools that have been previously overlooked and by identifying ongoing theoretical and methodological challenges for researchers.

CONCLUSION

An exclusive reliance on modernistic approaches in organizational culture research cannot yield a complete understanding of the phenomenon. Rather, the field could benefit from a variety of cultural approaches. In a similar vein, researchers need to be mindful of the terminology and the unit of analysis they use in their research, as these are the two largest research challenges.

Unit type differences in RN workgroup job satisfaction

Using cross-sectional data from the 2004 National Database of Nursing Quality Indicators (NDNQI) RN Satisfaction Survey, differences in RN workgroup job satisfaction were examined among 10 unit types--medical-surgical, step-down, critical care, pediatric, maternal-newborn, psychiatric, emergency department, rehabilitation, surgical services, and outpatient clinics and labs. The national sample included RN workgroups in 2,900 patient care units (55,516 RNs; 206 hospitals in 44 states). Workgroup satisfaction across all unit types was moderate. RN workgroups in pediatric units were the most satisfied, whereas those in surgical services and emergency department unit types were least satisfied. A consistent finding across all unit types was high satisfaction with the specific domains of nurse-to-nurse interaction, professional status, and professional development versus much lower satisfaction with task, decision making, and pay. Findings can be used to inform and develop investigations that examine specific aspects of the work environment for RN workgroups in various unit types.

Nursing staff mix models and outcomes

BACKGROUND

Recently, restructuring of the nursing workforce has been undertaken in a number of countries in an effort to provide efficient and cost-effective services to users. This often takes the form of the introduction of unregulated workers to carry out support roles with registered nurses. However, these changes have not been evaluated for efficacy or impact on nurses, patients or the health care system.

PURPOSE

The purpose of this study was to determine the relationship between staff mix models comprising regulated staff (Registered Nurses and Registered Practical Nurses) or regulated and unregulated staff (Registered Nurses and unregulated workers), and nursing and quality outcomes.

METHODS

This comparative correlational study was conducted in a random sample of 30 adult, acute care patient units within eight hospitals located in Toronto, Canada. Registered Nurses employed on 30 randomly selected hospital units, grouped by the two staff mix models (15 units per group), were surveyed using previously validated instruments to measure role conflict, role ambiguity, job satisfaction, perceived effectiveness of care and perceived quality of care.

RESULTS

Results indicated that Registered Nurses in this study experienced high levels of role conflict, regardless of the type of staff mix model within which they worked. Registered Nurses on units employing both Registered Nurses and unregulated workers reported higher levels of job satisfaction. On units employing both Registered Nurses and unregulated workers, Registered Nurses perceived that the quality of care was lower.

CONCLUSIONS

Staff mix model was related to Registered Nurses' perceptions of the quality of patient care. It was also evident that other variables within the work environment might have more influence on the outcomes examined than the independent variable of staff mix.

Making the most of collaboration: exploring the relationship between partnership synergy and partnership functioning

Considering the challenges inherent to collaboration and the time it takes to achieve measurable outcomes, partnerships need a way to determine, at an early stage, whether they are making the most of collaboration. The authors have developed a new measure, partnership synergy, which assesses the degree to which a partnership's collaborative process successfully combines its participants' perspectives, knowledge, and skills. This article reports the results of a national study designed to examine the relationship between partnership synergy and six dimensions of partnership functioning: leadership, administration and management, partnership efficiency, nonfinancial resources, partner involvement challenges, and community-related challenges. Data were collected from 815 informants in 63 partnerships. Results of regression analysis conducted with partnership-level data indicated that partnership synergy was most closely related to leadership effectiveness and partnership efficiency. Implications of these findings for research and practice are discussed.

Data aggregation: criteria for psychometric evaluation

Criteria for the examination of data collected at the individual level and aggregated to the group are proposed. These four empirical criteria relate to content validity, representativeness, reliability, and validity of the aggregated scores as indicators of group phenomena. An example of the use of the criteria is provided using data from a large research project on the implementation of a professional practice model on nine nursing units. The use of the criteria proposed provide a systematic approach to the evaluation of individual data for use at the group level.

Focus on psychometrics. Aggregating family data

Instruments to collect data about families are often administered to all or some individuals within the family. Researchers may wish to use these individual scores to describe the family. The purpose of this article is to describe the special issues with aggregation of data when only a small number of family members are used as respondents. A refinement of the definition of aggregation for family researchers is proposed to assist researchers to focus on specific issues when data are to reflect subgroups within the family. A few examples of changes in findings are reported to illustrate the effects of different aggregation schemes when two members of the family are used as respondents.