A unified approach to multi-item reliability

Overestimation of Internal Consistency by Coefficient Omega in Data Giving Rise to a Centroid-Like Factor Solution

Coefficient Omega measuring internal consistency is investigated for its deviations from expected outcomes when applied to correlational patterns that produce variable-focused factor solutions in confirmatory factor analysis. In these solutions, the factor loadings on the factor of the one-factor measurement model closely correspond to the correlations of one manifest variable with the other manifest variables, as is in centroid solutions. It is demonstrated that in such a situation, a heterogeneous correlational pattern leads to an Omega estimate larger than those for similarly heterogeneous and uniform patterns. A simulation study reveals that these deviations are restricted to datasets including small numbers of manifest variables and that the degree of heterogeneity determines the degree of deviation. We propose a method for identifying variable-focused factor solutions and how to deal with deviations.

Generalized reliability based on distances

The intraclass correlation coefficient (ICC) is a classical index of measurement reliability. With the advent of new and complex types of data for which the ICC is not defined, there is a need for new ways to assess reliability. To meet this need, we propose a new distance-based ICC (dbICC), defined in terms of arbitrary distances among observations. We introduce a bias correction to improve the coverage of bootstrap confidence intervals for the dbICC, and demonstrate its efficacy via simulation. We illustrate the proposed method by analyzing the test-retest reliability of brain connectivity matrices derived from a set of repeated functional magnetic resonance imaging scans. The Spearman-Brown formula, which shows how more intensive measurement increases reliability, is extended to encompass the dbICC.

Measuring Nutrition and Food Literacy in Adults: A Systematic Review and Appraisal of Existing Measurement Tools

Background:

Nutrition literacy (NL) and food literacy (FL) have emerged as key components in the promotion and maintenance of healthy dietary practices. However, a critical appraisal of existing tools is required to advance the operationalization and measurement of these constructs using instruments that demonstrate sound validity and reliability.

Methods:

Electronic databases were searched in January and July 2016, January 2017, and March 2018 for publications detailing the development and/or testing of NL or FL instruments. Instruments' psychometric properties were assessed using a structured methodological framework. We identified 2,563 new titles and abstracts, and short-listed 524 for full review. The extent to which key domains of NL were included in each measure was examined.

Key Results:

Thirteen instruments assessing NL underwent full evaluation; seven from the United States, and one each from Australia, Norway, Switzerland, Italy, Hong Kong, and Japan. Measures targeted general Spanish-, Italian-, or Cantonese-speaking adults; primary care patients, parent, and populations with breast cancer. Instruments ranged from 6 to 64 items, and they predominantly assessed functional NL rather than broader domains of NL. Substantial variation in methodological rigor was observed across measures.

Discussion:

Multidimensional and psychometrically sound measures that capture broader domains of NL and assess FL are needed.

Plain Language Summary:

This review systemically compiles, and critically appraises 13 existing measures that assess nutrition literacy and food literacy in an adult population. Substantial variation in methodological rigor was found across the measures, and most tools assessed nutrition literacy rather than food literacy. Findings from this current review may be useful to guide development of future measures that comprehensively capture nutrition literacy and food literacy. [HLRP: Health Literacy Research and Practice. 2018;2(3):e134–e160.]

The Nutrition Literacy Assessment Instrument is a Valid and Reliable Measure of Nutrition Literacy in Adults with Chronic Disease

OBJECTIVE

To test the reliability and validity of the Nutrition Literacy Assessment Instrument (NLit) in adult primary care and identify the relationship between nutrition literacy and diet quality.

DESIGN

This instrument validation study included a cross-sectional sample participating in up to 2 visits 1 month apart.

SETTING/PARTICIPANTS

A total of 429 adults with nutrition-related chronic disease were recruited from clinics and a patient registry affiliated with a Midwestern university medical center.

MAIN OUTCOME MEASURES

Nutrition literacy was measured by the NLit, which was composed of 6 subscales: nutrition and health, energy sources in food, food label and numeracy, household food measurement, food groups, and consumer skills. Diet quality was measured by Healthy Eating Index-2010 with nutrient data from Diet History Questionnaire II surveys.

ANALYSIS

The researchers measured factor validity and reliability by using binary confirmatory factor analysis; test-retest reliability was measured by Pearson r and the intraclass correlation coefficient, and relationships between nutrition literacy and diet quality were analyzed by linear regression.

RESULTS

The NLit demonstrated substantial factor validity and reliability (0.97; confidence interval, 0.96-0.98) and test-retest reliability (0.88; confidence interval, 0.85-0.90). Nutrition literacy was the most significant predictor of diet quality (β = .17; multivariate coefficient = 0.10; P < .001).

CONCLUSIONS

The NLit is a valid and reliable tool for measuring nutrition literacy in adult primary care patients.

Confirmatory Factor Analysis Alternative: Free, Accessible CBID Software

New software that performs Classical and Bayesian Instrument Development (CBID) is reported that seamlessly integrates expert (content validity) and participant data (construct validity) to produce entire reliability estimates with smaller sample requirements. The free CBID software can be accessed through a website and used by clinical investigators in new instrument development. Demonstrations are presented of the three approaches using the CBID software: (a) traditional confirmatory factor analysis (CFA), (b) Bayesian CFA using flat uninformative prior, and (c) Bayesian CFA using content expert data (informative prior). Outcomes of usability testing demonstrate the need to make the user-friendly, free CBID software available to interdisciplinary researchers. CBID has the potential to be a new and expeditious method for instrument development, adding to our current measurement toolbox. This allows for the development of new instruments for measuring determinants of health in smaller diverse populations or populations of rare diseases.

Common Sense Model Factors Affecting African Americans' Willingness to Consult a Healthcare Provider Regarding Symptoms of Mild Cognitive Impairment

OBJECTIVE

Although at increased risk for developing dementia compared with white patients, older African Americans are diagnosed later in the course of dementia. Using the common sense model (CSM) of illness perception, we sought to clarify processes promoting timely diagnosis of mild cognitive impairment (MCI) for African American patients.

DESIGN, SETTING, PARTICIPANTS

In-person, cross-sectional survey data were obtained from 187 African American (mean age: 60.44 years). Data were collected at social and health-focused community events in three southern Wisconsin cities.

MEASUREMENTS

The survey represented a compilation of published surveys querying CSM constructs focused on early detection of memory disorders, and willingness to discuss concerns about memory loss with healthcare providers. Derived CSM variables measuring perceived causes, consequences, and controllability of MCI were included in a structural equation model predicting the primary outcome: Willingness to discuss symptoms of MCI with a provider.

RESULTS

Two CSM factors influenced willingness to discuss symptoms of MCI with providers: Anticipation of beneficial consequences and perception of low harm associated with an MCI diagnosis predicted participants' willingness to discuss concerns about cognitive changes. No association was found between perceived controllability and causes of MCI, and willingness to discuss symptoms with providers.

CONCLUSIONS

These data suggest that allaying concerns about the deleterious effects of a diagnosis, and raising awareness of potential benefits, couldinfluence an African American patient's willingness to discuss symptoms of MCI with a provider. The findings offer guidance to designers of culturally congruent MCI education materials, and healthcare providers caring for older African Americans. .

Expediting Clinical and Translational Research via Bayesian Instrument Development

Developing valid and reliable instruments is crucial but costly and time-consuming in health care research and evaluation. The Food and Drug Administration and the National Institutes of Health have set up guidelines for developing patient-reported outcome instruments. However, the guidelines are not applicable to cases of small sample sizes. Instead of using an exact estimation procedure to examine psychometric properties, our Bayesian Instrument Development (BID) method integrates expert data and participant data into a single seamless analysis. Using a novel set of priors, we use simulated data to compare BID to classical instrument development procedures and test the stability of BID. To display BID to non-statisticians, a graphical user interface based on R and WINBUGS is developed and demonstrated with data on a small sample of heart failure patients. Costs were saved by eliminating the need for unnecessary continuation of data collection for larger samples as required by the classical instrument development approach.

Staff commitment to providing tobacco dependence in drug treatment: Reliability, validity, and results of a national survey

Although most people in treatment for illicit drug use smoke cigarettes, few facilities offer any form of treatment for tobacco dependence. One reason for this may be that drug treatment staff have varying levels of commitment to treat tobacco. We developed and validated a 14-item Tobacco Treatment Commitment Scale (TTCS), using 405 participants in leadership positions in drug treatment facilities. We first conducted a confirmatory factor analysis to evaluate 4 a priori domains suggested by our original set of 38 items-this did not produce a good fit (comparative fit index [CFI] = 0.782, root mean square error of approximation [RMSEA] = 0.067). We then conducted a series of exploratory factor analyses to produce a more precise and reliable scale. The final confirmatory factor analysis indicated a 3-factor solution, produced a good fit (CFI = 0.950, RMSEA = 0.058), and had substantial unified reliability of 0.975. The final TTCS contained 14 items in 3 domains: "Tobacco is less harmful than other drugs," "It's not our job to treat tobacco," and "Tobacco treatment will harm clients." These constructs account for most of the variance in the survey items and emerged as major sentiments driving staff commitment to providing tobacco services. The TTCS can be used to understand the role of staff attitudes in the adoption of tobacco services in this important treatment setting.

The index of tobacco treatment quality: development of a tool to assess evidence-based treatment in a national sample of drug treatment facilities

Background

Quitting smoking improves health and drug use outcomes among people in treatment for substance abuse. The twofold purpose of this study is to describe tobacco treatment provision across a representative sample of U.S. facilities and to use these data to develop the brief Index of Tobacco Treatment Quality (ITTQ).

Methods

We constructed survey items based on current tobacco treatment guidelines, existing surveys, expert input, and qualitative research. We administered the survey to a stratified sample of 405 facility administrators selected from all 3,800 U.S. adult outpatient facilities listed in the SAMHSA Inventory of Substance Abuse Treatment Services. We constructed the ITTQ with a subset of 7 items that have the strongest clinical evidence for smoking cessation.

Results

Most facilities (87.7%) reported that a majority of their clients were asked if they smoke cigarettes. Nearly half of facilities (48.6%) reported that a majority of their smoking clients were advised to quit. Fewer (23.3%) reported that a majority of their smoking clients received tobacco treatment counseling and even fewer facilities (18.3%) reported a majority of their smoking clients were advised to use quit smoking medications. The median facility ITTQ score was 2.57 (on a scale of 1–5) and the ITTQ displayed good internal consistency (Cronbach’s alpha = .844). Moreover, the ITTQ had substantial test-retest reliability (.856), and ordinal confirmatory factor analysis found that our one-factor model for ITTQ fit the data very well with a CFI of 0.997 and an RMSEA of 0.042.

Conclusions

The ITTQ is a brief and reliable tool for measuring tobacco treatment quality in substance abuse treatment facilities. Given the clear-cut room for improvement in tobacco treatment, the ITTQ could be an important tool for quality improvement by identifying service levels, facilitating goal setting, and measuring change.

Validation of the Excited Component of the Positive and Negative Syndrome Scale (PANSS-EC) in a naturalistic sample of 278 patients with acute psychosis and agitation in a psychiatric emergency room

Background

Despite the wide use of the Excited Component of the Positive and Negative Syndrome Scale (PANSS-EC) in a clinical setting to assess agitated patients, a validation study to evaluate its psychometric properties was missing.

Methods

Data from the observational NATURA study were used. This research describes trends in the use of treatments in patients with acute psychotic episodes and agitation seen in emergency departments. Exploratory principal component factor analysis was performed. Spearman's correlation and regression analyses (linear regression model) as well as equipercentile linking of Clinical Global Impression of Severity (CGI-S), Agitation and Calmness Evaluation Scale (ACES) and PANSS-EC items were conducted to examine the scale's diagnostic validity. Furthermore, reliability (Cronbach's alpha) and responsiveness were evaluated.

Results

Factor analysis resulted in one factor being retained according to eigenvalue ≥1. At admission, the PANSS-EC and CGI-S were found to be linearly related, with an average increase of 3.4 points (p < 0.001) on the PANSS-EC for each additional CGI-S point. The PANSS-EC and ACES were found to be linearly and inversely related, with an average decrease of 5.5 points (p < 0.001) on the PANSS-EC for each additional point. The equipercentile method shows the poor sensitivity of the ACES scale. Cronbach's alpha was 0.86 and effect size was 1.44.

Conclusions

The factorial analyses confirm the unifactorial structure of the PANSS-EC subscale. The PANSS-EC showed a strong linear correlation with rating scales such as CGI-S and ACES. PANSS-EC has also shown an excellent capacity to detect real changes in agitated patients.