Response-Scale Formats and Psychological Distances Between Categories

Psychometric benefits of self-chosen rating scales over given rating scales

Rating scales are susceptible to response styles that undermine the scale quality. Optimizing a rating scale can tailor it to individuals' cognitive abilities, thereby preventing the occurrence of response styles related to a suboptimal response format. However, the discrimination ability of individuals in a sample may vary, suggesting that different rating scales may be appropriate for different individuals. This study aims to examine (1) whether response styles can be avoided when individuals are allowed to choose a rating scale and (2) whether the psychometric properties of self-chosen rating scales improve compared to given rating scales. To address these objectives, data from the flourishing scale were used as an illustrative example. MTurk workers from Amazon's Mechanical Turk platform (N = 7042) completed an eight-item flourishing scale twice: (1) using a randomly assigned four-, six-, or 11-point rating scale, and (2) using a self-chosen rating scale. Applying the restrictive mixed generalized partial credit model (rmGPCM) allowed examination of category use across the conditions. Correlations with external variables were calculated to assess the effects of the rating scales on criterion validity. The results revealed consistent use of self-chosen rating scales, with approximately equal proportions of the three response styles. Ordinary response behavior was observed in 55-58% of individuals, which was an increase of 12-15% compared to assigned rating scales. The self-chosen rating scales also exhibited superior psychometric properties. The implications of these findings are discussed.

Evidence of validity of internal structure of the Functional Assessment of Chronic Illness Therapy-Spiritual Well-Being Scale (FACIT-Sp-12) in Brazilian adolescents with chronic health conditions

This study explored the evidence of validity of internal structure of the 12-item Functional Assessment of Chronic Illness Therapy—Spiritual Wellbeing Scale (FACIT-Sp-12) in Brazilian adolescents with chronic health conditions. The study involved 301 Brazilian adolescents with cancer, type 1 diabetes mellitus, or cystic fibrosis. Exploratory Factor Analysis (EFA), Confirmatory Factor Analysis (CFA), and Item Response Theory (IRT) were used to test the internal structure. Reliability was determined with Cronbach’s Alpha and McDonald’s Omega. The EFA suggested a one-dimensional scale structure in contrast to the original 2-factor model or the 3-factor model which were not reproduced in the current CFA. All quality indicators for the EFA one-factor exceeded the required criteria (FDI = 0.97, EAP = 0.97, SR = 3.96 and EPTD = 0.96, latent GH = 0.90. and the observed GH = 0.85). The FACIT-Sp-12 for adolescents yielded strong evidence for a 1-factor model and with good reliability.

Survey Satisficing Inflates Reliability and Validity Measures: An Experimental Comparison of College and Amazon Mechanical Turk Samples

This study examined the predictors and psychometric outcomes of survey satisficing, wherein respondents provide quick, "good enough" answers (satisficing) rather than carefully considered answers (optimizing). We administered surveys to university students and respondents-half of whom held college degrees-from a for-pay survey website, and we used an experimental method to randomly assign the participants to survey formats, which presumably differed in task difficulty. Based on satisficing theory, we predicted that ability, motivation, and task difficulty would predict satisficing behavior and that satisficing would artificially inflate internal consistency reliability and both convergent and discriminant validity correlations. Indeed, results indicated effects for task difficulty and motivation in predicting survey satisficing, and satisficing in the first part of the study was associated with improved internal consistency reliability and convergent validity but also worse discriminant validity in the second part of the study. Implications for research designs and improvements are discussed.

A Meta-Analytic Investigation of the Relationship Between Scale-Item Length, Label Format, and Reliability

Abstract. Using two meta-analytic datasets, we investigated the effect that two scale-item characteristics – number of item response categories and item response-category label format – have on the reliability of multi-item rating scales. The first dataset contained 289 reliability coefficients harvested from 100 samples that measured Big Five traits. The second dataset contained 2,524 reliability coefficients harvested from 381 samples that measured a wide variety of constructs in psychology, marketing, management, and education. We performed moderator analyses on the two datasets with the two item characteristics and their interaction. As expected, as the number of item response categories increased, so did reliability, but more importantly, there was a significant interaction between the number of item response categories and item response-category label format. Increasing the number of response categories increased reliabilities for scale-items with all response categories labeled more so than for other item response-category label formats. We explain that the interaction may be due to both statistical and psychological factors. The present results help to explain why findings on the relationships between the two scale-item characteristics and reliability have been mixed.