Psychometric evaluation of the Trust in Science and Scientists Scale

Reliable and valid measurement of trust in science and scientists is important. Assessing levels of such trust is important in determining attitudes and predicting behaviours in response to medical and scientific interventions targeted at managing public crises. However, trust is a complex phenomenon that has to be understood in relation to both distrust and mistrust. The Trust in Science and Scientists Scale has been adopted with increasing frequency in large-scale public health research. Detailed psychometric evaluation of the scale is overdue and makes meaningful comparisons between studies that use the scale difficult. Here, we examine the scale's dimensionality across five separate samples. We find that two factors emerge that are divided by their item polarity. Implications for scale use and trust in science measurement are discussed.

Safety climate and safety behaviors: Adapting and validating a positively worded safety climate scale through the time-lagged approach

BACKGROUND

Contemporary literature raises serious questions about the inclusion of negatively worded items in the safety climate scale. Despite these reservations, limited efforts have been made to address this shortcoming.

OBJECTIVE

The present study aims to adapt and empirically validate the ten-items group-level safety climate scale with the purpose of replacing negatively worded items with positively worded ones after a thorough validation process. The present study is one of the first to propose an empirically validated group-level safety climate scale that uses positive items to measure the safety climate construct.

METHODS

Study 1 was conducted using a sample of 135 participants. Study 2 used a time-lagged approach to validate the scale, with a sample of 173 production workers from six oil and gas organizations in Malaysia. The Partial Least Squares Structural Equation Modeling (PLS-SEM) method was used to test the hypothesized relationships.

RESULTS

In Study 1, the results of the exploratory factor analysis showed good reliability for the revised scale. In Study 2, the results of the PLS-SEM analysis demonstrated a positive relationship between safety climate and safety behaviors, thereby validating the revised and translated scale of safety climate.

CONCLUSION

The revised safety climate scale will not only improve data quality, but it will also increase response rates. Additionally, the revised scale will assist managers in understanding the true perceptions of safety climate in their organization, regardless of the cultural context in which the scale is used.

Effects of smartphone-based hypnotic intervention for undergraduate students' shyness during the COVID-19 pandemic: A randomized controlled trial

BACKGROUND

Since the outbreak of the COVID-19 pandemic, university students have been exposed to a heightened vulnerability towards developing psychological issues, such as psychological distress and shyness. Internet-based interventions offer a convenient avenue for scalability, thus prompting the development of a smartphone-based hypnotic intervention aimed at addressing shyness among university students.

OBJECTIVE

We devised an innovative smartphone-based hypnotic intervention called mHypnosis to examine its impact on shyness among undergraduate students. Furthermore, we aimed to investigate whether the apprehension of negative evaluations before treatment could serve as a predictor for the effectiveness of the intervention on shyness.

METHODS

Eighty students with high shyness scores were randomly assigned to the experimental group and the control group. Another 40 participants with low shyness score were selected as the baseline group. The Shyness Scale (SS), Fear of Negative evaluation scale (FNE), Self-Acceptance Questionnaire (SAQ), and Self-Esteem Scale (SES) were used to evaluate the effect of hypnotic intervention.

RESULTS

Before the intervention, the scores of the experimental and control groups on the SS, FNE, SAQ, and SES were higher than those in the baseline group (p < 0.05). There was no significant difference in scores between the experimental and control group (p > 0.05). After the intervention, the scores of the SS, FNE, SAQ, and SES were significantly lower in the experimental group than those in the control group (p < 0.05). The pretest score of FNE could predict the shyness score after hypnotic intervention (B = 0.35, p < 0.05).

CONCLUSION

Smartphone-based hypnotic intervention had a significant effect on ameliorating shyness during the COVID-19 pandemic; fear of negative evaluation can be a target for treating shyness.

Why Do Regular and Reversed Items Load on Separate Factors? Response Difficulty vs. Item Extremity

When constructing measurement scales, regular and reversed items are often used (e.g., "I am satisfied with my job"/"I am not satisfied with my job"). Some methodologists recommend excluding reversed items because they are more difficult to understand and therefore engender a second, artificial factor distinct from the regular-item factor. The current study compares two explanations for why a construct's dimensionality may become distorted: response difficulty and item extremity. Two types of reversed items were created: negation items ("The conditions of my life are not good") and polar opposites ("The conditions of my life are bad"), with the former type having higher response difficulty. When extreme wording was used (e.g., "excellent/terrible" instead of "good/bad"), negation items did not load on a factor distinct from regular items, but polar opposites did. Results thus support item extremity over response difficulty as an explanation for dimensionality distortion. Given that scale developers seldom check for extremity, it is unsurprising that regular and polar opposite items often load on distinct factors.

Estimating Local Structural Equation Models

Local structural equation models (LSEM) are structural equation models that study model parameters as a function of a moderator. This article reviews and extends LSEM estimation methods and discusses the implementation in the R package sirt. In previous studies, LSEM was fitted as a sequence of models separately evaluated as each value of the moderator variables. In this article, a joint estimation approach is proposed that is a simultaneous estimation method across all moderator values and also allows some model parameters to be invariant with respect to the moderator. Moreover, sufficient details on the main estimation functions in the R package sirt are provided. The practical implementation of LSEM is demonstrated using illustrative datasets and an empirical example. Moreover, two simulation studies investigate the statistical properties of parameter estimation and significance testing in LSEM.

It Takes Two to Tango: Links Between Traditional Beliefs About both Men's and Women's Gender Roles and Comfort Initiating Sex and Comfort Refusing Sex

Traditional gender roles dictate rigid rules and standards prescribing which behaviors, thoughts, and feelings are considered masculine and feminine within sexual contexts, and thus internalizing these beliefs (higher traditional gender ideology) may influence sexual attitudes. Prior theorizing has primarily focused on how women's traditional beliefs about women's gender roles (traditional femininity ideology) and men's traditional beliefs about men's gender roles (traditional masculinity ideology) influence their sexual assertiveness. Yet, men can hold traditional beliefs about women, and women can hold traditional beliefs about men, and these beliefs should have important implications for sexual assertiveness. We addressed this gap by testing how both heterosexual women's (n = 389) and men's (n = 393) traditional masculinity and femininity ideologies associate with their reported comfort initiating sex and comfort refusing sex in their relationships. When accounting for both sets of beliefs, women's traditional beliefs about men's and women's roles interacted to predict comfort initiating sex, but not comfort refusing sex. Men's traditional beliefs about men's roles predicted less comfort refusing sex, and their traditional beliefs about women's roles predicted less comfort initiating sex. This novel research underscores the importance of considering beliefs about both sets of traditional gender roles for understanding people's sexual attitudes.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11199-023-01366-w.

The Dimensionality of Reading Self-Concept: Examining Its Stability Using Local Structural Equation Models

For results from large-scale surveys to inform policy and practice appropriately, all participants must interpret and respond to items similarly. While organizers of surveys assessing student outcomes often ensure this for achievement measures, doing so for psychological questionnaires is also critical. We demonstrate this by examining the dimensionality of reading self-concept-a crucial psychological construct for several outcomes-across reading achievement levels. We use Programme for International Student Assessment 2018 data (N = 529,966) and local structural equation models (LSEMs) to do so. Results reveal that reading self-concept dimensions (assessed through reading competence and difficulty) vary across reading achievement levels. Students with low reading achievement show differentiated responses to the two item sets (high competence-high difficulty). In contrast, students with high reading achievement have reconciled responses (high competence-low difficulty). Our results highlight the value of LSEMs in examining factor structure generalizability of constructs in large-scale surveys and call for greater cognitive testing during item development.

Wording Effects in Assessment: Missing the Trees for the Forest

This article examines wording effects when positive and negative worded items are included in psychological assessment. Wordings effects have been analyzed in the literature using statistical approaches based on population homogeneity assumptions (i.e. CFA, SEM), commonly adopting the bifactor model to separate trait variance and wording effects. This article presents an alternative approach by explicitly modeling population heterogeneity through a latent profile model, based on the idea that a subset of individuals exhibits wording effects. This kind of mixture model allows simultaneously to classify respondents, substantively characterize the differences in their response profiles, and report respondents' results in a comparable manner. Using the Rosenberg's self-esteem scale data from the LISS Panel (N = 6,762) in three studies, we identify a subgroup of participants who respond differentially according to item-wording and examine the impact of its responses in the estimation of the RSES measurement model, in terms of global and individual fit, under one-factor and bifactor models.The results of these analyses support the interpretation of wording effects in terms of a theoretically-proposed differential pattern of response to positively and negatively worded items, introducing a valuable tool for examining the artifactual or substantive interpretations of such wording effects.

Testing the Nonlinearity Assumption Underlying the Use of Reverse-Keyed Items: A Logical Response Perspective

Researchers often assume a strong, linear relationship between regular- and reverse-keyed items, with responses on regular-keyed items (e.g., agree) perfectly mirroring those on reverse-keyed items (e.g., disagree). The current research challenges this received view and propounds a possible nonlinear relationship, partly due to the logical tendency of midlevel respondents to disagree with both types of items. In four examples (reported human height, job satisfaction, positive-negative affect, and self-esteem; total N = 50,544), a nonlinear model consistently explained additional item variance beyond a linear model. We further demonstrate that this relationship is moderated by item characteristics such as item extremity (job satisfaction) and item softening (self-esteem). Suboptimal modeling of the relationship may result in the apparent bidmensionality of a construct that characterizes regular- and reverse-keyed items as separate factors. User-friendly syntax for the examination of nonlinearity is provided to enhance the accessibility of the procedure.

A validation study of the Italian version of the Attitudes Toward Obese Persons (I-ATOP) questionnaire

OBJECTIVES

Weight bias is defined as negative attitudes towards, and beliefs about, others because of their weight. Like other forms of stigma, weight stigma has a harmful impact on health, including depressive symptoms, disordered eating, body image disturbances and poor quality of life. Several instruments measuring weight-related attitudes have been developed, such as the Attitudes Toward Obese Persons (ATOP). The purpose of this study was to test the psychometric properties of the Italian version of the ATOP (I-ATOP).

METHODS

The ATOP was adapted into Italian following the back-translation procedure. A total of 800 participants (M_age=31.40; 54.9% females) completed the I-ATOP alongside the Weight Bias Internalization Scale (WBIS). Exploratory factor analysis (EFA) was conducted on half of the sample, while the remaining half was selected to cross-validate the resulting solution via confirmatory factor analysis (CFA). Factorial invariance tests across gender were computed using multi-group CFA.

RESULTS

EFA suggested a one-factor structure with four items excluded due to their low standardised loadings. The trimmed model was cross-validated showing an acceptable fit to the data: MLRχ2 = 159.467 (df=81); RMSEA= 0.049; CFI= 0.939; TLI= 0.910; SRMR= 0.046. An omega coefficient of 0.818 confirmed the strong reliability of the I-ATOP. Convergent validity was demonstrated by a significant and moderate correlation with the WBIS. Configural, metric, and scalar invariance across gender were established.

CONCLUSIONS

I-ATOP demonstrated to be a valid and reliable instrument useful for both clinical and research practices, as well as to support the development of educational and therapeutic actions able to reduce the stigma among the general public.

Detecting Careless Responding in Survey Data Using Stochastic Gradient Boosting

Careless responding is a bias in survey responses that disregards the actual item content, constituting a threat to the factor structure, reliability, and validity of psychological measurements. Different approaches have been proposed to detect aberrant responses such as probing questions that directly assess test-taking behavior (e.g., bogus items), auxiliary or paradata (e.g., response times), or data-driven statistical techniques (e.g., Mahalanobis distance). In the present study, gradient boosted trees, a state-of-the-art machine learning technique, are introduced to identify careless respondents. The performance of the approach was compared with established techniques previously described in the literature (e.g., statistical outlier methods, consistency analyses, and response pattern functions) using simulated data and empirical data from a web-based study, in which diligent versus careless response behavior was experimentally induced. In the simulation study, gradient boosting machines outperformed traditional detection mechanisms in flagging aberrant responses. However, this advantage did not transfer to the empirical study. In terms of precision, the results of both traditional and the novel detection mechanisms were unsatisfactory, although the latter incorporated response times as additional information. The comparison between the results of the simulation and the online study showed that responses in real-world settings seem to be much more erratic than can be expected from the simulation studies. We critically discuss the generalizability of currently available detection methods and provide an outlook on future research on the detection of aberrant response patterns in survey research.

Response tendencies due to item wording using eye-tracking methodology accounting for individual differences and item characteristics

The discrepancy of the scores on responses to negatively and positively worded items has led to hypotheses of inattention, confusion, difficulty, and differential processing of negatively worded items. The present study, utilizing eye-tracking methodology, aimed to fill an explanatory gap regarding response behavior, providing observations on the item-level response process. It experimentally examined characteristics of the items (wording type, self-relevance) and characteristics of the respondents (neuroticism, verbal abilities, and mood) for their impact on response outcomes. A sample of 87 university students completed a computerized version of a questionnaire with items presented in four alternative wording types: positive, negative, negated positive, and negated negative; half of the items referred to attitudes toward the self and the other half to attitudes toward others. Participants' eye movements during item completion were recorded with the Gazepoint-GP3-HD desk-mounted eye tracker. In linear mixed effects models, wording type and self-relevance were found to relate to response time, time of viewing and revisits to the body of the items and the response options, indicating that there were effects at the stages of comprehension and selection of response. Neuroticism was associated with differential item responses, suggesting a role in later levels of the response process, the retrieval, judgment, and response selection stages. Eye-tracking measures can enhance the examination of response tendencies with regards to item content, item wording, and person characteristics.

Using Constrained Factor Mixture Analysis to Validate Mixed-Worded Psychological Scales: The Case of the Rosenberg Self-Esteem Scale in the Dominican Republic

A common method to collect information in the behavioral and health sciences is the self-report. However, the validity of self-reports is frequently threatened by response biases, particularly those associated with inconsistent responses to positively and negatively worded items of the same dimension, known as wording effects. Modeling strategies based on confirmatory factor analysis have traditionally been used to account for this response bias, but they have recently become under scrutiny due to their incorrect assumption of population homogeneity, inability to recover uncontaminated person scores or preserve structural validities, and their inherent ambiguity. Recently, two constrained factor mixture analysis (FMA) models have been proposed by Arias et al. (2020) and Steinmann et al. (2021) that can be used to identify and screen inconsistent response profiles. While these methods have shown promise, tests of their performance have been limited and they have not been directly compared. Thus the objective of the current study was to assess and compare their performance with data from the Dominican Republic of the Rosenberg Self-Esteem Scale (N = 632). Additionally, as this scale had not yet been studied for this population, another objective was to show how using constrained FMAs could help in the validation of mixed-worded scales. The results indicated that removing the inconsistent respondents identified by both FMAs (≈8%) reduced the amount of wording effects in the database. However, whereas the Steinmann et al. method only cleaned the data partially, the Arias et al. (2020) method was able to remove the great majority of the wording effects variance. Based on the screened data with the Arias et al. method, we evaluated the psychometric properties of the RSES for the Dominican population, and the results indicated that the scores had good validity and reliability properties. Given these findings, we recommend that researchers incorporate constrained FMAs into their toolbox and consider using them to screen out inconsistent respondents to mixed-worded scales.

To Reverse Item Orientation or Not to Reverse Item Orientation, That Is the Question

To investigate the effect of using negatively oriented items, we wrote semantic reversals of the items in the Rosenberg Self Esteem Scale, UCLA Loneliness Scale, and the General Belongingness Scale and used them to create four experimental conditions. Participants (N = 2,019) were recruited through Amazon's Mechanical Turk. Data were assessed for dimensionality, item functioning, instrument properties, and associations with other variables. Regarding dimensionality, although a two-factor model (positively vs. negatively oriented factors) exhibits better fit than a unidimensional model across all conditions, bifactor indices were used to argue that a unidimensional interpretation of the data can be employed. With respect to item functioning, factor loadings were found to be nearly invariant across conditions, but thresholds were not. Concerning instrument properties, inclusion of negatively oriented items results in lower mean scores and higher score variances. Instruments with both positively and negatively oriented items demonstrated lower reliability estimates than those with only one orientation. For associations with other variables, path coefficients in a model where loneliness mediates the effects of belongingness on life satisfaction and self-esteem were found to vary across conditions. Findings suggest that negatively oriented items have minor impact on instrument quality, but influence measurement model and path coefficients.

Bifactor analysis of the rosenberg self-esteem scale in morbid obesity

Self-esteem is a set of feelings and thoughts a person has about his/her own worth and competence, which is reflected in a positive or negative attitude toward himself/herself. This study examined the factor structure of the Rosenberg Self-Esteem Scale with scores of 288 obese brazilian women waiting for bariatric surgery. Psychometric studies have found controversial results regarding the factor structure of this measure with samples from different profiles. In this investigation, the unifactor, two-oblique-factor and bifactor models were tested by Confirmatory Factor Analysis. The results clearly support the existence of a single dimension for the scale, but also for the method effects relating to the positively and negatively worded items. Further investigations with the inclusion of obese males are suggested.

The Big Five Inventory-2 in China: A Comprehensive Psychometric Evaluation in Four Diverse Samples

The Big Five Inventory-2 (BFI-2) has received wide recognition since its publication because it strikes a good balance between content coverage and brevity. The current study translated the BFI-2 into Chinese, evaluated its psychometric properties in four diverse Chinese samples (college students, adult employees, adults treated for substance use, and adolescents), and compared its factor structure with those obtained from two U.S. samples. Across two studies, the Chinese BFI-2 demonstrated good reliability (Cronbach's α and test-retest reliability), structural validity, convergent/discriminant validity, and criterion-related validity at the domain level. At lower levels of analyses, some facets and negatively worded items functioned better among participants with higher than those with lower education levels. Implications, limitations, and future directions are discussed.

Cross-Cultural Adaptation and Validation of the Physical Disability Resiliency Scale in a Sample of Chinese With Physical Disability

This study adapted the Physical Disability Resilience Scale (PDRS) to Chinese conditions and evaluated the psychometric characteristics of the Chinese version in individuals with physical disability. A total of 438 individuals with physical disability were included in this study. The PDRS was translated to Chinese using a backward translation method. Construct validity, internal consistency reliability, and convergent validity were examined. Confirmatory factor analysis failed to replicate the original five-factor structure of the PDRS. After removing the Spirituality factor and an underperformed item (Item 22), exploratory factor analysis yielded four trait factors (i.e., Emotional and Cognitive Strategies, Physical Activity and Diet, Peer Support, and Support from Family and Friends) and a method-effect factor. A correlated trait-correlated method model that included the four trait factors and a method-effect factor reported better model fit than the four-factor model, which did not consider method effects. The four subscales of the revised PDRS showed adequate internal consistency. The convergent validity of the revised PDRS was established by the moderate-to-strong associations between its four subscales and theoretically related constructs. We conclude that the revised PDRS is a reliable and valid measure in assessing resilience among Chinese people with physical disability.

Validating the Workplace Dignity Scale

Workplace Dignity has long been the subject of scholarly enquiry, although until recently the body of research has been dominated by ethnographic work. Recently, Thomas and Lucas (2019) developed the first quantitative, direct measure of perceptions of workplace dignity: the Workplace Dignity Scale (WDS). Given the importance of understanding dignity in the workplace, this study sought to replicate the initial scale validation study conducted by Thomas and Lucas, so as to further test the validity of the WDS and the reliability of the scores it produces. Moreover, the current study contributes to the ongoing methodological reform of psychology towards a transparent and rigorous science by preregistering the method and analysis script prior to collecting data. A large sample of workers (N = 812) from the United States were recruited through Prolific Academic and completed an online questionnaire that included the WDS, as well as theoretically related scales (e.g., workplace incivility). Confirmatory factor analyses indicated that the model specified by Thomas and Lucas had reasonable global fit, although it did not meet all of our criteria for good fit, and estimates of reliability (ωt) indicated that responses to items making up the two subscales of the WDS, Dignity and Indignity, had high internal consistency. Nomological analyses revealed that the Dignity subscale of the WDS was significantly correlated in the expected directions with theoretically related variables. Furthermore, the Dignity and Indignity factors of the WDS were found to highly correlate with one another, and an exploratory analysis suggested that the Indignity factor might be a methodological artefact, posing questions as to whether the two factors are qualitatively different phenomena as was argued by Thomas and Lucas. It is concluded that the WDS is a promising tool for measuring workplace dignity although refinement of the proposed measurement model may be necessary.

Balanced and positively worded personality short-forms: Mini-IPIP validity and cross-cultural invariance

Background

The Mini-IPIP scales (Donellan et al., 2006) are possibly one of the most commonly used short inventories for measuring the Big Five Factors of personality. In this study, we aimed to investigate the psychometric properties of two Mini-IPIP Spanish short forms, one balanced and one positively wording (PW).

Method

Two samples, one from native Spanish speakers and another from native English speakers, made up a total of 940 participants in this study. The short forms were translated and adapted based on international guidelines. Reliability (internal and composite) and validity analyses (construct ESEM, concurrent, predictive and cross-cultural invariance through multi-group factorial models) were performed.

Results

For both the balanced scale and the PW one, modeling a method factor was not relevant. The reliability and validity indices of both forms were according to theory and prior studies’ findings: (a) personality factors were medium-high related to affective factors; (b) personality factors were less related to life satisfaction than affective factors; (c) life satisfaction was medium-high related to affective factors; (d) neuroticism appeared mainly related to all criteria variables; and (e) an acceptable level of invariance was achieved with regard to the English version.

Discussion

This study contributes to research on personality assessment by providing the first evidence regarding the psychometric properties of a PW short measure. These results suggest that PW short scales of personality used after data screening techniques may be appropriate for future studies (e.g., cross-cultural, content validity).

The structure of the General Health Questionnaire (GHQ-12): two meta-analytic factor analyses

The General Health Questionnaire (GHQ-12) is a popular measure of psychological distress. Despite its widespread use, an ongoing controversy pertains to its internal structure. Although the GHQ-12 was originally constructed to capture a unitary construct, empirical studies identified different factor structures. Therefore, this study examined the dimensionality of the GHQ-12 in two independent meta-analyses. The first meta-analysis used summary data published in 38 primary studies (total N = 76,473). Meta-analytic exploratory factor analyses identified two factors formed by negatively and positively worded items. The second meta-analysis included individual responses of 410,640 participants from 84 independent samples. Meta-analytic confirmatory factor analyses corroborated the two-dimensional structure of the GHQ-12. However, bifactor modelling showed that most of the variance was explained by a general factor. Therefore, subscale scores reflected rather limited unique variance. Overall, the two meta-analyses demonstrated that the GHQ-12 is essentially unidimensional. It is not recommended to use and interpret subscale scores because they primarily reflect general mental health rather than distinct constructs.