What are the minimal sample size requirements for Mokken scaling? An empirical example with the Warwick- Edinburgh Mental Well-Being Scale

An investigation into the factor structure of the Health of the Nation Outcome Scales for People with Learning Disabilities

BACKGROUND

The Health of the Nation Outcome Scales for People with Learning Disabilities (HoNOS-LD) is one of the most used outcome measures in learning disability services in the United Kingdom. There is relatively little known of the psychometric properties of the scales.

METHOD

A data set of HoNOS-LD scales from 571 people with learning disabilities was randomly split into two halves. Exploratory Mokken analysis was applied to the first dataset, and confirmatory scale factor analysis was applied to the second dataset to test the fit of scale structures.

RESULTS

Two-factor and three-factor solutions were explored in the Mokken analysis, with the three-factor option having somewhat better characteristics. One-factor, three-factor and seven-factor solutions were explored using confirmatory factor analysis; a three-factor solution with items 8, 16, 17 and 18 used separately offers the best characteristics.

CONCLUSIONS

The HoNOS-LD is best conceptualised as consisting of three scales, accounting for 14 items that can be labelled as 'Cognitive and Physical Functioning', 'Behaviour and Mood Disturbances' and 'Functional Difficulties'.

Assessment of the Psychometric Characteristics of the Italian Version of the Nurse Manager Actions Scale

Nurse managers play a vital role in healthcare organizations, wielding the ability to substantially enhance work environments, foster nurses' autonomy, and bolster retention within workplaces. In this context, this study focuses on the Nurse Manager Actions scale, aiming to evaluate its items' scalability as well as the scale's validity and reliability among nurses and nurse managers operating within the Italian healthcare context. The study protocol was not registered. To ensure linguistic and cultural alignment, an iterative and collaborative translation process was undertaken. Subsequently, a multi-center cross-sectional design was adopted. Using a web-survey approach, data were collected among 683 nurses and 188 nurse managers between August 2022 and January 2023. The Nurse Manager Actions scale was found to be a valid and reliable instrument in Italian after a Mokken Scale Analysis. For nurses (HT= 0.630, Molenaar-Sijtsma rho = 0.890), the scale included 6 items, while 11 items were confirmed for nurse managers (HT= 0.620, Molenaar-Sijtsma rho = 0.830). Nurse Manager Actions scale scores were correlated with increased satisfaction and decreased intention to leave for both nurses and nurse managers. The employed validation process enhanced the scale validity for use in Italy and provided a model for other researchers to follow when assessing similar measures in different populations. Measuring and empowering nurse manager actions in work contexts is essential to improve the general well-being and retention of nurses, especially in the current nursing shortage.

Assessment of differential item functioning of the PHQ-9, HADS-D and PROMIS-depression scales in persons with and without multiple sclerosis

OBJECTIVE

We tested for the presence of differential item functioning (DIF) in commonly used measures of depressive symptoms, in people with multiple sclerosis (MS) versus people with a psychiatric disorder without MS.

METHODS

Participants included individuals with MS, or with a lifetime history of a depressive or anxiety disorder (Dep/Anx) but no immune-mediated inflammatory disease. Participants completed the Patient Health Questionnaire (PHQ-9), Hospital Anxiety and Depression Scale (HADS), and the Patient Reported Outcome Measurement Information System (PROMIS)-Depression. We assessed unidimensionality of the measures using factor analysis. We evaluated DIF using logistic regression, with and without adjustment for age, gender and body mass index (BMI).

RESULTS

We included 555 participants (MS: 252, Dep/Anx: 303). Factor analysis showed that each depression symptom measure had acceptable evidence of unidimensionality. In unadjusted analyses comparing the MS versus Dep/Anx groups we identified multiple items with evidence of DIF, but few items showed DIF effects that were large enough to be clinically meaningful. We observed non-uniform DIF for one PHQ-9 item, and three HADS-D items. We also observed DIF with respect to gender (one HADS-D item), and BMI (one PHQ-9 item). For the MS versus Dep/Anx groups, we no longer observed DIF post-adjustment for age, gender and BMI. On unadjusted and adjusted analyses, we did not observe DIF for any PROMIS-D item.

CONCLUSION

Our findings suggest that DIF exists for the PHQ-9 and HADS-D with respect to gender and BMI in clinical samples that include people with MS whereas DIF was not observed for the PROMIS-Depression scale.

The effects of advanced factor analysis approaches on outcomes in randomised trials for depression: protocol for secondary analysis of individual participant data

BACKGROUND

Modern psychometric methods make it possible to eliminate nonperforming items and reduce measurement error. Application of these methods to existing outcome measures can reduce variability in scores, and may increase treatment effect sizes in depression treatment trials.

AIMS

We aim to determine whether using confirmatory factor analysis techniques can provide better estimates of the true effects of treatments, by conducting secondary analyses of individual patient data from randomised trials of antidepressant therapies.

METHOD

We will access individual patient data from antidepressant treatment trials through Clinicalstudydatarequest.com and Vivli.org, specifically targeting studies that used the Hamilton Rating Scale for Depression (HRSD) as the outcome measure. Exploratory and confirmatory factor analytic approaches will be used to determine pre-treatment (baseline) and post-treatment models of depression, in terms of the number of factors and weighted scores of each item. Differences in the derived factor scores between baseline and outcome measurements will yield an effect size for factor-informed depression change. The difference between the factor-informed effect size and each original trial effect size, calculated with total HRSD-17 scores, will be determined, and the differences modelled with meta-analytic approaches. Risk differences for proportions of patients who achieved remission will also be evaluated. Furthermore, measurement invariance methods will be used to assess potential gender differences.

CONCLUSIONS

Our approach will determine whether adopting advanced psychometric analyses can improve precision and better estimate effect sizes in antidepressant treatment trials. The proposed methods could have implications for future trials and other types of studies that use patient-reported outcome measures.

Pathways for primary care practice adoption of patient engagement strategies

OBJECTIVE

To identify potential orderings of primary care practice adoption of patient engagement strategies overall and separately for interpersonally and technologically oriented strategies.

DATA SOURCES

We analyzed physician practice survey data (n = 71) on the adoption of 12 patient engagement strategies.

STUDY DESIGN

Mokken scale analysis was used to assess latent traits among the patient engagement strategies.

DATA COLLECTION

Three groupings of patient engagement strategies were analyzed: (1) all 12 patient engagement strategies, (2) six interpersonally oriented strategies, and (3) six technologically oriented strategies.

PRINCIPAL FINDINGS

We did not find scalability among all 12 patient engagement strategies, however, separately analyzing the subgroups of six interpersonally and six technologically oriented strategies demonstrated scalability (Loevinger's H coefficient of scalability [range]: interpersonal strategies, H = 0.54 [0.49-0.60], technological strategies, H = 0.42 [0.31, 0.54]). Ordered patterns emerged in the adoption of strategies for both interpersonal and technological types.

CONCLUSIONS

Common pathways of practice adoption of patient engagement strategies were identified. Implementing interpersonally intensive patient engagement strategies may require different physician practice capabilities than technological strategies. Rather than simultaneously adopting multiple patient engagement strategies, gradual and purposeful practice adoption may improve the impact of these strategies and support sustainability.

Identifying Problematic Item Characteristics With Small Samples Using Mokken Scale Analysis

Researchers frequently use Mokken scale analysis (MSA), which is a nonparametric approach to item response theory, when they have relatively small samples of examinees. Researchers have provided some guidance regarding the minimum sample size for applications of MSA under various conditions. However, these studies have not focused on item-level measurement problems, such as violations of monotonicity or invariant item ordering (IIO). Moreover, these studies have focused on problems that occur for a complete sample of examinees. The current study uses a simulation study to consider the sensitivity of MSA item analysis procedures to problematic item characteristics that occur within limited ranges of the latent variable. Results generally support the use of MSA with small samples (N around 100 examinees) as long as multiple indicators of item quality are considered.

Parametric and Nonparametric Analysis of the Internal Structure of the Psychosocial Work Processes Questionnaire (PROPSIT) as Applied to Workers

The study of the dimensionality or internal structure of a measure has a definitional purpose with notable theoretical and practical implications; this aspect can be analyzed via both parametric and nonparametric approaches. The latter are probably used less often to validate constructs in the context of psychosocial work factors. The aim of the present manuscript was to employ both nonparametric (DETECT and AISP-Mokken) and parametric (semiconfirmatory factor analysis) procedures to analyze the internal structure of the Psychosocial Work Processes Questionnaire (PROPSIT) in the context of two samples of Peruvian workers located in the city of Lima, Perú, with one sample drawn from various work centers (n = 201) and the other comprising elementary education teachers (n = 158). The nonparametric results indicated that the content of the PROPSIT is sufficiently multidimensional to be able to describe a variety of psychosocial factors, while the parametric results require modification of the measurement model to obtain greater factorial congruence. In general, the analyses show a similar structure to those discussed by previous preliminary studies that have reported similar item-level performances. Some findings and considerations for future research are discussed.

Validation of the Connor-Davidson Resilience Scale-10 in South Africa: Item Response Theory and Classical Test Theory

Purpose

During times of crisis, and in particular during the current COVID-19 pandemic, resistance resources, such as resilience, are key to managing the negative impacts of the crisis. The Connor-Davidson Resilience Scale-10 has been extensively used to assess resilience. In the current study, we examined the reliability, validity and dimensionality of this instrument using Rasch analysis, Mokken analysis and classical test theory. Using both item response theory and classical test theory provides a more comprehensive overview of the psychometric properties of the Connor-Davidson Resilience Scale-10.

Participants and Methods

School teachers from South Africa (N = 355) constituted the sample for the study and they completed the following scales: Connor-Davidson Resilience Scale-10, the Center for Epidemiological Studies Depression Scale, State-Trait Anxiety Inventory-Trait Scale, the Beck Hopelessness Scale, and the Sense of Coherence Scale.

Results

Both item response theory and classical test theory provided strong evidence for the reliability and validity of the Connor-Davidson Resilience Scale-10. In addition, the analyses provided support for considering the scale as essentially unidimensional.

Conclusion

The obtained results provide additional support for the reliability and validity, of the Connor-Davidson Resilience Scale-10 for samples from diverse contexts.

Assessing medical mistrust in organ donation across countries using item response theory

Although medical mistrust (MM) may be an impediment to public health interventions, no MM scale has been validated across countries and the assessment of MM has not been explored using item response theory, which allows generalisation beyond the sampled data. We aimed to determine the dimensionality of a brief MM measure across four countries through Mokken analysis and Graded Response Modelling. Analysis of 1468 participants from UK (n = 1179), Ireland (n = 191), India (n = 49) and Malaysia (n = 49) demonstrated that MM items formed a hierarchical, unidimensional measure, which is very informative about high levels of MM. Possible item reduction and scoring changes were also demonstrated. This study demonstrates that this brief MM measure is suitable for international studies as it is unidimensional across countries, cross cultural, and shows that minor adjustments will not impact on the assessment of MM when using these items.

Construct Validity and Test-Retest Reliability of the Automated Vehicle User Perception Survey

Fully automated vehicles (AVs) hold promise toward providing numerous societal benefits including reducing road fatalities. However, we are uncertain about how individuals' perceptions will influence their ability to accept and adopt AVs. The 28-item Automated Vehicle User Perception Survey (AVUPS) is a visual analog scale that was previously constructed, with established face and content validity, to assess individuals' perceptions of AVs. In this study, we examined construct validity, via exploratory factor analysis and subsequent Mokken scale analyses. Next, internal consistency was assessed via Cronbach's alpha (α) and 2-week test-retest reliability was assessed via Spearman's rho (ρ) and intraclass correlation coefficient (ICC). The Mokken scale analyses resulted in a refined 20-item AVUPS and three Mokken subscales assessing specific domains of adults' perceptions of AVs: (a) Intention to use; (b) perceived barriers; and (c) well-being. The Mokken scale analysis showed that all item-coefficients of homogeneity (H) exceeded 0.3, indicating that the items reflect a single latent variable. The AVUPS indicated a strong Mokken scale (H scale = 0.51) with excellent internal consistency (α = 0.95) and test-retest reliability (ρ = 0.76, ICC = 0.95). Similarly, the three Mokken subscales ranged from moderate to strong (range H scale = 0.47-0.66) and had excellent internal consistency (range α = 0.84-0.94) and test-retest reliability (range ICC = 0.84-0.93). The AVUPS and three Mokken subscales of AV acceptance were validated in a moderate sample size (N = 312) of adults living in the United States. Two-week test-retest reliability was established using a subset of Amazon Mechanical Turk participants (N = 84). The AVUPS, or any combination of the three subscales, can be used to validly and reliably assess adults' perceptions before and after being exposed to AVs. The AVUPS can be used to quantify adults' acceptance of fully AVs.

A Mokken scale analysis of the Kessler-6 screening measure among Chinese older population: findings from a national survey

BACKGROUND

The aging population increases rapidly across the world. Timely and effective screening of their mental-health problems is important to individuals, families, and the whole society. The Kessler-6 screening measure (K6) is a very popular instrument for non-specific psychological distress. However, few studies have focused on the psychometric properties of this instrument in the older population.

METHODS

The present study employed Mokken scale analysis to evaluate its dimensionality and structure. This study also used differential item functioning (DIF) to examine whether the same structure existed across sex in a national representative sample of old Chinese people. Data were drawn from a public data set, the 2010 China Family Panel Studies (CFPS2010), and responses from a total of 6450 participants aged 60 years old and above (3136 males and 3314 females) were included in the final analysis.

RESULTS

Mokken scale analysis supported the unidimensional structure of the K6. Differential item functioning (DIF) analysis revealed that two of the six items ("Hopeless" and "Everything was an effort") were marked for DIF based on the Chi-square. However, their impacts were negligible in terms of McFadden's pseudo R2.

CONCLUSIONS

The K6 demonstrates adequate psychometric properties in the old Chinese population. The sum of all six items can be used as an indicator of non-specific psychological distress. Differences in the indicator across sex should be considered as a real difference in psychological distress between the female and the male.

Should the PBL tutor be present? A cross-sectional study of group effectiveness in synchronous and asynchronous settings

BACKGROUND

The tutorial group and its dynamics are a cornerstone of problem-based learning (PBL). The tutor's support varies according to the setting, and it is pertinent to explore group effectiveness in relation to different settings, for example online or campus-based. The PBL groups' effectiveness can partly be assessed in terms of cognitive and motivational aspects, using a self-report tool to measure PBL group effectiveness, the Tutorial Group Effectiveness Instrument (TGEI). This study's aim was to explore tutor participation in variations of online and campus-based tutorial groups in relation to group effectiveness. A secondary aim was to validate a tool for assessing tutorial group effectiveness in a Swedish context.

METHODS

A cross-sectional study was conducted with advanced-level nursing students studying to become specialised nurses or midwives at a Swedish university. The TGEI was used to measure motivational and cognitive aspects in addition to overall group effectiveness. The instrument's items were translated into Swedish and refined with an expert group and students. The responses were calculated descriptively and compared between groups using the Mann-Whitney U and Kruskal-Wallis tests. A psychometric evaluation was performed using the Mokken scale analysis. The subscale scores were compared between three different tutor settings: the tutor present face-to-face in the room, the tutor present online and the consultant tutor not present in the room and giving support asynchronously.

RESULTS

All the invited students (n = 221) participated in the study. There were no differences in motivational or cognitive aspects between students with or without prior PBL experience, nor between men and women. Higher scores were identified on cognitive aspects (22.6, 24.6 and 21.3; p < 0.001), motivational aspects (26.3, 27 and 24.5; p = 002) and group effectiveness (4.1, 4.3, 3.8, p = 0.02) for the two synchronously tutored groups compared to the asynchronously tutored group. The TGEI subscales showed adequate homogeneity.

CONCLUSIONS

The tutor's presence is productive for PBL group effectiveness. However, the tutor need not be in the actual room but can provide support in online settings as long as the tutoring is synchronous.

Conceptualising and operationalising resilience in older adults

Context: As a result of increases in life expectancy and decreases in fertility, the proportion of the population entering later life has increased dramatically in recent decades. When faced with age-related challenges, some older adults respond more positively to adversity than would be expected given the level of adversity that they have experienced, demonstrating 'resilience'. Objectives: Having a clear conceptual framework for resilience is a prerequisite to operationalising resilience in a research context. Methods: Here we compare and contrast several approaches to the operationalisation of resilience: psychometric-driven and data-driven (variable-centred and individual-centred) methods. Results: Psychometric-driven methods involve the administration of established questionnaires aimed at quantifying resilience. Data-driven techniques use statistical procedures to examine and/or operationalise resilience and can be broadly categorised into variable-centred methods, i.e. interaction and residuals, and individual-centred methods, i.e. categorical and latent class. Conclusions: The specific question(s) driving the research and the nature of the variables a researcher intends to use in their adversity-outcome dyad will largely dictate which methods are more (or less) appropriate in that circumstance. A measured approach to the ways in which resilience is investigated is warranted in order to facilitate the most useful application of this burgeoning field of research.