When better is the enemy of good: two cautionary tales of conceptual validity versus parsimony in clinical psychometric research

This paper presents an empirical challenge to the assumption that an item-response theory analysis always yields a better measure of a clinical construct. We summarize results from two measurement development studies that showed that such an analysis lost important content reflecting the conceptual model ("conceptual validity"). The cost of parsimony may thus be too high. Conceptual models that form the foundation of QOL measurement reflect the patient's experience. This experience may include concepts and items that are psychometrically "redundant" but capture distinct features of the concept. Good measurement is likely a balance between relying on IRT's quantitative metrics and recognizing the importance of conceptual validity and clinical utility.

An Initial Psychometric Evaluation of a Novel Upper Extremity Pediatric Stroke Hemiplegic Motor Impairment Scale

BACKGROUND

Our team designed an innovative, observation-based motor impairment measure-the Pediatric Stroke Hemiplegic Motor Impairment Scale (Pedi HEMIs). Here we present the results of a survey describing common practices in the pediatric stroke community and the initial psychometric properties of the upper extremity subscale of the Pedi HEMIs (Pedi HEMIs-UE).

METHODS

This is a cross-sectional study whereby participants completed a battery of assessments including the novel Pedi HEMIs-UE. Internal consistency was measured via Cronbach alpha (α). Intraclass correlation (ICC) was used to assess inter-rater reliability (IRR). Concurrent validity was investigated using Pearson or polychoric correlations and simple linear regressions.

RESULTS

The study sample consisted of 18 children aged 1.08 to 15 years. Two participants completed two sets of evaluations, totaling 20 data sets. Cronbach α, a measure of internal consistency, was on average 0.91 (range: 0.89 to 0.92). IRR was excellent with the six raters in almost perfect agreement (ICC = 0.91; 95% confidence interval [CI]: 0.83 to 0.96). Pearson correlation coefficient between the Pedi HEMIs-UE and logit Assisting Hand Assessment (AHA)/mini-AHA was -0.938 (95% CI: -0.979 to -0.827, P < 0.001), indicating excellent concurrent validity.

CONCLUSIONS

We found excellent feasibility, reliability, and validity of the Pedi HEMIs-UE in a convenience sample of youth with hemiparesis after stroke.

Measuring leadership in sport: Development and validation of the Identity Leadership Inventory - Youth (ILI-Y)

The social identity approach to leadership posits that leaders' effectiveness depends on their ability to represent, advance, create, and embed a shared sense of social identity among their followers. Although significant progress has been made in investigating the benefits of identity leadership in adult sports, research in youth sports is still in its infancy. One reason is the lack of a youth-centric inventory that adequately measures identity leadership in this population. To bridge this gap, we developed and validated a long (16 items) and short (5 items) version of the Identity Leadership Inventory for Youth Sport (ILI-Y or ILI-Y-Short-Form) through five studies conducted in three phases of research. Data were primarily collected in football in the United Kingdom, involving a total of 1096 participants. Results of Phase I of this study provided little to no evidence that the ILI - originally developed for adults - was understandable (Study 1) and had factor validity and internal consistency (Study 2) in a sample of youth athletes. Therefore, in Phase II, the ILI was revised, leading to the development of the ILI-Y, which was understandable for youth athletes (Study 3). Results from Phase II (Study 4) also indicated that the ILI-Y exhibited a unidimensional factor structure, which was subsequently confirmed in Phase III (Study 5). This last phase offered additional evidence for the discriminant, criterion, and incremental validity of the ILI-Y and its short form, along with their measurement invariance across genders and age groups, and internal consistency. This study provides sports psychology researchers and practitioners with a valid measure to assess identity leadership in youth sports.

The Perceived Contribution of Older People to Climate Change Impact, Mitigation, and Adaptation: Measurement Development and Validation

Background and Objectives

To improve the understanding of ageism toward older people in the context of climate change, the present study developed and validated a new measure that examines the perceived negative and positive contributions of older people to climate change impact, mitigation, and adaptation efforts.

Research Design and Methods

Four studies (N = 774) were conducted to develop a new measure and evaluate its reliability and validity, relying on exploratory factor analysis, reliability analysis, confirmatory factor analysis, multiple-group analysis (Australia and India), discriminative validity, and convergent and divergent validity.

Results

A 2-subscale measure covering older people's perceived negative contribution to climate change effects (5 items) and perceived positive contribution to adaptation and mitigation measures (3 items; eg, negative, and positive ageism in the context of climate change) was supported by the data. The measure has demonstrated adequate validity and reliability.

Discussion and Implications

The measure highlights a relatively neglected area in current climate change discourse and may assist in identifying ways to improve intergenerational solidarity as part of climate change adaptation and mitigation efforts toward building a world for all ages under a healthy climate, which allows for healthy aging and healthy longevity. These objectives are in line with the current mission posed by the UN Decade of Healthy Ageing.

New measures to assess the "Other" three pillars of food security-availability, utilization, and stability

BACKGROUND

In recent reviews of available measures, no existing measures assessed all four pillars of food security and most only assessed one or two pillars-predominantly the access pillar. The purpose of this study was to preliminarily develop novel measures of availability, utilization, and stability that are complementary to the USDA's household food security survey measure (HFSSM).

METHODS

A formative phase included an expert advisory group, literature scans, and interviews with individuals experiencing food insecurity. From April-June 2021, the new measures were piloted in five states (California, Florida, Maryland, North Carolina, and Washington). The cross-sectional pilot survey included the new measures (perceived limited availability, utilization barriers, and food insecurity stability), scales and items for validation (e.g., food security, and self-reported dietary and health outcomes), and demographic questions. Exploratory factor analysis was used to assess dimensionality, internal consistency was assessed using Kuder-Richardson formula 21 (KR21), and convergent and discriminant validity were assessed using Spearman's correlation coefficients. Also, a brief screener version was created for the utilization barriers measure that may be necessary for certain applications (e.g., clinical intake screening to inform referrals to assistance programs).

RESULTS

The analytic samples (perceived limited availability (n = 334); utilization barriers (n = 428); food insecurity stability (n = 445)) were around 45 years old on average, most households had children, over two-thirds were food insecure, over three-fourths were women, and the samples were racially/ethnically diverse. All items loaded highly and unambiguously to a factor (factor loadings range 0.525-0.903). Food insecurity stability showed a four-factor structure, utilization barriers showed a two-factor structure, and perceived limited availability showed a two-factor structure. KR21 metrics ranged from 0.72 to 0.84. Higher scores for the new measures were generally associated with increased food insecurity (rhos = 0.248-0.497), except for one of the food insecurity stability scores. Also, several of the measures were associated with statistically significantly worse health and dietary outcomes.

CONCLUSIONS

The findings support the reliability and construct validity of these new measures within a largely low-income and food insecure sample of households in the United States. Following further testing, such as Confirmatory Factor Analysis in future samples, these measures may be used in various applications to promote a more comprehensive understanding of the food insecurity experience. Such work can help inform novel intervention approaches to address food insecurity more fully.

Test anxiety in online exams: scale development and validity

Considering the increasing use of online tests, this study aims to develop an up-to-date and reliable scale to measure university students' online test anxiety. This study was designed by using mixed research model by combining qualitative and quantitative research methods together. The study consisted of four stages: planning, structuring, quantitative evaluation, reliability and validation. While in the first phase an extensive literature review was conducted, students' opinions were obtained to create an item pool in the second phase. In the third phase, the 29-item scale was administered to 442 university students for factor and reliability analysis. A total of nine items were dropped out from the pool. The Cronbach's alpha value was .98. Exploratory factor analysis revealed that the items loaded on two factors: the psychological and physiological anxiety factor (α = .95), the technical anxiety factor (α = .89). The two-factor solution accounted for more than 63% of the total variance. The final version of the scale was administered to 387 university students for confirmatory factor analysis in the fourth stage. The results proved that the scale had two factors and the fit indices were at an acceptable level. The reliability analysis was run and Cronbach's alpha values were .94 the whole scale, .93 for the psychological and physiological anxiety factor, and .90 for the technical anxiety factor. According to the result, it was concluded that the Test Anxiety Scale for Online Exams is a reliable and valid measurement tool in determining university students' online test anxiety. Finally, recommendations for future research are provided.

Fear of illness & virus evaluation (FIVE) COVID-19 scales for children-parent/caregiver-report development and validation

OBJECTIVE

Commonly-used youth anxiety measures may not comprehensively capture fears, worries, and experiences related to the pervasive impact of the COVID-19 pandemic. This study described the development of the Fear of Illness and Virus Evaluation (FIVE) scales and validated the caregiver-report version.

METHOD

After initial development, feedback was obtained from clinicians and researchers, who provided suggestions on item content/wording, reviewed edits, and provided support for the updated FIVE's content and face validity. Factor structure, measurement invariance, and psychometric properties were analyzed using data from a multi-site, longitudinal study of COVID-19-related effects on family functioning with 1599 caregivers from the United States and Canada.

RESULTS

Confirmatory factor analyses indicated a hierarchical five-factor structure best fit the data, resulting in a 31-item measure with four lower-order subscales: (1) Fears about Contamination and Illness; (2) Fears about Social Distancing, (3) Avoidance Behaviors, and (4) Mitigation Behaviors, and a higher-order factor, (5) Total Fears, indicated by the two fear-related lower-order subscales. Measurement invariance by country of residence, child age, and child sex was found. All subscales demonstrated strong internal consistency, appropriate item-scale discrimination, and no floor or ceiling effects. The Total Fears subscale demonstrated appropriate test-retest reliability. Concurrent validity supported by strong correlation with a youth anxiety measure.

DISCUSSION

The FIVE provides a psychometrically-sound measure of COVID-19-related fears and behaviors in youth in a caregiver-report format. Future research is necessary to evaluate correlates and longitudinal symptom patterns captured by the FIVE caregiver-report, as well as the validity and reliability of a youth self-report version of the FIVE.

Development of a support for Black Lives Matter measure among racially-ethnically diverse college students

This paper developed and validated a new measure of support for the Black Lives Matter (BLM) movement among a racially-ethnically diverse sample of college students. The measure focuses on the movement's principles of Black liberation, intersectionality, and alliance building. Participants included 1934 college students (75% female) from a large public Southwestern university. The factor structure was supported by exploratory and confirmatory factor analyses, resulting in an 18-item measure, Support for Black Lives Matter, with two underlying factors. Black Liberation includes 12 items representing support for BLM because of awareness of and challenging structural inequality and racism experienced by Black individuals. Intersectional Values includes six items representing support for BLM because it embraces and affirms marginalized populations within the Black community, especially disabled Blacks, queer Blacks, Black women, and Black families with children. Evidence of criterion-related validity was demonstrated with racial group differences in support of BLM factors. Evidence of convergent validity was supported by significant positive correlations between support for BLM factors and critical consciousness (including awareness of racism, classism, and heterosexism), and negative correlations between support for BLM factors and subtle racist attitudes toward Blacks. Measurement invariance was evident between White, Black, Asian American, Latinx, and Multiracial participants. Implications and suggestions for use of the new measure are discussed.

The Clinical Sustainability Assessment Tool: measuring organizational capacity to promote sustainability in healthcare

Background

Few validated assessment tools are available to increase understanding and measure factors associated with sustainment of clinical practices, an increasingly recognized need among clinicians. We describe the development of the Clinical Sustainability Assessment Tool (CSAT), designed to assess factors that contribute to sustainable practices in clinical settings.

Methods

Sixty-four participants from clinical and research fields participated in concept mapping and were recruited to brainstorm factors that lead to sustained clinical practices. Once repeated factors were removed, participants sorted items based on similarity and rated them by importance and feasibility. Using concept mapping analyses, items were grouped into meaningful domains to develop an initial tool. We then recruited pilot sites and early adopters, for a total of 286 practicing clinicians, to pilot and evaluate the tool. Individuals were recruited from clinical settings across pediatric and adult medical and surgical subspecialties. The data were analyzed using confirmatory factor analysis (CFA) to test hypothesized subscale structure in the instrument. We used root mean square error of approximation (RMSEA) and the standardized root mean square residual (SRMR) to assess fit and thus the ability of CSAT to measure the identified domains.

Results

The concept mapping produced sorted statements that were edited into items that could be responded to, resulting in the creation of a tool with seven determinant domains and 47 items. The pilot and CFA testing resulted in a final CSAT instrument made up 35 items, five per domain. CFA results demonstrated very good fit of the seven domain structure of the CSAT (RMSEA = 0.049; SRMR = 0.049). Usability testing indicated the CSAT is brief, easy to use, easy to learn, and does not require extensive training. Additionally, the measure scored highly (18/20) on the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). The seven final CSAT domains were engaged staff and leadership, engaged stakeholders, organizational readiness, workflow integration, implementation and training, monitoring and evaluation, and outcomes and effectiveness.

Conclusions

The CSAT is a new reliable assessment tool which allows for greater practical and scientific understanding of contextual factors that enable sustainable clinical practices over time.

Supplementary Information

The online version contains supplementary material available at 10.1186/s43058-021-00181-2.

Development and initial validation of the Spanish form of the adjustment scales for preschool intervention (ASPI)

The purpose of the study was to develop and validate a Spanish language form of the Adjustment Scales for Preschool Intervention (ASPI) for use by early childhood teachers. A multi-step, mixed method adaptation and translation process of the English form was used to increase the cultural relevance, linguistic comparability, and metric equivalence of the measure in its Spanish language form. Exploratory and confirmatory factor analyses in a diverse sample of preschool children from low-income families (N = 4077) revealed two higher order behavioral dimensions of Overactivity and Underactivity. ASPI dimension scores across Spanish and English language forms were horizontally equated through IRT. Criterion-related validity using multilevel models was established through concurrent associations with social, emotional, behavioral, and academic readiness skills. Policy and practice applications, and future directions for the use of scale scores are discussed.

Development and validation of a scale to measure diabetes burnout

Introduction

The aim of this study is to establish reliability and validity of the Diabetes Burnout Scale (DBS) among adults with type 1 diabetes (T1D).

Methods

We used a multi-stage, mixed methods approach to developing the DBS. First, the research team identified twenty-eight candidate items through a review of the literature and 117 qualitative narratives from adults living with T1D. Next, items were revised based on the expert (n = 20) and individual with T1D (n = 10) feedback. The resulting 18-item DBS measure along with validated measures of diabetes distress, depressive symptoms, and questions related to diabetes outcomes (i.e., last reported hemoglobin A1c [HbA1c] and Time-in-Range [TiR]) were completed by 1099 adults with T1D across the U.S. The sample was randomly divided into two subsets (n1 = 561, n2 = 538) for exploratory and confirmatory factor analyses (EFA/CFA) to determine the underlying structure of the DBS. Regression analyses examined the relationships of the DBS with self-reported glycemic control and socio-demographic characteristics.

Results

Based on the EFA three factors are identified and the DBS demonstrated strong internal consistency with Cronbach alphas (≥0.80). The validation and confirmatory analysis for the structure of the DBS provided consistent results with EFA. Higher burnout (overall DBS) was positively associated with diabetes distress (b = 0.74; p < 0.01) and depressive symptoms (b = 0.61; p < 0.01). Overall DBS, however was the strongest predictor for poorer HbA1c (r2 = 0.19; p < 0.01) and lower TiR (r2 = 0.17; p < 0.01) compared to diabetes distress and depressive symptoms.

Conclusion

The 12-item DBS is a reliable and valid scale to measure diabetes burnout in adults with diabetes. The results provide a weak to strong degree of association between the validated DBS scale, T1-DDS and PHQ-8. The DBS can contribute to advancement of diabetes science by measuring diabetes burnout and informing clinical interventions to improve psychosocial care in individuals with diabetes.

Developing the Person-Place Fit Measure for Older Adults: Broadening Place Domains

BACKGROUND AND OBJECTIVES

Literature calls for larger studies with empirically tested instruments about the meaning of place for older adults-studies that include a broader range of domains and neighborhood characteristics. Although rich narrative and qualitative data for small groups of older individuals exist, a measure with valid and reliable scores that includes the new, multiple domains about aging in place does not.

RESEARCH DESIGN AND METHODS

Findings are reported from a two-phase, nine-step, exploratory sequential mixed-methods process of measurement development for the Person-Place Fit Measure for Older Adults (PPFM-OA). In Phase I, a focus group (n = 8) and qualitative interviews (n = 77) with persons 65 and older were used to develop emerging domains of aging in place. Qualitative data about concepts and language informed the development of a quantitative item pool for the Mechanical Turk-distributed survey. Phase II included a Delphi process reducing the number of items in the PPFM-OA. Exploratory factor analysis (EFA) and reliability analysis further reduced the number of potential measure items.

RESULTS

Five factors emerged from the EFA. They were primary or basic needs/necessities (α = .84), neighborhood changes and moving (α = .88), identity and place attachment (α = .86), community value (α = .89), and services and resources (α = .78).

DISCUSSION AND IMPLICATIONS

The development of a measure, such as the PPFM-OA, is crucial as more programs and services are created to address aging in place but uniform data for planning and evaluation are lacking. These initial quantitative analyses are informative for the next step, a larger-scale, quantitative evaluation.

A New Multidimensional Questionnaire to Assess Awareness of Age-Related Change (AARC)

BACKGROUND AND OBJECTIVES

This two-study article describes the development and evaluation of a multidimensional questionnaire based on the subjective aging construct Awareness of Age-Related Change (AARC). AARC captures the inherent multidimensionality and complexity of aging attitudes, which are strongly linked to indicators of successful aging, including health and well-being.

RESEARCH DESIGN AND METHODS

In Study 1, we generated a large item pool related to subjective aging experiences and then evaluated the psychometric properties of a 189-item version of the AARC questionnaire in a sample of 396 adults aged 40-95 years. Based on findings from Study 1, we retained the best-performing items and arrived at a more parsimonious 50-item version (AARC-50). In Study 2, the psychometric properties of the 50-item version were examined in an expanded sample of 424 adults ages 40-98.

RESULTS

Factor analyses in Study 1 indicated a two-factor structure of the questionnaire, representing the awareness of positive (AARC-Gains) and negative (AARC-Losses) age-related changes across five behavioral domains. Confirmatory factor analysis in Study 2 further supported this two-factor structure. In both studies, the AARC questionnaire demonstrated strong psychometric properties, including scale and item reliability, convergent and divergent validity, and predictive validity.

DISCUSSION AND IMPLICATIONS

The availability of a reliable and valid assessment tool for measuring AARC-Gains and AARC-Losses allows researchers to capture detailed information about adults' positive and negative self-perceptions of aging across multiple behavioral domains, which are instrumental for promoting successful aging.

A qualitative study investigating the meaning of participation to improve the measurement of this construct

PURPOSE

The purpose of this study was to improve the measurement of participation. Research questions were as follows: (1) What constitutes participation according to adults? (2) Do they mention participation subdomains that are not covered in the Patient-Reported Outcomes Measurement Information System (PROMIS) item bank "Ability to Participate in Social Roles and Activities"?

METHODS

Semi-structured interviews were conducted with 46 adults from the general population. Interviews were thematically analysed using the International Classification of Functioning, Disability and Health (ICF) as conceptual framework. Thereafter, assigned codes were compared to PROMIS item bank.

RESULTS

Participants mentioned a variety of participation subdomains that were meaningful to them, such as socializing and employment. All subdomains could be classified into the ICF. The following subdomains were not covered by the PROMIS item bank: acquisition of necessities, education life, economic life, community life, and religion and spirituality. Also a distinction between remunerative (i.e. paid) and non-remunerative (i.e. unpaid) employment, and domestic life was missing. Several ICF sub-codes were not mentioned, such as ceremonies.

CONCLUSIONS

Many participation subdomains were mentioned to be meaningful. As several of these subdomains are not covered in the PROMIS item bank, it may benefit from extension with new (patient-)reported subdomains of participation.

Using spontaneous commentary of nursing home residents to develop resident-centered measurement tools: A case study

Nursing home (NH) residents routinely complete surveys that assess their health, well-being, preferences, and care needs. Such surveys reveal important information, however, are largely based on the concerns of providers as opposed to the concerns of residents. Thus, researchers must enhance efforts to ensure that these surveys are guided by the priorities, needs, and concerns of residents. We present a case study to demonstrate how spontaneous commentary of NH residents holds particular efficacy for ensuring that measurement tools are guided by the needs, concerns, and priorities of residents. Spontaneous comments from NH residents (N = 370) collected as part of a study developing the Preferences for Everyday Living Inventory for NH residents (PELI-NH) were used to refine the PELI-NH across key phases of measurement development. This work demonstrates how the spontaneous commentary of NH residents may contribute to the refinement of NH measurement tools, and allow researchers to base these tools on the needs and priorities of NH residents.

Infant Movement Motivation Questionnaire: development of a measure evaluating infant characteristics relating to motor development in the first year of life

This paper highlights the development and testing of the Infant Movement Motivation Questionnaire (IMMQ), an instrument designed to evaluate qualities of infant characteristics that relate specifically to early motor development. The measurement development process included three phases: item generation, pilot testing and evaluation of acceptability and feasibility for parents and exploratory factor analysis. The resultant 27-item questionnaire is designed for completion by parents and contains four factors including Activity, Exploration, Motivation and Adaptability. Overall, the internal consistency of the IMMQ is 0.89 (Cronbach's alpha), with test-retest reliability measured at 0.92 (ICC, with 95% CI 0.83-0.96). Further work could be done to strengthen the individual factors; however it is adequate for use in its full form. The IMMQ can be used for clinical or research purposes, as well as an educational tool for parents.

Combining agreement and frequency rating scales to optimize psychometrics in measuring behavioral health functioning

OBJECTIVE

The goal of this article was to investigate optimal functioning of using frequency vs. agreement rating scales in two subdomains of the newly developed Work Disability Functional Assessment Battery: the Mood & Emotions and Behavioral Control scales.

STUDY DESIGN AND SETTING

A psychometric study comparing rating scale performance embedded in a cross-sectional survey used for developing a new instrument to measure behavioral health functioning among adults applying for disability benefits in the United States was performed.

RESULTS

Within the sample of 1,017 respondents, the range of response category endorsement was similar for both frequency and agreement item types for both scales. There were fewer missing values in the frequency items than the agreement items. Both frequency and agreement items showed acceptable reliability. The frequency items demonstrated optimal effectiveness around the mean ± 1-2 standard deviation score range; the agreement items performed better at the extreme score ranges.

CONCLUSION

Findings suggest an optimal response format requires a mix of both agreement-based and frequency-based items. Frequency items perform better in the normal range of responses, capturing specific behaviors, reactions, or situations that may elicit a specific response. Agreement items do better for those whose scores are more extreme and capture subjective content related to general attitudes, behaviors, or feelings of work-related behavioral health functioning.