Methods to develop and evaluate attitudinal scales to measure farmer perceptions: Using sheep scab as an example

There is limited information on the attitudes and opinions of British sheep farmers when preventing and managing sheep scab in their flocks. Attitudes are examples of latent traits, which are not directly observable. They must be measured indirectly which can be achieved using measurement scales. Attitudinal measurement scales are used frequently in research, but are seldom evaluated prior to use, particularly in veterinary research. Therefore, the aim of this study was to identify and assess latent traits which influence the control of sheep scab with the use of exploratory factor analysis, and then further evaluate these factors with the use of item response theory to determine how they perform as scales. An online survey was completed by 726 British sheep farmers during October 2021 (725 useable responses), which included a section with 37 belief statements related to the opinions and attitudes of sheep farmers around the management of sheep scab. These belief statements were developed and adapted from previous research. During the analysis, they were subjected to exploratory factor analysis, which suggested that the scales performed reasonably well. The scales were then evaluated further by item response theory. The graded response models for each attitudinal scale were assessed by graphical evaluation. They demonstrated that the scales were able to distinguish those respondents with low perceptions of control, responsibility, and trust, and those who had high perceptions of luck. This study provides insights into some of the factors influencing the management of sheep scab, and how well these scales perform. Additionally, it further highlights the importance of evaluating attitudinal scales for use in research, and that exploratory factor analysis alone does not provide enough information on how well they perform.

The influence of item order of the Household Food Security Survey Module on the assessment of food insecurity in households with children

OBJECTIVE

Changes in the item order of the US Household Food Security Survey Module (USHFSSM) were performed throughout time. This study aimed to compare the psychometric properties of the general and specific factors of the 2000 and 2012 versions of the USHFSSM to measure the construct of food insecurity in two Portuguese samples of households with children.

DESIGN

Cross-sectional.

SETTING

Portugal.

PARTICIPANTS

An adaptation of the 2000 version was applied to 839 adults (from households with children aged 7-17 years) from the National Food, Nutrition and Physical Activity Survey 2015-2016, while the 2012 version was used among 2855 families from the Generation XXI birth cohort.

RESULTS

The 2000 version showed to have a stronger ωh than the 2012 version (0·89 v. 0·78 for the general factor), as well as eigenvalues higher than 1 for the general factor (eigenvalues equal to 9·54, 0·97 and 0·80, for the general factor, specific factor 1 and specific factor 2, respectively), while the 2012 version had also the contribution of specific factors to explain food insecurity (eigenvalues equal to 9·40, 2·40 and 1·20, for general factor and specific factors 1 and 2, respectively). Good internal consistency (ωt = 0·99, for both versions) was obtained.

CONCLUSIONS

In conclusion, the 2000 and 2012 versions of the USHFSSM showed good psychometric properties; however, the 2000 version has stronger general factor, while the 2012 version also has the contribution of specific factors.

Content, diagnostic, correlational, and genetic similarities between common measures of childhood aggressive behaviors and related psychiatric traits

BACKGROUND

Given the role of childhood aggressive behavior (AGG) in everyday child development, precise and accurate measurement is critical in clinical practice and research. This study aims to quantify agreement among widely used measures of childhood AGG regarding item content, clinical concordance, correlation, and underlying genetic construct.

METHODS

We analyzed data from 1254 Dutch twin pairs (age 8-10 years, 51.1% boys) from a general population sample for whom both parents completed the A-TAC, CBCL, and SDQ at the same occasion.

RESULTS

There was substantial variation in item content among AGG measures, ranging from .00 (i.e., mutually exclusive) to .50 (moderate agreement). Clinical concordance (i.e., do the same children score above a clinical threshold among AGG measures) was very weak to moderate with estimates ranging between .01 and .43 for mother-reports and between .12 and .42 for father-reports. Correlations among scales were weak to strong, ranging from .32 to .70 for mother-reports and from .32 to .64 for father-reports. We found weak to very strong genetic correlations among the measures, with estimates between .65 and .84 for mother-reports and between .30 and .87 for father-reports.

CONCLUSIONS

Our results demonstrated that degree of agreement between measures of AGG depends on the type (i.e., item content, clinical concordance, correlation, genetic correlation) of agreement considered. Because agreement was higher for correlations compared to clinical concordance (i.e., above or below a clinical cutoff), we propose the use of continuous scores to assess AGG, especially for combining data with different measures. Although item content can be different and agreement among observed measures may not be high, the genetic correlations indicate that the underlying genetic liability for childhood AGG is consistent across measures.

Methodological considerations when measuring oral health-related quality of life

BACKGROUND

Dental patient-reported outcomes (dPROs) and their measures, dPROMs, are fundamental for evidence-based dentistry. However when selecting, applying and evaluating an instrument with a focus on OHRQoL assessment for adults, several methodological considerations are essential to derive valid and meaningful results.

METHODS

In this review article, criteria for selecting the appropriate OHRQoL instrument, aspects of administering the instrument and how to evaluate resulting scores of single and multiple assessments are presented and critically assessed.

RESULTS

Oral disease-generic and dimension-generic instruments capturing the entire construct OHRQoL allow for best comparability of findings across different diseases, settings and populations, with the Oral Health Impact Profile (OHIP) being the most often used and methodologically best investigated one. It is available in several versions with the 5-item version being the one with the lowest burden for the patient. Responses are given on a 5-point ordinal rating scale, the recommended response scale for dPROMs. A 7-day recall period allows for assessment of short-term effects. Clinically relevant effects of item or instrument order or administration method on OHIP scores do not seem to be likely. OHIP summary and dimension scores can be compared to norms from general population or different patient populations. Change scores should be set into context with the minimal important difference of the instrument.

CONCLUSION

OHIP-5 has greatest potential to be used across all settings for assessment and evaluation of OHRQoL in adults. It allows a comprehensive characterising of patients suffering from oral diseases and of this impact using OHRQoL dimensions.

Initial Validation of the Parent-Report Child Adjustment and Parent Efficacy Scale (CAPES) in a Chinese Cultural Context

This study aimed to validate a parent-report measure of child adjustment and parenting efficacy (Child Adjustment and Parent Efficacy Scale [CAPES]) in a Chinese cultural context. Six hundred and fifty Chinese parents in China participated in the study and completed a set of questionnaires including the CAPES, the Strengths and Difficulties Questionnaire (SDQ), the Parenting Sense of Competence Scale (PSOC), and the Parenting and Family Adjustment Scales (PAFAS). The factorial, convergent, and concurrent validity as well as internal consistency were evaluated. The results provided support for a new 2-factor structure of child behavioral and emotional adjustment. The 1-factor structure of parenting efficacy was confirmed. The three constructs obtained satisfactory internal consistency. Moreover, the CAPES scores correlated significantly with the SDQ, PSOC, and PAFAS in the expected pattern, which indicated its excellent convergent and concurrent validity. Therefore, it is a promising measure, due to its brevity and comprehensiveness, with the first indications of satisfactory internal consistency and validity among Chinese parents. However, a revision of the instrument is needed to avoid the ordering effects of positive and negative items, and further hypothesis-driven research is also needed to validate the (revised) instrument in different Chinese samples.

Investigating diagnostic bias in autism spectrum conditions: An item response theory analysis of sex bias in the AQ-10

Diagnostic bias is a concern in autism spectrum conditions (ASC) where prevalence and presentation differ by sex. To ensure that females with ASC are not under-identified, it is important that ASC screening tools do not systematically underestimate autistic traits in females relative to males. We evaluated whether the AQ-10, a brief screen for ASC recommended by the National Institute of Clinical Excellence in cases of suspected ASC, exhibits such a bias. Using an item response theory approach, we evaluated differential item functioning and differential test functioning. We found that although individual items showed some sex bias, these biases at times favored males and at other times favored females. Thus, at the level of test scores the item-level biases cancelled out to give an unbiased overall score. Results support the continued use of the AQ-10 sum score in its current form; however, suggest that caution should be exercised when interpreting responses to individual items. The nature of the item level biases could serve as a guide for future research into how ASC affects males and females differently. Autism Res 2017, 10: 790-800. © 2016 International Society for Autism Research, Wiley Periodicals, Inc.

Validierung einer deutschen Version des Ruminative Thought Style Questionnaire (RTS-D)

Zusammenfassung. Theoretischer Hintergrund: Der Ruminative Thought Style Questionnaire (RTS) wurde entwickelt, um einen ruminativen Denkstil unabhängig von Inhalt, zeitlicher Orientierung und Valenz der Gedanken zu erfassen. Fragestellung: Überprüfung von Reliabilität und Validität einer deutschen Übersetzung des Fragebogens (RTS-D). Methode: Psychometrische Kennwerte und Faktorenstruktur der deutschsprachigen Version wurden an einer nicht-klinischen (N = 203) und einer klinischen Stichprobe (N = 201) überprüft. Ergebnisse: Die einfaktorielle Struktur der Originalversion konnte für die deutsche Übersetzung nicht bestätigt werden. Hingegen schnitt ein von Tanner, Voon, Hasking und Martin (2013) vorgeschlagenes vierfaktorielles Modell mit 15 Items zufriedenstellend ab. Die 15-Item-Version beinhaltet neben dem übergeordneten Faktor Ruminativer Denkstil die vier Subskalen Repetitives Denken, Kontrafaktisches/Hypothetisches Denken, Problemfokussiertes Denken und Antizipatorisches Denken. Die Befunde zu Reliabilität und Validität der gekürzten Fragebogenfassung erwiesen sich als zufriedenstellend bis gut. Schlussfolgerungen: Der RTS-D bildet Rumination als mehrdimensionales Konstrukt mit verschiedenen Facetten von grüblerischem Denken ab und kann sowohl im nicht-klinischen Setting als auch störungsübergreifend im klinischen Bereich eingesetzt werden. Weiterführende Untersuchungen insbesondere zur faktoriellen Struktur des Fragebogens werden jedoch empfohlen.

The Effect of Item Order on Psychological Aggression Reporting: An Examination With the Multidimensional Measure of Emotional Abuse

Psychological aggression between intimate partners has been found to occur at high rates in college populations, but the reliability and validity of the self-report measures used to gather this information have been criticized. Some studies indicate item order may affect self-reports of victimization and perpetration of violent behaviors. In this study, we administered standardized and randomized formats of the Multidimensional Measure of Emotional Abuse (MMEA; Murphy & Hoover, 1999) to study the impact question order had on self-reports of psychological aggression victimization and perpetration. Results demonstrated that women reported more frequent perpetration and victimization than men on some MMEA subscales. Furthermore, those who took the standardized format of the MMEA reported more frequent perpetration and victimization on some MMEA subscales than those who took the randomized format. However, no significant interaction effects were found between gender and format. Overall, these mixed results warrant further research on item order effects with self-report measures of intimate partner violence.

Development and preliminary validation of an indirect screener for drug use in the perinatal period

AIMS

This study sought to develop and begin validation of an indirect screener for identification of drug use during pregnancy, without reliance on direct disclosure.

DESIGN

Women were recruited from their hospital rooms after giving birth. Participation involved (i) completing a computerized assessment battery containing three types of items: direct (asking directly about drug use), semi-indirect (asking only about drug use prior to pregnancy) and indirect (with no mention of drug use), and (ii) providing urine and hair samples. An optimal subset of indirect items was developed and cross-validated based on ability to predict urine/hair test results.

SETTING

Obstetric unit of a university-affiliated hospital in Detroit.

PARTICIPANTS

Four hundred low-income, African American, post-partum women (300 in the developmental sample and 100 in the cross-validation sample); all available women were recruited without consideration of substance abuse risk or other characteristics.

MEASUREMENTS

Women first completed the series of direct and indirect items using a Tablet PC; they were then asked for separate consent to obtain urine and hair samples that were tested for evidence of illicit drug use.

FINDINGS

In the cross-validation sample, the brief screener consisting of six indirect items predicted toxicology results more accurately than direct questions about drug use (area under the ROC curve = 0.74, P < 0.001). Traditional direct screening questions were highly specific, but identified only a small minority of women who used drugs during the last trimester of pregnancy.

CONCLUSIONS

Indirect screening may increase the accuracy of mothers' self-reports of prenatal drug use.

The Mandarin Chinese shortened version of Oral Health Impact Profile for partially edentate patients with implant-supported prostheses

This study was to validate a mandarin Chinese version of Oral Health Impact Profile (OHIP-49) in China and to develop a shortened version of OHIP appropriate for use in partially dentate patients with implant-supported prostheses. The original 49 items of OHIP were translated into mandarin Chinese using a forward-backward method and administered to 580 subjects selected by stratified random sampling. Self-perceived oral health status and treatment need were also collected. Reliability and validity of the Chinese version of OHIP (OHIP-C49) were validated. A shortened version (OHIP-I) was derived from the OHIP-C49 by exploratory factor analysis (EFA) as well as expert-based approach in partially dentate patients (n=102) with implant-supported prostheses. For validation of the new modified shortened version, another independent sample of 97 partially dentate patients completed OHIP-I and their self-perceived oral health status at baseline and at least 3 months after dental implant rehabilitation. Five hundred and thirty-seven effectual questionnaires were reclaimed from the 580 subjects interviewed. Cronbach's alpha ranged from 0.78 to 0.96 and test-retest correlation coefficients ranged from 0.84 to 0.97 for subscale and summary scores. Construct validity was demonstrated by priori hypothesised associations between the OHIP-C49 scores and self-perceived oral health (P<0.001). The reliability and validity of OHIP-I were similar to which of the OHIP-C49, and the responsiveness appeared able to measure the effect of dental implant therapy effectively. The mandarin version of OHIP-49 showed sufficient psychometric properties for Chinese. The modified shortened version (OHIP-I) may be appropriate for the evaluation of implant therapy outcomes in partially dentate Chinese patients.

Instrument-order effects: using the Oral Health Impact Profile 49 and the Short Form 12

Whereas it is well known that the ordering of items can influence research outcomes considerably, very little literature addresses instrument-order effects. Therefore, the aim of this study was to evaluate the effect of changing the administrative order of the Short-Form-12 (SF-12) and the Oral Health Impact Profile-49 (OHIP-49). It was hypothesized that if the SF-12 was administered first, the results would show poorer scores on the SF-12 subscales, as responses would not be restrained to only the oral impacts described by the OHIP-49. Using the Mann-Whitney U-test no significant instrument-order effects were found, except for the Psychological discomfort scale of the OHIP-49, where subjects scored higher when receiving the OHIP-49 first. However, the effect size was negligible (-0.08). These results suggest that no instrument-order effects occurred. Nonetheless, more research dealing with different instruments is needed. This study was performed within a dental setting and we recommend that instrument-order effects should be studied outside this domain.

Item-order effects in the Oral Health Impact Profile (OHIP)

Items and subscales in quality of life questionnaires generally have a part-whole combination, making the content of one item more general than another. Consequently, order effects can occur. The aim of this study was to evaluate the effect of changing the item order of the Oral Health Impact Profile (OHIP). Two versions of the OHIP were randomly distributed amongst psychology freshmen. Subjects who filled out the version in which more general items were presented first showed higher subscale scores. Using the Mann-Whitney U-test, small, but statistically significant, differences between the two versions of the OHIP were found on the Psychological disability scale, the Social disability scale, and the Handicap scale. Subscale intercorrelations of both versions of the OHIP were compared to investigate whether item order also influences the factor structure. Statistically significant differences between subscale intercorrelations were found, indicating a different factor structure for both versions. In conclusion, the OHIP seems to be susceptible to order effects, implying that the original item order of the OHIP should be upheld, especially when considering comparing research outcomes with other studies.

Enhancing the validity and utility of randomized clinical trials in addictions treatment research: II. Participant samples and assessment

PURPOSE

This paper is the second in a series that describes strategies for optimizing the validity and utility of randomized clinical trials (RCTs) in addictions treatment research. Whereas the first paper focused on treatment implementation and research design, here we address issues pertaining to participant samples and assessment methods.

SCOPE

With respect to participant samples, sections focus on the definition of study populations; informed consent; sample size and statistical power; recruitment and enrollment; sample retention; and participant tracking systems. Assessment topics include eligibility screening and baseline assessment; treatment-related variables; outcome measures; the frequency of follow-up evaluation; and assessment process. A final section highlights the importance of pilot testing.

CONCLUSIONS

Sample recruitment and retention strategies are needed that safeguard both internal and external validity. Daily estimation assessment procedures are recommended because of their versatility for creating a range of outcome measures. Assessment batteries should include measures that permit the investigation of treatment processes and mechanisms of action.