Measurement reliability

Item Understanding of Common Quality of Life Measures for use with Autistic Adults

Quality of life (QOL) and life satisfaction are important research priorities for autistic adults. As such, we saw a need to evaluate individual items of commonly used subjective QOL scales to understand how they are interpreted and perceived by autistic adults. This study used cognitive interviews and repeated sampling to evaluate the accessibility, test-retest reliability and internal consistency of several common QOL measures in a sample of young autistic adults (n = 20; aged 19-32). Cognitive interviews suggested that the Satisfaction with Life Scale was well understood and demonstrated excellent internal consistency and test-retest reliability. While the WHOQoL-BREF and WHOQoL Disability Modules had adequate reliability, cognitive interviews suggested that additional instructions and examples would further enhance their accessibility for use with autistic adults.

Ataxia Rating Scales: Content Analysis by Linking to the International Classification of Functioning, Disability and Health

Ataxia management is mainly based on rehabilitation, symptomatic management, and functional improvement. Therefore, it is important to comprehensively assess ataxic symptoms and their impact on function. Recently, the movement disorders society recommended four generic ataxia rating scales: scale for assessment and rating of ataxia (SARA), international cooperative ataxia rating scales, Friedreich's ataxia rating scale (FARS), and unified multiple system atrophy rating scale (UMSARS). The aim of the study was to analyze and compare the content of the recommended ataxia rating scales by linking them to the international classification of functioning, disability and health (ICF). A total of 125 meaningful concepts from 93 items of the four included scales were linked to 57 different ICF categories. The ICF categories were distributed in body structure (n = 8), body function (n = 26), activity and participation (n = 20), and environmental factors (n = 3) components. UMSARS and FARS were the only ones that have addressed the body structure or environmental factors component. The content analysis of ataxia rating scales would help clinicians and researchers select the most appropriate scale and understand ataxic symptoms and their impact on function. It seems that SARA is the optimal scale for rapid assessment of ataxia or in busy clinical settings. UMSARS or FARS are more appropriate for the investigating the impact of ataxia on overall health, and monitoring ataxia progression and disability.

Questionnaire development on measuring parents’ anxiety about their children’s education: Empirical evidence of parental perceived anxiety data for primary and secondary school students in China

Background

With the implementation of the “double reduction” policy in China, parents of primary and secondary school students are experiencing a growing trend of educational anxiety that needs to be alleviated.

Objective

To manage the education anxiety risk of parents of primary and secondary school students, a measurement questionnaire of parents’ anxiety about their children’s education (MQPAE) was developed and its reliability and validity were evaluated.

Methods

A self-administered MQPAE was developed. An online crowdsourcing questionnaire platform was used to collect data on parents’ anxiety about their children’s education (PAE), and parents of primary and secondary school students in Hefei, China, were selected as the study population. The randomly extracted 5,747 questionnaires were gradually screened by discrete trend method, t-test, and Pearson’s correlation coefficient method for the initial screening of PAE items, based on which exploratory factor analysis (EFA) was conducted for the final screening of questionnaire items and the reliability of the questionnaire. The reliability of the questionnaire was assessed by internal consistency and Pearson’s correlation coefficient analysis. Confirmatory factor analysis (CFA) was conducted using 639 pre-selected data to investigate the validity of the questionnaire. Structural equation modeling was used to investigate the structural validity of the questionnaire, and average variance extracted (AVE), combined reliability (CR), and maximum of shared squared variance (MSV) were used to test for convergent and discriminant validity.

Results

Exploratory factor analysis extracted five factors with a cumulative variance contribution of 65.66%. The CFA showed that χ²/df = 4.306, CFI = 0.920, NFI = 0.898, RMSEA = 0.072<0.08, AGFI = 0.839>0.80, PNFI = 0.793 and PGFI = 0.708. The overall Cronbach’s α coefficient of the questionnaire was 0.956, and the factors’ Cronbach’s α coefficients were 0.926, 0.857, 0.913, 0.901, and 0.768, respectively. Repeated measurements of Pearson’s correlation coefficients were 0.908, 0.911, 0.873, 0.891, 0.907 and 0.885 (all p < 0.001). The AVE was greater than 0.5 and the CR was greater than 0.7, and the value of the MSV was less than the corresponding AVE.

Conclusion

The MQPAE has good reliability and validity and can be used in studies related to PAE of primary and secondary school students.

Psychometric Properties of the Five-Item Francis Attitude Towards Christianity Scale in Colombian Psychiatric Outpatients

This study aimed to know the psychometric properties of the five-item Francis scale of the attitude towards Christianity (Francis-5) by assessing psychiatric outpatients in Santa Marta, Colombia. A sample of 260 adult patients participated. The exploratory and confirmatory factor analyses revealed an excellent one-dimensional structure with high internal consistency. It was concluded that the Francis-5 exhibited good indicators of validity and reliability in Colombian psychiatric outpatients.

Environmental Education for Sustainability in Higher Education Institutions: Design of an Instrument for Its Evaluation

Higher Education Institutions (HEI) play a fundamental role in the transition towards Environmental Education for Sustainability (EES). As a consequence, one of the most critical challenges is the need to know their level of incorporation into the environmental agenda. Therefore, an instrument was made and validated to determine the level of incorporation of Environmental Education for Sustainability into the environmental agenda of HEIs. For its construction, the dimensions of Institutional Identity, Teaching, Research, Extension/dissemination, and Linkage were considered, relying on a total of 17 items. Its validation was carried out through an expert review and expert judgment, and a pilot test was carried out to adapt it to the target population. The main result was an instrument that integrates the substantive and procedural functions of HEIs. Following the expert review, the instrument was improved according to their suggestions. The expert judgment showed an adequate content validity (Aiken’s V > 0.80; LL > 0.60). The pilot test also suggested that the understanding of instructions and items was adequate with an optimal value of internal consistency (Cronbach’s alpha of 0.862). An instrument that determines the level of incorporation of the EES in the substantive and procedural functions of HEIs is presented, valid in content, and with adequate levels of clarity and understanding of the target population.

Identifying factors associated with overweight and obesity among intermediate school students aged 12-15 years in school settings: mixed-methodology protocol

INTRODUCTION

Childhood obesity is a public health concern globally. In Arab countries, the prevalence of overweight and obesity has increased significantly over the last three decades. The level of childhood overweight and obesity in the Kingdom of Saudi Arabia (KSA) is high and continues to increase. This study will explore factors associated with overweight and obesity among adolescents and identify barriers and enablers to the implementation of comprehensive school-based obesity prevention interventions.

METHODS AND ANALYSIS

Socioecological model will inform this mixed-methods study. The study will include three phases: (1) a scoping review of the literature; (2) the development of a student survey instrument and (3) a mixed-method study comprising a cross-sectional survey targeting students aged 12-15 years with the collection of the students' height and weight measurements; one-on-one interviews with physical education teachers and school principals; and the administration of school climate audits using the Health Promoting School framework. Reliability and validity of the survey instrument will be examined during survey development. Descriptive, inferential and thematic analysis will be employed using appropriate statistical software.

ETHICS AND DISSEMINATION

Ethical approval has been granted from the Curtin University of Human Research Ethics Committee (HR2020-0337) and from the KSA Ministry of Education (4181827686). School principals will provide permission to conduct the study in individual schools. Individual consent/assent will be obtained from students and their parents, and teachers. Study findings will be disseminated via peer-review publications, reports and conferences.

Are fatigue scales the same? A content comparison using the International Classification of Functioning, Disability and Health

BACKGROUND

Fatigue is a common impairment in a wide range of disorders. Numerous fatigue scales have been designed in an attempt to quantify this impairment without any clear distinction between them. The International Classification of Functioning, Disability and Health (ICF) is a useful tool for content comparison of measurement scales.

OBJECTIVE

To explore the content of generic fatigue scales using the ICF.

METHODS

Twenty generic fatigue scales were identified and linked to the ICF by two health care professionals according to the established linking rules. The contents of the 20 scales were compared and the inter-rater agreement was estimated using kappa coefficients.

RESULTS

The content of generic fatigue scales varies and was found to focus mostly on body functions, activities and participation components of the ICF with a moderate to high degree of inter-observer agreement.

CONCLUSIONS

The content comparison of fatigue questionnaires would assist clinicians and researchers in selecting the most appropriate measurement for use and precisely analyze the results of these measurements.

Climate of Respect Evaluation in ICUs: Development of an Instrument (ICU-CORE)

OBJECTIVES

To develop a valid, reliable measure that reflected the environment of respectfulness within the ICU setting.

DESIGN

We developed a preliminary survey instrument based on conceptual domains of respect identified through prior qualitative analyses of ICU patient, family member, and clinician perspectives. The initial instrument consisted of 21 items. After five cognitive interviews and 16 pilot surveys, we revised the instrument to include 23 items. We used standard psychometric methods to analyze the instrument.

SETTINGS

Eight ICUs serving adult patients affiliated with a large university health system.

SUBJECTS

ICU clinicians.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Based on 249 responses, we identified three factors and created subscales: General Respect, Respectful Behaviors, and Disrespectful Behaviors. The General Respect subscale had seven items (α = 0.932) and reflected how often patients in the ICU are treated with respect, in a dignified manner, as an individual, equally to all other patients, on the "same level" as the ICU team, as a person, and as you yourself would want to be treated. The Respectful Behaviors subscale had 10 items (α = 0.926) and reflected how often the ICU team responds to patient and/or family anxiety, makes an effort to get to know the patient and family as people, listens carefully, explains things thoroughly, gives the opportunity to provide input into care, protects patient modesty, greets when entering room, and talks to sedated patients. The subscale measuring disrespect has four items (α = 0.702) and reflects how often the ICU team dismisses family concerns, talks down to patients and families, speaks disrespectfully behind their backs, and gets frustrated with patients and families.

CONCLUSIONS

We created a reliable set of scales to measure the climate of respectfulness in intensive care settings. These measures can be used for ongoing quality improvement that aim to enhance the experience of ICU patients and their families.

Interrater and Intrarater Reliability of the Balance Computerized Adaptive Test in Patients With Stroke

OBJECTIVE

To examine the interrater and intrarater reliability of the Balance Computerized Adaptive Test (Balance CAT) in patients with chronic stroke having a wide range of balance functions.

DESIGN

Repeated assessments design (1wk apart).

SETTING

Seven teaching hospitals.

PARTICIPANTS

A pooled sample (N=102) including 2 independent groups of outpatients (n=50 for the interrater reliability study; n=52 for the intrarater reliability study) with chronic stroke.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURES

Balance CAT.

RESULTS

For the interrater reliability study, the values of intraclass correlation coefficient, minimal detectable change (MDC), and percentage of MDC (MDC%) for the Balance CAT were .84, 1.90, and 31.0%, respectively. For the intrarater reliability study, the values of intraclass correlation coefficient, MDC, and MDC% ranged from .89 to .91, from 1.14 to 1.26, and from 17.1% to 18.6%, respectively.

CONCLUSIONS

The Balance CAT showed sufficient intrarater reliability in patients with chronic stroke having balance functions ranging from sitting with support to independent walking. Although the Balance CAT may have good interrater reliability, we found substantial random measurement error between different raters. Accordingly, if the Balance CAT is used as an outcome measure in clinical or research settings, same raters are suggested over different time points to ensure reliable assessments.

Psychometric properties of three measures assessing advanced theory of mind: Evidence from people with schizophrenia

Patients with schizophrenia tend to have deficits in advanced Theory of Mind (ToM). The "Reading the mind in the eyes" test (RMET), the Faux Pas Task, and the Strange Stories are commonly used for assessing advanced ToM. However, most of the psychometric properties of these 3 measures in patients with schizophrenia are unknown. The aims of this study were to validate the psychometric properties of the 3 advanced ToM measures in patients with schizophrenia, including: (1) test-retest reliability; (2) random measurement error; (3) practice effect; (4) concurrent validity; and (5) ecological validity. We recruited 53 patients with schizophrenia, who completed the 3 measures twice, 4 weeks apart. The Revised Social Functioning Scale-Taiwan short version (R-SFST) was completed within 3 days of first session of assessments. We found that the intraclass correlation coefficients of the RMET, Strange Stories, and Faux Pas Task were 0.24, 0.5, and 0.76. All 3 advanced ToM measures had large random measurement error, trivial to small practice effects, poor concurrent validity, and low ecological validity. We recommend that the scores of the 3 advanced ToM measures be interpreted with caution because these measures may not provide reliable and valid results on patients' advanced ToM abilities.

A sustainable model for training teachers to use pivotal response training

The increase in the rate of autism diagnoses has created a growing demand for teachers who are trained to use effective interventions. The train-the-trainer model, which involves training supervisors to train others, may be ideal for providing cost-effective training and ongoing support to teachers. Although research supports interventions, such as pivotal response training, as evidence-based, dissemination to school environments has been problematic. This study assessed the benefits of using the train-the-trainer model to disseminate pivotal response training to school settings. A multiple-baseline design was conducted across three training groups, each consisting of one school staff member (trainer), three special education teachers, and six students. Trainers conducted the teacher-training workshop with high adherence to training protocol and met mastery criteria in their ability to implement pivotal response training, assess implementation of pivotal response training, and provide feedback to teachers. Six of the nine teachers mastered all components of pivotal response training. The remaining three teachers implemented 89% of the pivotal response training components correctly. The majority of trainers and teachers maintained their abilities at follow-up. These results support the use of the train-the-trainer model as an effective method of disseminating evidence-based practices in school settings.

Comparing top-down and bottom-up costing approaches for economic evaluation within social welfare

This study compares two approaches to the estimation of social welfare intervention costs: one "top-down" and the other "bottom-up" for a group of social welfare clients with severe problem behavior participating in a randomized trial. Intervention costs ranging over a two-year period were compared by intervention category (foster care placement, institutional placement, mentorship services, individual support services and structured support services), estimation method (price, micro costing, average cost) and treatment group (intervention, control). Analyses are based upon 2007 costs for 156 individuals receiving 404 interventions. Overall, both approaches were found to produce reliable estimates of intervention costs at the group level but not at the individual level. As choice of approach can greatly impact the estimate of mean difference, adjustment based on estimation approach should be incorporated into sensitivity analyses. Analysts must take care in assessing the purpose and perspective of the analysis when choosing a costing approach for use within economic evaluation.

Content comparison of self-report measures used in vestibular rehabilitation based on the international classification of functioning, disability and health

BACKGROUND

Physical therapists should understand the content included in self-report questionnaires in order to choose the appropriate questionnaire for examination and follow-up purposes. The International Classification of Functioning, Disability and Health (ICF) provides a universal and standard language for the description of health and health-related states and can be used for the content comparison of self-report questionnaires. Objective The purpose of this study was to describe and compare the contents covered by 8 self-report measures used in vestibular rehabilitation based on the linkage of their content to the ICF.

DESIGN

This was a cross-sectional study.

METHOD

Eight vestibular questionnaires were identified and linked to the ICF by 2 health care professionals according to established linking rules. Based on the linking, the contents of the 8 questionnaires were compared and the interobserver agreement between 2 raters was estimated using kappa coefficients.

RESULTS

A total of 312 meaningful concepts from the 164 items of the 8 vestibular questionnaires were identified and linked to the ICF. The meaningful concepts identified were linked to 51 different ICF components: 19 categories of the component "body functions," 30 categories of the component "activities and participation," and 2 categories of the component "environmental factors." Forty-two concepts could not be linked to any of the ICF components. The estimated kappa coefficients for 2 raters at the component and first and second levels of the ICF ranged from 0.83 to 0.96. Limitations The method of identifying vestibular measures was not based on a systematic search; instead, the most widely used instruments in the field of vestibular rehabilitation were selected. Thus, the study results are limited to the examined vestibular instruments.

CONCLUSION

Using the ICF as a theoretical framework was found to be useful for comparing the content of health status questionnaires, as well as for exploring the focus of the measures currently in use in vestibular rehabilitation.

Assessing linguistic and cultural equivalency of two Chinese-version sexual health instruments among Chinese immigrant youth

This study aimed to evaluate the linguistic and cultural equivalency of two Chinese-version instruments measuring sexual knowledge, attitudes, and behaviors among 10 bilingual and 2 monolingual Chinese immigrant youth aged 13-19 years. We used a rigorous design to translate, back-translate, and pilot test the instruments. Kappa coefficient, percentage agreement, and qualitative feedback from participants were used to examine reliability and validity of the instruments. Telephone interviews revealed that answer discrepancies in different language versions were due to external factors rather than lack of linguistic and cultural equivalency. This study offers preliminary evidence supporting the cultural and linguistic equivalence of two Chinese-version sexual scales.

Psychometric comparisons of 4 measures for assessing upper-extremity function in people with stroke

BACKGROUND

Functional limitation of the upper extremities is common in patients with stroke. An upper-extremity measure with sound psychometric properties is indispensable for clinical and research use.

OBJECTIVE

The purpose of this study was to compare the psychometric properties of 4 clinical measures for assessing upper-extremity motor function in people with stroke: the upper-extremity subscale of the Fugl-Meyer Motor Test (UE-FM), the upper-extremity subscale of the Stroke Rehabilitation Assessment of Movement, the Action Research Arm Test (ARAT), and the Wolf Motor Function Test.

DESIGN

This was a prospective, longitudinal study.

METHODS

Fifty-three people with stroke were evaluated with the 4 measures at 4 time points (14, 30, 90, and 180 days after stroke). Thirty-five participants completed all of the assessments. The ceiling and floor effects, validity (concurrent validity and predictive validity), and responsiveness of each measure were examined. Interrater reliability and test-retest reliability also were examined.

RESULTS

All measures, except for the UE-FM, had significant floor effects or ceiling effects at one or more time points. The Spearman rho correlation coefficient for each pair of the 4 measures was > or =.81, indicating high concurrent validity. The predictive validity of the 4 measures was satisfactory (Spearman rho, > or =.51). The responsiveness of the 4 measures at 14 to 180 days after stroke was moderate (.52 < or = effect size < or = .79). The 4 measures had good interrater reliability (intraclass correlation coefficient [ICC], > or =.92) and test-retest reliability (ICC, > or =.97). Only the minimal detectable changes of the UE-FM (8% of the highest possible score) and the ARAT (6%) were satisfactory.

LIMITATIONS

The sample size was too small to conduct data analysis according to type or severity of stroke. In addition, the timed component of the Wolf Motor Function Test was not used in this study.

CONCLUSIONS

All 4 measures showed sufficient validity, responsiveness, and reliability in participants with stroke. The UE-FM for assessing impairment and the ARAT for assessing disability had satisfactory minimal detectable changes, supporting their utility in clinical settings.

Team management of dysphagia in the institutional setting

The capacity to swallow effectively and safely is a basic human need, yet nearly 40% of Americans over age 60 experience dysphagia. Since the resources in acute-care hospitals often are unavailable in institutional settings, the daily involvement of nursing and the dietitian's screening and continuous assessment are critical to timely, effective dysphagia identification, referral, and management. Upon referral, the speech pathologist executes comprehensive evaluation, leading the design and implementation of a team treatment plan. Literature highlighting a sample of dysphagia screening tools and interventions, care transitions and aspiration prevention strategies is reviewed herein to guide current practice and future research.

Psychometric properties of the Peer Interactions in Primary School (PIPS) Questionnaire

BACKGROUND

Recently, national and international scientific and popular press has focused on bullying and victimization. Unfortunately, many interventions that address bullying and victimization are yet to be empirically validated. One problem is the lack of a psychometrically sound instrument for the measurement of bullying and victimization.

METHODS

To alleviate this shortcoming, the Peer Interactions in Primary School Questionnaire (PIPS) was developed and tested. Twenty-two questions designed to capture direct and indirect forms of bullying and victimization were created at a third-grade reading level. Psychometric data were collected from administration of the questionnaire to 270 students in third through sixth grades at three different elementary schools. An exploratory factor analysis yielded two factors (bullying and victimization).

RESULTS

Internal consistency for the questionnaire was high (Cronbach's alpha = .90). Intraclass correlation coefficients (ICCs) and Spearman's rho established that test-retest reliability was high for both scales: bullying (ICC = .84; rho = .76) and victimization (ICC = .88; rho = .87). Significant Kruskal-Wallis tests of relationships between PIPS scales and items on the Olweus Bullying/Victimization Questionnaire and the Strengths and Difficulties Questionnaire supported concurrent validity. Bullying and victimization were widespread, as 89.5% of children experienced some form of victimization and 59.0% of students participated in some form of bullying.

CONCLUSION

With these data, the PIPS is the first self-report bullying and victimization measure designed for elementary school use determined reliable (internally consistent and reproducible) and valid. The PIPS is a tool that could be used in the design and evaluation of school-based bullying/victimization interventions.

The laboratory school protocol: its origin, use, and new applications

OBJECTIVE

ADHD is the most common childhood psychiatric disorder, with impairments seen in home and academic settings. To investigate such impairments in a school-like setting, the laboratory school protocol (LSP) was developed at the University of California, Irvine.

METHOD

This model provides a rigorously controlled environment to examine pharmacodynamic and pharmacokinetic aspects of responses to treatment. A key principle of this methodology is to exercise tight control of the timing and context of measurements by establishing a cycle of activities repeated across each study day. In addition, the LSP approach has been extended to both younger and older populations than the typically studied school-aged group. This extension requires corresponding modifications in measures to characterize drug efficacy and to allow evaluation of ADHD symptoms in a highly standardized setting.

RESULTS

This article provides guidelines for employing the LSP for the assessment of medication effects for both preschool and adolescent/adult populations.

CONCLUSION

The LSP can be modified to form either an Adult Workplace Environment or a Preschool Assessment Laboratory.