Using generalizability theory to evaluate the comparative reliability of developmental measures in neurogenetic syndrome and low-risk populations

Applying the Generalizability Theory to Identify the Sources of Validity Evidence for the Quality of Communication Questionnaire

Effective doctor-patient-family communication is an integral and sensitive part of health care, assessing its quality is essential to identify aspects needing disclosure and, if necessary, improvement. Cross-sectional study aimed to analyze the sources of evidence of validity and the number of participants needed to reliably apply the Quality of Communication Questionnaire (QoC) through Generalizability Theory (GT). The mean age of the 150 patients hospitalized at the end of life was 50.5 (SD = 13.8) years, the mean hospital length of stay was 7.5 (SD = 10.2) days, 56.9% were male. Regarding the 105 patients' family members of patients whose mean length of hospital stay was 9.5 (SD = 9.1) days, their mean age was 42.2 (SD = 14.7) years, 69.5% were female. GT was used to quantify the minimum number of questionnaires needed, with the aim of reaching a reliable estimate of QoC with G-coefficients. To reach a reliability of .90, there is a need for 25 for the Eρ2 questionnaires and 35 for the Φ. The exact estimation identified the minimum number of questionnaires required for the evaluation of physicians by patients. To obtain a reliability of .90, there is a need for 30 and 40 questionnaires for the G-coefficients. A practical and fast application makes it possible to use QoC in its entirety or alone to evaluate general communication or communication about palliative care. Furthermore, based on these results, it was possible to identify which aspects were effective or ineffective in these contexts.

Understanding reliability of the observer-reported communication ability measure within Angelman syndrome through the lens of generalizability theory

AIMS

Caregivers rate improved communication ability as one of the most desired outcomes for successful interventions for individuals with Angelman syndrome (AS). When measuring communication ability in clinical trials, the reliability of such measures is critical for detecting significant changes over time. This study examined the reliability of the Observed-Reported Communication Ability (ORCA) measure completed by caregivers of individuals with AS.

METHODS

The ORCA measure was completed by 249 caregivers with 170 caregivers completing the ORCA measure again after 5-12 days. Generalizability theory was used to examine the following sources of measurement error in ORCA scores: concepts, subdomains, assessment points, and the interactions among those facets and the object of measurement: communication ability. Three generalizability studies were conducted to understand the reliability of the ORCA measure for different measurement designs. Decision studies were carried out to demonstrate the optimization of measurement procedures of the ORCA measure.

RESULTS

G and Phi coefficients of the original measurement design exceeded the 0.80 threshold considered sufficiently reliable to make relative and absolute decisions about the communication ability of individuals with AS based on their caregivers' observed scores. The optimization procedures indicated that increasing the number of communication concepts and/or assessment points leads to more reliable estimates of communication.

CONCLUSION

The ORCA measure was able to reliably distinguish different levels of communication ability among individuals with AS. Multiple assessment points and or more concepts would provide more precise estimates of an individual's communication ability but at the cost of survey fatigue.

Development and validation of a Decision-Making Ability Scale for postpartum urinary incontinence women engaging in pelvic floor physical therapy

OBJECTIVE

This study aimed to develop and validate a Decision-Making Ability Scale (DMA-S) for postpartum urinary incontinence (PPUI) women engaging in pelvic floor physical therapy (PFPT).

METHODS

Items were created in line with a review of the literature and exploratory qualitative study with 22 women. The items were submitted for expert opinion and a pilot implementation was made with 58 women with PPUI. Furthermore, the construct validity of the scale was tested with exploratory factor analysis (EFA) (n = 220) and confirmatory factor analysis (CFA) (n = 240). Internal consistency for the Chronbach's α and test-retest reliability for the intraclass correlation coefficient (ICC) were also investigated for the DMA-S in the study.

RESULTS

The results of the EFA indicated a Kaiser-Meyer-Olkin value of 0.85 and Bartlett's test of sphericity showed a χ2 value of 8352.101, p < 0.001. After removing one item with factor loading values below 0.50, the resulting factor structure accounted for 83.38% of the total variance. The fit indices of the scale model tested in the CFA were determined as χ2 /df = 1.08 < 3, root mean square error of approximation = 0.018 < 0.08, comparative fit index = 0.996 > 0.90, Tucker-Lewis index = 0.995 > 0.90, goodness-of-fit index (GFI) = 0.933 > 0.90, adjusted GFI = 0.916 > 0.90, and incremental fit index = 0.996 > 0.90. The Cronbach's α values were 0.95-0.97 for the subdimensions of the scale and 0.93 for the total scale. Data also showed a good test-retest stability (ICC = 0.984).

CONCLUSION

The DMA-S is a reliable and valid tool for assessing the decision-making ability for PPUI women engaging in PFPT.

Measuring affect in daily life: A multilevel psychometric evaluation of the PANAS-X across four ecological momentary assessment samples

While there is strong evidence for the psychometric reliability of the Positive and Negative Affect Schedule-Expanded Form (PANAS-X) in cross-sectional studies, the between- and within-person psychometric performance of the PANAS-X in an intensive longitudinal framework is less understood. As affect is thought to be dynamic and responsive to context, this study investigated the multilevel reliability of PANAS-X Positive Affect, Negative Affect, Fear, Sadness, and Hostility scales. Generalizability theory and structural equation modeling techniques (coefficient ω) were employed in four ecological momentary assessment samples (N = 309; 41,261 reports). Results demonstrate that the PANAS-X scales, including short versions of the Positive and Negative Affect scales, can reliably detect between-person differences. PANAS-X scales also were able to reliably measure within-person change, though these estimates may be impacted by scale content and study design. These results support the use of the PANAS-X in daily life research to intensively measure affect in the natural environment. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Development and Validation of the Breast Cancer Scale QLICP-BR V2.0 Based on Classical Test Theory and Generalizability Theory

Objective

The aim of this study was to develop and validate the breast cancer scale among the system of quality-of-life instruments for cancer patients (QLICP-BR V2.0).

Methods

Programmed decision procedures and theories on instrument development were applied to develop QLICP-BR V2.0. A total of 246 breast cancer inpatients were investigated using QLICP-BR V2.0 from hospital admission until discharge. The reliability, validity, and responsiveness of the QLICP-BR V2.0 scale were evaluated by using the classical test theory combined with the generalizability theory (GT), including correlation analysis, multi-trait scaling analysis, factor analyses, t-tests, and also multivariate generalizability theory analysis.

Results

The test–retest reliability of the total scale is 0.79, the Cronbach coefficient is 0.85, and the intra-class correlations coefficient is 0.88. The item–domain correlation analysis showed that the correlation coefficient between items and their own domain is greater than that with other domains except of item GSO4. The exploratory factor analysis showed that three principal components are obtained in the specific module. The outcome of the factor analysis coincides substantially with our theoretical conception. The score difference of each domain of the scale and the total scale before and after treatment is statistically significant (P &lt; 0.05), with the standardized response mean of the total scale being 0.61. According to GT, the generalization coefficient of the scores in the 5 domains is between 0.626 and 0.768, and the reliability index is between 0.557 and 0.695.

Conclusion

QLICP-BR V2.0 exhibited reasonable degrees of validity, reliability, and responsiveness according to classical test and the generalizability theory. The number of items in the scale is appropriate.

Development, reliability and validity of infectious disease specialist Nurse's Core competence scale

Aim

This study aims to develop an instrument to measure infectious disease specialist nurses’ core competence and examining the scale’s validity and reliability.

Background

With the increase of infectious diseases, more and more attention has been paid to infectious disease nursing care. The core competence of the infectious disease specialist nurses is directly related to the quality of nursing work. In previous researches, infectious disease specialist nurses’ core competence was measured by the tools developed for general nurses instead of specialized tools, which made it difficult to clarify the core competence of nurses in infectious diseases department.

Methods

Preliminary items were developed through literature review, theoretical research, qualitative interview and Delphi method. The confirmed 47 items were applied in the two rounds of data collection. Evaluation data on 516 infectious disease specialist nurses’ core competence in the first round were utilized to preliminarily evaluate and explore the scale’s constrution, while evaluation data on 497 infectious disease specialist nurses’ core competence in the second round were utilized to do reliability analysis and validity analysis. In this study, factor analysis, Cronbach’s α, Pearson correlation coefficients were all adopted.

Results

The final scale is composed of 34 items and 5 factors, and adopted the 5-point scoring method. The factors are Professional Development Abilities, Infection Prevention and Control Abilities, Nursing Abilities for Infectious Diseases, Professionalism and Humanistic Accomplishment, and Responsiveness to Emergency Infectious Diseases. The explanatory variance of the five factors was 75.569%. The reliability and validity of the scale is well validated. The internal consistency, split-half reliability and test-retest reliability were 0.806, 0.966 and 0.831 respectively. The scale has good structural validity and content validity. The content validity was 0.869. Discrimination analysis showed that there were significant differences in the scores of core competence and its five dimensions among infectious disease specialist nurses of different ages, working years in infectious diseases, titles, educational background, marital status and wages (all P < 0.05).

Conclusions

The proposed scale takes on high reliability and validity, and is suitable for assessing the infectious disease specialist nurses’ core competence.

Relevance to clinical practice

This scale provides a reference for clinical assessment of infectious disease nursing.